2024-08-18T15:06:57.045 INFO:root:teuthology version: 1.2.0 2024-08-18T15:06:57.066 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-08-18T15:06:57.164 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-08-16_22:40:05-rgw-reef-distro-default-smithi/7858171 branch: reef description: rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zones supported-random-distro$/{centos_latest} tasks/test_multi} email: ceph-qa@ceph.com first_in_suite: false job_id: '7858171' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-08-16_22:40:05-rgw-reef-distro-default-smithi no_nested_subset: false os_type: centos os_version: 9.stream overrides: admin_socket: branch: reef ceph: conf: client: debug rgw: 20 debug rgw sync: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: testing rgw crypt s3 kms encryption keys: testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= rgw curl low speed time: 300 rgw data log num shards: 4 rgw md log max shards: 4 rgw sync data inject err probability: 0.1 rgw sync log trim interval: 0 rgw sync meta inject err probability: 0.1 rgw sync obj etag verify: true setgroup: ceph setuser: ceph client.0: rgw data notify interval msec: 0 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 rocksdb delete range threshold: 0 osd.0: osd_max_omap_entries_per_request: 10 osd.1: osd_max_omap_entries_per_request: 1000 osd.2: osd_max_omap_entries_per_request: 10000 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_AVAILABILITY\) - \(PG_DEGRADED\) - \(POOL_APP_NOT_ENABLED\) - not have an application enabled sha1: 03f4c3e85ce172201896147a7aef87ba256187f2 wait-for-scrub: false ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 03f4c3e85ce172201896147a7aef87ba256187f2 rgw: compression type: random frontend: beast rgw-multisite: realm: is default: true name: test-realm zonegroups: - enabled_features: - resharding endpoints: - c1.client.0 is_default: true is_master: true name: test-zonegroup zones: - endpoints: - c1.client.0 is_default: true is_master: true name: test-zone1 - endpoints: - c2.client.0 is_default: true name: test-zone2 - endpoints: - c1.client.1 name: test-zone3 rgw-multisite-tests: args: - tests.py selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 workunit: branch: reef sha1: 03f4c3e85ce172201896147a7aef87ba256187f2 owner: scheduled_teuthology@teuthology priority: 930 repo: https://git.ceph.com/ceph.git roles: - - c1.mon.a - c1.mgr.x - c1.osd.0 - c1.osd.1 - c1.osd.2 - c1.client.0 - c1.client.1 - - c2.mon.a - c2.mgr.x - c2.osd.0 - c2.osd.1 - c2.osd.2 - c2.client.0 - c2.client.1 seed: 4794 sha1: 03f4c3e85ce172201896147a7aef87ba256187f2 sleep_before_teardown: 0 subset: 0/1 suite: rgw suite_branch: reef suite_path: /home/teuthworker/src/git.ceph.com_ceph_03f4c3e85ce172201896147a7aef87ba256187f2/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 03f4c3e85ce172201896147a7aef87ba256187f2 targets: smithi134.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIE9XgycsFVMPLcbi8grvCs2X7jnzF+W6OvDtQvdPSikD8DSyZ6GPlvctMpS+baiu/U3LWz8MtsURSjI/icSDh0= smithi193.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCC+k2hQSoI+gvHGxnhhbNPw+ovFp8VjwRVMU4OOFllP2RKhpxQhCB1OIlvSYy7oNCv6WGBH8zahtKKyQE/oYyQ= tasks: - install: null - ceph: cluster: c1 - ceph: cluster: c2 - rgw: c1.client.0: port: 8000 c1.client.1: port: 8001 c2.client.0: port: 8000 c2.client.1: port: 8001 - rgw-multisite: null - rgw-multisite-tests: config: reconfigure_delay: 60 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 67db276914d29505c9707d04f3a3b8fb9d8aee4b timestamp: 2024-08-16_22:40:05 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-08-18T15:06:57.164 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_03f4c3e85ce172201896147a7aef87ba256187f2/qa; will attempt to use it 2024-08-18T15:06:57.165 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_03f4c3e85ce172201896147a7aef87ba256187f2/qa/tasks 2024-08-18T15:06:57.165 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-08-18T15:06:57.167 INFO:teuthology.task.internal:Checking packages... 2024-08-18T15:06:57.188 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '03f4c3e85ce172201896147a7aef87ba256187f2' 2024-08-18T15:06:57.188 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-08-18T15:06:57.188 INFO:teuthology.packaging:ref: None 2024-08-18T15:06:57.188 INFO:teuthology.packaging:tag: None 2024-08-18T15:06:57.188 INFO:teuthology.packaging:branch: reef 2024-08-18T15:06:57.188 INFO:teuthology.packaging:sha1: 03f4c3e85ce172201896147a7aef87ba256187f2 2024-08-18T15:06:57.189 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=reef 2024-08-18T15:06:57.475 INFO:teuthology.task.internal:Found packages for ceph version 18.2.4-719.gced649f8 2024-08-18T15:06:57.477 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-08-18T15:06:57.483 INFO:teuthology.task.internal:no buildpackages task found 2024-08-18T15:06:57.483 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-08-18T15:06:57.636 INFO:teuthology.task.internal:Saving configuration 2024-08-18T15:06:57.650 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-08-18T15:06:57.699 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-08-18T15:06:57.732 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi134.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-08-16_22:40:05-rgw-reef-distro-default-smithi/7858171', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-08-18 15:01:46.139850', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIE9XgycsFVMPLcbi8grvCs2X7jnzF+W6OvDtQvdPSikD8DSyZ6GPlvctMpS+baiu/U3LWz8MtsURSjI/icSDh0='} 2024-08-18T15:06:57.765 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi193.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-08-16_22:40:05-rgw-reef-distro-default-smithi/7858171', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-08-18 15:01:46.142609', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCC+k2hQSoI+gvHGxnhhbNPw+ovFp8VjwRVMU4OOFllP2RKhpxQhCB1OIlvSYy7oNCv6WGBH8zahtKKyQE/oYyQ='} 2024-08-18T15:06:57.765 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-08-18T15:06:57.772 INFO:teuthology.task.internal:roles: ubuntu@smithi134.front.sepia.ceph.com - ['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'] 2024-08-18T15:06:57.772 INFO:teuthology.task.internal:roles: ubuntu@smithi193.front.sepia.ceph.com - ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1'] 2024-08-18T15:06:57.772 INFO:teuthology.run_tasks:Running task console_log... 2024-08-18T15:06:57.845 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f15f9fc3be0>, signals=[15]) 2024-08-18T15:06:57.845 INFO:teuthology.run_tasks:Running task internal.connect... 2024-08-18T15:06:57.851 INFO:teuthology.task.internal:Opening connections... 2024-08-18T15:06:57.851 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi134.front.sepia.ceph.com 2024-08-18T15:06:57.853 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-18T15:06:57.936 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi193.front.sepia.ceph.com 2024-08-18T15:06:57.937 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi193.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-18T15:06:58.039 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-08-18T15:06:58.045 DEBUG:teuthology.orchestra.run.smithi134:> uname -m 2024-08-18T15:06:58.063 INFO:teuthology.orchestra.run.smithi134.stdout:x86_64 2024-08-18T15:06:58.063 DEBUG:teuthology.orchestra.run.smithi134:> cat /etc/os-release 2024-08-18T15:06:58.120 INFO:teuthology.orchestra.run.smithi134.stdout:NAME="CentOS Stream" 2024-08-18T15:06:58.120 INFO:teuthology.orchestra.run.smithi134.stdout:VERSION="9" 2024-08-18T15:06:58.120 INFO:teuthology.orchestra.run.smithi134.stdout:ID="centos" 2024-08-18T15:06:58.120 INFO:teuthology.orchestra.run.smithi134.stdout:ID_LIKE="rhel fedora" 2024-08-18T15:06:58.120 INFO:teuthology.orchestra.run.smithi134.stdout:VERSION_ID="9" 2024-08-18T15:06:58.120 INFO:teuthology.orchestra.run.smithi134.stdout:PLATFORM_ID="platform:el9" 2024-08-18T15:06:58.120 INFO:teuthology.orchestra.run.smithi134.stdout:PRETTY_NAME="CentOS Stream 9" 2024-08-18T15:06:58.120 INFO:teuthology.orchestra.run.smithi134.stdout:ANSI_COLOR="0;31" 2024-08-18T15:06:58.121 INFO:teuthology.orchestra.run.smithi134.stdout:LOGO="fedora-logo-icon" 2024-08-18T15:06:58.121 INFO:teuthology.orchestra.run.smithi134.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-08-18T15:06:58.121 INFO:teuthology.orchestra.run.smithi134.stdout:HOME_URL="https://centos.org/" 2024-08-18T15:06:58.121 INFO:teuthology.orchestra.run.smithi134.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-08-18T15:06:58.121 INFO:teuthology.orchestra.run.smithi134.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-08-18T15:06:58.121 INFO:teuthology.orchestra.run.smithi134.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-08-18T15:06:58.122 INFO:teuthology.lock.ops:Updating smithi134.front.sepia.ceph.com on lock server 2024-08-18T15:06:58.267 DEBUG:teuthology.orchestra.run.smithi193:> uname -m 2024-08-18T15:06:58.286 INFO:teuthology.orchestra.run.smithi193.stdout:x86_64 2024-08-18T15:06:58.286 DEBUG:teuthology.orchestra.run.smithi193:> cat /etc/os-release 2024-08-18T15:06:58.343 INFO:teuthology.orchestra.run.smithi193.stdout:NAME="CentOS Stream" 2024-08-18T15:06:58.343 INFO:teuthology.orchestra.run.smithi193.stdout:VERSION="9" 2024-08-18T15:06:58.343 INFO:teuthology.orchestra.run.smithi193.stdout:ID="centos" 2024-08-18T15:06:58.343 INFO:teuthology.orchestra.run.smithi193.stdout:ID_LIKE="rhel fedora" 2024-08-18T15:06:58.343 INFO:teuthology.orchestra.run.smithi193.stdout:VERSION_ID="9" 2024-08-18T15:06:58.343 INFO:teuthology.orchestra.run.smithi193.stdout:PLATFORM_ID="platform:el9" 2024-08-18T15:06:58.343 INFO:teuthology.orchestra.run.smithi193.stdout:PRETTY_NAME="CentOS Stream 9" 2024-08-18T15:06:58.343 INFO:teuthology.orchestra.run.smithi193.stdout:ANSI_COLOR="0;31" 2024-08-18T15:06:58.344 INFO:teuthology.orchestra.run.smithi193.stdout:LOGO="fedora-logo-icon" 2024-08-18T15:06:58.344 INFO:teuthology.orchestra.run.smithi193.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-08-18T15:06:58.344 INFO:teuthology.orchestra.run.smithi193.stdout:HOME_URL="https://centos.org/" 2024-08-18T15:06:58.344 INFO:teuthology.orchestra.run.smithi193.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-08-18T15:06:58.344 INFO:teuthology.orchestra.run.smithi193.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-08-18T15:06:58.344 INFO:teuthology.orchestra.run.smithi193.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-08-18T15:06:58.344 INFO:teuthology.lock.ops:Updating smithi193.front.sepia.ceph.com on lock server 2024-08-18T15:06:58.379 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-08-18T15:06:58.388 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-08-18T15:06:58.411 INFO:teuthology.task.internal:Checking for old test directory... 2024-08-18T15:06:58.412 DEBUG:teuthology.orchestra.run.smithi134:> test '!' -e /home/ubuntu/cephtest 2024-08-18T15:06:58.415 DEBUG:teuthology.orchestra.run.smithi193:> test '!' -e /home/ubuntu/cephtest 2024-08-18T15:06:58.430 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-08-18T15:06:58.440 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-08-18T15:06:58.441 DEBUG:teuthology.orchestra.run.smithi134:> test -z $(ls -A /var/lib/ceph) 2024-08-18T15:06:58.471 DEBUG:teuthology.orchestra.run.smithi193:> test -z $(ls -A /var/lib/ceph) 2024-08-18T15:06:58.509 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-08-18T15:06:58.581 INFO:teuthology.run_tasks:Running task kernel... 2024-08-18T15:06:58.595 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-08-18T15:06:58.595 INFO:teuthology.task.kernel:config {'c1.mon.a': {'kdb': 1, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-08-18T15:06:58.595 DEBUG:teuthology.orchestra.run.smithi134:> test -f /run/.containerenv -o -f /.dockerenv 2024-08-18T15:06:58.596 DEBUG:teuthology.orchestra.run.smithi193:> test -f /run/.containerenv -o -f /.dockerenv 2024-08-18T15:06:58.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-18T15:06:58.612 DEBUG:teuthology.orchestra.run.smithi193:> uname -r 2024-08-18T15:06:58.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-18T15:06:58.613 DEBUG:teuthology.orchestra.run.smithi134:> uname -r 2024-08-18T15:06:58.672 INFO:teuthology.orchestra.run.smithi193.stdout:5.14.0-437.el9.x86_64 2024-08-18T15:06:58.672 INFO:teuthology.task.kernel:Running kernel on smithi193: 5.14.0-437.el9.x86_64 2024-08-18T15:06:58.672 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum install -y kernel 2024-08-18T15:06:58.673 INFO:teuthology.orchestra.run.smithi134.stdout:5.14.0-437.el9.x86_64 2024-08-18T15:06:58.673 INFO:teuthology.task.kernel:Running kernel on smithi134: 5.14.0-437.el9.x86_64 2024-08-18T15:06:58.674 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum install -y kernel 2024-08-18T15:07:01.101 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - BaseOS 10 MB/s | 8.2 MB 00:00 2024-08-18T15:07:02.676 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS Stream 9 - BaseOS 3.5 MB/s | 8.2 MB 00:02 2024-08-18T15:07:03.052 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - AppStream 23 MB/s | 20 MB 00:00 2024-08-18T15:07:07.661 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - CRB 8.0 MB/s | 6.6 MB 00:00 2024-08-18T15:07:09.320 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - Extras packages 43 kB/s | 18 kB 00:00 2024-08-18T15:07:10.440 INFO:teuthology.orchestra.run.smithi134.stdout:Extra Packages for Enterprise Linux 23 MB/s | 22 MB 00:00 2024-08-18T15:07:15.879 INFO:teuthology.orchestra.run.smithi134.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-08-18T15:07:17.836 INFO:teuthology.orchestra.run.smithi134.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-08-18T15:07:17.836 INFO:teuthology.orchestra.run.smithi134.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-08-18T15:07:17.921 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-08-18T15:07:17.929 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-08-18T15:07:17.929 INFO:teuthology.orchestra.run.smithi134.stdout: Package Architecture Version Repository Size 2024-08-18T15:07:17.929 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-08-18T15:07:17.929 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-08-18T15:07:17.929 INFO:teuthology.orchestra.run.smithi134.stdout: kernel x86_64 5.14.0-480.el9 baseos 1.5 M 2024-08-18T15:07:17.929 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-08-18T15:07:17.929 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-core x86_64 5.14.0-480.el9 baseos 17 M 2024-08-18T15:07:17.929 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-modules x86_64 5.14.0-480.el9 baseos 36 M 2024-08-18T15:07:17.929 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-modules-core x86_64 5.14.0-480.el9 baseos 30 M 2024-08-18T15:07:17.929 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:07:17.929 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-08-18T15:07:17.930 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-08-18T15:07:17.930 INFO:teuthology.orchestra.run.smithi134.stdout:Install 4 Packages 2024-08-18T15:07:17.930 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:07:17.930 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 84 M 2024-08-18T15:07:17.930 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 126 M 2024-08-18T15:07:17.930 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-08-18T15:07:19.297 INFO:teuthology.orchestra.run.smithi134.stdout:(1/4): kernel-5.14.0-480.el9.x86_64.rpm 1.2 MB/s | 1.5 MB 00:01 2024-08-18T15:07:19.781 INFO:teuthology.orchestra.run.smithi134.stdout:(2/4): kernel-core-5.14.0-480.el9.x86_64.rpm 9.8 MB/s | 17 MB 00:01 2024-08-18T15:07:20.498 INFO:teuthology.orchestra.run.smithi134.stdout:(3/4): kernel-modules-5.14.0-480.el9.x86_64.rpm 15 MB/s | 36 MB 00:02 2024-08-18T15:07:21.116 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS Stream 9 - AppStream 1.2 MB/s | 20 MB 00:17 2024-08-18T15:07:22.281 INFO:teuthology.orchestra.run.smithi134.stdout:(4/4): kernel-modules-core-5.14.0-480.el9.x86_6 10 MB/s | 30 MB 00:02 2024-08-18T15:07:22.281 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-08-18T15:07:22.282 INFO:teuthology.orchestra.run.smithi134.stdout:Total 19 MB/s | 84 MB 00:04 2024-08-18T15:07:23.015 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-08-18T15:07:23.370 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-08-18T15:07:23.370 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-08-18T15:07:24.361 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-08-18T15:07:24.362 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-08-18T15:07:25.683 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS Stream 9 - CRB 9.2 MB/s | 6.6 MB 00:00 2024-08-18T15:07:26.799 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-08-18T15:07:27.092 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS Stream 9 - Extras packages 103 kB/s | 18 kB 00:00 2024-08-18T15:07:27.504 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : kernel-modules-core-5.14.0-480.el9.x86_64 1/4 2024-08-18T15:07:27.598 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : kernel-core-5.14.0-480.el9.x86_64 2/4 2024-08-18T15:07:28.371 INFO:teuthology.orchestra.run.smithi193.stdout:Extra Packages for Enterprise Linux 19 MB/s | 22 MB 00:01 2024-08-18T15:07:30.256 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-core-5.14.0-480.el9.x86_64 2/4 2024-08-18T15:07:30.399 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : kernel-modules-5.14.0-480.el9.x86_64 3/4 2024-08-18T15:07:33.677 INFO:teuthology.orchestra.run.smithi193.stdout:lab-extras 41 kB/s | 1.7 kB 00:00 2024-08-18T15:07:35.241 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-modules-5.14.0-480.el9.x86_64 3/4 2024-08-18T15:07:35.308 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : kernel-5.14.0-480.el9.x86_64 4/4 2024-08-18T15:07:35.625 INFO:teuthology.orchestra.run.smithi193.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-08-18T15:07:35.625 INFO:teuthology.orchestra.run.smithi193.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-08-18T15:07:35.717 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-08-18T15:07:35.725 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-08-18T15:07:35.725 INFO:teuthology.orchestra.run.smithi193.stdout: Package Architecture Version Repository Size 2024-08-18T15:07:35.726 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-08-18T15:07:35.726 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-08-18T15:07:35.726 INFO:teuthology.orchestra.run.smithi193.stdout: kernel x86_64 5.14.0-480.el9 baseos 1.5 M 2024-08-18T15:07:35.726 INFO:teuthology.orchestra.run.smithi193.stdout:Installing dependencies: 2024-08-18T15:07:35.726 INFO:teuthology.orchestra.run.smithi193.stdout: kernel-core x86_64 5.14.0-480.el9 baseos 17 M 2024-08-18T15:07:35.726 INFO:teuthology.orchestra.run.smithi193.stdout: kernel-modules x86_64 5.14.0-480.el9 baseos 36 M 2024-08-18T15:07:35.726 INFO:teuthology.orchestra.run.smithi193.stdout: kernel-modules-core x86_64 5.14.0-480.el9 baseos 30 M 2024-08-18T15:07:35.726 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:07:35.726 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-08-18T15:07:35.726 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-08-18T15:07:35.726 INFO:teuthology.orchestra.run.smithi193.stdout:Install 4 Packages 2024-08-18T15:07:35.726 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:07:35.727 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 84 M 2024-08-18T15:07:35.727 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 126 M 2024-08-18T15:07:35.727 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-08-18T15:07:36.270 INFO:teuthology.orchestra.run.smithi193.stdout:(1/4): kernel-5.14.0-480.el9.x86_64.rpm 3.6 MB/s | 1.5 MB 00:00 2024-08-18T15:07:38.636 INFO:teuthology.orchestra.run.smithi193.stdout:(2/4): kernel-core-5.14.0-480.el9.x86_64.rpm 6.1 MB/s | 17 MB 00:02 2024-08-18T15:07:40.194 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-modules-core-5.14.0-480.el9.x86_64 4/4 2024-08-18T15:07:41.095 INFO:teuthology.orchestra.run.smithi193.stdout:(3/4): kernel-modules-core-5.14.0-480.el9.x86_6 6.2 MB/s | 30 MB 00:04 2024-08-18T15:07:44.069 INFO:teuthology.orchestra.run.smithi193.stdout:(4/4): kernel-modules-5.14.0-480.el9.x86_64.rpm 4.4 MB/s | 36 MB 00:08 2024-08-18T15:07:44.070 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-08-18T15:07:44.070 INFO:teuthology.orchestra.run.smithi193.stdout:Total 10 MB/s | 84 MB 00:08 2024-08-18T15:07:44.809 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-08-18T15:07:45.143 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-08-18T15:07:45.144 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-08-18T15:07:46.134 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-08-18T15:07:46.135 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-08-18T15:07:48.560 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-08-18T15:07:49.292 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : kernel-modules-core-5.14.0-480.el9.x86_64 1/4 2024-08-18T15:07:49.386 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : kernel-core-5.14.0-480.el9.x86_64 2/4 2024-08-18T15:07:52.018 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: kernel-core-5.14.0-480.el9.x86_64 2/4 2024-08-18T15:07:52.170 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : kernel-modules-5.14.0-480.el9.x86_64 3/4 2024-08-18T15:07:57.031 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: kernel-modules-5.14.0-480.el9.x86_64 3/4 2024-08-18T15:07:57.245 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : kernel-5.14.0-480.el9.x86_64 4/4 2024-08-18T15:08:02.138 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: kernel-modules-core-5.14.0-480.el9.x86_64 4/4 2024-08-18T15:08:09.741 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-core-5.14.0-480.el9.x86_64 4/4 2024-08-18T15:08:09.782 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-modules-5.14.0-480.el9.x86_64 4/4 2024-08-18T15:08:10.185 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-5.14.0-480.el9.x86_64 4/4 2024-08-18T15:08:10.185 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : kernel-5.14.0-480.el9.x86_64 1/4 2024-08-18T15:08:10.185 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : kernel-core-5.14.0-480.el9.x86_64 2/4 2024-08-18T15:08:10.185 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : kernel-modules-5.14.0-480.el9.x86_64 3/4 2024-08-18T15:08:10.984 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : kernel-modules-core-5.14.0-480.el9.x86_64 4/4 2024-08-18T15:08:10.984 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:08:10.984 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-08-18T15:08:10.984 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-5.14.0-480.el9.x86_64 2024-08-18T15:08:10.984 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-core-5.14.0-480.el9.x86_64 2024-08-18T15:08:10.984 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-modules-5.14.0-480.el9.x86_64 2024-08-18T15:08:10.984 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-modules-core-5.14.0-480.el9.x86_64 2024-08-18T15:08:10.984 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:08:10.984 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-08-18T15:08:11.406 DEBUG:teuthology.orchestra.run.smithi134:> rpm -q kernel | sort -rV | head -n 1 2024-08-18T15:08:11.483 INFO:teuthology.orchestra.run.smithi134.stdout:kernel-5.14.0-480.el9.x86_64 2024-08-18T15:08:11.483 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-480.el9.x86_64 2024-08-18T15:08:11.484 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-480.el9.x86_64 2024-08-18T15:08:11.484 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-08-18T15:08:11.484 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-08-18T15:08:11.484 INFO:teuthology.task.kernel:Installing distro kernel on c1.mon.a... 2024-08-18T15:08:11.484 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi134.front.sepia.ceph.com, path=None, version=distro) 2024-08-18T15:08:11.484 DEBUG:teuthology.orchestra.run.smithi134:> rpm -q kernel | sort -rV | head -n 1 2024-08-18T15:08:11.541 INFO:teuthology.orchestra.run.smithi134.stdout:kernel-5.14.0-480.el9.x86_64 2024-08-18T15:08:11.541 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-480.el9.x86_64 2024-08-18T15:08:11.542 DEBUG:teuthology.orchestra.run.smithi134:> sudo rpm -qi grub2-tools 2024-08-18T15:08:11.577 INFO:teuthology.orchestra.run.smithi134.stdout:Name : grub2-tools 2024-08-18T15:08:11.577 INFO:teuthology.orchestra.run.smithi134.stdout:Epoch : 1 2024-08-18T15:08:11.577 INFO:teuthology.orchestra.run.smithi134.stdout:Version : 2.06 2024-08-18T15:08:11.577 INFO:teuthology.orchestra.run.smithi134.stdout:Release : 68.el9 2024-08-18T15:08:11.577 INFO:teuthology.orchestra.run.smithi134.stdout:Architecture: x86_64 2024-08-18T15:08:11.577 INFO:teuthology.orchestra.run.smithi134.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-08-18T15:08:11.577 INFO:teuthology.orchestra.run.smithi134.stdout:Group : Unspecified 2024-08-18T15:08:11.577 INFO:teuthology.orchestra.run.smithi134.stdout:Size : 8264254 2024-08-18T15:08:11.577 INFO:teuthology.orchestra.run.smithi134.stdout:License : GPLv3+ 2024-08-18T15:08:11.577 INFO:teuthology.orchestra.run.smithi134.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-08-18T15:08:11.578 INFO:teuthology.orchestra.run.smithi134.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-08-18T15:08:11.578 INFO:teuthology.orchestra.run.smithi134.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-08-18T15:08:11.578 INFO:teuthology.orchestra.run.smithi134.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-08-18T15:08:11.578 INFO:teuthology.orchestra.run.smithi134.stdout:Packager : builder@centos.org 2024-08-18T15:08:11.578 INFO:teuthology.orchestra.run.smithi134.stdout:Vendor : CentOS 2024-08-18T15:08:11.578 INFO:teuthology.orchestra.run.smithi134.stdout:URL : http://www.gnu.org/software/grub/ 2024-08-18T15:08:11.578 INFO:teuthology.orchestra.run.smithi134.stdout:Summary : Support tools for GRUB. 2024-08-18T15:08:11.578 INFO:teuthology.orchestra.run.smithi134.stdout:Description : 2024-08-18T15:08:11.578 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:08:11.578 INFO:teuthology.orchestra.run.smithi134.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-08-18T15:08:11.578 INFO:teuthology.orchestra.run.smithi134.stdout:customizable bootloader with modular architecture. It supports a rich 2024-08-18T15:08:11.578 INFO:teuthology.orchestra.run.smithi134.stdout:variety of kernel formats, file systems, computer architectures and 2024-08-18T15:08:11.579 INFO:teuthology.orchestra.run.smithi134.stdout:hardware devices. 2024-08-18T15:08:11.579 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:08:11.579 INFO:teuthology.orchestra.run.smithi134.stdout:This subpackage provides tools for support of all platforms. 2024-08-18T15:08:11.580 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-08-18T15:08:11.580 INFO:teuthology.task.kernel:Updating grub on smithi134 to boot 5.14.0-480.el9.x86_64 2024-08-18T15:08:11.580 DEBUG:teuthology.orchestra.run.smithi134:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-08-18T15:08:12.217 INFO:teuthology.orchestra.run.smithi134.stderr:Generating grub configuration file ... 2024-08-18T15:08:13.730 INFO:teuthology.orchestra.run.smithi134.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-08-18T15:08:13.761 INFO:teuthology.orchestra.run.smithi134.stderr:done 2024-08-18T15:08:13.763 DEBUG:teuthology.orchestra.run.smithi134:> mktemp 2024-08-18T15:08:13.818 INFO:teuthology.orchestra.run.smithi134.stdout:/tmp/tmp.jXpY9jpAtT 2024-08-18T15:08:13.818 DEBUG:teuthology.orchestra.run.smithi134:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.jXpY9jpAtT 2024-08-18T15:08:13.884 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod 0666 /tmp/tmp.jXpY9jpAtT 2024-08-18T15:08:14.020 DEBUG:teuthology.orchestra.remote:smithi134:/tmp/tmp.jXpY9jpAtT is 6KB 2024-08-18T15:08:14.068 DEBUG:teuthology.orchestra.run.smithi134:> rm -fr /tmp/tmp.jXpY9jpAtT 2024-08-18T15:08:14.084 DEBUG:teuthology.orchestra.run.smithi134:> sudo /bin/ls /boot/loader/entries || true 2024-08-18T15:08:14.149 INFO:teuthology.orchestra.run.smithi134.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-08-18T15:08:14.149 INFO:teuthology.orchestra.run.smithi134.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-08-18T15:08:14.150 INFO:teuthology.orchestra.run.smithi134.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-08-18T15:08:14.150 INFO:teuthology.orchestra.run.smithi134.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-480.el9.x86_64.conf 2024-08-18T15:08:14.151 DEBUG:teuthology.orchestra.run.smithi134:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-480.el9.x86_64 2024-08-18T15:08:14.288 DEBUG:teuthology.orchestra.run.smithi134:> sudo shutdown -r now 2024-08-18T15:08:31.723 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: kernel-core-5.14.0-480.el9.x86_64 4/4 2024-08-18T15:08:31.763 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: kernel-modules-5.14.0-480.el9.x86_64 4/4 2024-08-18T15:08:32.152 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: kernel-5.14.0-480.el9.x86_64 4/4 2024-08-18T15:08:32.152 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : kernel-5.14.0-480.el9.x86_64 1/4 2024-08-18T15:08:32.152 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : kernel-core-5.14.0-480.el9.x86_64 2/4 2024-08-18T15:08:32.152 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : kernel-modules-5.14.0-480.el9.x86_64 3/4 2024-08-18T15:08:32.927 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : kernel-modules-core-5.14.0-480.el9.x86_64 4/4 2024-08-18T15:08:32.927 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:08:32.927 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-08-18T15:08:32.927 INFO:teuthology.orchestra.run.smithi193.stdout: kernel-5.14.0-480.el9.x86_64 2024-08-18T15:08:32.928 INFO:teuthology.orchestra.run.smithi193.stdout: kernel-core-5.14.0-480.el9.x86_64 2024-08-18T15:08:32.928 INFO:teuthology.orchestra.run.smithi193.stdout: kernel-modules-5.14.0-480.el9.x86_64 2024-08-18T15:08:32.928 INFO:teuthology.orchestra.run.smithi193.stdout: kernel-modules-core-5.14.0-480.el9.x86_64 2024-08-18T15:08:32.928 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:08:32.928 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-08-18T15:08:33.315 DEBUG:teuthology.orchestra.run.smithi193:> rpm -q kernel | sort -rV | head -n 1 2024-08-18T15:08:33.391 INFO:teuthology.orchestra.run.smithi193.stdout:kernel-5.14.0-480.el9.x86_64 2024-08-18T15:08:33.392 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-480.el9.x86_64 2024-08-18T15:08:33.393 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-480.el9.x86_64 2024-08-18T15:08:33.393 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-08-18T15:08:33.393 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-08-18T15:08:33.393 INFO:teuthology.task.kernel:Installing distro kernel on c2.mon.a... 2024-08-18T15:08:33.393 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi193.front.sepia.ceph.com, path=None, version=distro) 2024-08-18T15:08:33.393 DEBUG:teuthology.orchestra.run.smithi193:> rpm -q kernel | sort -rV | head -n 1 2024-08-18T15:08:33.449 INFO:teuthology.orchestra.run.smithi193.stdout:kernel-5.14.0-480.el9.x86_64 2024-08-18T15:08:33.449 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-480.el9.x86_64 2024-08-18T15:08:33.449 DEBUG:teuthology.orchestra.run.smithi193:> sudo rpm -qi grub2-tools 2024-08-18T15:08:33.488 INFO:teuthology.orchestra.run.smithi193.stdout:Name : grub2-tools 2024-08-18T15:08:33.489 INFO:teuthology.orchestra.run.smithi193.stdout:Epoch : 1 2024-08-18T15:08:33.489 INFO:teuthology.orchestra.run.smithi193.stdout:Version : 2.06 2024-08-18T15:08:33.489 INFO:teuthology.orchestra.run.smithi193.stdout:Release : 68.el9 2024-08-18T15:08:33.489 INFO:teuthology.orchestra.run.smithi193.stdout:Architecture: x86_64 2024-08-18T15:08:33.489 INFO:teuthology.orchestra.run.smithi193.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-08-18T15:08:33.489 INFO:teuthology.orchestra.run.smithi193.stdout:Group : Unspecified 2024-08-18T15:08:33.489 INFO:teuthology.orchestra.run.smithi193.stdout:Size : 8264254 2024-08-18T15:08:33.489 INFO:teuthology.orchestra.run.smithi193.stdout:License : GPLv3+ 2024-08-18T15:08:33.489 INFO:teuthology.orchestra.run.smithi193.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-08-18T15:08:33.489 INFO:teuthology.orchestra.run.smithi193.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-08-18T15:08:33.489 INFO:teuthology.orchestra.run.smithi193.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-08-18T15:08:33.489 INFO:teuthology.orchestra.run.smithi193.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-08-18T15:08:33.489 INFO:teuthology.orchestra.run.smithi193.stdout:Packager : builder@centos.org 2024-08-18T15:08:33.489 INFO:teuthology.orchestra.run.smithi193.stdout:Vendor : CentOS 2024-08-18T15:08:33.490 INFO:teuthology.orchestra.run.smithi193.stdout:URL : http://www.gnu.org/software/grub/ 2024-08-18T15:08:33.490 INFO:teuthology.orchestra.run.smithi193.stdout:Summary : Support tools for GRUB. 2024-08-18T15:08:33.490 INFO:teuthology.orchestra.run.smithi193.stdout:Description : 2024-08-18T15:08:33.490 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:08:33.490 INFO:teuthology.orchestra.run.smithi193.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-08-18T15:08:33.490 INFO:teuthology.orchestra.run.smithi193.stdout:customizable bootloader with modular architecture. It supports a rich 2024-08-18T15:08:33.490 INFO:teuthology.orchestra.run.smithi193.stdout:variety of kernel formats, file systems, computer architectures and 2024-08-18T15:08:33.490 INFO:teuthology.orchestra.run.smithi193.stdout:hardware devices. 2024-08-18T15:08:33.490 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:08:33.490 INFO:teuthology.orchestra.run.smithi193.stdout:This subpackage provides tools for support of all platforms. 2024-08-18T15:08:33.491 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-08-18T15:08:33.492 INFO:teuthology.task.kernel:Updating grub on smithi193 to boot 5.14.0-480.el9.x86_64 2024-08-18T15:08:33.492 DEBUG:teuthology.orchestra.run.smithi193:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-08-18T15:08:34.131 INFO:teuthology.orchestra.run.smithi193.stderr:Generating grub configuration file ... 2024-08-18T15:08:35.545 INFO:teuthology.orchestra.run.smithi193.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-08-18T15:08:35.581 INFO:teuthology.orchestra.run.smithi193.stderr:done 2024-08-18T15:08:35.584 DEBUG:teuthology.orchestra.run.smithi193:> mktemp 2024-08-18T15:08:35.599 INFO:teuthology.orchestra.run.smithi193.stdout:/tmp/tmp.YreTW7vhqL 2024-08-18T15:08:35.599 DEBUG:teuthology.orchestra.run.smithi193:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.YreTW7vhqL 2024-08-18T15:08:35.664 DEBUG:teuthology.orchestra.run.smithi193:> sudo chmod 0666 /tmp/tmp.YreTW7vhqL 2024-08-18T15:08:35.804 DEBUG:teuthology.orchestra.remote:smithi193:/tmp/tmp.YreTW7vhqL is 6KB 2024-08-18T15:08:35.853 DEBUG:teuthology.orchestra.run.smithi193:> rm -fr /tmp/tmp.YreTW7vhqL 2024-08-18T15:08:35.867 DEBUG:teuthology.orchestra.run.smithi193:> sudo /bin/ls /boot/loader/entries || true 2024-08-18T15:08:35.929 INFO:teuthology.orchestra.run.smithi193.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-08-18T15:08:35.929 INFO:teuthology.orchestra.run.smithi193.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-08-18T15:08:35.929 INFO:teuthology.orchestra.run.smithi193.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-08-18T15:08:35.929 INFO:teuthology.orchestra.run.smithi193.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-480.el9.x86_64.conf 2024-08-18T15:08:35.930 DEBUG:teuthology.orchestra.run.smithi193:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-480.el9.x86_64 2024-08-18T15:08:36.091 DEBUG:teuthology.orchestra.run.smithi193:> sudo shutdown -r now 2024-08-18T15:08:44.296 INFO:teuthology.task.kernel:Checking client c1.mon.a for new kernel version... 2024-08-18T15:08:44.297 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-08-18T15:08:44.297 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-18T15:09:06.114 INFO:teuthology.task.kernel:Checking client c2.mon.a for new kernel version... 2024-08-18T15:09:06.115 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi193.front.sepia.ceph.com' 2024-08-18T15:09:06.116 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi193.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-18T15:09:44.357 DEBUG:teuthology.orchestra.remote:timed out 2024-08-18T15:09:53.366 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-08-18T15:09:53.368 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-18T15:09:53.717 DEBUG:teuthology.orchestra.run.smithi134:> true 2024-08-18T15:09:54.281 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-08-18T15:09:54.282 INFO:teuthology.task.kernel:Checking kernel version of c1.mon.a, want "5.14.0-480.el9.x86_64"... 2024-08-18T15:09:54.282 DEBUG:teuthology.orchestra.run.smithi134:> uname -r 2024-08-18T15:09:54.297 INFO:teuthology.orchestra.run.smithi134.stdout:5.14.0-480.el9.x86_64 2024-08-18T15:09:54.297 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-480.el9.x86_64 vs 5.14.0-480.el9.x86_64 2024-08-18T15:09:54.297 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-08-18T15:09:54.297 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-08-18T15:09:55.298 INFO:teuthology.task.kernel:Enabling kdb on c1.mon.a... 2024-08-18T15:09:55.299 DEBUG:teuthology.orchestra.run.smithi134:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-08-18T15:09:55.368 INFO:teuthology.orchestra.run.smithi134.stdout:ttyS1 2024-08-18T15:09:55.402 DEBUG:teuthology.parallel:result is None 2024-08-18T15:10:06.144 DEBUG:teuthology.orchestra.remote:timed out 2024-08-18T15:10:15.145 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi193.front.sepia.ceph.com' 2024-08-18T15:10:15.145 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi193.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-18T15:10:15.528 DEBUG:teuthology.orchestra.run.smithi193:> true 2024-08-18T15:10:15.963 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi193.front.sepia.ceph.com' 2024-08-18T15:10:15.963 INFO:teuthology.task.kernel:Checking kernel version of c2.mon.a, want "5.14.0-480.el9.x86_64"... 2024-08-18T15:10:15.964 DEBUG:teuthology.orchestra.run.smithi193:> uname -r 2024-08-18T15:10:15.978 INFO:teuthology.orchestra.run.smithi193.stdout:5.14.0-480.el9.x86_64 2024-08-18T15:10:15.979 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-480.el9.x86_64 vs 5.14.0-480.el9.x86_64 2024-08-18T15:10:15.979 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-08-18T15:10:15.979 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-08-18T15:10:16.980 INFO:teuthology.task.kernel:Enabling kdb on c2.mon.a... 2024-08-18T15:10:16.981 DEBUG:teuthology.orchestra.run.smithi193:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-08-18T15:10:17.047 INFO:teuthology.orchestra.run.smithi193.stdout:ttyS1 2024-08-18T15:10:17.081 DEBUG:teuthology.parallel:result is None 2024-08-18T15:10:17.081 INFO:teuthology.run_tasks:Running task internal.base... 2024-08-18T15:10:17.092 INFO:teuthology.task.internal:Creating test directory... 2024-08-18T15:10:17.092 DEBUG:teuthology.orchestra.run.smithi134:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-08-18T15:10:17.096 DEBUG:teuthology.orchestra.run.smithi193:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-08-18T15:10:17.113 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-08-18T15:10:17.124 INFO:teuthology.run_tasks:Running task internal.archive... 2024-08-18T15:10:17.171 INFO:teuthology.task.internal:Creating archive directory... 2024-08-18T15:10:17.172 DEBUG:teuthology.orchestra.run.smithi134:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-08-18T15:10:17.174 DEBUG:teuthology.orchestra.run.smithi193:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-08-18T15:10:17.256 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-08-18T15:10:17.266 INFO:teuthology.task.internal:Enabling coredump saving... 2024-08-18T15:10:17.266 DEBUG:teuthology.orchestra.run.smithi134:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-08-18T15:10:17.269 DEBUG:teuthology.orchestra.run.smithi193:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-08-18T15:10:17.299 INFO:teuthology.orchestra.run.smithi134.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-18T15:10:17.300 INFO:teuthology.orchestra.run.smithi193.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-18T15:10:17.311 INFO:teuthology.orchestra.run.smithi193.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-18T15:10:17.311 INFO:teuthology.orchestra.run.smithi134.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-18T15:10:17.313 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-08-18T15:10:17.326 INFO:teuthology.task.internal:Configuring sudo... 2024-08-18T15:10:17.326 DEBUG:teuthology.orchestra.run.smithi134:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-08-18T15:10:17.356 DEBUG:teuthology.orchestra.run.smithi193:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-08-18T15:10:17.382 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-08-18T15:10:17.395 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-08-18T15:10:17.395 DEBUG:teuthology.orchestra.run.smithi134:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-08-18T15:10:17.425 DEBUG:teuthology.orchestra.run.smithi193:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-08-18T15:10:17.440 DEBUG:teuthology.orchestra.run.smithi134:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-18T15:10:17.523 DEBUG:teuthology.orchestra.run.smithi134:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-18T15:10:17.574 DEBUG:teuthology.orchestra.run.smithi134:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-18T15:10:17.650 DEBUG:teuthology.orchestra.run.smithi134:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-18T15:10:17.698 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-08-18T15:10:17.698 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-08-18T15:10:17.764 DEBUG:teuthology.orchestra.run.smithi193:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-18T15:10:17.801 DEBUG:teuthology.orchestra.run.smithi193:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-18T15:10:17.862 DEBUG:teuthology.orchestra.run.smithi193:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-18T15:10:17.947 DEBUG:teuthology.orchestra.run.smithi193:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-18T15:10:17.992 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-08-18T15:10:17.992 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-08-18T15:10:18.060 DEBUG:teuthology.orchestra.run.smithi134:> sudo service rsyslog restart 2024-08-18T15:10:18.062 DEBUG:teuthology.orchestra.run.smithi193:> sudo service rsyslog restart 2024-08-18T15:10:18.107 INFO:teuthology.orchestra.run.smithi134.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-08-18T15:10:18.154 INFO:teuthology.orchestra.run.smithi193.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-08-18T15:10:18.607 INFO:teuthology.run_tasks:Running task internal.timer... 2024-08-18T15:10:18.615 INFO:teuthology.task.internal:Starting timer... 2024-08-18T15:10:18.615 INFO:teuthology.run_tasks:Running task pcp... 2024-08-18T15:10:18.653 INFO:teuthology.run_tasks:Running task selinux... 2024-08-18T15:10:18.735 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-08-18T15:10:18.735 DEBUG:teuthology.orchestra.run.smithi134:> sudo service auditd rotate 2024-08-18T15:10:18.813 INFO:teuthology.orchestra.run.smithi134.stdout:Rotating logs: 2024-08-18T15:10:18.815 DEBUG:teuthology.orchestra.run.smithi193:> sudo service auditd rotate 2024-08-18T15:10:18.922 INFO:teuthology.orchestra.run.smithi193.stdout:Rotating logs: 2024-08-18T15:10:18.924 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-08-18T15:10:18.924 DEBUG:teuthology.orchestra.run.smithi134:> /usr/sbin/getenforce 2024-08-18T15:10:18.956 INFO:teuthology.orchestra.run.smithi134.stdout:Permissive 2024-08-18T15:10:18.956 DEBUG:teuthology.orchestra.run.smithi193:> /usr/sbin/getenforce 2024-08-18T15:10:18.979 INFO:teuthology.orchestra.run.smithi193.stdout:Permissive 2024-08-18T15:10:18.979 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi134.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi193.front.sepia.ceph.com': 'permissive'} 2024-08-18T15:10:18.981 DEBUG:teuthology.orchestra.run.smithi134:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-08-18T15:10:19.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-18T15:10:19.026 DEBUG:teuthology.orchestra.run.smithi193:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-08-18T15:10:19.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-18T15:10:19.052 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-08-18T15:10:19.053 DEBUG:teuthology.orchestra.run.smithi134:> sudo /usr/sbin/setenforce permissive 2024-08-18T15:10:19.094 DEBUG:teuthology.orchestra.run.smithi193:> sudo /usr/sbin/setenforce permissive 2024-08-18T15:10:19.232 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-08-18T15:10:19.241 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-08-18T15:10:19.252 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-08-18T15:10:19.253 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi134.front.sepia.ceph.com,smithi193.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-08-18T15:15:00.963 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi134.front.sepia.ceph.com'), Remote(name='ubuntu@smithi193.front.sepia.ceph.com')] 2024-08-18T15:15:00.965 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-08-18T15:15:00.966 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-18T15:15:01.044 DEBUG:teuthology.orchestra.run.smithi134:> true 2024-08-18T15:15:01.127 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-08-18T15:15:01.127 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi193.front.sepia.ceph.com' 2024-08-18T15:15:01.128 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi193.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-18T15:15:01.208 DEBUG:teuthology.orchestra.run.smithi193:> true 2024-08-18T15:15:01.290 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi193.front.sepia.ceph.com' 2024-08-18T15:15:01.290 INFO:teuthology.run_tasks:Running task clock... 2024-08-18T15:15:01.300 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-08-18T15:15:01.300 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-08-18T15:15:01.300 DEBUG:teuthology.orchestra.run.smithi134:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-18T15:15:01.303 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-08-18T15:15:01.303 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-18T15:15:01.330 INFO:teuthology.orchestra.run.smithi134.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-08-18T15:15:01.344 INFO:teuthology.orchestra.run.smithi134.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-08-18T15:15:01.359 INFO:teuthology.orchestra.run.smithi193.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-08-18T15:15:01.368 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: ntpd: command not found 2024-08-18T15:15:01.374 INFO:teuthology.orchestra.run.smithi193.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-08-18T15:15:01.379 INFO:teuthology.orchestra.run.smithi134.stdout:506 Cannot talk to daemon 2024-08-18T15:15:01.393 INFO:teuthology.orchestra.run.smithi134.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-08-18T15:15:01.398 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: ntpd: command not found 2024-08-18T15:15:01.407 INFO:teuthology.orchestra.run.smithi134.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-08-18T15:15:01.409 INFO:teuthology.orchestra.run.smithi193.stdout:506 Cannot talk to daemon 2024-08-18T15:15:01.424 INFO:teuthology.orchestra.run.smithi193.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-08-18T15:15:01.439 INFO:teuthology.orchestra.run.smithi193.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-08-18T15:15:01.471 INFO:teuthology.orchestra.run.smithi134.stderr:bash: line 1: ntpq: command not found 2024-08-18T15:15:01.473 INFO:teuthology.orchestra.run.smithi134.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-08-18T15:15:01.474 INFO:teuthology.orchestra.run.smithi134.stdout:=============================================================================== 2024-08-18T15:15:01.474 INFO:teuthology.orchestra.run.smithi134.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-18T15:15:01.474 INFO:teuthology.orchestra.run.smithi134.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-18T15:15:01.474 INFO:teuthology.orchestra.run.smithi134.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-18T15:15:01.474 INFO:teuthology.orchestra.run.smithi134.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-18T15:15:01.493 INFO:teuthology.orchestra.run.smithi193.stderr:bash: line 1: ntpq: command not found 2024-08-18T15:15:01.496 INFO:teuthology.orchestra.run.smithi193.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-08-18T15:15:01.496 INFO:teuthology.orchestra.run.smithi193.stdout:=============================================================================== 2024-08-18T15:15:01.496 INFO:teuthology.orchestra.run.smithi193.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-18T15:15:01.496 INFO:teuthology.orchestra.run.smithi193.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-18T15:15:01.496 INFO:teuthology.orchestra.run.smithi193.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-18T15:15:01.497 INFO:teuthology.orchestra.run.smithi193.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-18T15:15:01.498 INFO:teuthology.run_tasks:Running task install... 2024-08-18T15:15:01.507 DEBUG:teuthology.task.install:project ceph 2024-08-18T15:15:01.507 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '03f4c3e85ce172201896147a7aef87ba256187f2'}} 2024-08-18T15:15:01.508 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '03f4c3e85ce172201896147a7aef87ba256187f2'} 2024-08-18T15:15:01.508 INFO:teuthology.task.install:Using flavor: default 2024-08-18T15:15:01.514 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-08-18T15:15:01.514 INFO:teuthology.task.install:extra packages: [] 2024-08-18T15:15:01.515 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '03f4c3e85ce172201896147a7aef87ba256187f2', 'tag': None, 'wait_for_package': False} 2024-08-18T15:15:01.515 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=03f4c3e85ce172201896147a7aef87ba256187f2 2024-08-18T15:15:01.517 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '03f4c3e85ce172201896147a7aef87ba256187f2', 'tag': None, 'wait_for_package': False} 2024-08-18T15:15:01.517 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=03f4c3e85ce172201896147a7aef87ba256187f2 2024-08-18T15:15:01.663 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/reef/03f4c3e85ce172201896147a7aef87ba256187f2/centos/9/flavors/default/ 2024-08-18T15:15:01.663 INFO:teuthology.task.install.rpm:Package version is 18.2.4-717.g03f4c3e8 2024-08-18T15:15:01.678 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/reef/03f4c3e85ce172201896147a7aef87ba256187f2/centos/9/flavors/default/ 2024-08-18T15:15:01.678 INFO:teuthology.task.install.rpm:Package version is 18.2.4-717.g03f4c3e8 2024-08-18T15:15:01.773 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/reef/03f4c3e85ce172201896147a7aef87ba256187f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/reef/03f4c3e85ce172201896147a7aef87ba256187f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/reef/03f4c3e85ce172201896147a7aef87ba256187f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-08-18T15:15:01.774 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-08-18T15:15:01.774 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-08-18T15:15:01.791 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/reef/03f4c3e85ce172201896147a7aef87ba256187f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/reef/03f4c3e85ce172201896147a7aef87ba256187f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/reef/03f4c3e85ce172201896147a7aef87ba256187f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-08-18T15:15:01.791 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-08-18T15:15:01.791 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-08-18T15:15:01.800 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-08-18T15:15:01.800 DEBUG:teuthology.orchestra.run.smithi134:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/03f4c3e85ce172201896147a7aef87ba256187f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-08-18T15:15:01.817 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-08-18T15:15:01.817 DEBUG:teuthology.orchestra.run.smithi193:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/03f4c3e85ce172201896147a7aef87ba256187f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-08-18T15:15:01.866 DEBUG:teuthology.orchestra.run.smithi134:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-08-18T15:15:01.883 DEBUG:teuthology.orchestra.run.smithi193:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-08-18T15:15:01.942 DEBUG:teuthology.orchestra.run.smithi134:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-08-18T15:15:01.958 DEBUG:teuthology.orchestra.run.smithi193:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-08-18T15:15:02.006 INFO:teuthology.orchestra.run.smithi134.stdout:check_obsoletes = 1 2024-08-18T15:15:02.008 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum clean all 2024-08-18T15:15:02.022 INFO:teuthology.orchestra.run.smithi193.stdout:check_obsoletes = 1 2024-08-18T15:15:02.024 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum clean all 2024-08-18T15:15:02.241 INFO:teuthology.orchestra.run.smithi134.stdout:45 files removed 2024-08-18T15:15:02.256 INFO:teuthology.orchestra.run.smithi193.stdout:45 files removed 2024-08-18T15:15:02.268 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-radosgw 2024-08-18T15:15:02.283 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph-radosgw 2024-08-18T15:15:02.858 INFO:teuthology.orchestra.run.smithi134.stdout:ceph packages for x86_64 247 kB/s | 77 kB 00:00 2024-08-18T15:15:02.881 INFO:teuthology.orchestra.run.smithi193.stdout:ceph packages for x86_64 243 kB/s | 77 kB 00:00 2024-08-18T15:15:03.184 INFO:teuthology.orchestra.run.smithi134.stdout:ceph noarch packages 42 kB/s | 11 kB 00:00 2024-08-18T15:15:03.198 INFO:teuthology.orchestra.run.smithi193.stdout:ceph noarch packages 43 kB/s | 11 kB 00:00 2024-08-18T15:15:03.509 INFO:teuthology.orchestra.run.smithi134.stdout:ceph source packages 6.7 kB/s | 1.9 kB 00:00 2024-08-18T15:15:03.524 INFO:teuthology.orchestra.run.smithi193.stdout:ceph source packages 6.7 kB/s | 1.9 kB 00:00 2024-08-18T15:15:04.026 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - BaseOS 17 MB/s | 8.2 MB 00:00 2024-08-18T15:15:04.290 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.2 MB 00:00 2024-08-18T15:15:06.019 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - AppStream 22 MB/s | 20 MB 00:00 2024-08-18T15:15:07.028 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS Stream 9 - AppStream 12 MB/s | 20 MB 00:01 2024-08-18T15:15:10.636 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - CRB 7.5 MB/s | 6.6 MB 00:00 2024-08-18T15:15:12.029 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS Stream 9 - CRB 5.1 MB/s | 6.6 MB 00:01 2024-08-18T15:15:12.254 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - Extras packages 77 kB/s | 18 kB 00:00 2024-08-18T15:15:13.663 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS Stream 9 - Extras packages 63 kB/s | 18 kB 00:00 2024-08-18T15:15:13.758 INFO:teuthology.orchestra.run.smithi134.stdout:Extra Packages for Enterprise Linux 16 MB/s | 22 MB 00:01 2024-08-18T15:15:14.809 INFO:teuthology.orchestra.run.smithi193.stdout:Extra Packages for Enterprise Linux 22 MB/s | 22 MB 00:01 2024-08-18T15:15:19.072 INFO:teuthology.orchestra.run.smithi134.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-08-18T15:15:20.348 INFO:teuthology.orchestra.run.smithi193.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-08-18T15:15:21.406 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-08-18T15:15:21.409 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-08-18T15:15:21.409 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-08-18T15:15:21.409 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-08-18T15:15:21.409 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-08-18T15:15:21.409 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-radosgw x86_64 2:18.2.4-717.g03f4c3e8.el9 ceph 7.7 M 2024-08-18T15:15:21.409 INFO:teuthology.orchestra.run.smithi134.stdout:Upgrading: 2024-08-18T15:15:21.409 INFO:teuthology.orchestra.run.smithi134.stdout: librados2 x86_64 2:18.2.4-717.g03f4c3e8.el9 ceph 3.3 M 2024-08-18T15:15:21.410 INFO:teuthology.orchestra.run.smithi134.stdout: librbd1 x86_64 2:18.2.4-717.g03f4c3e8.el9 ceph 3.0 M 2024-08-18T15:15:21.410 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-08-18T15:15:21.410 INFO:teuthology.orchestra.run.smithi134.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-08-18T15:15:21.410 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-base x86_64 2:18.2.4-717.g03f4c3e8.el9 ceph 5.1 M 2024-08-18T15:15:21.410 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-common x86_64 2:18.2.4-717.g03f4c3e8.el9 ceph 18 M 2024-08-18T15:15:21.410 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-selinux x86_64 2:18.2.4-717.g03f4c3e8.el9 ceph 25 k 2024-08-18T15:15:21.410 INFO:teuthology.orchestra.run.smithi134.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-08-18T15:15:21.410 INFO:teuthology.orchestra.run.smithi134.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-08-18T15:15:21.410 INFO:teuthology.orchestra.run.smithi134.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-08-18T15:15:21.410 INFO:teuthology.orchestra.run.smithi134.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-08-18T15:15:21.410 INFO:teuthology.orchestra.run.smithi134.stdout: libcephfs2 x86_64 2:18.2.4-717.g03f4c3e8.el9 ceph 713 k 2024-08-18T15:15:21.410 INFO:teuthology.orchestra.run.smithi134.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-08-18T15:15:21.410 INFO:teuthology.orchestra.run.smithi134.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-08-18T15:15:21.411 INFO:teuthology.orchestra.run.smithi134.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-08-18T15:15:21.411 INFO:teuthology.orchestra.run.smithi134.stdout: libradosstriper1 x86_64 2:18.2.4-717.g03f4c3e8.el9 ceph 477 k 2024-08-18T15:15:21.411 INFO:teuthology.orchestra.run.smithi134.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-08-18T15:15:21.411 INFO:teuthology.orchestra.run.smithi134.stdout: librgw2 x86_64 2:18.2.4-717.g03f4c3e8.el9 ceph 4.5 M 2024-08-18T15:15:21.411 INFO:teuthology.orchestra.run.smithi134.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-08-18T15:15:21.411 INFO:teuthology.orchestra.run.smithi134.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-08-18T15:15:21.411 INFO:teuthology.orchestra.run.smithi134.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-08-18T15:15:21.411 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ceph-argparse x86_64 2:18.2.4-717.g03f4c3e8.el9 ceph 45 k 2024-08-18T15:15:21.411 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ceph-common x86_64 2:18.2.4-717.g03f4c3e8.el9 ceph 129 k 2024-08-18T15:15:21.411 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cephfs x86_64 2:18.2.4-717.g03f4c3e8.el9 ceph 188 k 2024-08-18T15:15:21.411 INFO:teuthology.orchestra.run.smithi134.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-08-18T15:15:21.411 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rados x86_64 2:18.2.4-717.g03f4c3e8.el9 ceph 370 k 2024-08-18T15:15:21.411 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rbd x86_64 2:18.2.4-717.g03f4c3e8.el9 ceph 348 k 2024-08-18T15:15:21.411 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rgw x86_64 2:18.2.4-717.g03f4c3e8.el9 ceph 116 k 2024-08-18T15:15:21.412 INFO:teuthology.orchestra.run.smithi134.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-08-18T15:15:21.412 INFO:teuthology.orchestra.run.smithi134.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-08-18T15:15:21.412 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:15:21.412 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-08-18T15:15:21.412 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-08-18T15:15:21.412 INFO:teuthology.orchestra.run.smithi134.stdout:Install 28 Packages 2024-08-18T15:15:21.412 INFO:teuthology.orchestra.run.smithi134.stdout:Upgrade 2 Packages 2024-08-18T15:15:21.412 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:15:21.413 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 53 M 2024-08-18T15:15:21.413 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-08-18T15:15:22.664 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-08-18T15:15:22.680 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-08-18T15:15:22.681 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repository Size 2024-08-18T15:15:22.681 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-08-18T15:15:22.681 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-08-18T15:15:22.681 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-radosgw x86_64 2:18.2.4-717.g03f4c3e8.el9 ceph 7.7 M 2024-08-18T15:15:22.681 INFO:teuthology.orchestra.run.smithi193.stdout:Upgrading: 2024-08-18T15:15:22.681 INFO:teuthology.orchestra.run.smithi193.stdout: librados2 x86_64 2:18.2.4-717.g03f4c3e8.el9 ceph 3.3 M 2024-08-18T15:15:22.681 INFO:teuthology.orchestra.run.smithi193.stdout: librbd1 x86_64 2:18.2.4-717.g03f4c3e8.el9 ceph 3.0 M 2024-08-18T15:15:22.681 INFO:teuthology.orchestra.run.smithi193.stdout:Installing dependencies: 2024-08-18T15:15:22.681 INFO:teuthology.orchestra.run.smithi193.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-08-18T15:15:22.681 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-base x86_64 2:18.2.4-717.g03f4c3e8.el9 ceph 5.1 M 2024-08-18T15:15:22.681 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-common x86_64 2:18.2.4-717.g03f4c3e8.el9 ceph 18 M 2024-08-18T15:15:22.681 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-selinux x86_64 2:18.2.4-717.g03f4c3e8.el9 ceph 25 k 2024-08-18T15:15:22.681 INFO:teuthology.orchestra.run.smithi193.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-08-18T15:15:22.682 INFO:teuthology.orchestra.run.smithi193.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-08-18T15:15:22.682 INFO:teuthology.orchestra.run.smithi193.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-08-18T15:15:22.682 INFO:teuthology.orchestra.run.smithi193.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-08-18T15:15:22.682 INFO:teuthology.orchestra.run.smithi193.stdout: libcephfs2 x86_64 2:18.2.4-717.g03f4c3e8.el9 ceph 713 k 2024-08-18T15:15:22.682 INFO:teuthology.orchestra.run.smithi193.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-08-18T15:15:22.682 INFO:teuthology.orchestra.run.smithi193.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-08-18T15:15:22.682 INFO:teuthology.orchestra.run.smithi193.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-08-18T15:15:22.682 INFO:teuthology.orchestra.run.smithi193.stdout: libradosstriper1 x86_64 2:18.2.4-717.g03f4c3e8.el9 ceph 477 k 2024-08-18T15:15:22.682 INFO:teuthology.orchestra.run.smithi193.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-08-18T15:15:22.682 INFO:teuthology.orchestra.run.smithi193.stdout: librgw2 x86_64 2:18.2.4-717.g03f4c3e8.el9 ceph 4.5 M 2024-08-18T15:15:22.682 INFO:teuthology.orchestra.run.smithi193.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-08-18T15:15:22.682 INFO:teuthology.orchestra.run.smithi193.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-08-18T15:15:22.682 INFO:teuthology.orchestra.run.smithi193.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-08-18T15:15:22.682 INFO:teuthology.orchestra.run.smithi193.stdout: python3-ceph-argparse x86_64 2:18.2.4-717.g03f4c3e8.el9 ceph 45 k 2024-08-18T15:15:22.683 INFO:teuthology.orchestra.run.smithi193.stdout: python3-ceph-common x86_64 2:18.2.4-717.g03f4c3e8.el9 ceph 129 k 2024-08-18T15:15:22.683 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cephfs x86_64 2:18.2.4-717.g03f4c3e8.el9 ceph 188 k 2024-08-18T15:15:22.683 INFO:teuthology.orchestra.run.smithi193.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-08-18T15:15:22.683 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rados x86_64 2:18.2.4-717.g03f4c3e8.el9 ceph 370 k 2024-08-18T15:15:22.683 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rbd x86_64 2:18.2.4-717.g03f4c3e8.el9 ceph 348 k 2024-08-18T15:15:22.683 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rgw x86_64 2:18.2.4-717.g03f4c3e8.el9 ceph 116 k 2024-08-18T15:15:22.683 INFO:teuthology.orchestra.run.smithi193.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-08-18T15:15:22.683 INFO:teuthology.orchestra.run.smithi193.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-08-18T15:15:22.683 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:15:22.683 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-08-18T15:15:22.683 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-08-18T15:15:22.683 INFO:teuthology.orchestra.run.smithi193.stdout:Install 28 Packages 2024-08-18T15:15:22.683 INFO:teuthology.orchestra.run.smithi193.stdout:Upgrade 2 Packages 2024-08-18T15:15:22.683 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:15:22.684 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 53 M 2024-08-18T15:15:22.684 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-08-18T15:15:22.806 INFO:teuthology.orchestra.run.smithi134.stdout:(1/30): ceph-base-18.2.4-717.g03f4c3e8.el9.x86_ 7.3 MB/s | 5.1 MB 00:00 2024-08-18T15:15:22.848 INFO:teuthology.orchestra.run.smithi134.stdout:(2/30): ceph-selinux-18.2.4-717.g03f4c3e8.el9.x 603 kB/s | 25 kB 00:00 2024-08-18T15:15:22.933 INFO:teuthology.orchestra.run.smithi134.stdout:(3/30): ceph-radosgw-18.2.4-717.g03f4c3e8.el9.x 9.5 MB/s | 7.7 MB 00:00 2024-08-18T15:15:22.965 INFO:teuthology.orchestra.run.smithi134.stdout:(4/30): libcephfs2-18.2.4-717.g03f4c3e8.el9.x86 6.0 MB/s | 713 kB 00:00 2024-08-18T15:15:22.990 INFO:teuthology.orchestra.run.smithi134.stdout:(5/30): libradosstriper1-18.2.4-717.g03f4c3e8.e 8.0 MB/s | 477 kB 00:00 2024-08-18T15:15:23.049 INFO:teuthology.orchestra.run.smithi134.stdout:(6/30): python3-ceph-argparse-18.2.4-717.g03f4c 773 kB/s | 45 kB 00:00 2024-08-18T15:15:23.133 INFO:teuthology.orchestra.run.smithi134.stdout:(7/30): python3-ceph-common-18.2.4-717.g03f4c3e 1.5 MB/s | 129 kB 00:00 2024-08-18T15:15:23.308 INFO:teuthology.orchestra.run.smithi134.stdout:(8/30): ceph-common-18.2.4-717.g03f4c3e8.el9.x8 15 MB/s | 18 MB 00:01 2024-08-18T15:15:23.375 INFO:teuthology.orchestra.run.smithi134.stdout:(9/30): librgw2-18.2.4-717.g03f4c3e8.el9.x86_64 11 MB/s | 4.5 MB 00:00 2024-08-18T15:15:23.400 INFO:teuthology.orchestra.run.smithi134.stdout:(10/30): python3-cephfs-18.2.4-717.g03f4c3e8.el 702 kB/s | 188 kB 00:00 2024-08-18T15:15:23.434 INFO:teuthology.orchestra.run.smithi134.stdout:(11/30): python3-rados-18.2.4-717.g03f4c3e8.el9 2.9 MB/s | 370 kB 00:00 2024-08-18T15:15:23.459 INFO:teuthology.orchestra.run.smithi134.stdout:(12/30): python3-rbd-18.2.4-717.g03f4c3e8.el9.x 4.1 MB/s | 348 kB 00:00 2024-08-18T15:15:23.484 INFO:teuthology.orchestra.run.smithi134.stdout:(13/30): python3-rgw-18.2.4-717.g03f4c3e8.el9.x 1.4 MB/s | 116 kB 00:00 2024-08-18T15:15:23.626 INFO:teuthology.orchestra.run.smithi134.stdout:(14/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 319 kB/s | 45 kB 00:00 2024-08-18T15:15:23.652 INFO:teuthology.orchestra.run.smithi134.stdout:(15/30): boost-program-options-1.75.0-8.el9.x86 491 kB/s | 107 kB 00:00 2024-08-18T15:15:23.665 INFO:teuthology.orchestra.run.smithi193.stdout:(1/30): ceph-base-18.2.4-717.g03f4c3e8.el9.x86_ 9.5 MB/s | 5.1 MB 00:00 2024-08-18T15:15:23.677 INFO:teuthology.orchestra.run.smithi134.stdout:(16/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 736 kB/s | 160 kB 00:00 2024-08-18T15:15:23.727 INFO:teuthology.orchestra.run.smithi134.stdout:(17/30): python3-prettytable-0.7.2-27.el9.noarc 847 kB/s | 42 kB 00:00 2024-08-18T15:15:23.761 INFO:teuthology.orchestra.run.smithi134.stdout:(18/30): librdkafka-1.6.1-102.el9.x86_64.rpm 4.8 MB/s | 662 kB 00:00 2024-08-18T15:15:23.791 INFO:teuthology.orchestra.run.smithi193.stdout:(2/30): ceph-radosgw-18.2.4-717.g03f4c3e8.el9.x 12 MB/s | 7.7 MB 00:00 2024-08-18T15:15:23.811 INFO:teuthology.orchestra.run.smithi134.stdout:(19/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.8 MB/s | 292 kB 00:00 2024-08-18T15:15:23.816 INFO:teuthology.orchestra.run.smithi193.stdout:(3/30): ceph-selinux-18.2.4-717.g03f4c3e8.el9.x 167 kB/s | 25 kB 00:00 2024-08-18T15:15:23.858 INFO:teuthology.orchestra.run.smithi193.stdout:(4/30): libcephfs2-18.2.4-717.g03f4c3e8.el9.x86 10 MB/s | 713 kB 00:00 2024-08-18T15:15:23.883 INFO:teuthology.orchestra.run.smithi193.stdout:(5/30): libradosstriper1-18.2.4-717.g03f4c3e8.e 7.0 MB/s | 477 kB 00:00 2024-08-18T15:15:23.895 INFO:teuthology.orchestra.run.smithi134.stdout:(20/30): fmt-8.1.1-5.el9.x86_64.rpm 662 kB/s | 111 kB 00:00 2024-08-18T15:15:23.920 INFO:teuthology.orchestra.run.smithi134.stdout:(21/30): libarrow-doc-9.0.0-11.el9.noarch.rpm 1.1 MB/s | 26 kB 00:00 2024-08-18T15:15:23.945 INFO:teuthology.orchestra.run.smithi134.stdout:(22/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.9 MB/s | 309 kB 00:00 2024-08-18T15:15:23.950 INFO:teuthology.orchestra.run.smithi193.stdout:(6/30): python3-ceph-argparse-18.2.4-717.g03f4c 673 kB/s | 45 kB 00:00 2024-08-18T15:15:23.970 INFO:teuthology.orchestra.run.smithi134.stdout:(23/30): liboath-2.6.7-2.el9.x86_64.rpm 964 kB/s | 49 kB 00:00 2024-08-18T15:15:23.996 INFO:teuthology.orchestra.run.smithi134.stdout:(24/30): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-08-18T15:15:24.025 INFO:teuthology.orchestra.run.smithi193.stdout:(7/30): python3-ceph-common-18.2.4-717.g03f4c3e 1.7 MB/s | 129 kB 00:00 2024-08-18T15:15:24.029 INFO:teuthology.orchestra.run.smithi134.stdout:(25/30): re2-20211101-3.el9.x86_64.rpm 5.6 MB/s | 192 kB 00:00 2024-08-18T15:15:24.063 INFO:teuthology.orchestra.run.smithi134.stdout:(26/30): parquet-libs-9.0.0-11.el9.x86_64.rpm 8.9 MB/s | 839 kB 00:00 2024-08-18T15:15:24.092 INFO:teuthology.orchestra.run.smithi193.stdout:(8/30): librgw2-18.2.4-717.g03f4c3e8.el9.x86_64 19 MB/s | 4.5 MB 00:00 2024-08-18T15:15:24.118 INFO:teuthology.orchestra.run.smithi193.stdout:(9/30): python3-cephfs-18.2.4-717.g03f4c3e8.el9 2.0 MB/s | 188 kB 00:00 2024-08-18T15:15:24.163 INFO:teuthology.orchestra.run.smithi134.stdout:(27/30): thrift-0.15.0-2.el9.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2024-08-18T15:15:24.168 INFO:teuthology.orchestra.run.smithi193.stdout:(10/30): python3-rados-18.2.4-717.g03f4c3e8.el9 4.8 MB/s | 370 kB 00:00 2024-08-18T15:15:24.207 INFO:teuthology.orchestra.run.smithi193.stdout:(11/30): python3-rbd-18.2.4-717.g03f4c3e8.el9.x 3.8 MB/s | 348 kB 00:00 2024-08-18T15:15:24.227 INFO:teuthology.orchestra.run.smithi193.stdout:(12/30): python3-rgw-18.2.4-717.g03f4c3e8.el9.x 1.9 MB/s | 116 kB 00:00 2024-08-18T15:15:24.272 INFO:teuthology.orchestra.run.smithi134.stdout:(28/30): librados2-18.2.4-717.g03f4c3e8.el9.x86 16 MB/s | 3.3 MB 00:00 2024-08-18T15:15:24.347 INFO:teuthology.orchestra.run.smithi134.stdout:(29/30): libarrow-9.0.0-11.el9.x86_64.rpm 8.7 MB/s | 4.4 MB 00:00 2024-08-18T15:15:24.414 INFO:teuthology.orchestra.run.smithi134.stdout:(30/30): librbd1-18.2.4-717.g03f4c3e8.el9.x86_6 12 MB/s | 3.0 MB 00:00 2024-08-18T15:15:24.419 INFO:teuthology.orchestra.run.smithi193.stdout:(13/30): ceph-common-18.2.4-717.g03f4c3e8.el9.x 14 MB/s | 18 MB 00:01 2024-08-18T15:15:24.419 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-08-18T15:15:24.419 INFO:teuthology.orchestra.run.smithi134.stdout:Total 18 MB/s | 53 MB 00:03 2024-08-18T15:15:24.487 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-08-18T15:15:24.521 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-08-18T15:15:24.521 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-08-18T15:15:25.044 INFO:teuthology.orchestra.run.smithi193.stdout:(14/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 72 kB/s | 45 kB 00:00 2024-08-18T15:15:25.058 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-08-18T15:15:25.058 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-08-18T15:15:25.061 INFO:teuthology.orchestra.run.smithi193.stdout:(15/30): boost-program-options-1.75.0-8.el9.x86 125 kB/s | 107 kB 00:00 2024-08-18T15:15:25.095 INFO:teuthology.orchestra.run.smithi193.stdout:(16/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 185 kB/s | 160 kB 00:00 2024-08-18T15:15:25.703 INFO:teuthology.orchestra.run.smithi193.stdout:(17/30): python3-prettytable-0.7.2-27.el9.noarc 69 kB/s | 42 kB 00:00 2024-08-18T15:15:25.770 INFO:teuthology.orchestra.run.smithi193.stdout:(18/30): fmt-8.1.1-5.el9.x86_64.rpm 1.6 MB/s | 111 kB 00:00 2024-08-18T15:15:25.803 INFO:teuthology.orchestra.run.smithi193.stdout:(19/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 9.1 MB/s | 309 kB 00:00 2024-08-18T15:15:25.862 INFO:teuthology.orchestra.run.smithi193.stdout:(20/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 365 kB/s | 292 kB 00:00 2024-08-18T15:15:25.946 INFO:teuthology.orchestra.run.smithi193.stdout:(21/30): librdkafka-1.6.1-102.el9.x86_64.rpm 735 kB/s | 662 kB 00:00 2024-08-18T15:15:25.963 INFO:teuthology.orchestra.run.smithi193.stdout:(22/30): libarrow-doc-9.0.0-11.el9.noarch.rpm 259 kB/s | 26 kB 00:00 2024-08-18T15:15:26.021 INFO:teuthology.orchestra.run.smithi193.stdout:(23/30): libunwind-1.6.2-1.el9.x86_64.rpm 2.0 MB/s | 67 kB 00:00 2024-08-18T15:15:26.055 INFO:teuthology.orchestra.run.smithi193.stdout:(24/30): liboath-2.6.7-2.el9.x86_64.rpm 447 kB/s | 49 kB 00:00 2024-08-18T15:15:26.122 INFO:teuthology.orchestra.run.smithi193.stdout:(25/30): libarrow-9.0.0-11.el9.x86_64.rpm 14 MB/s | 4.4 MB 00:00 2024-08-18T15:15:26.147 INFO:teuthology.orchestra.run.smithi193.stdout:(26/30): re2-20211101-3.el9.x86_64.rpm 2.0 MB/s | 192 kB 00:00 2024-08-18T15:15:26.206 INFO:teuthology.orchestra.run.smithi193.stdout:(27/30): parquet-libs-9.0.0-11.el9.x86_64.rpm 4.5 MB/s | 839 kB 00:00 2024-08-18T15:15:26.248 INFO:teuthology.orchestra.run.smithi193.stdout:(28/30): thrift-0.15.0-2.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-08-18T15:15:26.367 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-08-18T15:15:26.454 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2024-08-18T15:15:26.456 INFO:teuthology.orchestra.run.smithi193.stdout:(29/30): librbd1-18.2.4-717.g03f4c3e8.el9.x86_6 12 MB/s | 3.0 MB 00:00 2024-08-18T15:15:26.495 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/32 2024-08-18T15:15:26.523 INFO:teuthology.orchestra.run.smithi193.stdout:(30/30): librados2-18.2.4-717.g03f4c3e8.el9.x86 8.8 MB/s | 3.3 MB 00:00 2024-08-18T15:15:26.529 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-08-18T15:15:26.529 INFO:teuthology.orchestra.run.smithi193.stdout:Total 14 MB/s | 53 MB 00:03 2024-08-18T15:15:26.559 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/32 2024-08-18T15:15:26.600 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-08-18T15:15:26.634 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-08-18T15:15:26.635 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-08-18T15:15:26.872 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2024-08-18T15:15:26.977 INFO:teuthology.orchestra.run.smithi134.stdout: Upgrading : librados2-2:18.2.4-717.g03f4c3e8.el9.x86_64 5/32 2024-08-18T15:15:27.073 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librados2-2:18.2.4-717.g03f4c3e8.el9.x86_64 5/32 2024-08-18T15:15:27.145 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-08-18T15:15:27.145 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-08-18T15:15:27.161 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-rados-2:18.2.4-717.g03f4c3e8.el9.x86_64 6/32 2024-08-18T15:15:27.187 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libcephfs2-2:18.2.4-717.g03f4c3e8.el9.x86_64 7/32 2024-08-18T15:15:27.249 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: libcephfs2-2:18.2.4-717.g03f4c3e8.el9.x86_64 7/32 2024-08-18T15:15:27.278 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2024-08-18T15:15:27.323 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-ceph-argparse-2:18.2.4-717.g03f4c3e8.el9.x 9/32 2024-08-18T15:15:27.387 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cephfs-2:18.2.4-717.g03f4c3e8.el9.x86_64 10/32 2024-08-18T15:15:27.414 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libradosstriper1-2:18.2.4-717.g03f4c3e8.el9.x86_64 11/32 2024-08-18T15:15:27.471 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: libradosstriper1-2:18.2.4-717.g03f4c3e8.el9.x86_64 11/32 2024-08-18T15:15:27.577 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/32 2024-08-18T15:15:27.612 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 13/32 2024-08-18T15:15:27.654 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2024-08-18T15:15:27.694 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2024-08-18T15:15:27.721 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 16/32 2024-08-18T15:15:27.757 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2024-08-18T15:15:27.790 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2024-08-18T15:15:28.098 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2024-08-18T15:15:28.173 INFO:teuthology.orchestra.run.smithi134.stdout: Upgrading : librbd1-2:18.2.4-717.g03f4c3e8.el9.x86_64 20/32 2024-08-18T15:15:28.264 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librbd1-2:18.2.4-717.g03f4c3e8.el9.x86_64 20/32 2024-08-18T15:15:28.291 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-rbd-2:18.2.4-717.g03f4c3e8.el9.x86_64 21/32 2024-08-18T15:15:28.327 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2024-08-18T15:15:28.509 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-08-18T15:15:28.589 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2024-08-18T15:15:28.620 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/32 2024-08-18T15:15:28.674 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/32 2024-08-18T15:15:28.724 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 23/32 2024-08-18T15:15:28.755 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : librgw2-2:18.2.4-717.g03f4c3e8.el9.x86_64 24/32 2024-08-18T15:15:28.811 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librgw2-2:18.2.4-717.g03f4c3e8.el9.x86_64 24/32 2024-08-18T15:15:28.886 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-rgw-2:18.2.4-717.g03f4c3e8.el9.x86_64 25/32 2024-08-18T15:15:28.916 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-ceph-common-2:18.2.4-717.g03f4c3e8.el9.x86 26/32 2024-08-18T15:15:29.045 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2024-08-18T15:15:29.102 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : librados2-2:18.2.4-717.g03f4c3e8.el9.x86_64 5/32 2024-08-18T15:15:29.197 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: librados2-2:18.2.4-717.g03f4c3e8.el9.x86_64 5/32 2024-08-18T15:15:29.286 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-rados-2:18.2.4-717.g03f4c3e8.el9.x86_64 6/32 2024-08-18T15:15:29.320 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libcephfs2-2:18.2.4-717.g03f4c3e8.el9.x86_64 7/32 2024-08-18T15:15:29.383 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: libcephfs2-2:18.2.4-717.g03f4c3e8.el9.x86_64 7/32 2024-08-18T15:15:29.411 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2024-08-18T15:15:29.457 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-ceph-argparse-2:18.2.4-717.g03f4c3e8.el9.x 9/32 2024-08-18T15:15:29.521 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cephfs-2:18.2.4-717.g03f4c3e8.el9.x86_64 10/32 2024-08-18T15:15:29.547 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libradosstriper1-2:18.2.4-717.g03f4c3e8.el9.x86_64 11/32 2024-08-18T15:15:29.614 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: libradosstriper1-2:18.2.4-717.g03f4c3e8.el9.x86_64 11/32 2024-08-18T15:15:29.727 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/32 2024-08-18T15:15:29.755 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 13/32 2024-08-18T15:15:29.807 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2024-08-18T15:15:29.828 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2024-08-18T15:15:29.855 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 16/32 2024-08-18T15:15:29.891 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2024-08-18T15:15:29.924 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2024-08-18T15:15:30.223 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2024-08-18T15:15:30.270 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : librbd1-2:18.2.4-717.g03f4c3e8.el9.x86_64 20/32 2024-08-18T15:15:30.364 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: librbd1-2:18.2.4-717.g03f4c3e8.el9.x86_64 20/32 2024-08-18T15:15:30.392 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-rbd-2:18.2.4-717.g03f4c3e8.el9.x86_64 21/32 2024-08-18T15:15:30.428 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2024-08-18T15:15:30.825 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 23/32 2024-08-18T15:15:30.856 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : librgw2-2:18.2.4-717.g03f4c3e8.el9.x86_64 24/32 2024-08-18T15:15:30.912 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: librgw2-2:18.2.4-717.g03f4c3e8.el9.x86_64 24/32 2024-08-18T15:15:30.987 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-rgw-2:18.2.4-717.g03f4c3e8.el9.x86_64 25/32 2024-08-18T15:15:31.017 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-ceph-common-2:18.2.4-717.g03f4c3e8.el9.x86 26/32 2024-08-18T15:15:31.028 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-common-2:18.2.4-717.g03f4c3e8.el9.x86_64 27/32 2024-08-18T15:15:31.057 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-common-2:18.2.4-717.g03f4c3e8.el9.x86_64 27/32 2024-08-18T15:15:31.596 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-common-2:18.2.4-717.g03f4c3e8.el9.x86_64 27/32 2024-08-18T15:15:31.636 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-base-2:18.2.4-717.g03f4c3e8.el9.x86_64 28/32 2024-08-18T15:15:31.723 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-base-2:18.2.4-717.g03f4c3e8.el9.x86_64 28/32 2024-08-18T15:15:31.723 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-08-18T15:15:31.723 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-08-18T15:15:31.724 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:15:31.762 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-selinux-2:18.2.4-717.g03f4c3e8.el9.x86_64 29/32 2024-08-18T15:15:32.969 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-common-2:18.2.4-717.g03f4c3e8.el9.x86_64 27/32 2024-08-18T15:15:33.005 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-common-2:18.2.4-717.g03f4c3e8.el9.x86_64 27/32 2024-08-18T15:15:33.533 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-common-2:18.2.4-717.g03f4c3e8.el9.x86_64 27/32 2024-08-18T15:15:33.575 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-base-2:18.2.4-717.g03f4c3e8.el9.x86_64 28/32 2024-08-18T15:15:33.653 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-base-2:18.2.4-717.g03f4c3e8.el9.x86_64 28/32 2024-08-18T15:15:33.653 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-08-18T15:15:33.653 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-08-18T15:15:33.653 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:15:33.685 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-selinux-2:18.2.4-717.g03f4c3e8.el9.x86_64 29/32 2024-08-18T15:15:41.433 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-selinux-2:18.2.4-717.g03f4c3e8.el9.x86_64 29/32 2024-08-18T15:15:41.433 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /sys 2024-08-18T15:15:41.434 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /proc 2024-08-18T15:15:41.434 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /mnt 2024-08-18T15:15:41.434 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /var/tmp 2024-08-18T15:15:41.434 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /home 2024-08-18T15:15:41.434 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /root 2024-08-18T15:15:41.434 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /tmp 2024-08-18T15:15:41.434 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:15:42.182 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-radosgw-2:18.2.4-717.g03f4c3e8.el9.x86_64 30/32 2024-08-18T15:15:42.207 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-radosgw-2:18.2.4-717.g03f4c3e8.el9.x86_64 30/32 2024-08-18T15:15:42.207 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-18T15:15:42.207 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-08-18T15:15:42.207 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-08-18T15:15:42.207 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-08-18T15:15:42.207 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:15:42.208 INFO:teuthology.orchestra.run.smithi134.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-08-18T15:15:42.326 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-08-18T15:15:42.327 INFO:teuthology.orchestra.run.smithi134.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2024-08-18T15:15:43.059 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-selinux-2:18.2.4-717.g03f4c3e8.el9.x86_64 29/32 2024-08-18T15:15:43.059 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /sys 2024-08-18T15:15:43.059 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /proc 2024-08-18T15:15:43.059 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /mnt 2024-08-18T15:15:43.059 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /var/tmp 2024-08-18T15:15:43.059 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /home 2024-08-18T15:15:43.059 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /root 2024-08-18T15:15:43.059 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /tmp 2024-08-18T15:15:43.059 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:15:43.437 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2024-08-18T15:15:43.437 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-base-2:18.2.4-717.g03f4c3e8.el9.x86_64 1/32 2024-08-18T15:15:43.437 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-common-2:18.2.4-717.g03f4c3e8.el9.x86_64 2/32 2024-08-18T15:15:43.437 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-radosgw-2:18.2.4-717.g03f4c3e8.el9.x86_64 3/32 2024-08-18T15:15:43.437 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-selinux-2:18.2.4-717.g03f4c3e8.el9.x86_64 4/32 2024-08-18T15:15:43.437 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libcephfs2-2:18.2.4-717.g03f4c3e8.el9.x86_64 5/32 2024-08-18T15:15:43.437 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libradosstriper1-2:18.2.4-717.g03f4c3e8.el9.x86_64 6/32 2024-08-18T15:15:43.437 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librgw2-2:18.2.4-717.g03f4c3e8.el9.x86_64 7/32 2024-08-18T15:15:43.438 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-ceph-argparse-2:18.2.4-717.g03f4c3e8.el9.x 8/32 2024-08-18T15:15:43.438 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-ceph-common-2:18.2.4-717.g03f4c3e8.el9.x86 9/32 2024-08-18T15:15:43.438 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cephfs-2:18.2.4-717.g03f4c3e8.el9.x86_64 10/32 2024-08-18T15:15:43.438 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-rados-2:18.2.4-717.g03f4c3e8.el9.x86_64 11/32 2024-08-18T15:15:43.438 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-rbd-2:18.2.4-717.g03f4c3e8.el9.x86_64 12/32 2024-08-18T15:15:43.438 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-rgw-2:18.2.4-717.g03f4c3e8.el9.x86_64 13/32 2024-08-18T15:15:43.438 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2024-08-18T15:15:43.438 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2024-08-18T15:15:43.438 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/32 2024-08-18T15:15:43.438 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/32 2024-08-18T15:15:43.438 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/32 2024-08-18T15:15:43.438 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/32 2024-08-18T15:15:43.439 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2024-08-18T15:15:43.439 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2024-08-18T15:15:43.439 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 22/32 2024-08-18T15:15:43.439 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 23/32 2024-08-18T15:15:43.439 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2024-08-18T15:15:43.439 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2024-08-18T15:15:43.439 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 26/32 2024-08-18T15:15:43.439 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : re2-1:20211101-3.el9.x86_64 27/32 2024-08-18T15:15:43.440 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2024-08-18T15:15:43.440 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librados2-2:18.2.4-717.g03f4c3e8.el9.x86_64 29/32 2024-08-18T15:15:43.440 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2024-08-18T15:15:43.440 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librbd1-2:18.2.4-717.g03f4c3e8.el9.x86_64 31/32 2024-08-18T15:15:43.803 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-radosgw-2:18.2.4-717.g03f4c3e8.el9.x86_64 30/32 2024-08-18T15:15:43.831 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-radosgw-2:18.2.4-717.g03f4c3e8.el9.x86_64 30/32 2024-08-18T15:15:43.831 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-18T15:15:43.831 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-08-18T15:15:43.831 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-08-18T15:15:43.831 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-08-18T15:15:43.831 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:15:43.834 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-08-18T15:15:43.906 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-08-18T15:15:43.906 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2024-08-18T15:15:44.205 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2024-08-18T15:15:44.205 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:15:44.205 INFO:teuthology.orchestra.run.smithi134.stdout:Upgraded: 2024-08-18T15:15:44.205 INFO:teuthology.orchestra.run.smithi134.stdout: librados2-2:18.2.4-717.g03f4c3e8.el9.x86_64 2024-08-18T15:15:44.205 INFO:teuthology.orchestra.run.smithi134.stdout: librbd1-2:18.2.4-717.g03f4c3e8.el9.x86_64 2024-08-18T15:15:44.205 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-08-18T15:15:44.205 INFO:teuthology.orchestra.run.smithi134.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-08-18T15:15:44.205 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-base-2:18.2.4-717.g03f4c3e8.el9.x86_64 2024-08-18T15:15:44.205 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-common-2:18.2.4-717.g03f4c3e8.el9.x86_64 2024-08-18T15:15:44.205 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-radosgw-2:18.2.4-717.g03f4c3e8.el9.x86_64 2024-08-18T15:15:44.205 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-selinux-2:18.2.4-717.g03f4c3e8.el9.x86_64 2024-08-18T15:15:44.206 INFO:teuthology.orchestra.run.smithi134.stdout: fmt-8.1.1-5.el9.x86_64 2024-08-18T15:15:44.206 INFO:teuthology.orchestra.run.smithi134.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-08-18T15:15:44.206 INFO:teuthology.orchestra.run.smithi134.stdout: libarrow-9.0.0-11.el9.x86_64 2024-08-18T15:15:44.206 INFO:teuthology.orchestra.run.smithi134.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-08-18T15:15:44.206 INFO:teuthology.orchestra.run.smithi134.stdout: libcephfs2-2:18.2.4-717.g03f4c3e8.el9.x86_64 2024-08-18T15:15:44.206 INFO:teuthology.orchestra.run.smithi134.stdout: liboath-2.6.7-2.el9.x86_64 2024-08-18T15:15:44.206 INFO:teuthology.orchestra.run.smithi134.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-08-18T15:15:44.206 INFO:teuthology.orchestra.run.smithi134.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-08-18T15:15:44.206 INFO:teuthology.orchestra.run.smithi134.stdout: libradosstriper1-2:18.2.4-717.g03f4c3e8.el9.x86_64 2024-08-18T15:15:44.206 INFO:teuthology.orchestra.run.smithi134.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-08-18T15:15:44.207 INFO:teuthology.orchestra.run.smithi134.stdout: librgw2-2:18.2.4-717.g03f4c3e8.el9.x86_64 2024-08-18T15:15:44.207 INFO:teuthology.orchestra.run.smithi134.stdout: libunwind-1.6.2-1.el9.x86_64 2024-08-18T15:15:44.207 INFO:teuthology.orchestra.run.smithi134.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-08-18T15:15:44.207 INFO:teuthology.orchestra.run.smithi134.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-08-18T15:15:44.207 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ceph-argparse-2:18.2.4-717.g03f4c3e8.el9.x86_64 2024-08-18T15:15:44.207 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ceph-common-2:18.2.4-717.g03f4c3e8.el9.x86_64 2024-08-18T15:15:44.207 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cephfs-2:18.2.4-717.g03f4c3e8.el9.x86_64 2024-08-18T15:15:44.207 INFO:teuthology.orchestra.run.smithi134.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-08-18T15:15:44.207 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rados-2:18.2.4-717.g03f4c3e8.el9.x86_64 2024-08-18T15:15:44.208 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rbd-2:18.2.4-717.g03f4c3e8.el9.x86_64 2024-08-18T15:15:44.208 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rgw-2:18.2.4-717.g03f4c3e8.el9.x86_64 2024-08-18T15:15:44.208 INFO:teuthology.orchestra.run.smithi134.stdout: re2-1:20211101-3.el9.x86_64 2024-08-18T15:15:44.208 INFO:teuthology.orchestra.run.smithi134.stdout: thrift-0.15.0-2.el9.x86_64 2024-08-18T15:15:44.208 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:15:44.208 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-08-18T15:15:44.628 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-test 2024-08-18T15:15:45.040 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2024-08-18T15:15:45.041 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-base-2:18.2.4-717.g03f4c3e8.el9.x86_64 1/32 2024-08-18T15:15:45.041 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-common-2:18.2.4-717.g03f4c3e8.el9.x86_64 2/32 2024-08-18T15:15:45.041 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-radosgw-2:18.2.4-717.g03f4c3e8.el9.x86_64 3/32 2024-08-18T15:15:45.041 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-selinux-2:18.2.4-717.g03f4c3e8.el9.x86_64 4/32 2024-08-18T15:15:45.041 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libcephfs2-2:18.2.4-717.g03f4c3e8.el9.x86_64 5/32 2024-08-18T15:15:45.041 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libradosstriper1-2:18.2.4-717.g03f4c3e8.el9.x86_64 6/32 2024-08-18T15:15:45.041 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librgw2-2:18.2.4-717.g03f4c3e8.el9.x86_64 7/32 2024-08-18T15:15:45.041 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-ceph-argparse-2:18.2.4-717.g03f4c3e8.el9.x 8/32 2024-08-18T15:15:45.041 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-ceph-common-2:18.2.4-717.g03f4c3e8.el9.x86 9/32 2024-08-18T15:15:45.041 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cephfs-2:18.2.4-717.g03f4c3e8.el9.x86_64 10/32 2024-08-18T15:15:45.041 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-rados-2:18.2.4-717.g03f4c3e8.el9.x86_64 11/32 2024-08-18T15:15:45.041 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-rbd-2:18.2.4-717.g03f4c3e8.el9.x86_64 12/32 2024-08-18T15:15:45.041 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-rgw-2:18.2.4-717.g03f4c3e8.el9.x86_64 13/32 2024-08-18T15:15:45.042 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2024-08-18T15:15:45.042 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2024-08-18T15:15:45.042 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/32 2024-08-18T15:15:45.042 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/32 2024-08-18T15:15:45.042 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/32 2024-08-18T15:15:45.042 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/32 2024-08-18T15:15:45.042 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2024-08-18T15:15:45.042 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2024-08-18T15:15:45.042 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 22/32 2024-08-18T15:15:45.043 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 23/32 2024-08-18T15:15:45.043 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2024-08-18T15:15:45.043 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2024-08-18T15:15:45.043 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 26/32 2024-08-18T15:15:45.043 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : re2-1:20211101-3.el9.x86_64 27/32 2024-08-18T15:15:45.043 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2024-08-18T15:15:45.043 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librados2-2:18.2.4-717.g03f4c3e8.el9.x86_64 29/32 2024-08-18T15:15:45.043 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2024-08-18T15:15:45.043 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librbd1-2:18.2.4-717.g03f4c3e8.el9.x86_64 31/32 2024-08-18T15:15:45.183 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:00:26 ago on Sun 18 Aug 2024 03:15:19 PM UTC. 2024-08-18T15:15:45.308 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-08-18T15:15:45.308 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-08-18T15:15:45.309 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-08-18T15:15:45.309 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-08-18T15:15:45.309 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-08-18T15:15:45.309 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-test x86_64 2:18.2.4-717.g03f4c3e8.el9 ceph 40 M 2024-08-18T15:15:45.309 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-08-18T15:15:45.309 INFO:teuthology.orchestra.run.smithi134.stdout: libcephsqlite x86_64 2:18.2.4-717.g03f4c3e8.el9 ceph 166 k 2024-08-18T15:15:45.309 INFO:teuthology.orchestra.run.smithi134.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-08-18T15:15:45.309 INFO:teuthology.orchestra.run.smithi134.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-08-18T15:15:45.309 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:15:45.309 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-08-18T15:15:45.309 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-08-18T15:15:45.309 INFO:teuthology.orchestra.run.smithi134.stdout:Install 4 Packages 2024-08-18T15:15:45.310 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:15:45.310 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 40 M 2024-08-18T15:15:45.310 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 163 M 2024-08-18T15:15:45.310 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-08-18T15:15:45.666 INFO:teuthology.orchestra.run.smithi134.stdout:(1/4): libcephsqlite-18.2.4-717.g03f4c3e8.el9.x 684 kB/s | 166 kB 00:00 2024-08-18T15:15:45.932 INFO:teuthology.orchestra.run.smithi134.stdout:(2/4): socat-1.7.4.1-6.el9.x86_64.rpm 597 kB/s | 304 kB 00:00 2024-08-18T15:15:46.007 INFO:teuthology.orchestra.run.smithi134.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 186 kB/s | 64 kB 00:00 2024-08-18T15:15:46.030 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2024-08-18T15:15:46.031 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:15:46.031 INFO:teuthology.orchestra.run.smithi193.stdout:Upgraded: 2024-08-18T15:15:46.031 INFO:teuthology.orchestra.run.smithi193.stdout: librados2-2:18.2.4-717.g03f4c3e8.el9.x86_64 2024-08-18T15:15:46.031 INFO:teuthology.orchestra.run.smithi193.stdout: librbd1-2:18.2.4-717.g03f4c3e8.el9.x86_64 2024-08-18T15:15:46.031 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-08-18T15:15:46.031 INFO:teuthology.orchestra.run.smithi193.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-08-18T15:15:46.031 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-base-2:18.2.4-717.g03f4c3e8.el9.x86_64 2024-08-18T15:15:46.031 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-common-2:18.2.4-717.g03f4c3e8.el9.x86_64 2024-08-18T15:15:46.031 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-radosgw-2:18.2.4-717.g03f4c3e8.el9.x86_64 2024-08-18T15:15:46.031 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-selinux-2:18.2.4-717.g03f4c3e8.el9.x86_64 2024-08-18T15:15:46.031 INFO:teuthology.orchestra.run.smithi193.stdout: fmt-8.1.1-5.el9.x86_64 2024-08-18T15:15:46.032 INFO:teuthology.orchestra.run.smithi193.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-08-18T15:15:46.032 INFO:teuthology.orchestra.run.smithi193.stdout: libarrow-9.0.0-11.el9.x86_64 2024-08-18T15:15:46.032 INFO:teuthology.orchestra.run.smithi193.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-08-18T15:15:46.032 INFO:teuthology.orchestra.run.smithi193.stdout: libcephfs2-2:18.2.4-717.g03f4c3e8.el9.x86_64 2024-08-18T15:15:46.032 INFO:teuthology.orchestra.run.smithi193.stdout: liboath-2.6.7-2.el9.x86_64 2024-08-18T15:15:46.032 INFO:teuthology.orchestra.run.smithi193.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-08-18T15:15:46.032 INFO:teuthology.orchestra.run.smithi193.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-08-18T15:15:46.032 INFO:teuthology.orchestra.run.smithi193.stdout: libradosstriper1-2:18.2.4-717.g03f4c3e8.el9.x86_64 2024-08-18T15:15:46.032 INFO:teuthology.orchestra.run.smithi193.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-08-18T15:15:46.032 INFO:teuthology.orchestra.run.smithi193.stdout: librgw2-2:18.2.4-717.g03f4c3e8.el9.x86_64 2024-08-18T15:15:46.032 INFO:teuthology.orchestra.run.smithi193.stdout: libunwind-1.6.2-1.el9.x86_64 2024-08-18T15:15:46.032 INFO:teuthology.orchestra.run.smithi193.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-08-18T15:15:46.032 INFO:teuthology.orchestra.run.smithi193.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-08-18T15:15:46.032 INFO:teuthology.orchestra.run.smithi193.stdout: python3-ceph-argparse-2:18.2.4-717.g03f4c3e8.el9.x86_64 2024-08-18T15:15:46.033 INFO:teuthology.orchestra.run.smithi193.stdout: python3-ceph-common-2:18.2.4-717.g03f4c3e8.el9.x86_64 2024-08-18T15:15:46.033 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cephfs-2:18.2.4-717.g03f4c3e8.el9.x86_64 2024-08-18T15:15:46.033 INFO:teuthology.orchestra.run.smithi193.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-08-18T15:15:46.033 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rados-2:18.2.4-717.g03f4c3e8.el9.x86_64 2024-08-18T15:15:46.033 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rbd-2:18.2.4-717.g03f4c3e8.el9.x86_64 2024-08-18T15:15:46.033 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rgw-2:18.2.4-717.g03f4c3e8.el9.x86_64 2024-08-18T15:15:46.033 INFO:teuthology.orchestra.run.smithi193.stdout: re2-1:20211101-3.el9.x86_64 2024-08-18T15:15:46.033 INFO:teuthology.orchestra.run.smithi193.stdout: thrift-0.15.0-2.el9.x86_64 2024-08-18T15:15:46.033 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:15:46.033 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-08-18T15:15:46.431 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph-test 2024-08-18T15:15:46.741 INFO:teuthology.orchestra.run.smithi134.stdout:(4/4): ceph-test-18.2.4-717.g03f4c3e8.el9.x86_6 30 MB/s | 40 MB 00:01 2024-08-18T15:15:46.743 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-08-18T15:15:46.744 INFO:teuthology.orchestra.run.smithi134.stdout:Total 28 MB/s | 40 MB 00:01 2024-08-18T15:15:46.770 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-08-18T15:15:46.789 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-08-18T15:15:46.789 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-08-18T15:15:46.982 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:00:26 ago on Sun 18 Aug 2024 03:15:20 PM UTC. 2024-08-18T15:15:47.105 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-08-18T15:15:47.106 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-08-18T15:15:47.109 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-08-18T15:15:47.110 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-08-18T15:15:47.110 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repository Size 2024-08-18T15:15:47.110 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-08-18T15:15:47.110 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-08-18T15:15:47.110 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-test x86_64 2:18.2.4-717.g03f4c3e8.el9 ceph 40 M 2024-08-18T15:15:47.110 INFO:teuthology.orchestra.run.smithi193.stdout:Installing dependencies: 2024-08-18T15:15:47.110 INFO:teuthology.orchestra.run.smithi193.stdout: libcephsqlite x86_64 2:18.2.4-717.g03f4c3e8.el9 ceph 166 k 2024-08-18T15:15:47.111 INFO:teuthology.orchestra.run.smithi193.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-08-18T15:15:47.111 INFO:teuthology.orchestra.run.smithi193.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-08-18T15:15:47.111 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:15:47.111 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-08-18T15:15:47.111 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-08-18T15:15:47.111 INFO:teuthology.orchestra.run.smithi193.stdout:Install 4 Packages 2024-08-18T15:15:47.111 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:15:47.112 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 40 M 2024-08-18T15:15:47.112 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 163 M 2024-08-18T15:15:47.112 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-08-18T15:15:47.328 INFO:teuthology.orchestra.run.smithi193.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.9 MB/s | 304 kB 00:00 2024-08-18T15:15:47.370 INFO:teuthology.orchestra.run.smithi193.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-08-18T15:15:47.437 INFO:teuthology.orchestra.run.smithi193.stdout:(3/4): libcephsqlite-18.2.4-717.g03f4c3e8.el9.x 632 kB/s | 166 kB 00:00 2024-08-18T15:15:47.576 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-08-18T15:15:47.666 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-08-18T15:15:47.707 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-08-18T15:15:47.735 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libcephsqlite-2:18.2.4-717.g03f4c3e8.el9.x86_64 3/4 2024-08-18T15:15:48.471 INFO:teuthology.orchestra.run.smithi193.stdout:(4/4): ceph-test-18.2.4-717.g03f4c3e8.el9.x86_6 31 MB/s | 40 MB 00:01 2024-08-18T15:15:48.473 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-08-18T15:15:48.473 INFO:teuthology.orchestra.run.smithi193.stdout:Total 30 MB/s | 40 MB 00:01 2024-08-18T15:15:48.500 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-08-18T15:15:48.520 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-08-18T15:15:48.520 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-08-18T15:15:48.835 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-08-18T15:15:48.836 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-08-18T15:15:49.307 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-08-18T15:15:49.371 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-08-18T15:15:49.411 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-08-18T15:15:49.439 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libcephsqlite-2:18.2.4-717.g03f4c3e8.el9.x86_64 3/4 2024-08-18T15:15:51.421 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: libcephsqlite-2:18.2.4-717.g03f4c3e8.el9.x86_64 3/4 2024-08-18T15:15:51.507 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-test-2:18.2.4-717.g03f4c3e8.el9.x86_64 4/4 2024-08-18T15:15:53.102 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: libcephsqlite-2:18.2.4-717.g03f4c3e8.el9.x86_64 3/4 2024-08-18T15:15:53.186 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-test-2:18.2.4-717.g03f4c3e8.el9.x86_64 4/4 2024-08-18T15:15:53.584 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-test-2:18.2.4-717.g03f4c3e8.el9.x86_64 4/4 2024-08-18T15:15:53.584 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-test-2:18.2.4-717.g03f4c3e8.el9.x86_64 1/4 2024-08-18T15:15:53.584 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libcephsqlite-2:18.2.4-717.g03f4c3e8.el9.x86_64 2/4 2024-08-18T15:15:53.584 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-08-18T15:15:53.863 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-08-18T15:15:53.863 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:15:53.863 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-08-18T15:15:53.863 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-test-2:18.2.4-717.g03f4c3e8.el9.x86_64 2024-08-18T15:15:53.863 INFO:teuthology.orchestra.run.smithi134.stdout: libcephsqlite-2:18.2.4-717.g03f4c3e8.el9.x86_64 2024-08-18T15:15:53.863 INFO:teuthology.orchestra.run.smithi134.stdout: socat-1.7.4.1-6.el9.x86_64 2024-08-18T15:15:53.863 INFO:teuthology.orchestra.run.smithi134.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-08-18T15:15:53.863 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:15:53.863 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-08-18T15:15:54.134 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph 2024-08-18T15:15:54.685 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:00:35 ago on Sun 18 Aug 2024 03:15:19 PM UTC. 2024-08-18T15:15:54.815 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-08-18T15:15:54.818 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-08-18T15:15:54.818 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-08-18T15:15:54.818 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-08-18T15:15:54.818 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-08-18T15:15:54.818 INFO:teuthology.orchestra.run.smithi134.stdout: ceph x86_64 2:18.2.4-717.g03f4c3e8.el9 ceph 6.5 k 2024-08-18T15:15:54.818 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-08-18T15:15:54.818 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mds x86_64 2:18.2.4-717.g03f4c3e8.el9 ceph 2.1 M 2024-08-18T15:15:54.818 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr x86_64 2:18.2.4-717.g03f4c3e8.el9 ceph 1.5 M 2024-08-18T15:15:54.818 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-modules-core noarch 2:18.2.4-717.g03f4c3e8.el9 ceph-noarch 246 k 2024-08-18T15:15:54.818 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mon x86_64 2:18.2.4-717.g03f4c3e8.el9 ceph 4.7 M 2024-08-18T15:15:54.819 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-osd x86_64 2:18.2.4-717.g03f4c3e8.el9 ceph 17 M 2024-08-18T15:15:54.819 INFO:teuthology.orchestra.run.smithi134.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-08-18T15:15:54.819 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-08-18T15:15:54.819 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-08-18T15:15:54.819 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-08-18T15:15:54.819 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-08-18T15:15:54.819 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-08-18T15:15:54.819 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-08-18T15:15:54.819 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-08-18T15:15:54.819 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-08-18T15:15:54.819 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-08-18T15:15:54.819 INFO:teuthology.orchestra.run.smithi134.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-08-18T15:15:54.820 INFO:teuthology.orchestra.run.smithi134.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-08-18T15:15:54.820 INFO:teuthology.orchestra.run.smithi134.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-08-18T15:15:54.820 INFO:teuthology.orchestra.run.smithi134.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-08-18T15:15:54.820 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-08-18T15:15:54.820 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-08-18T15:15:54.820 INFO:teuthology.orchestra.run.smithi134.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-08-18T15:15:54.820 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-08-18T15:15:54.820 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-08-18T15:15:54.820 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-08-18T15:15:54.820 INFO:teuthology.orchestra.run.smithi134.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-08-18T15:15:54.820 INFO:teuthology.orchestra.run.smithi134.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-08-18T15:15:54.820 INFO:teuthology.orchestra.run.smithi134.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-08-18T15:15:54.820 INFO:teuthology.orchestra.run.smithi134.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-08-18T15:15:54.820 INFO:teuthology.orchestra.run.smithi134.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-08-18T15:15:54.821 INFO:teuthology.orchestra.run.smithi134.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-08-18T15:15:54.821 INFO:teuthology.orchestra.run.smithi134.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-08-18T15:15:54.821 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:15:54.821 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-08-18T15:15:54.821 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-08-18T15:15:54.821 INFO:teuthology.orchestra.run.smithi134.stdout:Install 33 Packages 2024-08-18T15:15:54.821 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:15:54.822 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 29 M 2024-08-18T15:15:54.822 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 104 M 2024-08-18T15:15:54.822 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-08-18T15:15:55.224 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-test-2:18.2.4-717.g03f4c3e8.el9.x86_64 4/4 2024-08-18T15:15:55.224 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-test-2:18.2.4-717.g03f4c3e8.el9.x86_64 1/4 2024-08-18T15:15:55.224 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libcephsqlite-2:18.2.4-717.g03f4c3e8.el9.x86_64 2/4 2024-08-18T15:15:55.224 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-08-18T15:15:55.260 INFO:teuthology.orchestra.run.smithi134.stdout:(1/33): ceph-18.2.4-717.g03f4c3e8.el9.x86_64.rp 37 kB/s | 6.5 kB 00:00 2024-08-18T15:15:55.401 INFO:teuthology.orchestra.run.smithi134.stdout:(2/33): ceph-mgr-18.2.4-717.g03f4c3e8.el9.x86_6 4.6 MB/s | 1.5 MB 00:00 2024-08-18T15:15:55.519 INFO:teuthology.orchestra.run.smithi134.stdout:(3/33): ceph-mds-18.2.4-717.g03f4c3e8.el9.x86_6 4.8 MB/s | 2.1 MB 00:00 2024-08-18T15:15:55.522 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-08-18T15:15:55.522 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:15:55.522 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-08-18T15:15:55.522 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-test-2:18.2.4-717.g03f4c3e8.el9.x86_64 2024-08-18T15:15:55.522 INFO:teuthology.orchestra.run.smithi193.stdout: libcephsqlite-2:18.2.4-717.g03f4c3e8.el9.x86_64 2024-08-18T15:15:55.522 INFO:teuthology.orchestra.run.smithi193.stdout: socat-1.7.4.1-6.el9.x86_64 2024-08-18T15:15:55.522 INFO:teuthology.orchestra.run.smithi193.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-08-18T15:15:55.522 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:15:55.522 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-08-18T15:15:55.594 INFO:teuthology.orchestra.run.smithi134.stdout:(4/33): ceph-mgr-modules-core-18.2.4-717.g03f4c 3.2 MB/s | 246 kB 00:00 2024-08-18T15:15:55.702 INFO:teuthology.orchestra.run.smithi134.stdout:(5/33): ceph-mon-18.2.4-717.g03f4c3e8.el9.x86_6 11 MB/s | 4.7 MB 00:00 2024-08-18T15:15:55.769 INFO:teuthology.orchestra.run.smithi134.stdout:(6/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.4 MB/s | 253 kB 00:00 2024-08-18T15:15:55.804 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph 2024-08-18T15:15:55.811 INFO:teuthology.orchestra.run.smithi134.stdout:(7/33): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 106 kB 00:00 2024-08-18T15:15:55.853 INFO:teuthology.orchestra.run.smithi134.stdout:(8/33): python3-pycparser-2.20-6.el9.noarch.rpm 3.2 MB/s | 135 kB 00:00 2024-08-18T15:15:55.887 INFO:teuthology.orchestra.run.smithi134.stdout:(9/33): python3-cryptography-36.0.1-4.el9.x86_6 6.7 MB/s | 1.2 MB 00:00 2024-08-18T15:15:55.912 INFO:teuthology.orchestra.run.smithi134.stdout:(10/33): python3-requests-2.25.1-8.el9.noarch.r 2.1 MB/s | 125 kB 00:00 2024-08-18T15:15:56.079 INFO:teuthology.orchestra.run.smithi134.stdout:(11/33): ceph-osd-18.2.4-717.g03f4c3e8.el9.x86_ 24 MB/s | 17 MB 00:00 2024-08-18T15:15:56.104 INFO:teuthology.orchestra.run.smithi134.stdout:(12/33): python3-urllib3-1.26.5-6.el9.noarch.rp 988 kB/s | 215 kB 00:00 2024-08-18T15:15:56.221 INFO:teuthology.orchestra.run.smithi134.stdout:(13/33): python3-markupsafe-1.1.1-12.el9.x86_64 245 kB/s | 35 kB 00:00 2024-08-18T15:15:56.246 INFO:teuthology.orchestra.run.smithi134.stdout:(14/33): python3-pytz-2021.1-5.el9.noarch.rpm 360 kB/s | 51 kB 00:00 2024-08-18T15:15:56.263 INFO:teuthology.orchestra.run.smithi134.stdout:(15/33): python3-mako-1.1.4-6.el9.noarch.rpm 491 kB/s | 172 kB 00:00 2024-08-18T15:15:56.347 INFO:teuthology.orchestra.run.smithi134.stdout:(16/33): python3-toml-0.10.2-6.el9.noarch.rpm 331 kB/s | 42 kB 00:00 2024-08-18T15:15:56.361 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:00:36 ago on Sun 18 Aug 2024 03:15:20 PM UTC. 2024-08-18T15:15:56.364 INFO:teuthology.orchestra.run.smithi134.stdout:(17/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 371 kB/s | 43 kB 00:00 2024-08-18T15:15:56.406 INFO:teuthology.orchestra.run.smithi134.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 617 kB/s | 11 kB 00:00 2024-08-18T15:15:56.440 INFO:teuthology.orchestra.run.smithi134.stdout:(19/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 976 kB/s | 172 kB 00:00 2024-08-18T15:15:56.456 INFO:teuthology.orchestra.run.smithi134.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 368 kB/s | 18 kB 00:00 2024-08-18T15:15:56.473 INFO:teuthology.orchestra.run.smithi134.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 698 kB/s | 23 kB 00:00 2024-08-18T15:15:56.490 INFO:teuthology.orchestra.run.smithi134.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 578 kB/s | 19 kB 00:00 2024-08-18T15:15:56.494 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-08-18T15:15:56.497 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-08-18T15:15:56.497 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repository Size 2024-08-18T15:15:56.497 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-08-18T15:15:56.497 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-08-18T15:15:56.497 INFO:teuthology.orchestra.run.smithi193.stdout: ceph x86_64 2:18.2.4-717.g03f4c3e8.el9 ceph 6.5 k 2024-08-18T15:15:56.497 INFO:teuthology.orchestra.run.smithi193.stdout:Installing dependencies: 2024-08-18T15:15:56.497 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mds x86_64 2:18.2.4-717.g03f4c3e8.el9 ceph 2.1 M 2024-08-18T15:15:56.497 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr x86_64 2:18.2.4-717.g03f4c3e8.el9 ceph 1.5 M 2024-08-18T15:15:56.497 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-modules-core noarch 2:18.2.4-717.g03f4c3e8.el9 ceph-noarch 246 k 2024-08-18T15:15:56.498 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mon x86_64 2:18.2.4-717.g03f4c3e8.el9 ceph 4.7 M 2024-08-18T15:15:56.498 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-osd x86_64 2:18.2.4-717.g03f4c3e8.el9 ceph 17 M 2024-08-18T15:15:56.498 INFO:teuthology.orchestra.run.smithi193.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-08-18T15:15:56.498 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-08-18T15:15:56.498 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-08-18T15:15:56.498 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-08-18T15:15:56.498 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-08-18T15:15:56.498 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-08-18T15:15:56.498 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-08-18T15:15:56.498 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-08-18T15:15:56.498 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-08-18T15:15:56.498 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-08-18T15:15:56.498 INFO:teuthology.orchestra.run.smithi193.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-08-18T15:15:56.499 INFO:teuthology.orchestra.run.smithi193.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-08-18T15:15:56.499 INFO:teuthology.orchestra.run.smithi193.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-08-18T15:15:56.499 INFO:teuthology.orchestra.run.smithi193.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-08-18T15:15:56.499 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-08-18T15:15:56.499 INFO:teuthology.orchestra.run.smithi193.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-08-18T15:15:56.499 INFO:teuthology.orchestra.run.smithi193.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-08-18T15:15:56.499 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-08-18T15:15:56.499 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-08-18T15:15:56.499 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-08-18T15:15:56.499 INFO:teuthology.orchestra.run.smithi193.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-08-18T15:15:56.499 INFO:teuthology.orchestra.run.smithi193.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-08-18T15:15:56.499 INFO:teuthology.orchestra.run.smithi193.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-08-18T15:15:56.499 INFO:teuthology.orchestra.run.smithi193.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-08-18T15:15:56.499 INFO:teuthology.orchestra.run.smithi193.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-08-18T15:15:56.500 INFO:teuthology.orchestra.run.smithi193.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-08-18T15:15:56.500 INFO:teuthology.orchestra.run.smithi193.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-08-18T15:15:56.500 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:15:56.500 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-08-18T15:15:56.500 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-08-18T15:15:56.500 INFO:teuthology.orchestra.run.smithi193.stdout:Install 33 Packages 2024-08-18T15:15:56.500 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:15:56.500 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 29 M 2024-08-18T15:15:56.501 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 104 M 2024-08-18T15:15:56.501 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-08-18T15:15:56.507 INFO:teuthology.orchestra.run.smithi134.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 583 kB/s | 20 kB 00:00 2024-08-18T15:15:56.533 INFO:teuthology.orchestra.run.smithi134.stdout:(24/33): python3-logutils-0.3.5-21.el9.noarch.r 1.1 MB/s | 46 kB 00:00 2024-08-18T15:15:56.587 INFO:teuthology.orchestra.run.smithi134.stdout:(25/33): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-08-18T15:15:56.608 INFO:teuthology.orchestra.run.smithi134.stdout:(26/33): python3-portend-3.1.0-2.el9.noarch.rpm 761 kB/s | 16 kB 00:00 2024-08-18T15:15:56.633 INFO:teuthology.orchestra.run.smithi134.stdout:(27/33): python3-cherrypy-18.6.1-2.el9.noarch.r 1.2 MB/s | 358 kB 00:00 2024-08-18T15:15:56.659 INFO:teuthology.orchestra.run.smithi134.stdout:(28/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.8 MB/s | 90 kB 00:00 2024-08-18T15:15:56.684 INFO:teuthology.orchestra.run.smithi134.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 711 kB/s | 36 kB 00:00 2024-08-18T15:15:56.718 INFO:teuthology.orchestra.run.smithi134.stdout:(30/33): python3-pecan-1.4.2-3.el9.noarch.rpm 1.4 MB/s | 272 kB 00:00 2024-08-18T15:15:56.743 INFO:teuthology.orchestra.run.smithi134.stdout:(31/33): python3-zc-lockfile-2.0-10.el9.noarch. 799 kB/s | 20 kB 00:00 2024-08-18T15:15:56.768 INFO:teuthology.orchestra.run.smithi134.stdout:(32/33): python3-webob-1.8.7-6.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2024-08-18T15:15:56.794 INFO:teuthology.orchestra.run.smithi134.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.8 MB/s | 427 kB 00:00 2024-08-18T15:15:56.798 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-08-18T15:15:56.798 INFO:teuthology.orchestra.run.smithi134.stdout:Total 15 MB/s | 29 MB 00:01 2024-08-18T15:15:56.910 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-08-18T15:15:56.946 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-08-18T15:15:56.946 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-08-18T15:15:57.313 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-08-18T15:15:57.314 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-08-18T15:15:57.497 INFO:teuthology.orchestra.run.smithi193.stdout:(1/33): ceph-18.2.4-717.g03f4c3e8.el9.x86_64.rp 37 kB/s | 6.5 kB 00:00 2024-08-18T15:15:57.672 INFO:teuthology.orchestra.run.smithi193.stdout:(2/33): ceph-mds-18.2.4-717.g03f4c3e8.el9.x86_6 5.9 MB/s | 2.1 MB 00:00 2024-08-18T15:15:57.731 INFO:teuthology.orchestra.run.smithi193.stdout:(3/33): ceph-mgr-18.2.4-717.g03f4c3e8.el9.x86_6 3.5 MB/s | 1.5 MB 00:00 2024-08-18T15:15:57.822 INFO:teuthology.orchestra.run.smithi193.stdout:(4/33): ceph-mgr-modules-core-18.2.4-717.g03f4c 2.6 MB/s | 246 kB 00:00 2024-08-18T15:15:57.956 INFO:teuthology.orchestra.run.smithi193.stdout:(5/33): ceph-mon-18.2.4-717.g03f4c3e8.el9.x86_6 10 MB/s | 4.7 MB 00:00 2024-08-18T15:15:58.073 INFO:teuthology.orchestra.run.smithi193.stdout:(6/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.0 MB/s | 253 kB 00:00 2024-08-18T15:15:58.232 INFO:teuthology.orchestra.run.smithi193.stdout:(7/33): ceph-osd-18.2.4-717.g03f4c3e8.el9.x86_6 30 MB/s | 17 MB 00:00 2024-08-18T15:15:58.257 INFO:teuthology.orchestra.run.smithi193.stdout:(8/33): python3-ply-3.11-14.el9.noarch.rpm 580 kB/s | 106 kB 00:00 2024-08-18T15:15:58.316 INFO:teuthology.orchestra.run.smithi193.stdout:(9/33): python3-requests-2.25.1-8.el9.noarch.rp 2.1 MB/s | 125 kB 00:00 2024-08-18T15:15:58.357 INFO:teuthology.orchestra.run.smithi193.stdout:(10/33): python3-cryptography-36.0.1-4.el9.x86_ 3.1 MB/s | 1.2 MB 00:00 2024-08-18T15:15:58.382 INFO:teuthology.orchestra.run.smithi193.stdout:(11/33): python3-urllib3-1.26.5-6.el9.noarch.rp 3.1 MB/s | 215 kB 00:00 2024-08-18T15:15:58.408 INFO:teuthology.orchestra.run.smithi193.stdout:(12/33): python3-pycparser-2.20-6.el9.noarch.rp 769 kB/s | 135 kB 00:00 2024-08-18T15:15:58.474 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-08-18T15:15:58.491 INFO:teuthology.orchestra.run.smithi193.stdout:(13/33): python3-mako-1.1.4-6.el9.noarch.rpm 1.3 MB/s | 172 kB 00:00 2024-08-18T15:15:58.525 INFO:teuthology.orchestra.run.smithi193.stdout:(14/33): python3-markupsafe-1.1.1-12.el9.x86_64 244 kB/s | 35 kB 00:00 2024-08-18T15:15:58.546 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-08-18T15:15:58.575 INFO:teuthology.orchestra.run.smithi193.stdout:(15/33): python3-pytz-2021.1-5.el9.noarch.rpm 305 kB/s | 51 kB 00:00 2024-08-18T15:15:58.579 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-08-18T15:15:58.601 INFO:teuthology.orchestra.run.smithi193.stdout:(16/33): python3-toml-0.10.2-6.el9.noarch.rpm 382 kB/s | 42 kB 00:00 2024-08-18T15:15:58.611 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-08-18T15:15:58.651 INFO:teuthology.orchestra.run.smithi193.stdout:(17/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 346 kB/s | 43 kB 00:00 2024-08-18T15:15:58.701 INFO:teuthology.orchestra.run.smithi193.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 454 kB/s | 11 kB 00:00 2024-08-18T15:15:58.702 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-08-18T15:15:58.735 INFO:teuthology.orchestra.run.smithi193.stdout:(19/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.1 MB/s | 172 kB 00:00 2024-08-18T15:15:58.760 INFO:teuthology.orchestra.run.smithi193.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 314 kB/s | 18 kB 00:00 2024-08-18T15:15:58.781 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-08-18T15:15:58.785 INFO:teuthology.orchestra.run.smithi193.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 462 kB/s | 23 kB 00:00 2024-08-18T15:15:58.810 INFO:teuthology.orchestra.run.smithi193.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 388 kB/s | 19 kB 00:00 2024-08-18T15:15:58.822 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-08-18T15:15:58.838 INFO:teuthology.orchestra.run.smithi193.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 391 kB/s | 20 kB 00:00 2024-08-18T15:15:58.857 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-08-18T15:15:58.861 INFO:teuthology.orchestra.run.smithi193.stdout:(24/33): python3-logutils-0.3.5-21.el9.noarch.r 921 kB/s | 46 kB 00:00 2024-08-18T15:15:58.886 INFO:teuthology.orchestra.run.smithi193.stdout:(25/33): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-08-18T15:15:58.896 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/33 2024-08-18T15:15:58.911 INFO:teuthology.orchestra.run.smithi193.stdout:(26/33): python3-cherrypy-18.6.1-2.el9.noarch.r 1.2 MB/s | 358 kB 00:00 2024-08-18T15:15:58.930 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/33 2024-08-18T15:15:58.937 INFO:teuthology.orchestra.run.smithi193.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 326 kB/s | 16 kB 00:00 2024-08-18T15:15:58.962 INFO:teuthology.orchestra.run.smithi193.stdout:(28/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2024-08-18T15:15:58.963 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/33 2024-08-18T15:15:58.987 INFO:teuthology.orchestra.run.smithi193.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 709 kB/s | 36 kB 00:00 2024-08-18T15:15:59.012 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/33 2024-08-18T15:15:59.013 INFO:teuthology.orchestra.run.smithi193.stdout:(30/33): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-08-18T15:15:59.038 INFO:teuthology.orchestra.run.smithi193.stdout:(31/33): python3-zc-lockfile-2.0-10.el9.noarch. 795 kB/s | 20 kB 00:00 2024-08-18T15:15:59.047 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/33 2024-08-18T15:15:59.063 INFO:teuthology.orchestra.run.smithi193.stdout:(32/33): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-08-18T15:15:59.082 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/33 2024-08-18T15:15:59.088 INFO:teuthology.orchestra.run.smithi193.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.1 MB/s | 427 kB 00:00 2024-08-18T15:15:59.092 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-08-18T15:15:59.092 INFO:teuthology.orchestra.run.smithi193.stdout:Total 11 MB/s | 29 MB 00:02 2024-08-18T15:15:59.120 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/33 2024-08-18T15:15:59.193 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 15/33 2024-08-18T15:15:59.206 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-08-18T15:15:59.242 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-08-18T15:15:59.243 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-08-18T15:15:59.331 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 16/33 2024-08-18T15:15:59.411 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 17/33 2024-08-18T15:15:59.473 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 18/33 2024-08-18T15:15:59.512 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 19/33 2024-08-18T15:15:59.569 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-ply-3.11-14.el9.noarch 20/33 2024-08-18T15:15:59.614 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-08-18T15:15:59.614 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-08-18T15:15:59.634 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 21/33 2024-08-18T15:15:59.818 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 22/33 2024-08-18T15:15:59.866 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 23/33 2024-08-18T15:15:59.944 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 24/33 2024-08-18T15:16:00.033 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 25/33 2024-08-18T15:16:00.074 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 26/33 2024-08-18T15:16:00.189 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 27/33 2024-08-18T15:16:00.349 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-modules-core-2:18.2.4-717.g03f4c3e8.el9.n 28/33 2024-08-18T15:16:00.376 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-2:18.2.4-717.g03f4c3e8.el9.x86_64 29/33 2024-08-18T15:16:00.402 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-2:18.2.4-717.g03f4c3e8.el9.x86_64 29/33 2024-08-18T15:16:00.402 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-18T15:16:00.402 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-08-18T15:16:00.402 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-08-18T15:16:00.402 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-08-18T15:16:00.402 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:16:00.808 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-08-18T15:16:00.866 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-08-18T15:16:00.899 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-08-18T15:16:00.977 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-08-18T15:16:01.024 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-08-18T15:16:01.102 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-08-18T15:16:01.151 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-08-18T15:16:01.198 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-08-18T15:16:01.224 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/33 2024-08-18T15:16:01.251 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/33 2024-08-18T15:16:01.286 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/33 2024-08-18T15:16:01.324 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/33 2024-08-18T15:16:01.360 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/33 2024-08-18T15:16:01.394 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/33 2024-08-18T15:16:01.432 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/33 2024-08-18T15:16:01.505 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 15/33 2024-08-18T15:16:01.650 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 16/33 2024-08-18T15:16:01.723 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 17/33 2024-08-18T15:16:01.776 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 18/33 2024-08-18T15:16:01.815 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 19/33 2024-08-18T15:16:01.864 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-ply-3.11-14.el9.noarch 20/33 2024-08-18T15:16:01.870 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-osd-2:18.2.4-717.g03f4c3e8.el9.x86_64 30/33 2024-08-18T15:16:01.899 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-osd-2:18.2.4-717.g03f4c3e8.el9.x86_64 30/33 2024-08-18T15:16:01.899 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-18T15:16:01.899 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-08-18T15:16:01.899 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-08-18T15:16:01.899 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-08-18T15:16:01.900 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:16:01.923 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 21/33 2024-08-18T15:16:02.114 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 22/33 2024-08-18T15:16:02.194 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 23/33 2024-08-18T15:16:02.263 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 24/33 2024-08-18T15:16:02.337 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mon-2:18.2.4-717.g03f4c3e8.el9.x86_64 31/33 2024-08-18T15:16:02.352 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 25/33 2024-08-18T15:16:02.361 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mon-2:18.2.4-717.g03f4c3e8.el9.x86_64 31/33 2024-08-18T15:16:02.361 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-18T15:16:02.361 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-08-18T15:16:02.361 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-08-18T15:16:02.361 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-08-18T15:16:02.361 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:16:02.393 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 26/33 2024-08-18T15:16:02.508 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 27/33 2024-08-18T15:16:02.571 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mds-2:18.2.4-717.g03f4c3e8.el9.x86_64 32/33 2024-08-18T15:16:02.595 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mds-2:18.2.4-717.g03f4c3e8.el9.x86_64 32/33 2024-08-18T15:16:02.595 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-18T15:16:02.595 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-08-18T15:16:02.595 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-08-18T15:16:02.595 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-08-18T15:16:02.595 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:16:02.659 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-2:18.2.4-717.g03f4c3e8.el9.x86_64 33/33 2024-08-18T15:16:02.672 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mgr-modules-core-2:18.2.4-717.g03f4c3e8.el9.n 28/33 2024-08-18T15:16:02.695 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mgr-2:18.2.4-717.g03f4c3e8.el9.x86_64 29/33 2024-08-18T15:16:02.721 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mgr-2:18.2.4-717.g03f4c3e8.el9.x86_64 29/33 2024-08-18T15:16:02.721 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-18T15:16:02.722 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-08-18T15:16:02.722 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-08-18T15:16:02.722 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-08-18T15:16:02.722 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:16:04.129 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-osd-2:18.2.4-717.g03f4c3e8.el9.x86_64 30/33 2024-08-18T15:16:04.157 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-osd-2:18.2.4-717.g03f4c3e8.el9.x86_64 30/33 2024-08-18T15:16:04.157 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-18T15:16:04.157 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-08-18T15:16:04.157 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-08-18T15:16:04.157 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-08-18T15:16:04.157 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:16:04.307 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-2:18.2.4-717.g03f4c3e8.el9.x86_64 33/33 2024-08-18T15:16:04.307 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-2:18.2.4-717.g03f4c3e8.el9.x86_64 1/33 2024-08-18T15:16:04.307 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mds-2:18.2.4-717.g03f4c3e8.el9.x86_64 2/33 2024-08-18T15:16:04.307 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-2:18.2.4-717.g03f4c3e8.el9.x86_64 3/33 2024-08-18T15:16:04.307 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mon-2:18.2.4-717.g03f4c3e8.el9.x86_64 4/33 2024-08-18T15:16:04.307 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-osd-2:18.2.4-717.g03f4c3e8.el9.x86_64 5/33 2024-08-18T15:16:04.308 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-modules-core-2:18.2.4-717.g03f4c3e8.el9.n 6/33 2024-08-18T15:16:04.308 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/33 2024-08-18T15:16:04.308 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/33 2024-08-18T15:16:04.308 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/33 2024-08-18T15:16:04.308 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/33 2024-08-18T15:16:04.308 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/33 2024-08-18T15:16:04.308 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/33 2024-08-18T15:16:04.308 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/33 2024-08-18T15:16:04.308 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/33 2024-08-18T15:16:04.308 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 15/33 2024-08-18T15:16:04.308 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 16/33 2024-08-18T15:16:04.308 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-08-18T15:16:04.309 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-08-18T15:16:04.309 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-08-18T15:16:04.309 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-08-18T15:16:04.309 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-08-18T15:16:04.309 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-08-18T15:16:04.309 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-08-18T15:16:04.309 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-08-18T15:16:04.309 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-08-18T15:16:04.309 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-08-18T15:16:04.309 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-08-18T15:16:04.309 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-08-18T15:16:04.309 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-08-18T15:16:04.309 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-08-18T15:16:04.310 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-08-18T15:16:04.310 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-08-18T15:16:04.597 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mon-2:18.2.4-717.g03f4c3e8.el9.x86_64 31/33 2024-08-18T15:16:04.622 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mon-2:18.2.4-717.g03f4c3e8.el9.x86_64 31/33 2024-08-18T15:16:04.622 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-18T15:16:04.622 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-08-18T15:16:04.622 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-08-18T15:16:04.622 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-08-18T15:16:04.622 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:16:04.831 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mds-2:18.2.4-717.g03f4c3e8.el9.x86_64 32/33 2024-08-18T15:16:04.855 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mds-2:18.2.4-717.g03f4c3e8.el9.x86_64 32/33 2024-08-18T15:16:04.855 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-18T15:16:04.855 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-08-18T15:16:04.855 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-08-18T15:16:04.855 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-08-18T15:16:04.855 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:16:04.919 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-2:18.2.4-717.g03f4c3e8.el9.x86_64 33/33 2024-08-18T15:16:05.297 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-08-18T15:16:05.297 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:16:05.297 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-08-18T15:16:05.297 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-2:18.2.4-717.g03f4c3e8.el9.x86_64 2024-08-18T15:16:05.297 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mds-2:18.2.4-717.g03f4c3e8.el9.x86_64 2024-08-18T15:16:05.298 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-2:18.2.4-717.g03f4c3e8.el9.x86_64 2024-08-18T15:16:05.298 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-modules-core-2:18.2.4-717.g03f4c3e8.el9.noarch 2024-08-18T15:16:05.298 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mon-2:18.2.4-717.g03f4c3e8.el9.x86_64 2024-08-18T15:16:05.298 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-osd-2:18.2.4-717.g03f4c3e8.el9.x86_64 2024-08-18T15:16:05.298 INFO:teuthology.orchestra.run.smithi134.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-08-18T15:16:05.298 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-08-18T15:16:05.298 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-08-18T15:16:05.298 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-08-18T15:16:05.298 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-08-18T15:16:05.298 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-08-18T15:16:05.298 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-08-18T15:16:05.298 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-08-18T15:16:05.298 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-08-18T15:16:05.299 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-08-18T15:16:05.299 INFO:teuthology.orchestra.run.smithi134.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-08-18T15:16:05.299 INFO:teuthology.orchestra.run.smithi134.stdout: python3-mako-1.1.4-6.el9.noarch 2024-08-18T15:16:05.299 INFO:teuthology.orchestra.run.smithi134.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-08-18T15:16:05.299 INFO:teuthology.orchestra.run.smithi134.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-08-18T15:16:05.299 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-08-18T15:16:05.299 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ply-3.11-14.el9.noarch 2024-08-18T15:16:05.299 INFO:teuthology.orchestra.run.smithi134.stdout: python3-portend-3.1.0-2.el9.noarch 2024-08-18T15:16:05.299 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-08-18T15:16:05.299 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pycparser-2.20-6.el9.noarch 2024-08-18T15:16:05.299 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pytz-2021.1-5.el9.noarch 2024-08-18T15:16:05.299 INFO:teuthology.orchestra.run.smithi134.stdout: python3-requests-2.25.1-8.el9.noarch 2024-08-18T15:16:05.300 INFO:teuthology.orchestra.run.smithi134.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-08-18T15:16:05.300 INFO:teuthology.orchestra.run.smithi134.stdout: python3-toml-0.10.2-6.el9.noarch 2024-08-18T15:16:05.300 INFO:teuthology.orchestra.run.smithi134.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-08-18T15:16:05.300 INFO:teuthology.orchestra.run.smithi134.stdout: python3-webob-1.8.7-6.el9.noarch 2024-08-18T15:16:05.300 INFO:teuthology.orchestra.run.smithi134.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-08-18T15:16:05.300 INFO:teuthology.orchestra.run.smithi134.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-08-18T15:16:05.300 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:16:05.300 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-08-18T15:16:05.684 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-base 2024-08-18T15:16:06.243 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:00:47 ago on Sun 18 Aug 2024 03:15:19 PM UTC. 2024-08-18T15:16:06.329 INFO:teuthology.orchestra.run.smithi134.stdout:Package ceph-base-2:18.2.4-717.g03f4c3e8.el9.x86_64 is already installed. 2024-08-18T15:16:06.373 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-08-18T15:16:06.374 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-08-18T15:16:06.374 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-08-18T15:16:06.447 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install cephadm 2024-08-18T15:16:06.695 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-2:18.2.4-717.g03f4c3e8.el9.x86_64 33/33 2024-08-18T15:16:06.695 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-2:18.2.4-717.g03f4c3e8.el9.x86_64 1/33 2024-08-18T15:16:06.695 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mds-2:18.2.4-717.g03f4c3e8.el9.x86_64 2/33 2024-08-18T15:16:06.695 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mgr-2:18.2.4-717.g03f4c3e8.el9.x86_64 3/33 2024-08-18T15:16:06.695 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mon-2:18.2.4-717.g03f4c3e8.el9.x86_64 4/33 2024-08-18T15:16:06.695 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-osd-2:18.2.4-717.g03f4c3e8.el9.x86_64 5/33 2024-08-18T15:16:06.695 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mgr-modules-core-2:18.2.4-717.g03f4c3e8.el9.n 6/33 2024-08-18T15:16:06.696 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/33 2024-08-18T15:16:06.696 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/33 2024-08-18T15:16:06.696 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/33 2024-08-18T15:16:06.696 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/33 2024-08-18T15:16:06.696 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/33 2024-08-18T15:16:06.696 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/33 2024-08-18T15:16:06.696 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/33 2024-08-18T15:16:06.696 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/33 2024-08-18T15:16:06.696 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 15/33 2024-08-18T15:16:06.696 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 16/33 2024-08-18T15:16:06.696 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-08-18T15:16:06.696 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-08-18T15:16:06.696 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-08-18T15:16:06.696 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-08-18T15:16:06.697 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-08-18T15:16:06.697 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-08-18T15:16:06.697 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-08-18T15:16:06.697 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-08-18T15:16:06.697 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-08-18T15:16:06.697 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-08-18T15:16:06.697 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-08-18T15:16:06.697 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-08-18T15:16:06.698 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-08-18T15:16:06.698 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-08-18T15:16:06.698 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-08-18T15:16:06.698 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-08-18T15:16:06.993 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:00:47 ago on Sun 18 Aug 2024 03:15:19 PM UTC. 2024-08-18T15:16:07.120 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-08-18T15:16:07.121 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-08-18T15:16:07.121 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-08-18T15:16:07.121 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-08-18T15:16:07.121 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-08-18T15:16:07.121 INFO:teuthology.orchestra.run.smithi134.stdout: cephadm noarch 2:18.2.4-717.g03f4c3e8.el9 ceph-noarch 224 k 2024-08-18T15:16:07.121 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:16:07.122 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-08-18T15:16:07.122 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-08-18T15:16:07.122 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-08-18T15:16:07.122 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:16:07.122 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 224 k 2024-08-18T15:16:07.122 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 216 k 2024-08-18T15:16:07.122 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-08-18T15:16:07.346 INFO:teuthology.orchestra.run.smithi134.stdout:cephadm-18.2.4-717.g03f4c3e8.el9.noarch.rpm 1.0 MB/s | 224 kB 00:00 2024-08-18T15:16:07.347 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-08-18T15:16:07.347 INFO:teuthology.orchestra.run.smithi134.stdout:Total 997 kB/s | 224 kB 00:00 2024-08-18T15:16:07.347 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-08-18T15:16:07.351 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-08-18T15:16:07.352 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-08-18T15:16:07.354 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-08-18T15:16:07.354 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-08-18T15:16:07.424 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-08-18T15:16:07.448 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-08-18T15:16:07.448 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:16:07.449 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-08-18T15:16:07.449 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-2:18.2.4-717.g03f4c3e8.el9.x86_64 2024-08-18T15:16:07.449 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mds-2:18.2.4-717.g03f4c3e8.el9.x86_64 2024-08-18T15:16:07.449 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-2:18.2.4-717.g03f4c3e8.el9.x86_64 2024-08-18T15:16:07.449 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-modules-core-2:18.2.4-717.g03f4c3e8.el9.noarch 2024-08-18T15:16:07.449 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mon-2:18.2.4-717.g03f4c3e8.el9.x86_64 2024-08-18T15:16:07.449 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-osd-2:18.2.4-717.g03f4c3e8.el9.x86_64 2024-08-18T15:16:07.449 INFO:teuthology.orchestra.run.smithi193.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-08-18T15:16:07.449 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-08-18T15:16:07.449 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-08-18T15:16:07.449 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-08-18T15:16:07.449 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-08-18T15:16:07.449 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-08-18T15:16:07.450 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-08-18T15:16:07.450 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-08-18T15:16:07.450 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-08-18T15:16:07.450 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-08-18T15:16:07.450 INFO:teuthology.orchestra.run.smithi193.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-08-18T15:16:07.450 INFO:teuthology.orchestra.run.smithi193.stdout: python3-mako-1.1.4-6.el9.noarch 2024-08-18T15:16:07.450 INFO:teuthology.orchestra.run.smithi193.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-08-18T15:16:07.450 INFO:teuthology.orchestra.run.smithi193.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-08-18T15:16:07.450 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-08-18T15:16:07.450 INFO:teuthology.orchestra.run.smithi193.stdout: python3-ply-3.11-14.el9.noarch 2024-08-18T15:16:07.450 INFO:teuthology.orchestra.run.smithi193.stdout: python3-portend-3.1.0-2.el9.noarch 2024-08-18T15:16:07.450 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-08-18T15:16:07.450 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pycparser-2.20-6.el9.noarch 2024-08-18T15:16:07.451 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pytz-2021.1-5.el9.noarch 2024-08-18T15:16:07.451 INFO:teuthology.orchestra.run.smithi193.stdout: python3-requests-2.25.1-8.el9.noarch 2024-08-18T15:16:07.451 INFO:teuthology.orchestra.run.smithi193.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-08-18T15:16:07.451 INFO:teuthology.orchestra.run.smithi193.stdout: python3-toml-0.10.2-6.el9.noarch 2024-08-18T15:16:07.451 INFO:teuthology.orchestra.run.smithi193.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-08-18T15:16:07.451 INFO:teuthology.orchestra.run.smithi193.stdout: python3-webob-1.8.7-6.el9.noarch 2024-08-18T15:16:07.451 INFO:teuthology.orchestra.run.smithi193.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-08-18T15:16:07.451 INFO:teuthology.orchestra.run.smithi193.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-08-18T15:16:07.451 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:16:07.451 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-08-18T15:16:07.794 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph-base 2024-08-18T15:16:07.875 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: cephadm-2:18.2.4-717.g03f4c3e8.el9.noarch 1/1 2024-08-18T15:16:07.958 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : cephadm-2:18.2.4-717.g03f4c3e8.el9.noarch 1/1 2024-08-18T15:16:08.287 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: cephadm-2:18.2.4-717.g03f4c3e8.el9.noarch 1/1 2024-08-18T15:16:08.348 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:00:48 ago on Sun 18 Aug 2024 03:15:20 PM UTC. 2024-08-18T15:16:08.435 INFO:teuthology.orchestra.run.smithi193.stdout:Package ceph-base-2:18.2.4-717.g03f4c3e8.el9.x86_64 is already installed. 2024-08-18T15:16:08.477 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-08-18T15:16:08.478 INFO:teuthology.orchestra.run.smithi193.stdout:Nothing to do. 2024-08-18T15:16:08.478 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-08-18T15:16:08.531 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : cephadm-2:18.2.4-717.g03f4c3e8.el9.noarch 1/1 2024-08-18T15:16:08.531 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:16:08.531 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-08-18T15:16:08.531 INFO:teuthology.orchestra.run.smithi134.stdout: cephadm-2:18.2.4-717.g03f4c3e8.el9.noarch 2024-08-18T15:16:08.531 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:16:08.531 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-08-18T15:16:08.555 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install cephadm 2024-08-18T15:16:08.789 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-immutable-object-cache 2024-08-18T15:16:09.098 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:00:49 ago on Sun 18 Aug 2024 03:15:20 PM UTC. 2024-08-18T15:16:09.226 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-08-18T15:16:09.227 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-08-18T15:16:09.227 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repository Size 2024-08-18T15:16:09.227 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-08-18T15:16:09.227 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-08-18T15:16:09.227 INFO:teuthology.orchestra.run.smithi193.stdout: cephadm noarch 2:18.2.4-717.g03f4c3e8.el9 ceph-noarch 224 k 2024-08-18T15:16:09.227 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:16:09.227 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-08-18T15:16:09.227 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-08-18T15:16:09.227 INFO:teuthology.orchestra.run.smithi193.stdout:Install 1 Package 2024-08-18T15:16:09.228 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:16:09.228 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 224 k 2024-08-18T15:16:09.228 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 216 k 2024-08-18T15:16:09.228 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-08-18T15:16:09.351 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:00:50 ago on Sun 18 Aug 2024 03:15:19 PM UTC. 2024-08-18T15:16:09.463 INFO:teuthology.orchestra.run.smithi193.stdout:cephadm-18.2.4-717.g03f4c3e8.el9.noarch.rpm 955 kB/s | 224 kB 00:00 2024-08-18T15:16:09.464 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-08-18T15:16:09.464 INFO:teuthology.orchestra.run.smithi193.stdout:Total 949 kB/s | 224 kB 00:00 2024-08-18T15:16:09.464 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-08-18T15:16:09.470 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-08-18T15:16:09.470 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-08-18T15:16:09.472 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-08-18T15:16:09.473 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-08-18T15:16:09.483 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-08-18T15:16:09.484 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-08-18T15:16:09.484 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-08-18T15:16:09.484 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-08-18T15:16:09.484 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-08-18T15:16:09.484 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-immutable-object-cache x86_64 2:18.2.4-717.g03f4c3e8.el9 ceph 143 k 2024-08-18T15:16:09.484 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:16:09.484 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-08-18T15:16:09.484 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-08-18T15:16:09.484 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-08-18T15:16:09.484 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:16:09.485 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 143 k 2024-08-18T15:16:09.485 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 395 k 2024-08-18T15:16:09.485 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-08-18T15:16:09.545 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-08-18T15:16:09.686 INFO:teuthology.orchestra.run.smithi134.stdout:ceph-immutable-object-cache-18.2.4-717.g03f4c3e 712 kB/s | 143 kB 00:00 2024-08-18T15:16:09.687 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-08-18T15:16:09.687 INFO:teuthology.orchestra.run.smithi134.stdout:Total 706 kB/s | 143 kB 00:00 2024-08-18T15:16:09.687 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-08-18T15:16:09.694 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-08-18T15:16:09.694 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-08-18T15:16:09.751 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-08-18T15:16:09.751 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-08-18T15:16:09.902 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-08-18T15:16:09.987 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-immutable-object-cache-2:18.2.4-717.g03f4c3e8.e 1/1 2024-08-18T15:16:10.001 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: cephadm-2:18.2.4-717.g03f4c3e8.el9.noarch 1/1 2024-08-18T15:16:10.012 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.4-717.g03f4c3e8.e 1/1 2024-08-18T15:16:10.013 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-18T15:16:10.013 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-08-18T15:16:10.013 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:16:10.075 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : cephadm-2:18.2.4-717.g03f4c3e8.el9.noarch 1/1 2024-08-18T15:16:10.408 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: cephadm-2:18.2.4-717.g03f4c3e8.el9.noarch 1/1 2024-08-18T15:16:10.647 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : cephadm-2:18.2.4-717.g03f4c3e8.el9.noarch 1/1 2024-08-18T15:16:10.648 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:16:10.648 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-08-18T15:16:10.648 INFO:teuthology.orchestra.run.smithi193.stdout: cephadm-2:18.2.4-717.g03f4c3e8.el9.noarch 2024-08-18T15:16:10.648 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:16:10.648 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-08-18T15:16:10.880 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph-immutable-object-cache 2024-08-18T15:16:10.935 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-immutable-object-cache-2:18.2.4-717.g03f4c3e8.e 1/1 2024-08-18T15:16:10.935 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:16:10.935 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-08-18T15:16:10.935 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-immutable-object-cache-2:18.2.4-717.g03f4c3e8.el9.x86_64 2024-08-18T15:16:10.935 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:16:10.935 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-08-18T15:16:11.178 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-mgr 2024-08-18T15:16:11.438 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:00:51 ago on Sun 18 Aug 2024 03:15:20 PM UTC. 2024-08-18T15:16:11.566 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-08-18T15:16:11.567 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-08-18T15:16:11.567 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repo Size 2024-08-18T15:16:11.567 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-08-18T15:16:11.567 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-08-18T15:16:11.567 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-immutable-object-cache x86_64 2:18.2.4-717.g03f4c3e8.el9 ceph 143 k 2024-08-18T15:16:11.568 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:16:11.568 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-08-18T15:16:11.568 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-08-18T15:16:11.568 INFO:teuthology.orchestra.run.smithi193.stdout:Install 1 Package 2024-08-18T15:16:11.568 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:16:11.568 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 143 k 2024-08-18T15:16:11.568 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 395 k 2024-08-18T15:16:11.568 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-08-18T15:16:11.746 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:00:52 ago on Sun 18 Aug 2024 03:15:19 PM UTC. 2024-08-18T15:16:11.795 INFO:teuthology.orchestra.run.smithi193.stdout:ceph-immutable-object-cache-18.2.4-717.g03f4c3e 630 kB/s | 143 kB 00:00 2024-08-18T15:16:11.795 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-08-18T15:16:11.796 INFO:teuthology.orchestra.run.smithi193.stdout:Total 625 kB/s | 143 kB 00:00 2024-08-18T15:16:11.796 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-08-18T15:16:11.803 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-08-18T15:16:11.804 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-08-18T15:16:11.839 INFO:teuthology.orchestra.run.smithi134.stdout:Package ceph-mgr-2:18.2.4-717.g03f4c3e8.el9.x86_64 is already installed. 2024-08-18T15:16:11.860 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-08-18T15:16:11.861 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-08-18T15:16:11.882 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-08-18T15:16:11.883 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-08-18T15:16:11.884 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-08-18T15:16:11.959 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-mgr-dashboard 2024-08-18T15:16:12.007 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-08-18T15:16:12.096 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-immutable-object-cache-2:18.2.4-717.g03f4c3e8.e 1/1 2024-08-18T15:16:12.121 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.4-717.g03f4c3e8.e 1/1 2024-08-18T15:16:12.121 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-18T15:16:12.121 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-08-18T15:16:12.121 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:16:12.505 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:00:53 ago on Sun 18 Aug 2024 03:15:19 PM UTC. 2024-08-18T15:16:12.634 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-08-18T15:16:12.635 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-08-18T15:16:12.635 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-08-18T15:16:12.635 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-08-18T15:16:12.635 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-08-18T15:16:12.635 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-dashboard noarch 2:18.2.4-717.g03f4c3e8.el9 ceph-noarch 3.5 M 2024-08-18T15:16:12.635 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-08-18T15:16:12.635 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-grafana-dashboards noarch 2:18.2.4-717.g03f4c3e8.el9 ceph-noarch 24 k 2024-08-18T15:16:12.635 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-prometheus-alerts noarch 2:18.2.4-717.g03f4c3e8.el9 ceph-noarch 16 k 2024-08-18T15:16:12.636 INFO:teuthology.orchestra.run.smithi134.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-08-18T15:16:12.636 INFO:teuthology.orchestra.run.smithi134.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-08-18T15:16:12.636 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:16:12.636 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-08-18T15:16:12.636 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-08-18T15:16:12.636 INFO:teuthology.orchestra.run.smithi134.stdout:Install 5 Packages 2024-08-18T15:16:12.636 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:16:12.636 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 3.8 M 2024-08-18T15:16:12.637 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 87 M 2024-08-18T15:16:12.637 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-08-18T15:16:13.004 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-immutable-object-cache-2:18.2.4-717.g03f4c3e8.e 1/1 2024-08-18T15:16:13.004 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:16:13.004 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-08-18T15:16:13.004 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-immutable-object-cache-2:18.2.4-717.g03f4c3e8.el9.x86_64 2024-08-18T15:16:13.004 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:16:13.004 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-08-18T15:16:13.248 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph-mgr 2024-08-18T15:16:13.267 INFO:teuthology.orchestra.run.smithi134.stdout:(1/5): ceph-grafana-dashboards-18.2.4-717.g03f4 128 kB/s | 24 kB 00:00 2024-08-18T15:16:13.292 INFO:teuthology.orchestra.run.smithi134.stdout:(2/5): ceph-prometheus-alerts-18.2.4-717.g03f4c 76 kB/s | 16 kB 00:00 2024-08-18T15:16:13.384 INFO:teuthology.orchestra.run.smithi134.stdout:(3/5): python3-repoze-lru-0.7-16.el9.noarch.rpm 263 kB/s | 31 kB 00:00 2024-08-18T15:16:13.426 INFO:teuthology.orchestra.run.smithi134.stdout:(4/5): python3-routes-2.5.1-5.el9.noarch.rpm 1.4 MB/s | 188 kB 00:00 2024-08-18T15:16:13.543 INFO:teuthology.orchestra.run.smithi134.stdout:(5/5): ceph-mgr-dashboard-18.2.4-717.g03f4c3e8. 7.6 MB/s | 3.5 MB 00:00 2024-08-18T15:16:13.547 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-08-18T15:16:13.547 INFO:teuthology.orchestra.run.smithi134.stdout:Total 4.2 MB/s | 3.8 MB 00:00 2024-08-18T15:16:13.547 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-08-18T15:16:13.554 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-08-18T15:16:13.554 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-08-18T15:16:13.615 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-08-18T15:16:13.616 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-08-18T15:16:13.811 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:00:53 ago on Sun 18 Aug 2024 03:15:20 PM UTC. 2024-08-18T15:16:13.897 INFO:teuthology.orchestra.run.smithi193.stdout:Package ceph-mgr-2:18.2.4-717.g03f4c3e8.el9.x86_64 is already installed. 2024-08-18T15:16:13.915 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-08-18T15:16:13.939 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-08-18T15:16:13.940 INFO:teuthology.orchestra.run.smithi193.stdout:Nothing to do. 2024-08-18T15:16:13.941 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-08-18T15:16:13.991 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/5 2024-08-18T15:16:14.012 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph-mgr-dashboard 2024-08-18T15:16:14.023 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/5 2024-08-18T15:16:14.051 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-prometheus-alerts-2:18.2.4-717.g03f4c3e8.el9.no 3/5 2024-08-18T15:16:14.596 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:00:54 ago on Sun 18 Aug 2024 03:15:20 PM UTC. 2024-08-18T15:16:14.732 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-08-18T15:16:14.733 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-08-18T15:16:14.733 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repository Size 2024-08-18T15:16:14.733 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-08-18T15:16:14.733 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-08-18T15:16:14.733 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-dashboard noarch 2:18.2.4-717.g03f4c3e8.el9 ceph-noarch 3.5 M 2024-08-18T15:16:14.733 INFO:teuthology.orchestra.run.smithi193.stdout:Installing dependencies: 2024-08-18T15:16:14.734 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-grafana-dashboards noarch 2:18.2.4-717.g03f4c3e8.el9 ceph-noarch 24 k 2024-08-18T15:16:14.734 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-prometheus-alerts noarch 2:18.2.4-717.g03f4c3e8.el9 ceph-noarch 16 k 2024-08-18T15:16:14.734 INFO:teuthology.orchestra.run.smithi193.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-08-18T15:16:14.734 INFO:teuthology.orchestra.run.smithi193.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-08-18T15:16:14.734 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:16:14.734 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-08-18T15:16:14.734 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-08-18T15:16:14.734 INFO:teuthology.orchestra.run.smithi193.stdout:Install 5 Packages 2024-08-18T15:16:14.734 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:16:14.735 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 3.8 M 2024-08-18T15:16:14.735 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 87 M 2024-08-18T15:16:14.735 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-08-18T15:16:15.098 INFO:teuthology.orchestra.run.smithi193.stdout:(1/5): ceph-grafana-dashboards-18.2.4-717.g03f4 86 kB/s | 24 kB 00:00 2024-08-18T15:16:15.098 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-grafana-dashboards-2:18.2.4-717.g03f4c3e8.el9.n 4/5 2024-08-18T15:16:15.138 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-dashboard-2:18.2.4-717.g03f4c3e8.el9.noarch 5/5 2024-08-18T15:16:15.148 INFO:teuthology.orchestra.run.smithi193.stdout:(2/5): ceph-prometheus-alerts-18.2.4-717.g03f4c 49 kB/s | 16 kB 00:00 2024-08-18T15:16:15.198 INFO:teuthology.orchestra.run.smithi193.stdout:(3/5): python3-repoze-lru-0.7-16.el9.noarch.rpm 307 kB/s | 31 kB 00:00 2024-08-18T15:16:15.240 INFO:teuthology.orchestra.run.smithi193.stdout:(4/5): python3-routes-2.5.1-5.el9.noarch.rpm 2.1 MB/s | 188 kB 00:00 2024-08-18T15:16:15.749 INFO:teuthology.orchestra.run.smithi193.stdout:(5/5): ceph-mgr-dashboard-18.2.4-717.g03f4c3e8. 3.8 MB/s | 3.5 MB 00:00 2024-08-18T15:16:15.752 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-08-18T15:16:15.753 INFO:teuthology.orchestra.run.smithi193.stdout:Total 3.7 MB/s | 3.8 MB 00:01 2024-08-18T15:16:15.753 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-08-18T15:16:15.760 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-08-18T15:16:15.760 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-08-18T15:16:15.820 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-08-18T15:16:15.821 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-08-18T15:16:16.132 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-08-18T15:16:16.213 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/5 2024-08-18T15:16:16.254 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/5 2024-08-18T15:16:16.259 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.4-717.g03f4c3e8.el9.noarch 5/5 2024-08-18T15:16:16.260 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-grafana-dashboards-2:18.2.4-717.g03f4c3e8.el9.n 1/5 2024-08-18T15:16:16.260 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-dashboard-2:18.2.4-717.g03f4c3e8.el9.noarch 2/5 2024-08-18T15:16:16.260 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-prometheus-alerts-2:18.2.4-717.g03f4c3e8.el9.no 3/5 2024-08-18T15:16:16.260 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 4/5 2024-08-18T15:16:16.283 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-prometheus-alerts-2:18.2.4-717.g03f4c3e8.el9.no 3/5 2024-08-18T15:16:16.518 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 5/5 2024-08-18T15:16:16.518 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:16:16.518 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-08-18T15:16:16.518 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-grafana-dashboards-2:18.2.4-717.g03f4c3e8.el9.noarch 2024-08-18T15:16:16.518 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-dashboard-2:18.2.4-717.g03f4c3e8.el9.noarch 2024-08-18T15:16:16.518 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-prometheus-alerts-2:18.2.4-717.g03f4c3e8.el9.noarch 2024-08-18T15:16:16.518 INFO:teuthology.orchestra.run.smithi134.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-08-18T15:16:16.518 INFO:teuthology.orchestra.run.smithi134.stdout: python3-routes-2.5.1-5.el9.noarch 2024-08-18T15:16:16.519 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:16:16.519 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-08-18T15:16:16.754 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-mgr-diskprediction-local 2024-08-18T15:16:17.300 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:00:58 ago on Sun 18 Aug 2024 03:15:19 PM UTC. 2024-08-18T15:16:17.325 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-grafana-dashboards-2:18.2.4-717.g03f4c3e8.el9.n 4/5 2024-08-18T15:16:17.371 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mgr-dashboard-2:18.2.4-717.g03f4c3e8.el9.noarch 5/5 2024-08-18T15:16:17.428 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-08-18T15:16:17.428 INFO:teuthology.orchestra.run.smithi134.stdout:====================================================================================== 2024-08-18T15:16:17.429 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-08-18T15:16:17.429 INFO:teuthology.orchestra.run.smithi134.stdout:====================================================================================== 2024-08-18T15:16:17.429 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-08-18T15:16:17.429 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.4-717.g03f4c3e8.el9 ceph-noarch 7.4 M 2024-08-18T15:16:17.429 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-08-18T15:16:17.429 INFO:teuthology.orchestra.run.smithi134.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-08-18T15:16:17.429 INFO:teuthology.orchestra.run.smithi134.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-08-18T15:16:17.429 INFO:teuthology.orchestra.run.smithi134.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-08-18T15:16:17.429 INFO:teuthology.orchestra.run.smithi134.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-08-18T15:16:17.430 INFO:teuthology.orchestra.run.smithi134.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-08-18T15:16:17.430 INFO:teuthology.orchestra.run.smithi134.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-08-18T15:16:17.430 INFO:teuthology.orchestra.run.smithi134.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-08-18T15:16:17.430 INFO:teuthology.orchestra.run.smithi134.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-08-18T15:16:17.430 INFO:teuthology.orchestra.run.smithi134.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-08-18T15:16:17.430 INFO:teuthology.orchestra.run.smithi134.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-08-18T15:16:17.430 INFO:teuthology.orchestra.run.smithi134.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-08-18T15:16:17.430 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:16:17.430 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-08-18T15:16:17.430 INFO:teuthology.orchestra.run.smithi134.stdout:====================================================================================== 2024-08-18T15:16:17.430 INFO:teuthology.orchestra.run.smithi134.stdout:Install 12 Packages 2024-08-18T15:16:17.431 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:16:17.431 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 38 M 2024-08-18T15:16:17.431 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 209 M 2024-08-18T15:16:17.431 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-08-18T15:16:17.733 INFO:teuthology.orchestra.run.smithi134.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.6 MB/s | 195 kB 00:00 2024-08-18T15:16:17.767 INFO:teuthology.orchestra.run.smithi134.stdout:(2/12): libgfortran-11.4.1-3.el9.x86_64.rpm 5.1 MB/s | 803 kB 00:00 2024-08-18T15:16:17.792 INFO:teuthology.orchestra.run.smithi134.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 557 kB/s | 33 kB 00:00 2024-08-18T15:16:17.834 INFO:teuthology.orchestra.run.smithi134.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 438 kB/s | 18 kB 00:00 2024-08-18T15:16:17.876 INFO:teuthology.orchestra.run.smithi134.stdout:(5/12): openblas-0.3.21-2.el9.x86_64.rpm 841 kB/s | 35 kB 00:00 2024-08-18T15:16:17.926 INFO:teuthology.orchestra.run.smithi134.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 19 MB/s | 3.0 MB 00:00 2024-08-18T15:16:17.968 INFO:teuthology.orchestra.run.smithi134.stdout:(7/12): python3-devel-3.9.18-3.el9.x86_64.rpm 5.8 MB/s | 244 kB 00:00 2024-08-18T15:16:18.119 INFO:teuthology.orchestra.run.smithi134.stdout:(8/12): ceph-mgr-diskprediction-local-18.2.4-71 15 MB/s | 7.4 MB 00:00 2024-08-18T15:16:18.227 INFO:teuthology.orchestra.run.smithi134.stdout:(9/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 20 MB/s | 5.1 MB 00:00 2024-08-18T15:16:18.302 INFO:teuthology.orchestra.run.smithi134.stdout:(10/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 1.3 MB/s | 250 kB 00:00 2024-08-18T15:16:18.377 INFO:teuthology.orchestra.run.smithi134.stdout:(11/12): openblas-openmp-0.3.21-2.el9.x86_64.rp 9.5 MB/s | 4.7 MB 00:00 2024-08-18T15:16:18.407 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.4-717.g03f4c3e8.el9.noarch 5/5 2024-08-18T15:16:18.408 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-grafana-dashboards-2:18.2.4-717.g03f4c3e8.el9.n 1/5 2024-08-18T15:16:18.408 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mgr-dashboard-2:18.2.4-717.g03f4c3e8.el9.noarch 2/5 2024-08-18T15:16:18.408 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-prometheus-alerts-2:18.2.4-717.g03f4c3e8.el9.no 3/5 2024-08-18T15:16:18.408 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 4/5 2024-08-18T15:16:18.677 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 5/5 2024-08-18T15:16:18.677 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:16:18.678 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-08-18T15:16:18.678 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-grafana-dashboards-2:18.2.4-717.g03f4c3e8.el9.noarch 2024-08-18T15:16:18.678 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-dashboard-2:18.2.4-717.g03f4c3e8.el9.noarch 2024-08-18T15:16:18.678 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-prometheus-alerts-2:18.2.4-717.g03f4c3e8.el9.noarch 2024-08-18T15:16:18.678 INFO:teuthology.orchestra.run.smithi193.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-08-18T15:16:18.678 INFO:teuthology.orchestra.run.smithi193.stdout: python3-routes-2.5.1-5.el9.noarch 2024-08-18T15:16:18.678 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:16:18.678 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-08-18T15:16:18.744 INFO:teuthology.orchestra.run.smithi134.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 31 MB/s | 16 MB 00:00 2024-08-18T15:16:18.745 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-08-18T15:16:18.745 INFO:teuthology.orchestra.run.smithi134.stdout:Total 29 MB/s | 38 MB 00:01 2024-08-18T15:16:18.898 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph-mgr-diskprediction-local 2024-08-18T15:16:19.096 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-08-18T15:16:19.122 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-08-18T15:16:19.123 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-08-18T15:16:19.443 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:00:59 ago on Sun 18 Aug 2024 03:15:20 PM UTC. 2024-08-18T15:16:19.570 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-08-18T15:16:19.571 INFO:teuthology.orchestra.run.smithi193.stdout:====================================================================================== 2024-08-18T15:16:19.571 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repository Size 2024-08-18T15:16:19.571 INFO:teuthology.orchestra.run.smithi193.stdout:====================================================================================== 2024-08-18T15:16:19.571 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-08-18T15:16:19.571 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.4-717.g03f4c3e8.el9 ceph-noarch 7.4 M 2024-08-18T15:16:19.572 INFO:teuthology.orchestra.run.smithi193.stdout:Installing dependencies: 2024-08-18T15:16:19.572 INFO:teuthology.orchestra.run.smithi193.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-08-18T15:16:19.572 INFO:teuthology.orchestra.run.smithi193.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-08-18T15:16:19.572 INFO:teuthology.orchestra.run.smithi193.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-08-18T15:16:19.572 INFO:teuthology.orchestra.run.smithi193.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-08-18T15:16:19.573 INFO:teuthology.orchestra.run.smithi193.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-08-18T15:16:19.573 INFO:teuthology.orchestra.run.smithi193.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-08-18T15:16:19.573 INFO:teuthology.orchestra.run.smithi193.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-08-18T15:16:19.573 INFO:teuthology.orchestra.run.smithi193.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-08-18T15:16:19.573 INFO:teuthology.orchestra.run.smithi193.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-08-18T15:16:19.573 INFO:teuthology.orchestra.run.smithi193.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-08-18T15:16:19.574 INFO:teuthology.orchestra.run.smithi193.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-08-18T15:16:19.574 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:16:19.574 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-08-18T15:16:19.574 INFO:teuthology.orchestra.run.smithi193.stdout:====================================================================================== 2024-08-18T15:16:19.574 INFO:teuthology.orchestra.run.smithi193.stdout:Install 12 Packages 2024-08-18T15:16:19.575 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:16:19.576 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 38 M 2024-08-18T15:16:19.576 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 209 M 2024-08-18T15:16:19.576 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-08-18T15:16:19.620 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-08-18T15:16:19.621 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-08-18T15:16:20.283 INFO:teuthology.orchestra.run.smithi193.stdout:(1/12): ceph-mgr-diskprediction-local-18.2.4-71 15 MB/s | 7.4 MB 00:00 2024-08-18T15:16:20.306 INFO:teuthology.orchestra.run.smithi193.stdout:(2/12): libquadmath-11.4.1-3.el9.x86_64.rpm 388 kB/s | 195 kB 00:00 2024-08-18T15:16:20.367 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-08-18T15:16:20.415 INFO:teuthology.orchestra.run.smithi193.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 247 kB/s | 33 kB 00:00 2024-08-18T15:16:20.428 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-08-18T15:16:20.466 INFO:teuthology.orchestra.run.smithi193.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 362 kB/s | 18 kB 00:00 2024-08-18T15:16:20.474 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-08-18T15:16:20.516 INFO:teuthology.orchestra.run.smithi193.stdout:(5/12): openblas-0.3.21-2.el9.x86_64.rpm 705 kB/s | 35 kB 00:00 2024-08-18T15:16:20.549 INFO:teuthology.orchestra.run.smithi193.stdout:(6/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.1 MB/s | 803 kB 00:00 2024-08-18T15:16:20.575 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-08-18T15:16:20.610 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-08-18T15:16:20.725 INFO:teuthology.orchestra.run.smithi193.stdout:(7/12): python3-devel-3.9.18-3.el9.x86_64.rpm 1.4 MB/s | 244 kB 00:00 2024-08-18T15:16:20.787 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-08-18T15:16:20.832 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-08-18T15:16:20.917 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-08-18T15:16:21.042 INFO:teuthology.orchestra.run.smithi193.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 4.1 MB/s | 3.0 MB 00:00 2024-08-18T15:16:21.125 INFO:teuthology.orchestra.run.smithi193.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.9 MB/s | 250 kB 00:00 2024-08-18T15:16:21.570 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-08-18T15:16:21.696 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-08-18T15:16:21.742 INFO:teuthology.orchestra.run.smithi193.stdout:(10/12): openblas-openmp-0.3.21-2.el9.x86_64.rp 3.9 MB/s | 4.7 MB 00:01 2024-08-18T15:16:22.009 INFO:teuthology.orchestra.run.smithi193.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 4.0 MB/s | 5.1 MB 00:01 2024-08-18T15:16:22.922 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-08-18T15:16:23.601 INFO:teuthology.orchestra.run.smithi193.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 6.6 MB/s | 16 MB 00:02 2024-08-18T15:16:23.602 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-08-18T15:16:23.602 INFO:teuthology.orchestra.run.smithi193.stdout:Total 9.4 MB/s | 38 MB 00:04 2024-08-18T15:16:23.950 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-08-18T15:16:23.953 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-08-18T15:16:23.974 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.4-717.g03f4c3 12/12 2024-08-18T15:16:23.980 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-08-18T15:16:23.980 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-08-18T15:16:24.482 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-08-18T15:16:24.483 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-08-18T15:16:25.260 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-08-18T15:16:25.320 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-08-18T15:16:25.378 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-08-18T15:16:25.474 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-08-18T15:16:25.508 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-08-18T15:16:25.688 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-08-18T15:16:25.724 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-08-18T15:16:25.806 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-08-18T15:16:26.433 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.4-717.g03f4c3 12/12 2024-08-18T15:16:26.434 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.4-717.g03f4c3 1/12 2024-08-18T15:16:26.434 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-08-18T15:16:26.434 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-08-18T15:16:26.434 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-08-18T15:16:26.434 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-08-18T15:16:26.434 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-08-18T15:16:26.434 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-08-18T15:16:26.434 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-08-18T15:16:26.434 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-08-18T15:16:26.434 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-08-18T15:16:26.434 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-08-18T15:16:26.460 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-08-18T15:16:26.579 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-08-18T15:16:26.818 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-08-18T15:16:26.818 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:16:26.818 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-08-18T15:16:26.818 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-diskprediction-local-2:18.2.4-717.g03f4c3e8.el9.noarch 2024-08-18T15:16:26.819 INFO:teuthology.orchestra.run.smithi134.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-08-18T15:16:26.819 INFO:teuthology.orchestra.run.smithi134.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-08-18T15:16:26.819 INFO:teuthology.orchestra.run.smithi134.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-08-18T15:16:26.819 INFO:teuthology.orchestra.run.smithi134.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-08-18T15:16:26.819 INFO:teuthology.orchestra.run.smithi134.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-08-18T15:16:26.819 INFO:teuthology.orchestra.run.smithi134.stdout: openblas-0.3.21-2.el9.x86_64 2024-08-18T15:16:26.819 INFO:teuthology.orchestra.run.smithi134.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-08-18T15:16:26.819 INFO:teuthology.orchestra.run.smithi134.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-08-18T15:16:26.819 INFO:teuthology.orchestra.run.smithi134.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-08-18T15:16:26.819 INFO:teuthology.orchestra.run.smithi134.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-08-18T15:16:26.819 INFO:teuthology.orchestra.run.smithi134.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-08-18T15:16:26.819 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:16:26.819 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-08-18T15:16:27.210 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-mgr-rook 2024-08-18T15:16:27.772 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:08 ago on Sun 18 Aug 2024 03:15:19 PM UTC. 2024-08-18T15:16:27.817 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-08-18T15:16:27.901 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-08-18T15:16:27.902 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-08-18T15:16:27.903 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-08-18T15:16:27.903 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-08-18T15:16:27.903 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-08-18T15:16:27.903 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-rook noarch 2:18.2.4-717.g03f4c3e8.el9 ceph-noarch 49 k 2024-08-18T15:16:27.903 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-08-18T15:16:27.904 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-08-18T15:16:27.904 INFO:teuthology.orchestra.run.smithi134.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-08-18T15:16:27.904 INFO:teuthology.orchestra.run.smithi134.stdout: python3-google-auth noarch 1:2.33.0-1.el9 epel 220 k 2024-08-18T15:16:27.904 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-08-18T15:16:27.904 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-08-18T15:16:27.905 INFO:teuthology.orchestra.run.smithi134.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-08-18T15:16:27.905 INFO:teuthology.orchestra.run.smithi134.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-08-18T15:16:27.905 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-08-18T15:16:27.905 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-08-18T15:16:27.905 INFO:teuthology.orchestra.run.smithi134.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-08-18T15:16:27.905 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-08-18T15:16:27.906 INFO:teuthology.orchestra.run.smithi134.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-08-18T15:16:27.906 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:16:27.906 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-08-18T15:16:27.906 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-08-18T15:16:27.906 INFO:teuthology.orchestra.run.smithi134.stdout:Install 13 Packages 2024-08-18T15:16:27.906 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:16:27.907 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 2.2 M 2024-08-18T15:16:27.908 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 26 M 2024-08-18T15:16:27.908 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-08-18T15:16:28.506 INFO:teuthology.orchestra.run.smithi134.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 140 kB/s | 19 kB 00:00 2024-08-18T15:16:28.532 INFO:teuthology.orchestra.run.smithi134.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 158 kB/s | 26 kB 00:00 2024-08-18T15:16:28.557 INFO:teuthology.orchestra.run.smithi134.stdout:(3/13): ceph-mgr-rook-18.2.4-717.g03f4c3e8.el9. 263 kB/s | 49 kB 00:00 2024-08-18T15:16:28.624 INFO:teuthology.orchestra.run.smithi134.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.9 MB/s | 222 kB 00:00 2024-08-18T15:16:28.649 INFO:teuthology.orchestra.run.smithi134.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.3 MB/s | 159 kB 00:00 2024-08-18T15:16:28.699 INFO:teuthology.orchestra.run.smithi134.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 713 kB/s | 54 kB 00:00 2024-08-18T15:16:28.750 INFO:teuthology.orchestra.run.smithi134.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 321 kB/s | 32 kB 00:00 2024-08-18T15:16:28.783 INFO:teuthology.orchestra.run.smithi134.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 173 kB/s | 14 kB 00:00 2024-08-18T15:16:28.808 INFO:teuthology.orchestra.run.smithi134.stdout:(9/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.1 MB/s | 279 kB 00:00 2024-08-18T15:16:28.859 INFO:teuthology.orchestra.run.smithi134.stdout:(10/13): python3-google-auth-2.33.0-1.el9.noarc 2.0 MB/s | 220 kB 00:00 2024-08-18T15:16:28.867 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-08-18T15:16:28.900 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.4-717.g03f4c3 12/12 2024-08-18T15:16:28.909 INFO:teuthology.orchestra.run.smithi134.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 587 kB/s | 59 kB 00:00 2024-08-18T15:16:28.934 INFO:teuthology.orchestra.run.smithi134.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.2 MB/s | 90 kB 00:00 2024-08-18T15:16:28.968 INFO:teuthology.orchestra.run.smithi134.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 5.5 MB/s | 1.0 MB 00:00 2024-08-18T15:16:28.970 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-08-18T15:16:28.970 INFO:teuthology.orchestra.run.smithi134.stdout:Total 2.1 MB/s | 2.2 MB 00:01 2024-08-18T15:16:29.033 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-08-18T15:16:29.048 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-08-18T15:16:29.048 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-08-18T15:16:29.133 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-08-18T15:16:29.133 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-08-18T15:16:29.630 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-08-18T15:16:29.807 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-08-18T15:16:29.860 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-08-18T15:16:29.924 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-08-18T15:16:29.961 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-08-18T15:16:30.003 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-08-18T15:16:30.103 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-08-18T15:16:30.214 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-google-auth-1:2.33.0-1.el9.noarch 7/13 2024-08-18T15:16:30.271 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-08-18T15:16:30.978 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-08-18T15:16:31.026 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-08-18T15:16:31.060 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-08-18T15:16:31.100 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-08-18T15:16:31.130 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-rook-2:18.2.4-717.g03f4c3e8.el9.noarch 13/13 2024-08-18T15:16:31.424 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.4-717.g03f4c3 12/12 2024-08-18T15:16:31.424 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.4-717.g03f4c3 1/12 2024-08-18T15:16:31.424 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-08-18T15:16:31.424 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-08-18T15:16:31.424 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-08-18T15:16:31.424 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-08-18T15:16:31.425 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-08-18T15:16:31.425 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-08-18T15:16:31.425 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-08-18T15:16:31.425 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-08-18T15:16:31.425 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-08-18T15:16:31.425 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-08-18T15:16:31.745 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-08-18T15:16:31.745 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:16:31.745 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-08-18T15:16:31.745 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-diskprediction-local-2:18.2.4-717.g03f4c3e8.el9.noarch 2024-08-18T15:16:31.745 INFO:teuthology.orchestra.run.smithi193.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-08-18T15:16:31.745 INFO:teuthology.orchestra.run.smithi193.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-08-18T15:16:31.746 INFO:teuthology.orchestra.run.smithi193.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-08-18T15:16:31.746 INFO:teuthology.orchestra.run.smithi193.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-08-18T15:16:31.746 INFO:teuthology.orchestra.run.smithi193.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-08-18T15:16:31.746 INFO:teuthology.orchestra.run.smithi193.stdout: openblas-0.3.21-2.el9.x86_64 2024-08-18T15:16:31.746 INFO:teuthology.orchestra.run.smithi193.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-08-18T15:16:31.746 INFO:teuthology.orchestra.run.smithi193.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-08-18T15:16:31.746 INFO:teuthology.orchestra.run.smithi193.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-08-18T15:16:31.746 INFO:teuthology.orchestra.run.smithi193.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-08-18T15:16:31.746 INFO:teuthology.orchestra.run.smithi193.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-08-18T15:16:31.746 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:16:31.746 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-08-18T15:16:31.893 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.4-717.g03f4c3e8.el9.noarch 13/13 2024-08-18T15:16:31.893 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-rook-2:18.2.4-717.g03f4c3e8.el9.noarch 1/13 2024-08-18T15:16:31.893 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-08-18T15:16:31.893 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-08-18T15:16:31.893 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-08-18T15:16:31.893 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-08-18T15:16:31.893 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-08-18T15:16:31.893 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-08-18T15:16:31.893 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-08-18T15:16:31.893 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-08-18T15:16:31.893 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-google-auth-1:2.33.0-1.el9.noarch 10/13 2024-08-18T15:16:31.893 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-08-18T15:16:31.894 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-08-18T15:16:32.109 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph-mgr-rook 2024-08-18T15:16:32.225 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-08-18T15:16:32.225 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:16:32.225 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-08-18T15:16:32.225 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-rook-2:18.2.4-717.g03f4c3e8.el9.noarch 2024-08-18T15:16:32.225 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-08-18T15:16:32.225 INFO:teuthology.orchestra.run.smithi134.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-08-18T15:16:32.225 INFO:teuthology.orchestra.run.smithi134.stdout: python3-google-auth-1:2.33.0-1.el9.noarch 2024-08-18T15:16:32.225 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-08-18T15:16:32.225 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-08-18T15:16:32.225 INFO:teuthology.orchestra.run.smithi134.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-08-18T15:16:32.225 INFO:teuthology.orchestra.run.smithi134.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-08-18T15:16:32.226 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-08-18T15:16:32.226 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-08-18T15:16:32.226 INFO:teuthology.orchestra.run.smithi134.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-08-18T15:16:32.226 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rsa-4.9-2.el9.noarch 2024-08-18T15:16:32.226 INFO:teuthology.orchestra.run.smithi134.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-08-18T15:16:32.226 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:16:32.226 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-08-18T15:16:32.452 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-mgr-cephadm 2024-08-18T15:16:32.716 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:01:12 ago on Sun 18 Aug 2024 03:15:20 PM UTC. 2024-08-18T15:16:32.845 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-08-18T15:16:32.846 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-08-18T15:16:32.846 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repository Size 2024-08-18T15:16:32.846 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-08-18T15:16:32.846 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-08-18T15:16:32.846 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-rook noarch 2:18.2.4-717.g03f4c3e8.el9 ceph-noarch 49 k 2024-08-18T15:16:32.846 INFO:teuthology.orchestra.run.smithi193.stdout:Installing dependencies: 2024-08-18T15:16:32.846 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-08-18T15:16:32.846 INFO:teuthology.orchestra.run.smithi193.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-08-18T15:16:32.846 INFO:teuthology.orchestra.run.smithi193.stdout: python3-google-auth noarch 1:2.33.0-1.el9 epel 220 k 2024-08-18T15:16:32.846 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-08-18T15:16:32.846 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-08-18T15:16:32.847 INFO:teuthology.orchestra.run.smithi193.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-08-18T15:16:32.847 INFO:teuthology.orchestra.run.smithi193.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-08-18T15:16:32.847 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-08-18T15:16:32.847 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-08-18T15:16:32.847 INFO:teuthology.orchestra.run.smithi193.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-08-18T15:16:32.847 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-08-18T15:16:32.847 INFO:teuthology.orchestra.run.smithi193.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-08-18T15:16:32.847 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:16:32.847 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-08-18T15:16:32.847 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-08-18T15:16:32.847 INFO:teuthology.orchestra.run.smithi193.stdout:Install 13 Packages 2024-08-18T15:16:32.847 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:16:32.848 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 2.2 M 2024-08-18T15:16:32.848 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 26 M 2024-08-18T15:16:32.848 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-08-18T15:16:33.033 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:14 ago on Sun 18 Aug 2024 03:15:19 PM UTC. 2024-08-18T15:16:33.167 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-08-18T15:16:33.168 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-08-18T15:16:33.168 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-08-18T15:16:33.168 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-08-18T15:16:33.168 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-08-18T15:16:33.168 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-cephadm noarch 2:18.2.4-717.g03f4c3e8.el9 ceph-noarch 139 k 2024-08-18T15:16:33.168 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-08-18T15:16:33.168 INFO:teuthology.orchestra.run.smithi134.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-08-18T15:16:33.168 INFO:teuthology.orchestra.run.smithi134.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-08-18T15:16:33.168 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-08-18T15:16:33.169 INFO:teuthology.orchestra.run.smithi134.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-08-18T15:16:33.169 INFO:teuthology.orchestra.run.smithi134.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-08-18T15:16:33.169 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:16:33.169 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-08-18T15:16:33.169 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-08-18T15:16:33.169 INFO:teuthology.orchestra.run.smithi134.stdout:Install 6 Packages 2024-08-18T15:16:33.169 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:16:33.169 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 7.0 M 2024-08-18T15:16:33.170 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 33 M 2024-08-18T15:16:33.170 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-08-18T15:16:33.335 INFO:teuthology.orchestra.run.smithi193.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 139 kB/s | 19 kB 00:00 2024-08-18T15:16:33.360 INFO:teuthology.orchestra.run.smithi193.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 157 kB/s | 26 kB 00:00 2024-08-18T15:16:33.385 INFO:teuthology.orchestra.run.smithi193.stdout:(3/13): ceph-mgr-rook-18.2.4-717.g03f4c3e8.el9. 261 kB/s | 49 kB 00:00 2024-08-18T15:16:33.452 INFO:teuthology.orchestra.run.smithi193.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.9 MB/s | 222 kB 00:00 2024-08-18T15:16:33.485 INFO:teuthology.orchestra.run.smithi193.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.2 MB/s | 159 kB 00:00 2024-08-18T15:16:33.510 INFO:teuthology.orchestra.run.smithi193.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 915 kB/s | 54 kB 00:00 2024-08-18T15:16:33.602 INFO:teuthology.orchestra.run.smithi193.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.3 MB/s | 279 kB 00:00 2024-08-18T15:16:33.628 INFO:teuthology.orchestra.run.smithi193.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 120 kB/s | 14 kB 00:00 2024-08-18T15:16:33.653 INFO:teuthology.orchestra.run.smithi193.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 192 kB/s | 32 kB 00:00 2024-08-18T15:16:33.703 INFO:teuthology.orchestra.run.smithi193.stdout:(10/13): python3-google-auth-2.33.0-1.el9.noarc 2.2 MB/s | 220 kB 00:00 2024-08-18T15:16:33.729 INFO:teuthology.orchestra.run.smithi193.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 1.1 MB/s | 59 kB 00:00 2024-08-18T15:16:33.754 INFO:teuthology.orchestra.run.smithi193.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.7 MB/s | 90 kB 00:00 2024-08-18T15:16:33.770 INFO:teuthology.orchestra.run.smithi134.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.5 MB/s | 248 kB 00:00 2024-08-18T15:16:33.837 INFO:teuthology.orchestra.run.smithi134.stdout:(2/6): ceph-mgr-cephadm-18.2.4-717.g03f4c3e8.el 611 kB/s | 139 kB 00:00 2024-08-18T15:16:33.879 INFO:teuthology.orchestra.run.smithi193.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 4.1 MB/s | 1.0 MB 00:00 2024-08-18T15:16:33.882 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-08-18T15:16:33.882 INFO:teuthology.orchestra.run.smithi193.stdout:Total 2.1 MB/s | 2.2 MB 00:01 2024-08-18T15:16:33.945 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-08-18T15:16:33.960 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-08-18T15:16:33.960 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-08-18T15:16:33.970 INFO:teuthology.orchestra.run.smithi134.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 433 kB/s | 58 kB 00:00 2024-08-18T15:16:34.004 INFO:teuthology.orchestra.run.smithi134.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 2.4 MB/s | 74 kB 00:00 2024-08-18T15:16:34.045 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-08-18T15:16:34.045 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-08-18T15:16:34.054 INFO:teuthology.orchestra.run.smithi134.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.9 MB/s | 548 kB 00:00 2024-08-18T15:16:34.557 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-08-18T15:16:34.726 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-08-18T15:16:34.788 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-08-18T15:16:34.844 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-08-18T15:16:34.881 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-08-18T15:16:34.923 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-08-18T15:16:35.032 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-08-18T15:16:35.121 INFO:teuthology.orchestra.run.smithi134.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 3.9 MB/s | 6.0 MB 00:01 2024-08-18T15:16:35.124 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-08-18T15:16:35.124 INFO:teuthology.orchestra.run.smithi134.stdout:Total 3.6 MB/s | 7.0 MB 00:01 2024-08-18T15:16:35.143 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-google-auth-1:2.33.0-1.el9.noarch 7/13 2024-08-18T15:16:35.191 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-08-18T15:16:35.198 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-08-18T15:16:35.210 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-08-18T15:16:35.210 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-08-18T15:16:35.308 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-08-18T15:16:35.310 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-08-18T15:16:35.610 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-08-18T15:16:35.795 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-08-18T15:16:35.847 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-08-18T15:16:35.906 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-08-18T15:16:35.964 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-08-18T15:16:35.998 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-08-18T15:16:36.046 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-08-18T15:16:36.067 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mgr-rook-2:18.2.4-717.g03f4c3e8.el9.noarch 13/13 2024-08-18T15:16:36.362 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-08-18T15:16:36.449 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-08-18T15:16:36.520 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-08-18T15:16:36.558 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-cephadm-2:18.2.4-717.g03f4c3e8.el9.noarch 6/6 2024-08-18T15:16:36.884 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.4-717.g03f4c3e8.el9.noarch 13/13 2024-08-18T15:16:36.885 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mgr-rook-2:18.2.4-717.g03f4c3e8.el9.noarch 1/13 2024-08-18T15:16:36.885 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-08-18T15:16:36.885 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-08-18T15:16:36.885 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-08-18T15:16:36.885 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-08-18T15:16:36.885 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-08-18T15:16:36.885 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-08-18T15:16:36.885 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-08-18T15:16:36.885 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-08-18T15:16:36.885 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-google-auth-1:2.33.0-1.el9.noarch 10/13 2024-08-18T15:16:36.885 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-08-18T15:16:36.885 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-08-18T15:16:37.234 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-08-18T15:16:37.234 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:16:37.234 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-08-18T15:16:37.234 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-rook-2:18.2.4-717.g03f4c3e8.el9.noarch 2024-08-18T15:16:37.234 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-08-18T15:16:37.234 INFO:teuthology.orchestra.run.smithi193.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-08-18T15:16:37.234 INFO:teuthology.orchestra.run.smithi193.stdout: python3-google-auth-1:2.33.0-1.el9.noarch 2024-08-18T15:16:37.234 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-08-18T15:16:37.234 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-08-18T15:16:37.234 INFO:teuthology.orchestra.run.smithi193.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-08-18T15:16:37.234 INFO:teuthology.orchestra.run.smithi193.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-08-18T15:16:37.235 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-08-18T15:16:37.235 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-08-18T15:16:37.235 INFO:teuthology.orchestra.run.smithi193.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-08-18T15:16:37.235 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rsa-4.9-2.el9.noarch 2024-08-18T15:16:37.235 INFO:teuthology.orchestra.run.smithi193.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-08-18T15:16:37.235 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:16:37.235 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-08-18T15:16:37.371 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.4-717.g03f4c3e8.el9.noarch 6/6 2024-08-18T15:16:37.372 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-cephadm-2:18.2.4-717.g03f4c3e8.el9.noarch 1/6 2024-08-18T15:16:37.372 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-08-18T15:16:37.372 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-08-18T15:16:37.372 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-08-18T15:16:37.372 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-08-18T15:16:37.505 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph-mgr-cephadm 2024-08-18T15:16:37.646 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-08-18T15:16:37.647 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:16:37.647 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-08-18T15:16:37.647 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-cephadm-2:18.2.4-717.g03f4c3e8.el9.noarch 2024-08-18T15:16:37.647 INFO:teuthology.orchestra.run.smithi134.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-08-18T15:16:37.647 INFO:teuthology.orchestra.run.smithi134.stdout: python3-babel-2.9.1-2.el9.noarch 2024-08-18T15:16:37.647 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-08-18T15:16:37.647 INFO:teuthology.orchestra.run.smithi134.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-08-18T15:16:37.647 INFO:teuthology.orchestra.run.smithi134.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-08-18T15:16:37.647 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:16:37.647 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-08-18T15:16:37.911 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-fuse 2024-08-18T15:16:38.055 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:01:18 ago on Sun 18 Aug 2024 03:15:20 PM UTC. 2024-08-18T15:16:38.182 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-08-18T15:16:38.182 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-08-18T15:16:38.182 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repository Size 2024-08-18T15:16:38.182 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-08-18T15:16:38.183 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-08-18T15:16:38.183 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-cephadm noarch 2:18.2.4-717.g03f4c3e8.el9 ceph-noarch 139 k 2024-08-18T15:16:38.183 INFO:teuthology.orchestra.run.smithi193.stdout:Installing dependencies: 2024-08-18T15:16:38.183 INFO:teuthology.orchestra.run.smithi193.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-08-18T15:16:38.183 INFO:teuthology.orchestra.run.smithi193.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-08-18T15:16:38.183 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-08-18T15:16:38.183 INFO:teuthology.orchestra.run.smithi193.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-08-18T15:16:38.183 INFO:teuthology.orchestra.run.smithi193.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-08-18T15:16:38.183 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:16:38.183 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-08-18T15:16:38.183 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-08-18T15:16:38.183 INFO:teuthology.orchestra.run.smithi193.stdout:Install 6 Packages 2024-08-18T15:16:38.183 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:16:38.184 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 7.0 M 2024-08-18T15:16:38.184 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 33 M 2024-08-18T15:16:38.184 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-08-18T15:16:38.461 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:19 ago on Sun 18 Aug 2024 03:15:19 PM UTC. 2024-08-18T15:16:38.565 INFO:teuthology.orchestra.run.smithi193.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.6 MB/s | 248 kB 00:00 2024-08-18T15:16:38.589 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-08-18T15:16:38.589 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-08-18T15:16:38.589 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-08-18T15:16:38.590 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-08-18T15:16:38.590 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-08-18T15:16:38.590 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-fuse x86_64 2:18.2.4-717.g03f4c3e8.el9 ceph 853 k 2024-08-18T15:16:38.590 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-08-18T15:16:38.590 INFO:teuthology.orchestra.run.smithi134.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-08-18T15:16:38.590 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:16:38.590 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-08-18T15:16:38.590 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-08-18T15:16:38.590 INFO:teuthology.orchestra.run.smithi134.stdout:Install 2 Packages 2024-08-18T15:16:38.590 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:16:38.591 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 933 k 2024-08-18T15:16:38.591 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 2.7 M 2024-08-18T15:16:38.591 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-08-18T15:16:38.641 INFO:teuthology.orchestra.run.smithi193.stdout:(2/6): ceph-mgr-cephadm-18.2.4-717.g03f4c3e8.el 604 kB/s | 139 kB 00:00 2024-08-18T15:16:38.708 INFO:teuthology.orchestra.run.smithi193.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 865 kB/s | 58 kB 00:00 2024-08-18T15:16:38.800 INFO:teuthology.orchestra.run.smithi193.stdout:(4/6): python3-babel-2.9.1-2.el9.noarch.rpm 15 MB/s | 6.0 MB 00:00 2024-08-18T15:16:38.825 INFO:teuthology.orchestra.run.smithi193.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 637 kB/s | 74 kB 00:00 2024-08-18T15:16:38.850 INFO:teuthology.orchestra.run.smithi193.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.9 MB/s | 548 kB 00:00 2024-08-18T15:16:38.852 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-08-18T15:16:38.852 INFO:teuthology.orchestra.run.smithi193.stdout:Total 10 MB/s | 7.0 MB 00:00 2024-08-18T15:16:38.867 INFO:teuthology.orchestra.run.smithi134.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 497 kB/s | 80 kB 00:00 2024-08-18T15:16:38.922 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-08-18T15:16:38.933 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-08-18T15:16:38.933 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-08-18T15:16:39.025 INFO:teuthology.orchestra.run.smithi134.stdout:(2/2): ceph-fuse-18.2.4-717.g03f4c3e8.el9.x86_6 2.6 MB/s | 853 kB 00:00 2024-08-18T15:16:39.026 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-08-18T15:16:39.026 INFO:teuthology.orchestra.run.smithi134.stdout:Total 2.1 MB/s | 933 kB 00:00 2024-08-18T15:16:39.032 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-08-18T15:16:39.033 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-08-18T15:16:39.038 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-08-18T15:16:39.059 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-08-18T15:16:39.059 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-08-18T15:16:39.137 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-08-18T15:16:39.137 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-08-18T15:16:39.329 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-08-18T15:16:39.337 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-08-18T15:16:39.448 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-08-18T15:16:39.515 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-fuse-2:18.2.4-717.g03f4c3e8.el9.x86_64 2/2 2024-08-18T15:16:39.516 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-08-18T15:16:39.568 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-08-18T15:16:40.085 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-08-18T15:16:40.171 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-08-18T15:16:40.208 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-fuse-2:18.2.4-717.g03f4c3e8.el9.x86_64 2/2 2024-08-18T15:16:40.208 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-fuse-2:18.2.4-717.g03f4c3e8.el9.x86_64 1/2 2024-08-18T15:16:40.233 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-08-18T15:16:40.256 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mgr-cephadm-2:18.2.4-717.g03f4c3e8.el9.noarch 6/6 2024-08-18T15:16:40.497 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-08-18T15:16:40.497 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:16:40.498 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-08-18T15:16:40.498 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-fuse-2:18.2.4-717.g03f4c3e8.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-08-18T15:16:40.498 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:16:40.498 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-08-18T15:16:40.791 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-volume 2024-08-18T15:16:40.973 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.4-717.g03f4c3e8.el9.noarch 6/6 2024-08-18T15:16:40.973 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mgr-cephadm-2:18.2.4-717.g03f4c3e8.el9.noarch 1/6 2024-08-18T15:16:40.973 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-08-18T15:16:40.973 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-08-18T15:16:40.973 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-08-18T15:16:40.974 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-08-18T15:16:41.249 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-08-18T15:16:41.249 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:16:41.249 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-08-18T15:16:41.249 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-cephadm-2:18.2.4-717.g03f4c3e8.el9.noarch 2024-08-18T15:16:41.249 INFO:teuthology.orchestra.run.smithi193.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-08-18T15:16:41.250 INFO:teuthology.orchestra.run.smithi193.stdout: python3-babel-2.9.1-2.el9.noarch 2024-08-18T15:16:41.250 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-08-18T15:16:41.250 INFO:teuthology.orchestra.run.smithi193.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-08-18T15:16:41.250 INFO:teuthology.orchestra.run.smithi193.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-08-18T15:16:41.250 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:16:41.250 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-08-18T15:16:41.353 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:22 ago on Sun 18 Aug 2024 03:15:19 PM UTC. 2024-08-18T15:16:41.482 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph-fuse 2024-08-18T15:16:41.485 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-08-18T15:16:41.485 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-08-18T15:16:41.486 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-08-18T15:16:41.486 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-08-18T15:16:41.486 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-08-18T15:16:41.486 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-volume noarch 2:18.2.4-717.g03f4c3e8.el9 ceph-noarch 264 k 2024-08-18T15:16:41.486 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-08-18T15:16:41.486 INFO:teuthology.orchestra.run.smithi134.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-08-18T15:16:41.486 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:16:41.486 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-08-18T15:16:41.486 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-08-18T15:16:41.486 INFO:teuthology.orchestra.run.smithi134.stdout:Install 2 Packages 2024-08-18T15:16:41.486 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:16:41.487 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 342 k 2024-08-18T15:16:41.487 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 1.5 M 2024-08-18T15:16:41.487 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-08-18T15:16:42.023 INFO:teuthology.orchestra.run.smithi134.stdout:(1/2): ceph-volume-18.2.4-717.g03f4c3e8.el9.noa 1.0 MB/s | 264 kB 00:00 2024-08-18T15:16:42.030 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:01:22 ago on Sun 18 Aug 2024 03:15:20 PM UTC. 2024-08-18T15:16:42.159 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-08-18T15:16:42.159 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-08-18T15:16:42.160 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repository Size 2024-08-18T15:16:42.160 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-08-18T15:16:42.160 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-08-18T15:16:42.160 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-fuse x86_64 2:18.2.4-717.g03f4c3e8.el9 ceph 853 k 2024-08-18T15:16:42.160 INFO:teuthology.orchestra.run.smithi193.stdout:Installing dependencies: 2024-08-18T15:16:42.160 INFO:teuthology.orchestra.run.smithi193.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-08-18T15:16:42.160 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:16:42.160 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-08-18T15:16:42.160 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-08-18T15:16:42.160 INFO:teuthology.orchestra.run.smithi193.stdout:Install 2 Packages 2024-08-18T15:16:42.160 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:16:42.161 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 933 k 2024-08-18T15:16:42.161 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 2.7 M 2024-08-18T15:16:42.161 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-08-18T15:16:42.181 INFO:teuthology.orchestra.run.smithi134.stdout:(2/2): python3-packaging-20.9-5.el9.noarch.rpm 184 kB/s | 77 kB 00:00 2024-08-18T15:16:42.182 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-08-18T15:16:42.182 INFO:teuthology.orchestra.run.smithi134.stdout:Total 491 kB/s | 342 kB 00:00 2024-08-18T15:16:42.194 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-08-18T15:16:42.204 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-08-18T15:16:42.204 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-08-18T15:16:42.234 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-08-18T15:16:42.235 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-08-18T15:16:42.369 INFO:teuthology.orchestra.run.smithi193.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 469 kB/s | 80 kB 00:00 2024-08-18T15:16:42.381 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-08-18T15:16:42.503 INFO:teuthology.orchestra.run.smithi193.stdout:(2/2): ceph-fuse-18.2.4-717.g03f4c3e8.el9.x86_6 2.7 MB/s | 853 kB 00:00 2024-08-18T15:16:42.503 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-08-18T15:16:42.504 INFO:teuthology.orchestra.run.smithi193.stdout:Total 2.7 MB/s | 933 kB 00:00 2024-08-18T15:16:42.515 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-08-18T15:16:42.538 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-08-18T15:16:42.538 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-08-18T15:16:42.555 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-08-18T15:16:42.592 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-volume-2:18.2.4-717.g03f4c3e8.el9.noarch 2/2 2024-08-18T15:16:42.607 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-volume-2:18.2.4-717.g03f4c3e8.el9.noarch 2/2 2024-08-18T15:16:42.608 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-18T15:16:42.608 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-08-18T15:16:42.608 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:16:42.616 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-08-18T15:16:42.617 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-08-18T15:16:42.798 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-08-18T15:16:42.934 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-08-18T15:16:43.011 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-fuse-2:18.2.4-717.g03f4c3e8.el9.x86_64 2/2 2024-08-18T15:16:43.303 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-volume-2:18.2.4-717.g03f4c3e8.el9.noarch 1/2 2024-08-18T15:16:43.588 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-08-18T15:16:43.589 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:16:43.589 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-08-18T15:16:43.589 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-volume-2:18.2.4-717.g03f4c3e8.el9.noarch 2024-08-18T15:16:43.589 INFO:teuthology.orchestra.run.smithi134.stdout: python3-packaging-20.9-5.el9.noarch 2024-08-18T15:16:43.589 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:16:43.589 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-08-18T15:16:43.624 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-fuse-2:18.2.4-717.g03f4c3e8.el9.x86_64 2/2 2024-08-18T15:16:43.624 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-fuse-2:18.2.4-717.g03f4c3e8.el9.x86_64 1/2 2024-08-18T15:16:43.867 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-08-18T15:16:43.868 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:16:43.868 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-08-18T15:16:43.868 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-fuse-2:18.2.4-717.g03f4c3e8.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-08-18T15:16:43.868 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:16:43.868 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-08-18T15:16:43.957 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install librados-devel 2024-08-18T15:16:44.139 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph-volume 2024-08-18T15:16:44.516 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:25 ago on Sun 18 Aug 2024 03:15:19 PM UTC. 2024-08-18T15:16:44.648 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-08-18T15:16:44.648 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-08-18T15:16:44.649 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-08-18T15:16:44.649 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-08-18T15:16:44.649 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-08-18T15:16:44.649 INFO:teuthology.orchestra.run.smithi134.stdout: librados-devel x86_64 2:18.2.4-717.g03f4c3e8.el9 ceph 127 k 2024-08-18T15:16:44.649 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:16:44.649 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-08-18T15:16:44.649 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-08-18T15:16:44.649 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-08-18T15:16:44.649 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:16:44.649 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 127 k 2024-08-18T15:16:44.649 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 456 k 2024-08-18T15:16:44.649 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-08-18T15:16:44.704 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:01:24 ago on Sun 18 Aug 2024 03:15:20 PM UTC. 2024-08-18T15:16:44.836 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-08-18T15:16:44.836 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-08-18T15:16:44.837 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repository Size 2024-08-18T15:16:44.837 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-08-18T15:16:44.837 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-08-18T15:16:44.837 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-volume noarch 2:18.2.4-717.g03f4c3e8.el9 ceph-noarch 264 k 2024-08-18T15:16:44.837 INFO:teuthology.orchestra.run.smithi193.stdout:Installing dependencies: 2024-08-18T15:16:44.837 INFO:teuthology.orchestra.run.smithi193.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-08-18T15:16:44.837 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:16:44.837 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-08-18T15:16:44.837 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-08-18T15:16:44.837 INFO:teuthology.orchestra.run.smithi193.stdout:Install 2 Packages 2024-08-18T15:16:44.837 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:16:44.838 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 342 k 2024-08-18T15:16:44.838 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 1.5 M 2024-08-18T15:16:44.838 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-08-18T15:16:44.873 INFO:teuthology.orchestra.run.smithi134.stdout:librados-devel-18.2.4-717.g03f4c3e8.el9.x86_64. 568 kB/s | 127 kB 00:00 2024-08-18T15:16:44.873 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-08-18T15:16:44.873 INFO:teuthology.orchestra.run.smithi134.stdout:Total 564 kB/s | 127 kB 00:00 2024-08-18T15:16:44.874 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-08-18T15:16:44.879 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-08-18T15:16:44.879 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-08-18T15:16:44.941 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-08-18T15:16:44.941 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-08-18T15:16:45.086 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-08-18T15:16:45.195 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : librados-devel-2:18.2.4-717.g03f4c3e8.el9.x86_64 1/1 2024-08-18T15:16:45.199 INFO:teuthology.orchestra.run.smithi193.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 858 kB/s | 77 kB 00:00 2024-08-18T15:16:45.366 INFO:teuthology.orchestra.run.smithi193.stdout:(2/2): ceph-volume-18.2.4-717.g03f4c3e8.el9.noa 1.0 MB/s | 264 kB 00:00 2024-08-18T15:16:45.367 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-08-18T15:16:45.367 INFO:teuthology.orchestra.run.smithi193.stdout:Total 644 kB/s | 342 kB 00:00 2024-08-18T15:16:45.379 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-08-18T15:16:45.389 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-08-18T15:16:45.389 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-08-18T15:16:45.422 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-08-18T15:16:45.422 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-08-18T15:16:45.574 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-08-18T15:16:45.613 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librados-devel-2:18.2.4-717.g03f4c3e8.el9.x86_64 1/1 2024-08-18T15:16:45.757 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-08-18T15:16:45.794 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-volume-2:18.2.4-717.g03f4c3e8.el9.noarch 2/2 2024-08-18T15:16:45.810 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-volume-2:18.2.4-717.g03f4c3e8.el9.noarch 2/2 2024-08-18T15:16:45.810 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-18T15:16:45.810 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-08-18T15:16:45.810 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:16:45.881 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librados-devel-2:18.2.4-717.g03f4c3e8.el9.x86_64 1/1 2024-08-18T15:16:45.881 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:16:45.881 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-08-18T15:16:45.881 INFO:teuthology.orchestra.run.smithi134.stdout: librados-devel-2:18.2.4-717.g03f4c3e8.el9.x86_64 2024-08-18T15:16:45.881 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:16:45.881 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-08-18T15:16:46.152 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install libcephfs2 2024-08-18T15:16:46.554 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-volume-2:18.2.4-717.g03f4c3e8.el9.noarch 1/2 2024-08-18T15:16:46.711 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:27 ago on Sun 18 Aug 2024 03:15:19 PM UTC. 2024-08-18T15:16:46.799 INFO:teuthology.orchestra.run.smithi134.stdout:Package libcephfs2-2:18.2.4-717.g03f4c3e8.el9.x86_64 is already installed. 2024-08-18T15:16:46.841 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-08-18T15:16:46.842 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-08-18T15:16:46.842 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-08-18T15:16:46.874 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-08-18T15:16:46.874 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:16:46.874 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-08-18T15:16:46.874 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-volume-2:18.2.4-717.g03f4c3e8.el9.noarch 2024-08-18T15:16:46.874 INFO:teuthology.orchestra.run.smithi193.stdout: python3-packaging-20.9-5.el9.noarch 2024-08-18T15:16:46.874 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:16:46.874 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-08-18T15:16:46.920 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install libcephfs-devel 2024-08-18T15:16:47.170 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install librados-devel 2024-08-18T15:16:47.474 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:28 ago on Sun 18 Aug 2024 03:15:19 PM UTC. 2024-08-18T15:16:47.603 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-08-18T15:16:47.603 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-08-18T15:16:47.603 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-08-18T15:16:47.603 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-08-18T15:16:47.604 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-08-18T15:16:47.604 INFO:teuthology.orchestra.run.smithi134.stdout: libcephfs-devel x86_64 2:18.2.4-717.g03f4c3e8.el9 ceph 31 k 2024-08-18T15:16:47.604 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:16:47.604 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-08-18T15:16:47.604 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-08-18T15:16:47.604 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-08-18T15:16:47.604 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:16:47.604 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 31 k 2024-08-18T15:16:47.604 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 139 k 2024-08-18T15:16:47.604 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-08-18T15:16:47.730 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:01:27 ago on Sun 18 Aug 2024 03:15:20 PM UTC. 2024-08-18T15:16:47.772 INFO:teuthology.orchestra.run.smithi134.stdout:libcephfs-devel-18.2.4-717.g03f4c3e8.el9.x86_64 185 kB/s | 31 kB 00:00 2024-08-18T15:16:47.772 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-08-18T15:16:47.772 INFO:teuthology.orchestra.run.smithi134.stdout:Total 184 kB/s | 31 kB 00:00 2024-08-18T15:16:47.772 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-08-18T15:16:47.776 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-08-18T15:16:47.776 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-08-18T15:16:47.794 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-08-18T15:16:47.794 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-08-18T15:16:47.862 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-08-18T15:16:47.863 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-08-18T15:16:47.863 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repo Size 2024-08-18T15:16:47.863 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-08-18T15:16:47.863 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-08-18T15:16:47.863 INFO:teuthology.orchestra.run.smithi193.stdout: librados-devel x86_64 2:18.2.4-717.g03f4c3e8.el9 ceph 127 k 2024-08-18T15:16:47.864 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:16:47.864 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-08-18T15:16:47.864 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-08-18T15:16:47.864 INFO:teuthology.orchestra.run.smithi193.stdout:Install 1 Package 2024-08-18T15:16:47.864 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:16:47.864 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 127 k 2024-08-18T15:16:47.864 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 456 k 2024-08-18T15:16:47.864 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-08-18T15:16:47.876 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-08-18T15:16:47.985 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libcephfs-devel-2:18.2.4-717.g03f4c3e8.el9.x86_64 1/1 2024-08-18T15:16:48.073 INFO:teuthology.orchestra.run.smithi193.stdout:librados-devel-18.2.4-717.g03f4c3e8.el9.x86_64. 607 kB/s | 127 kB 00:00 2024-08-18T15:16:48.073 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-08-18T15:16:48.074 INFO:teuthology.orchestra.run.smithi193.stdout:Total 603 kB/s | 127 kB 00:00 2024-08-18T15:16:48.074 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-08-18T15:16:48.079 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-08-18T15:16:48.079 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-08-18T15:16:48.141 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-08-18T15:16:48.142 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-08-18T15:16:48.287 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-08-18T15:16:48.337 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: libcephfs-devel-2:18.2.4-717.g03f4c3e8.el9.x86_64 1/1 2024-08-18T15:16:48.387 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : librados-devel-2:18.2.4-717.g03f4c3e8.el9.x86_64 1/1 2024-08-18T15:16:48.553 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libcephfs-devel-2:18.2.4-717.g03f4c3e8.el9.x86_64 1/1 2024-08-18T15:16:48.553 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:16:48.553 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-08-18T15:16:48.553 INFO:teuthology.orchestra.run.smithi134.stdout: libcephfs-devel-2:18.2.4-717.g03f4c3e8.el9.x86_64 2024-08-18T15:16:48.553 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:16:48.553 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-08-18T15:16:48.756 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install librados2 2024-08-18T15:16:48.805 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: librados-devel-2:18.2.4-717.g03f4c3e8.el9.x86_64 1/1 2024-08-18T15:16:49.057 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librados-devel-2:18.2.4-717.g03f4c3e8.el9.x86_64 1/1 2024-08-18T15:16:49.057 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:16:49.057 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-08-18T15:16:49.057 INFO:teuthology.orchestra.run.smithi193.stdout: librados-devel-2:18.2.4-717.g03f4c3e8.el9.x86_64 2024-08-18T15:16:49.057 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:16:49.057 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-08-18T15:16:49.283 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install libcephfs2 2024-08-18T15:16:49.303 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:30 ago on Sun 18 Aug 2024 03:15:19 PM UTC. 2024-08-18T15:16:49.389 INFO:teuthology.orchestra.run.smithi134.stdout:Package librados2-2:18.2.4-717.g03f4c3e8.el9.x86_64 is already installed. 2024-08-18T15:16:49.432 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-08-18T15:16:49.432 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-08-18T15:16:49.433 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-08-18T15:16:49.507 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install librbd1 2024-08-18T15:16:49.838 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:01:29 ago on Sun 18 Aug 2024 03:15:20 PM UTC. 2024-08-18T15:16:49.924 INFO:teuthology.orchestra.run.smithi193.stdout:Package libcephfs2-2:18.2.4-717.g03f4c3e8.el9.x86_64 is already installed. 2024-08-18T15:16:49.966 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-08-18T15:16:49.968 INFO:teuthology.orchestra.run.smithi193.stdout:Nothing to do. 2024-08-18T15:16:49.968 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-08-18T15:16:50.037 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install libcephfs-devel 2024-08-18T15:16:50.052 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:31 ago on Sun 18 Aug 2024 03:15:19 PM UTC. 2024-08-18T15:16:50.140 INFO:teuthology.orchestra.run.smithi134.stdout:Package librbd1-2:18.2.4-717.g03f4c3e8.el9.x86_64 is already installed. 2024-08-18T15:16:50.182 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-08-18T15:16:50.183 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-08-18T15:16:50.183 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-08-18T15:16:50.261 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install python3-rados 2024-08-18T15:16:50.585 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:01:30 ago on Sun 18 Aug 2024 03:15:20 PM UTC. 2024-08-18T15:16:50.712 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-08-18T15:16:50.713 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-08-18T15:16:50.713 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repo Size 2024-08-18T15:16:50.713 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-08-18T15:16:50.713 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-08-18T15:16:50.714 INFO:teuthology.orchestra.run.smithi193.stdout: libcephfs-devel x86_64 2:18.2.4-717.g03f4c3e8.el9 ceph 31 k 2024-08-18T15:16:50.714 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:16:50.714 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-08-18T15:16:50.714 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-08-18T15:16:50.714 INFO:teuthology.orchestra.run.smithi193.stdout:Install 1 Package 2024-08-18T15:16:50.714 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:16:50.714 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 31 k 2024-08-18T15:16:50.714 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 139 k 2024-08-18T15:16:50.714 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-08-18T15:16:50.806 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:31 ago on Sun 18 Aug 2024 03:15:19 PM UTC. 2024-08-18T15:16:50.890 INFO:teuthology.orchestra.run.smithi193.stdout:libcephfs-devel-18.2.4-717.g03f4c3e8.el9.x86_64 177 kB/s | 31 kB 00:00 2024-08-18T15:16:50.891 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-08-18T15:16:50.891 INFO:teuthology.orchestra.run.smithi193.stdout:Total 175 kB/s | 31 kB 00:00 2024-08-18T15:16:50.891 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-08-18T15:16:50.891 INFO:teuthology.orchestra.run.smithi134.stdout:Package python3-rados-2:18.2.4-717.g03f4c3e8.el9.x86_64 is already installed. 2024-08-18T15:16:50.895 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-08-18T15:16:50.895 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-08-18T15:16:50.920 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-08-18T15:16:50.920 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-08-18T15:16:50.933 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-08-18T15:16:50.934 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-08-18T15:16:50.935 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-08-18T15:16:51.003 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-08-18T15:16:51.010 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install python3-rgw 2024-08-18T15:16:51.128 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libcephfs-devel-2:18.2.4-717.g03f4c3e8.el9.x86_64 1/1 2024-08-18T15:16:51.507 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: libcephfs-devel-2:18.2.4-717.g03f4c3e8.el9.x86_64 1/1 2024-08-18T15:16:51.558 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:32 ago on Sun 18 Aug 2024 03:15:19 PM UTC. 2024-08-18T15:16:51.644 INFO:teuthology.orchestra.run.smithi134.stdout:Package python3-rgw-2:18.2.4-717.g03f4c3e8.el9.x86_64 is already installed. 2024-08-18T15:16:51.686 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-08-18T15:16:51.686 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-08-18T15:16:51.687 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-08-18T15:16:51.731 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libcephfs-devel-2:18.2.4-717.g03f4c3e8.el9.x86_64 1/1 2024-08-18T15:16:51.731 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:16:51.732 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-08-18T15:16:51.732 INFO:teuthology.orchestra.run.smithi193.stdout: libcephfs-devel-2:18.2.4-717.g03f4c3e8.el9.x86_64 2024-08-18T15:16:51.732 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:16:51.732 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-08-18T15:16:51.766 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install python3-cephfs 2024-08-18T15:16:51.933 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install librados2 2024-08-18T15:16:52.315 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:33 ago on Sun 18 Aug 2024 03:15:19 PM UTC. 2024-08-18T15:16:52.401 INFO:teuthology.orchestra.run.smithi134.stdout:Package python3-cephfs-2:18.2.4-717.g03f4c3e8.el9.x86_64 is already installed. 2024-08-18T15:16:52.443 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-08-18T15:16:52.444 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-08-18T15:16:52.444 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-08-18T15:16:52.483 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:01:32 ago on Sun 18 Aug 2024 03:15:20 PM UTC. 2024-08-18T15:16:52.522 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install python3-rbd 2024-08-18T15:16:52.570 INFO:teuthology.orchestra.run.smithi193.stdout:Package librados2-2:18.2.4-717.g03f4c3e8.el9.x86_64 is already installed. 2024-08-18T15:16:52.613 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-08-18T15:16:52.614 INFO:teuthology.orchestra.run.smithi193.stdout:Nothing to do. 2024-08-18T15:16:52.614 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-08-18T15:16:52.688 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install librbd1 2024-08-18T15:16:53.071 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:34 ago on Sun 18 Aug 2024 03:15:19 PM UTC. 2024-08-18T15:16:53.158 INFO:teuthology.orchestra.run.smithi134.stdout:Package python3-rbd-2:18.2.4-717.g03f4c3e8.el9.x86_64 is already installed. 2024-08-18T15:16:53.199 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-08-18T15:16:53.200 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-08-18T15:16:53.200 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-08-18T15:16:53.238 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:01:33 ago on Sun 18 Aug 2024 03:15:20 PM UTC. 2024-08-18T15:16:53.286 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install rbd-fuse 2024-08-18T15:16:53.324 INFO:teuthology.orchestra.run.smithi193.stdout:Package librbd1-2:18.2.4-717.g03f4c3e8.el9.x86_64 is already installed. 2024-08-18T15:16:53.367 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-08-18T15:16:53.368 INFO:teuthology.orchestra.run.smithi193.stdout:Nothing to do. 2024-08-18T15:16:53.368 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-08-18T15:16:53.444 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install python3-rados 2024-08-18T15:16:53.835 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:34 ago on Sun 18 Aug 2024 03:15:19 PM UTC. 2024-08-18T15:16:53.964 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-08-18T15:16:53.965 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-08-18T15:16:53.965 INFO:teuthology.orchestra.run.smithi134.stdout: Package Architecture Version Repository Size 2024-08-18T15:16:53.965 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-08-18T15:16:53.965 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-08-18T15:16:53.965 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-fuse x86_64 2:18.2.4-717.g03f4c3e8.el9 ceph 87 k 2024-08-18T15:16:53.965 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:16:53.965 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-08-18T15:16:53.965 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-08-18T15:16:53.965 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-08-18T15:16:53.965 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:16:53.965 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 87 k 2024-08-18T15:16:53.966 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 227 k 2024-08-18T15:16:53.966 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-08-18T15:16:53.993 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:01:33 ago on Sun 18 Aug 2024 03:15:20 PM UTC. 2024-08-18T15:16:54.078 INFO:teuthology.orchestra.run.smithi193.stdout:Package python3-rados-2:18.2.4-717.g03f4c3e8.el9.x86_64 is already installed. 2024-08-18T15:16:54.121 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-08-18T15:16:54.122 INFO:teuthology.orchestra.run.smithi193.stdout:Nothing to do. 2024-08-18T15:16:54.122 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-08-18T15:16:54.177 INFO:teuthology.orchestra.run.smithi134.stdout:rbd-fuse-18.2.4-717.g03f4c3e8.el9.x86_64.rpm 410 kB/s | 87 kB 00:00 2024-08-18T15:16:54.177 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-08-18T15:16:54.178 INFO:teuthology.orchestra.run.smithi134.stdout:Total 407 kB/s | 87 kB 00:00 2024-08-18T15:16:54.178 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-08-18T15:16:54.185 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-08-18T15:16:54.185 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-08-18T15:16:54.197 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install python3-rgw 2024-08-18T15:16:54.247 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-08-18T15:16:54.248 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-08-18T15:16:54.405 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-08-18T15:16:54.508 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : rbd-fuse-2:18.2.4-717.g03f4c3e8.el9.x86_64 1/1 2024-08-18T15:16:54.744 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:01:34 ago on Sun 18 Aug 2024 03:15:20 PM UTC. 2024-08-18T15:16:54.829 INFO:teuthology.orchestra.run.smithi193.stdout:Package python3-rgw-2:18.2.4-717.g03f4c3e8.el9.x86_64 is already installed. 2024-08-18T15:16:54.871 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-08-18T15:16:54.871 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: rbd-fuse-2:18.2.4-717.g03f4c3e8.el9.x86_64 1/1 2024-08-18T15:16:54.872 INFO:teuthology.orchestra.run.smithi193.stdout:Nothing to do. 2024-08-18T15:16:54.872 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-08-18T15:16:54.934 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install python3-cephfs 2024-08-18T15:16:55.121 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : rbd-fuse-2:18.2.4-717.g03f4c3e8.el9.x86_64 1/1 2024-08-18T15:16:55.121 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:16:55.122 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-08-18T15:16:55.122 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-fuse-2:18.2.4-717.g03f4c3e8.el9.x86_64 2024-08-18T15:16:55.122 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:16:55.122 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-08-18T15:16:55.369 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install rbd-mirror 2024-08-18T15:16:55.488 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:01:35 ago on Sun 18 Aug 2024 03:15:20 PM UTC. 2024-08-18T15:16:55.574 INFO:teuthology.orchestra.run.smithi193.stdout:Package python3-cephfs-2:18.2.4-717.g03f4c3e8.el9.x86_64 is already installed. 2024-08-18T15:16:55.616 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-08-18T15:16:55.617 INFO:teuthology.orchestra.run.smithi193.stdout:Nothing to do. 2024-08-18T15:16:55.617 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-08-18T15:16:55.687 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install python3-rbd 2024-08-18T15:16:55.926 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:36 ago on Sun 18 Aug 2024 03:15:19 PM UTC. 2024-08-18T15:16:56.053 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-08-18T15:16:56.053 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-08-18T15:16:56.054 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-08-18T15:16:56.054 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-08-18T15:16:56.054 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-08-18T15:16:56.054 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-mirror x86_64 2:18.2.4-717.g03f4c3e8.el9 ceph 3.0 M 2024-08-18T15:16:56.054 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:16:56.054 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-08-18T15:16:56.054 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-08-18T15:16:56.054 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-08-18T15:16:56.054 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:16:56.055 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 3.0 M 2024-08-18T15:16:56.055 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 12 M 2024-08-18T15:16:56.055 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-08-18T15:16:56.233 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:01:36 ago on Sun 18 Aug 2024 03:15:20 PM UTC. 2024-08-18T15:16:56.318 INFO:teuthology.orchestra.run.smithi193.stdout:Package python3-rbd-2:18.2.4-717.g03f4c3e8.el9.x86_64 is already installed. 2024-08-18T15:16:56.360 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-08-18T15:16:56.361 INFO:teuthology.orchestra.run.smithi193.stdout:Nothing to do. 2024-08-18T15:16:56.361 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-08-18T15:16:56.405 INFO:teuthology.orchestra.run.smithi134.stdout:rbd-mirror-18.2.4-717.g03f4c3e8.el9.x86_64.rpm 8.5 MB/s | 3.0 MB 00:00 2024-08-18T15:16:56.405 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-08-18T15:16:56.406 INFO:teuthology.orchestra.run.smithi134.stdout:Total 8.5 MB/s | 3.0 MB 00:00 2024-08-18T15:16:56.406 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-08-18T15:16:56.415 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-08-18T15:16:56.415 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-08-18T15:16:56.429 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install rbd-fuse 2024-08-18T15:16:56.494 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-08-18T15:16:56.495 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-08-18T15:16:56.912 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-08-18T15:16:56.983 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:01:36 ago on Sun 18 Aug 2024 03:15:20 PM UTC. 2024-08-18T15:16:56.989 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : rbd-mirror-2:18.2.4-717.g03f4c3e8.el9.x86_64 1/1 2024-08-18T15:16:57.013 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: rbd-mirror-2:18.2.4-717.g03f4c3e8.el9.x86_64 1/1 2024-08-18T15:16:57.013 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-18T15:16:57.014 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-08-18T15:16:57.014 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-08-18T15:16:57.014 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-08-18T15:16:57.014 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:16:57.112 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-08-18T15:16:57.113 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-08-18T15:16:57.113 INFO:teuthology.orchestra.run.smithi193.stdout: Package Architecture Version Repository Size 2024-08-18T15:16:57.113 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-08-18T15:16:57.113 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-08-18T15:16:57.113 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-fuse x86_64 2:18.2.4-717.g03f4c3e8.el9 ceph 87 k 2024-08-18T15:16:57.113 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:16:57.113 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-08-18T15:16:57.113 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-08-18T15:16:57.113 INFO:teuthology.orchestra.run.smithi193.stdout:Install 1 Package 2024-08-18T15:16:57.113 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:16:57.113 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 87 k 2024-08-18T15:16:57.114 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 227 k 2024-08-18T15:16:57.114 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-08-18T15:16:57.329 INFO:teuthology.orchestra.run.smithi193.stdout:rbd-fuse-18.2.4-717.g03f4c3e8.el9.x86_64.rpm 402 kB/s | 87 kB 00:00 2024-08-18T15:16:57.329 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-08-18T15:16:57.330 INFO:teuthology.orchestra.run.smithi193.stdout:Total 399 kB/s | 87 kB 00:00 2024-08-18T15:16:57.330 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-08-18T15:16:57.337 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-08-18T15:16:57.337 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-08-18T15:16:57.398 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-08-18T15:16:57.399 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-08-18T15:16:57.540 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-08-18T15:16:57.644 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : rbd-fuse-2:18.2.4-717.g03f4c3e8.el9.x86_64 1/1 2024-08-18T15:16:58.015 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: rbd-fuse-2:18.2.4-717.g03f4c3e8.el9.x86_64 1/1 2024-08-18T15:16:58.081 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : rbd-mirror-2:18.2.4-717.g03f4c3e8.el9.x86_64 1/1 2024-08-18T15:16:58.081 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:16:58.081 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-08-18T15:16:58.081 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-mirror-2:18.2.4-717.g03f4c3e8.el9.x86_64 2024-08-18T15:16:58.081 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:16:58.081 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-08-18T15:16:58.289 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : rbd-fuse-2:18.2.4-717.g03f4c3e8.el9.x86_64 1/1 2024-08-18T15:16:58.290 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:16:58.290 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-08-18T15:16:58.290 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-fuse-2:18.2.4-717.g03f4c3e8.el9.x86_64 2024-08-18T15:16:58.290 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:16:58.290 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-08-18T15:16:58.342 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install rbd-nbd 2024-08-18T15:16:58.524 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install rbd-mirror 2024-08-18T15:16:58.898 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:39 ago on Sun 18 Aug 2024 03:15:19 PM UTC. 2024-08-18T15:16:59.030 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-08-18T15:16:59.031 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-08-18T15:16:59.031 INFO:teuthology.orchestra.run.smithi134.stdout: Package Architecture Version Repository Size 2024-08-18T15:16:59.031 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-08-18T15:16:59.031 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-08-18T15:16:59.031 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-nbd x86_64 2:18.2.4-717.g03f4c3e8.el9 ceph 172 k 2024-08-18T15:16:59.031 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:16:59.031 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-08-18T15:16:59.032 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-08-18T15:16:59.032 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-08-18T15:16:59.032 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:16:59.032 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 172 k 2024-08-18T15:16:59.032 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 494 k 2024-08-18T15:16:59.032 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-08-18T15:16:59.085 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:01:39 ago on Sun 18 Aug 2024 03:15:20 PM UTC. 2024-08-18T15:16:59.216 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-08-18T15:16:59.217 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-08-18T15:16:59.217 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repo Size 2024-08-18T15:16:59.217 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-08-18T15:16:59.217 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-08-18T15:16:59.217 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-mirror x86_64 2:18.2.4-717.g03f4c3e8.el9 ceph 3.0 M 2024-08-18T15:16:59.217 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:16:59.217 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-08-18T15:16:59.218 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-08-18T15:16:59.218 INFO:teuthology.orchestra.run.smithi193.stdout:Install 1 Package 2024-08-18T15:16:59.218 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:16:59.218 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 3.0 M 2024-08-18T15:16:59.218 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 12 M 2024-08-18T15:16:59.218 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-08-18T15:16:59.265 INFO:teuthology.orchestra.run.smithi134.stdout:rbd-nbd-18.2.4-717.g03f4c3e8.el9.x86_64.rpm 739 kB/s | 172 kB 00:00 2024-08-18T15:16:59.266 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-08-18T15:16:59.266 INFO:teuthology.orchestra.run.smithi134.stdout:Total 734 kB/s | 172 kB 00:00 2024-08-18T15:16:59.266 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-08-18T15:16:59.274 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-08-18T15:16:59.274 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-08-18T15:16:59.336 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-08-18T15:16:59.337 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-08-18T15:16:59.488 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-08-18T15:16:59.562 INFO:teuthology.orchestra.run.smithi193.stdout:rbd-mirror-18.2.4-717.g03f4c3e8.el9.x86_64.rpm 8.7 MB/s | 3.0 MB 00:00 2024-08-18T15:16:59.562 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-08-18T15:16:59.563 INFO:teuthology.orchestra.run.smithi193.stdout:Total 8.7 MB/s | 3.0 MB 00:00 2024-08-18T15:16:59.563 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-08-18T15:16:59.572 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-08-18T15:16:59.572 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-08-18T15:16:59.578 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : rbd-nbd-2:18.2.4-717.g03f4c3e8.el9.x86_64 1/1 2024-08-18T15:16:59.652 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-08-18T15:16:59.652 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-08-18T15:16:59.956 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: rbd-nbd-2:18.2.4-717.g03f4c3e8.el9.x86_64 1/1 2024-08-18T15:17:00.076 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-08-18T15:17:00.155 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : rbd-mirror-2:18.2.4-717.g03f4c3e8.el9.x86_64 1/1 2024-08-18T15:17:00.180 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: rbd-mirror-2:18.2.4-717.g03f4c3e8.el9.x86_64 1/1 2024-08-18T15:17:00.180 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-18T15:17:00.180 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-08-18T15:17:00.180 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-08-18T15:17:00.180 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-08-18T15:17:00.180 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:17:00.218 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : rbd-nbd-2:18.2.4-717.g03f4c3e8.el9.x86_64 1/1 2024-08-18T15:17:00.223 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:17:00.223 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-08-18T15:17:00.223 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-nbd-2:18.2.4-717.g03f4c3e8.el9.x86_64 2024-08-18T15:17:00.223 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:17:00.223 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-08-18T15:17:00.488 DEBUG:teuthology.parallel:result is None 2024-08-18T15:17:01.229 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : rbd-mirror-2:18.2.4-717.g03f4c3e8.el9.x86_64 1/1 2024-08-18T15:17:01.229 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:17:01.229 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-08-18T15:17:01.229 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-mirror-2:18.2.4-717.g03f4c3e8.el9.x86_64 2024-08-18T15:17:01.229 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:17:01.229 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-08-18T15:17:01.458 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install rbd-nbd 2024-08-18T15:17:02.017 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:01:42 ago on Sun 18 Aug 2024 03:15:20 PM UTC. 2024-08-18T15:17:02.147 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-08-18T15:17:02.148 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-08-18T15:17:02.148 INFO:teuthology.orchestra.run.smithi193.stdout: Package Architecture Version Repository Size 2024-08-18T15:17:02.148 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-08-18T15:17:02.148 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-08-18T15:17:02.148 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-nbd x86_64 2:18.2.4-717.g03f4c3e8.el9 ceph 172 k 2024-08-18T15:17:02.148 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:17:02.148 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-08-18T15:17:02.148 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-08-18T15:17:02.149 INFO:teuthology.orchestra.run.smithi193.stdout:Install 1 Package 2024-08-18T15:17:02.149 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:17:02.149 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 172 k 2024-08-18T15:17:02.149 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 494 k 2024-08-18T15:17:02.149 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-08-18T15:17:02.385 INFO:teuthology.orchestra.run.smithi193.stdout:rbd-nbd-18.2.4-717.g03f4c3e8.el9.x86_64.rpm 728 kB/s | 172 kB 00:00 2024-08-18T15:17:02.386 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-08-18T15:17:02.386 INFO:teuthology.orchestra.run.smithi193.stdout:Total 724 kB/s | 172 kB 00:00 2024-08-18T15:17:02.386 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-08-18T15:17:02.393 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-08-18T15:17:02.393 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-08-18T15:17:02.456 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-08-18T15:17:02.456 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-08-18T15:17:02.613 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-08-18T15:17:02.704 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : rbd-nbd-2:18.2.4-717.g03f4c3e8.el9.x86_64 1/1 2024-08-18T15:17:03.123 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: rbd-nbd-2:18.2.4-717.g03f4c3e8.el9.x86_64 1/1 2024-08-18T15:17:03.370 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : rbd-nbd-2:18.2.4-717.g03f4c3e8.el9.x86_64 1/1 2024-08-18T15:17:03.371 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:17:03.371 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-08-18T15:17:03.371 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-nbd-2:18.2.4-717.g03f4c3e8.el9.x86_64 2024-08-18T15:17:03.371 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:17:03.371 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-08-18T15:17:03.628 DEBUG:teuthology.parallel:result is None 2024-08-18T15:17:03.629 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=03f4c3e85ce172201896147a7aef87ba256187f2 2024-08-18T15:17:03.793 DEBUG:teuthology.orchestra.run.smithi134:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-08-18T15:17:03.819 INFO:teuthology.orchestra.run.smithi134.stdout:18.2.4-717.g03f4c3e8.el9 2024-08-18T15:17:03.820 INFO:teuthology.packaging:The installed version of ceph is 18.2.4-717.g03f4c3e8.el9 2024-08-18T15:17:03.820 INFO:teuthology.task.install:The correct ceph version 18.2.4-717.g03f4c3e8 is installed. 2024-08-18T15:17:03.822 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=03f4c3e85ce172201896147a7aef87ba256187f2 2024-08-18T15:17:03.979 DEBUG:teuthology.orchestra.run.smithi193:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-08-18T15:17:04.001 INFO:teuthology.orchestra.run.smithi193.stdout:18.2.4-717.g03f4c3e8.el9 2024-08-18T15:17:04.001 INFO:teuthology.packaging:The installed version of ceph is 18.2.4-717.g03f4c3e8.el9 2024-08-18T15:17:04.001 INFO:teuthology.task.install:The correct ceph version 18.2.4-717.g03f4c3e8 is installed. 2024-08-18T15:17:04.003 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-08-18T15:17:04.003 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-08-18T15:17:04.003 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-08-18T15:17:04.036 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-08-18T15:17:04.036 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-08-18T15:17:04.067 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-08-18T15:17:04.068 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-08-18T15:17:04.068 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/usr/bin/daemon-helper 2024-08-18T15:17:04.106 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-08-18T15:17:04.174 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-08-18T15:17:04.174 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/usr/bin/daemon-helper 2024-08-18T15:17:04.199 DEBUG:teuthology.orchestra.run.smithi193:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-08-18T15:17:04.264 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-08-18T15:17:04.264 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-08-18T15:17:04.264 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/usr/bin/adjust-ulimits 2024-08-18T15:17:04.292 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-08-18T15:17:04.360 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-08-18T15:17:04.360 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/usr/bin/adjust-ulimits 2024-08-18T15:17:04.384 DEBUG:teuthology.orchestra.run.smithi193:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-08-18T15:17:04.451 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-08-18T15:17:04.451 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-08-18T15:17:04.451 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/usr/bin/stdin-killer 2024-08-18T15:17:04.477 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-08-18T15:17:04.547 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-08-18T15:17:04.547 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/usr/bin/stdin-killer 2024-08-18T15:17:04.572 DEBUG:teuthology.orchestra.run.smithi193:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-08-18T15:17:04.637 INFO:teuthology.run_tasks:Running task ceph... 2024-08-18T15:17:04.712 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-08-18T15:17:04.714 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod 777 /var/log/ceph 2024-08-18T15:17:04.716 DEBUG:teuthology.orchestra.run.smithi193:> sudo chmod 777 /var/log/ceph 2024-08-18T15:17:04.740 INFO:tasks.ceph:Disabling ceph logrotate... 2024-08-18T15:17:04.741 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -f -- /etc/logrotate.d/ceph 2024-08-18T15:17:04.783 DEBUG:teuthology.orchestra.run.smithi193:> sudo rm -f -- /etc/logrotate.d/ceph 2024-08-18T15:17:04.807 INFO:tasks.ceph:Creating extra log directories... 2024-08-18T15:17:04.807 DEBUG:teuthology.orchestra.run.smithi134:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-08-18T15:17:04.849 DEBUG:teuthology.orchestra.run.smithi193:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-08-18T15:17:04.883 INFO:tasks.ceph:Creating ceph cluster c1... 2024-08-18T15:17:04.883 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'cpu_profile': set(), 'cluster': 'c1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-08-18T15:17:04.883 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-08-16_22:40:05-rgw-reef-distro-default-smithi/7858171', 'branch': 'reef', 'description': 'rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zones supported-random-distro$/{centos_latest} tasks/test_multi}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7858171', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'teuthology-2024-08-16_22:40:05-rgw-reef-distro-default-smithi', 'no_nested_subset': False, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'reef'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': '03f4c3e85ce172201896147a7aef87ba256187f2', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '03f4c3e85ce172201896147a7aef87ba256187f2'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'enabled_features': ['resharding'], 'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zonegroup', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zone1'}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'test-zone2'}, {'endpoints': ['c1.client.1'], 'name': 'test-zone3'}]}]}, 'rgw-multisite-tests': {'args': ['tests.py']}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'workunit': {'branch': 'reef', 'sha1': '03f4c3e85ce172201896147a7aef87ba256187f2'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 930, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'seed': 4794, 'sha1': '03f4c3e85ce172201896147a7aef87ba256187f2', 'sleep_before_teardown': 0, 'subset': '0/1', 'suite': 'rgw', 'suite_branch': 'reef', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_03f4c3e85ce172201896147a7aef87ba256187f2/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': '03f4c3e85ce172201896147a7aef87ba256187f2', 'targets': {'smithi134.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIE9XgycsFVMPLcbi8grvCs2X7jnzF+W6OvDtQvdPSikD8DSyZ6GPlvctMpS+baiu/U3LWz8MtsURSjI/icSDh0=', 'smithi193.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCC+k2hQSoI+gvHGxnhhbNPw+ovFp8VjwRVMU4OOFllP2RKhpxQhCB1OIlvSYy7oNCv6WGBH8zahtKKyQE/oYyQ='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': '03f4c3e85ce172201896147a7aef87ba256187f2', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2'}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '67db276914d29505c9707d04f3a3b8fb9d8aee4b', 'timestamp': '2024-08-16_22:40:05', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-08-18T15:17:04.884 DEBUG:teuthology.orchestra.run.smithi134:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2024-08-18T15:17:04.915 DEBUG:teuthology.orchestra.run.smithi193:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2024-08-18T15:17:04.930 DEBUG:teuthology.orchestra.run.smithi134:> sudo install -d -m0777 -- /var/run/ceph 2024-08-18T15:17:04.971 DEBUG:teuthology.orchestra.run.smithi193:> sudo install -d -m0777 -- /var/run/ceph 2024-08-18T15:17:04.995 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-08-18T15:17:04.995 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/scratch_devs of=/dev/stdout 2024-08-18T15:17:05.049 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-08-18T15:17:05.050 DEBUG:teuthology.orchestra.run.smithi134:> stat /dev/vg_nvme/lv_1 2024-08-18T15:17:05.105 INFO:teuthology.orchestra.run.smithi134.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-08-18T15:17:05.105 INFO:teuthology.orchestra.run.smithi134.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-18T15:17:05.105 INFO:teuthology.orchestra.run.smithi134.stdout:Device: 5h/5d Inode: 676 Links: 1 2024-08-18T15:17:05.105 INFO:teuthology.orchestra.run.smithi134.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-18T15:17:05.105 INFO:teuthology.orchestra.run.smithi134.stdout:Context: system_u:object_r:device_t:s0 2024-08-18T15:17:05.105 INFO:teuthology.orchestra.run.smithi134.stdout:Access: 2024-08-18 15:16:57.279599760 +0000 2024-08-18T15:17:05.105 INFO:teuthology.orchestra.run.smithi134.stdout:Modify: 2024-08-18 15:14:22.656846218 +0000 2024-08-18T15:17:05.105 INFO:teuthology.orchestra.run.smithi134.stdout:Change: 2024-08-18 15:14:22.656846218 +0000 2024-08-18T15:17:05.105 INFO:teuthology.orchestra.run.smithi134.stdout: Birth: 2024-08-18 15:14:22.656846218 +0000 2024-08-18T15:17:05.106 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-08-18T15:17:05.168 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records in 2024-08-18T15:17:05.169 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records out 2024-08-18T15:17:05.169 INFO:teuthology.orchestra.run.smithi134.stderr:512 bytes copied, 0.00023807 s, 2.2 MB/s 2024-08-18T15:17:05.170 DEBUG:teuthology.orchestra.run.smithi134:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-08-18T15:17:05.226 DEBUG:teuthology.orchestra.run.smithi134:> stat /dev/vg_nvme/lv_2 2024-08-18T15:17:05.281 INFO:teuthology.orchestra.run.smithi134.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-08-18T15:17:05.281 INFO:teuthology.orchestra.run.smithi134.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-18T15:17:05.282 INFO:teuthology.orchestra.run.smithi134.stdout:Device: 5h/5d Inode: 694 Links: 1 2024-08-18T15:17:05.282 INFO:teuthology.orchestra.run.smithi134.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-18T15:17:05.282 INFO:teuthology.orchestra.run.smithi134.stdout:Context: system_u:object_r:device_t:s0 2024-08-18T15:17:05.282 INFO:teuthology.orchestra.run.smithi134.stdout:Access: 2024-08-18 15:16:57.279599760 +0000 2024-08-18T15:17:05.282 INFO:teuthology.orchestra.run.smithi134.stdout:Modify: 2024-08-18 15:14:22.989838742 +0000 2024-08-18T15:17:05.282 INFO:teuthology.orchestra.run.smithi134.stdout:Change: 2024-08-18 15:14:22.989838742 +0000 2024-08-18T15:17:05.282 INFO:teuthology.orchestra.run.smithi134.stdout: Birth: 2024-08-18 15:14:22.989838742 +0000 2024-08-18T15:17:05.282 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-08-18T15:17:05.345 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records in 2024-08-18T15:17:05.345 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records out 2024-08-18T15:17:05.345 INFO:teuthology.orchestra.run.smithi134.stderr:512 bytes copied, 0.00021517 s, 2.4 MB/s 2024-08-18T15:17:05.346 DEBUG:teuthology.orchestra.run.smithi134:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-08-18T15:17:05.402 DEBUG:teuthology.orchestra.run.smithi134:> stat /dev/vg_nvme/lv_3 2024-08-18T15:17:05.457 INFO:teuthology.orchestra.run.smithi134.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-08-18T15:17:05.457 INFO:teuthology.orchestra.run.smithi134.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-18T15:17:05.458 INFO:teuthology.orchestra.run.smithi134.stdout:Device: 5h/5d Inode: 714 Links: 1 2024-08-18T15:17:05.458 INFO:teuthology.orchestra.run.smithi134.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-18T15:17:05.458 INFO:teuthology.orchestra.run.smithi134.stdout:Context: system_u:object_r:device_t:s0 2024-08-18T15:17:05.458 INFO:teuthology.orchestra.run.smithi134.stdout:Access: 2024-08-18 15:16:57.280599738 +0000 2024-08-18T15:17:05.458 INFO:teuthology.orchestra.run.smithi134.stdout:Modify: 2024-08-18 15:14:23.335830974 +0000 2024-08-18T15:17:05.458 INFO:teuthology.orchestra.run.smithi134.stdout:Change: 2024-08-18 15:14:23.335830974 +0000 2024-08-18T15:17:05.458 INFO:teuthology.orchestra.run.smithi134.stdout: Birth: 2024-08-18 15:14:23.335830974 +0000 2024-08-18T15:17:05.459 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-08-18T15:17:05.520 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records in 2024-08-18T15:17:05.520 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records out 2024-08-18T15:17:05.520 INFO:teuthology.orchestra.run.smithi134.stderr:512 bytes copied, 0.000244219 s, 2.1 MB/s 2024-08-18T15:17:05.521 DEBUG:teuthology.orchestra.run.smithi134:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-08-18T15:17:05.577 DEBUG:teuthology.orchestra.run.smithi134:> stat /dev/vg_nvme/lv_4 2024-08-18T15:17:05.632 INFO:teuthology.orchestra.run.smithi134.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-08-18T15:17:05.632 INFO:teuthology.orchestra.run.smithi134.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-18T15:17:05.632 INFO:teuthology.orchestra.run.smithi134.stdout:Device: 5h/5d Inode: 733 Links: 1 2024-08-18T15:17:05.632 INFO:teuthology.orchestra.run.smithi134.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-18T15:17:05.632 INFO:teuthology.orchestra.run.smithi134.stdout:Context: system_u:object_r:device_t:s0 2024-08-18T15:17:05.632 INFO:teuthology.orchestra.run.smithi134.stdout:Access: 2024-08-18 15:16:57.280599738 +0000 2024-08-18T15:17:05.633 INFO:teuthology.orchestra.run.smithi134.stdout:Modify: 2024-08-18 15:14:23.667823520 +0000 2024-08-18T15:17:05.633 INFO:teuthology.orchestra.run.smithi134.stdout:Change: 2024-08-18 15:14:23.667823520 +0000 2024-08-18T15:17:05.633 INFO:teuthology.orchestra.run.smithi134.stdout: Birth: 2024-08-18 15:14:23.667823520 +0000 2024-08-18T15:17:05.633 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-08-18T15:17:05.695 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records in 2024-08-18T15:17:05.695 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records out 2024-08-18T15:17:05.695 INFO:teuthology.orchestra.run.smithi134.stderr:512 bytes copied, 0.00020281 s, 2.5 MB/s 2024-08-18T15:17:05.696 DEBUG:teuthology.orchestra.run.smithi134:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-08-18T15:17:05.752 INFO:tasks.ceph:osd dev map: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2024-08-18T15:17:05.752 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi134.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2024-08-18T15:17:05.752 INFO:tasks.ceph:Generating config... 2024-08-18T15:17:05.753 INFO:tasks.ceph:[client] debug rgw = 20 2024-08-18T15:17:05.754 INFO:tasks.ceph:[client] debug rgw sync = 20 2024-08-18T15:17:05.754 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2024-08-18T15:17:05.754 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2024-08-18T15:17:05.754 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2024-08-18T15:17:05.754 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2024-08-18T15:17:05.754 INFO:tasks.ceph:[client] rgw data log num shards = 4 2024-08-18T15:17:05.754 INFO:tasks.ceph:[client] rgw md log max shards = 4 2024-08-18T15:17:05.754 INFO:tasks.ceph:[client] rgw sync data inject err probability = 0.1 2024-08-18T15:17:05.754 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2024-08-18T15:17:05.754 INFO:tasks.ceph:[client] rgw sync meta inject err probability = 0.1 2024-08-18T15:17:05.754 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2024-08-18T15:17:05.754 INFO:tasks.ceph:[client] setgroup = ceph 2024-08-18T15:17:05.754 INFO:tasks.ceph:[client] setuser = ceph 2024-08-18T15:17:05.755 INFO:tasks.ceph:[client.0] rgw data notify interval msec = 0 2024-08-18T15:17:05.755 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-08-18T15:17:05.755 INFO:tasks.ceph:[mgr] debug ms = 1 2024-08-18T15:17:05.755 INFO:tasks.ceph:[mon] debug mon = 20 2024-08-18T15:17:05.755 INFO:tasks.ceph:[mon] debug ms = 1 2024-08-18T15:17:05.755 INFO:tasks.ceph:[mon] debug paxos = 20 2024-08-18T15:17:05.755 INFO:tasks.ceph:[osd] debug ms = 1 2024-08-18T15:17:05.755 INFO:tasks.ceph:[osd] debug osd = 20 2024-08-18T15:17:05.755 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2024-08-18T15:17:05.755 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2024-08-18T15:17:05.755 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2024-08-18T15:17:05.755 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2024-08-18T15:17:05.756 INFO:tasks.ceph:Setting up c1.mon.a... 2024-08-18T15:17:05.756 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c1.keyring 2024-08-18T15:17:05.835 INFO:teuthology.orchestra.run.smithi134.stdout:creating /etc/ceph/c1.keyring 2024-08-18T15:17:05.837 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c1.keyring 2024-08-18T15:17:05.919 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod 0644 /etc/ceph/c1.keyring 2024-08-18T15:17:05.984 DEBUG:tasks.ceph:Ceph mon addresses: [('c1.mon.a', '172.21.15.134')] 2024-08-18T15:17:05.984 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.134'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'client.0': {'rgw data notify interval msec': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2024-08-18T15:17:05.985 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-08-18T15:17:05.985 DEBUG:teuthology.orchestra.run.smithi134:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-08-18T15:17:06.040 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.134 --print /home/ubuntu/cephtest/c1.monmap 2024-08-18T15:17:06.119 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:17:06.119 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:17:06.119 INFO:teuthology.orchestra.run.smithi134.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c1.monmap 2024-08-18T15:17:06.119 INFO:teuthology.orchestra.run.smithi134.stdout:monmaptool: generated fsid a11f7e7c-ddeb-4785-ae7a-25ca7716cfc1 2024-08-18T15:17:06.119 INFO:teuthology.orchestra.run.smithi134.stdout:setting min_mon_release = pacific 2024-08-18T15:17:06.120 INFO:teuthology.orchestra.run.smithi134.stdout:epoch 0 2024-08-18T15:17:06.120 INFO:teuthology.orchestra.run.smithi134.stdout:fsid a11f7e7c-ddeb-4785-ae7a-25ca7716cfc1 2024-08-18T15:17:06.120 INFO:teuthology.orchestra.run.smithi134.stdout:last_changed 2024-08-18T15:17:06.119931+0000 2024-08-18T15:17:06.120 INFO:teuthology.orchestra.run.smithi134.stdout:created 2024-08-18T15:17:06.119931+0000 2024-08-18T15:17:06.120 INFO:teuthology.orchestra.run.smithi134.stdout:min_mon_release 16 (pacific) 2024-08-18T15:17:06.120 INFO:teuthology.orchestra.run.smithi134.stdout:election_strategy: 1 2024-08-18T15:17:06.120 INFO:teuthology.orchestra.run.smithi134.stdout:0: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.a 2024-08-18T15:17:06.120 INFO:teuthology.orchestra.run.smithi134.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c1.monmap (1 monitors) 2024-08-18T15:17:06.121 DEBUG:teuthology.orchestra.run.smithi134:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-08-18T15:17:06.176 INFO:tasks.ceph:Writing /etc/ceph/c1.conf for FSID a11f7e7c-ddeb-4785-ae7a-25ca7716cfc1... 2024-08-18T15:17:06.177 DEBUG:teuthology.orchestra.run.smithi134:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2024-08-18T15:17:06.219 DEBUG:teuthology.orchestra.run.smithi193:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2024-08-18T15:17:06.259 INFO:teuthology.orchestra.run.smithi134.stdout:[global] 2024-08-18T15:17:06.259 INFO:teuthology.orchestra.run.smithi134.stdout: chdir = "" 2024-08-18T15:17:06.259 INFO:teuthology.orchestra.run.smithi134.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-08-18T15:17:06.259 INFO:teuthology.orchestra.run.smithi134.stdout: auth supported = cephx 2024-08-18T15:17:06.259 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:17:06.259 INFO:teuthology.orchestra.run.smithi134.stdout: filestore xattr use omap = true 2024-08-18T15:17:06.260 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:17:06.260 INFO:teuthology.orchestra.run.smithi134.stdout: mon clock drift allowed = 1.000 2024-08-18T15:17:06.260 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:17:06.260 INFO:teuthology.orchestra.run.smithi134.stdout: osd crush chooseleaf type = 0 2024-08-18T15:17:06.260 INFO:teuthology.orchestra.run.smithi134.stdout: auth debug = true 2024-08-18T15:17:06.260 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:17:06.260 INFO:teuthology.orchestra.run.smithi134.stdout: ms die on old message = true 2024-08-18T15:17:06.260 INFO:teuthology.orchestra.run.smithi134.stdout: ms die on bug = true 2024-08-18T15:17:06.260 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:17:06.260 INFO:teuthology.orchestra.run.smithi134.stdout: mon max pg per osd = 10000 # >= luminous 2024-08-18T15:17:06.260 INFO:teuthology.orchestra.run.smithi134.stdout: mon pg warn max object skew = 0 2024-08-18T15:17:06.260 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:17:06.260 INFO:teuthology.orchestra.run.smithi134.stdout: # disable pg_autoscaler by default for new pools 2024-08-18T15:17:06.260 INFO:teuthology.orchestra.run.smithi134.stdout: osd_pool_default_pg_autoscale_mode = off 2024-08-18T15:17:06.261 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:17:06.261 INFO:teuthology.orchestra.run.smithi134.stdout: osd pool default size = 2 2024-08-18T15:17:06.261 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:17:06.261 INFO:teuthology.orchestra.run.smithi134.stdout: mon osd allow primary affinity = true 2024-08-18T15:17:06.261 INFO:teuthology.orchestra.run.smithi134.stdout: mon osd allow pg remap = true 2024-08-18T15:17:06.261 INFO:teuthology.orchestra.run.smithi134.stdout: mon warn on legacy crush tunables = false 2024-08-18T15:17:06.261 INFO:teuthology.orchestra.run.smithi134.stdout: mon warn on crush straw calc version zero = false 2024-08-18T15:17:06.261 INFO:teuthology.orchestra.run.smithi134.stdout: mon warn on no sortbitwise = false 2024-08-18T15:17:06.261 INFO:teuthology.orchestra.run.smithi134.stdout: mon warn on osd down out interval zero = false 2024-08-18T15:17:06.261 INFO:teuthology.orchestra.run.smithi134.stdout: mon warn on too few osds = false 2024-08-18T15:17:06.261 INFO:teuthology.orchestra.run.smithi134.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-08-18T15:17:06.261 INFO:teuthology.orchestra.run.smithi134.stdout: mon_warn_on_pool_no_redundancy = false 2024-08-18T15:17:06.261 INFO:teuthology.orchestra.run.smithi134.stdout: mon_allow_pool_size_one = true 2024-08-18T15:17:06.261 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:17:06.262 INFO:teuthology.orchestra.run.smithi134.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-08-18T15:17:06.262 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:17:06.262 INFO:teuthology.orchestra.run.smithi134.stdout: osd default data pool replay window = 5 2024-08-18T15:17:06.262 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:17:06.262 INFO:teuthology.orchestra.run.smithi134.stdout: mon allow pool delete = true 2024-08-18T15:17:06.262 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:17:06.262 INFO:teuthology.orchestra.run.smithi134.stdout: mon cluster log file level = debug 2024-08-18T15:17:06.262 INFO:teuthology.orchestra.run.smithi134.stdout: debug asserts on shutdown = true 2024-08-18T15:17:06.262 INFO:teuthology.orchestra.run.smithi134.stdout: mon health detail to clog = false 2024-08-18T15:17:06.262 INFO:teuthology.orchestra.run.smithi134.stdout: mon host = 172.21.15.134 2024-08-18T15:17:06.262 INFO:teuthology.orchestra.run.smithi134.stdout: fsid = a11f7e7c-ddeb-4785-ae7a-25ca7716cfc1 2024-08-18T15:17:06.262 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:17:06.262 INFO:teuthology.orchestra.run.smithi134.stdout:[osd] 2024-08-18T15:17:06.263 INFO:teuthology.orchestra.run.smithi134.stdout: osd journal size = 100 2024-08-18T15:17:06.263 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:17:06.263 INFO:teuthology.orchestra.run.smithi134.stdout: osd scrub load threshold = 5.0 2024-08-18T15:17:06.263 INFO:teuthology.orchestra.run.smithi134.stdout: osd scrub max interval = 600 2024-08-18T15:17:06.263 INFO:teuthology.orchestra.run.smithi134.stdout: osd mclock profile = high_recovery_ops 2024-08-18T15:17:06.263 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:17:06.263 INFO:teuthology.orchestra.run.smithi134.stdout: osd recover clone overlap = true 2024-08-18T15:17:06.263 INFO:teuthology.orchestra.run.smithi134.stdout: osd recovery max chunk = 1048576 2024-08-18T15:17:06.263 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:17:06.263 INFO:teuthology.orchestra.run.smithi134.stdout: osd debug shutdown = true 2024-08-18T15:17:06.263 INFO:teuthology.orchestra.run.smithi134.stdout: osd debug op order = true 2024-08-18T15:17:06.263 INFO:teuthology.orchestra.run.smithi134.stdout: osd debug verify stray on activate = true 2024-08-18T15:17:06.263 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:17:06.264 INFO:teuthology.orchestra.run.smithi134.stdout: osd open classes on start = true 2024-08-18T15:17:06.264 INFO:teuthology.orchestra.run.smithi134.stdout: osd debug pg log writeout = true 2024-08-18T15:17:06.264 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:17:06.264 INFO:teuthology.orchestra.run.smithi134.stdout: osd deep scrub update digest min age = 30 2024-08-18T15:17:06.264 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:17:06.264 INFO:teuthology.orchestra.run.smithi134.stdout: osd map max advance = 10 2024-08-18T15:17:06.264 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:17:06.264 INFO:teuthology.orchestra.run.smithi134.stdout: journal zero on create = true 2024-08-18T15:17:06.264 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:17:06.264 INFO:teuthology.orchestra.run.smithi134.stdout: filestore ondisk finisher threads = 3 2024-08-18T15:17:06.264 INFO:teuthology.orchestra.run.smithi134.stdout: filestore apply finisher threads = 3 2024-08-18T15:17:06.264 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:17:06.264 INFO:teuthology.orchestra.run.smithi134.stdout: bdev debug aio = true 2024-08-18T15:17:06.265 INFO:teuthology.orchestra.run.smithi134.stdout: osd debug misdirected ops = true 2024-08-18T15:17:06.265 INFO:teuthology.orchestra.run.smithi134.stdout: debug ms = 1 2024-08-18T15:17:06.265 INFO:teuthology.orchestra.run.smithi134.stdout: debug osd = 20 2024-08-18T15:17:06.265 INFO:teuthology.orchestra.run.smithi134.stdout: rocksdb delete range threshold = 0 2024-08-18T15:17:06.265 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:17:06.265 INFO:teuthology.orchestra.run.smithi134.stdout:[mgr] 2024-08-18T15:17:06.265 INFO:teuthology.orchestra.run.smithi134.stdout: debug ms = 1 2024-08-18T15:17:06.265 INFO:teuthology.orchestra.run.smithi134.stdout: debug mgr = 20 2024-08-18T15:17:06.265 INFO:teuthology.orchestra.run.smithi134.stdout: debug mon = 20 2024-08-18T15:17:06.265 INFO:teuthology.orchestra.run.smithi134.stdout: debug auth = 20 2024-08-18T15:17:06.265 INFO:teuthology.orchestra.run.smithi134.stdout: mon reweight min pgs per osd = 4 2024-08-18T15:17:06.265 INFO:teuthology.orchestra.run.smithi134.stdout: mon reweight min bytes per osd = 10 2024-08-18T15:17:06.265 INFO:teuthology.orchestra.run.smithi134.stdout: mgr/telemetry/nag = false 2024-08-18T15:17:06.265 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:17:06.266 INFO:teuthology.orchestra.run.smithi134.stdout:[mon] 2024-08-18T15:17:06.266 INFO:teuthology.orchestra.run.smithi134.stdout: debug ms = 1 2024-08-18T15:17:06.266 INFO:teuthology.orchestra.run.smithi134.stdout: debug mon = 20 2024-08-18T15:17:06.266 INFO:teuthology.orchestra.run.smithi134.stdout: debug paxos = 20 2024-08-18T15:17:06.266 INFO:teuthology.orchestra.run.smithi134.stdout: debug auth = 20 2024-08-18T15:17:06.266 INFO:teuthology.orchestra.run.smithi134.stdout: mon data avail warn = 5 2024-08-18T15:17:06.266 INFO:teuthology.orchestra.run.smithi134.stdout: mon mgr mkfs grace = 240 2024-08-18T15:17:06.266 INFO:teuthology.orchestra.run.smithi134.stdout: mon reweight min pgs per osd = 4 2024-08-18T15:17:06.266 INFO:teuthology.orchestra.run.smithi134.stdout: mon osd reporter subtree level = osd 2024-08-18T15:17:06.266 INFO:teuthology.orchestra.run.smithi134.stdout: mon osd prime pg temp = true 2024-08-18T15:17:06.266 INFO:teuthology.orchestra.run.smithi134.stdout: mon reweight min bytes per osd = 10 2024-08-18T15:17:06.266 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:17:06.266 INFO:teuthology.orchestra.run.smithi134.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-08-18T15:17:06.266 INFO:teuthology.orchestra.run.smithi134.stdout: auth mon ticket ttl = 660 # 11m 2024-08-18T15:17:06.267 INFO:teuthology.orchestra.run.smithi134.stdout: auth service ticket ttl = 240 # 4m 2024-08-18T15:17:06.267 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:17:06.267 INFO:teuthology.orchestra.run.smithi134.stdout: # don't complain about insecure global_id in the test suite 2024-08-18T15:17:06.267 INFO:teuthology.orchestra.run.smithi134.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-08-18T15:17:06.267 INFO:teuthology.orchestra.run.smithi134.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-08-18T15:17:06.267 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:17:06.267 INFO:teuthology.orchestra.run.smithi134.stdout: # 1m isn't quite enough 2024-08-18T15:17:06.267 INFO:teuthology.orchestra.run.smithi134.stdout: mon_down_mkfs_grace = 2m 2024-08-18T15:17:06.267 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:17:06.267 INFO:teuthology.orchestra.run.smithi134.stdout: mon_warn_on_filestore_osds = false 2024-08-18T15:17:06.267 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:17:06.267 INFO:teuthology.orchestra.run.smithi134.stdout:[client] 2024-08-18T15:17:06.267 INFO:teuthology.orchestra.run.smithi134.stdout: rgw cache enabled = true 2024-08-18T15:17:06.267 INFO:teuthology.orchestra.run.smithi134.stdout: rgw enable ops log = true 2024-08-18T15:17:06.267 INFO:teuthology.orchestra.run.smithi134.stdout: rgw enable usage log = true 2024-08-18T15:17:06.267 INFO:teuthology.orchestra.run.smithi134.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-08-18T15:17:06.268 INFO:teuthology.orchestra.run.smithi134.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-08-18T15:17:06.268 INFO:teuthology.orchestra.run.smithi134.stdout: debug rgw = 20 2024-08-18T15:17:06.268 INFO:teuthology.orchestra.run.smithi134.stdout: debug rgw sync = 20 2024-08-18T15:17:06.268 INFO:teuthology.orchestra.run.smithi134.stdout: rgw crypt require ssl = False 2024-08-18T15:17:06.268 INFO:teuthology.orchestra.run.smithi134.stdout: rgw crypt s3 kms backend = testing 2024-08-18T15:17:06.268 INFO:teuthology.orchestra.run.smithi134.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2024-08-18T15:17:06.268 INFO:teuthology.orchestra.run.smithi134.stdout: rgw curl low speed time = 300 2024-08-18T15:17:06.268 INFO:teuthology.orchestra.run.smithi134.stdout: rgw data log num shards = 4 2024-08-18T15:17:06.268 INFO:teuthology.orchestra.run.smithi134.stdout: rgw md log max shards = 4 2024-08-18T15:17:06.268 INFO:teuthology.orchestra.run.smithi134.stdout: rgw sync data inject err probability = 0.1 2024-08-18T15:17:06.268 INFO:teuthology.orchestra.run.smithi134.stdout: rgw sync log trim interval = 0 2024-08-18T15:17:06.268 INFO:teuthology.orchestra.run.smithi134.stdout: rgw sync meta inject err probability = 0.1 2024-08-18T15:17:06.269 INFO:teuthology.orchestra.run.smithi134.stdout: rgw sync obj etag verify = True 2024-08-18T15:17:06.269 INFO:teuthology.orchestra.run.smithi134.stdout: setgroup = ceph 2024-08-18T15:17:06.269 INFO:teuthology.orchestra.run.smithi134.stdout: setuser = ceph 2024-08-18T15:17:06.269 INFO:teuthology.orchestra.run.smithi134.stdout:[mon.a] 2024-08-18T15:17:06.269 INFO:teuthology.orchestra.run.smithi134.stdout:[client.0] 2024-08-18T15:17:06.269 INFO:teuthology.orchestra.run.smithi134.stdout: rgw data notify interval msec = 0 2024-08-18T15:17:06.269 INFO:teuthology.orchestra.run.smithi134.stdout:[osd.0] 2024-08-18T15:17:06.269 INFO:teuthology.orchestra.run.smithi134.stdout: osd_max_omap_entries_per_request = 10 2024-08-18T15:17:06.269 INFO:teuthology.orchestra.run.smithi134.stdout:[osd.1] 2024-08-18T15:17:06.269 INFO:teuthology.orchestra.run.smithi134.stdout: osd_max_omap_entries_per_request = 1000 2024-08-18T15:17:06.269 INFO:teuthology.orchestra.run.smithi134.stdout:[osd.2] 2024-08-18T15:17:06.269 INFO:teuthology.orchestra.run.smithi134.stdout: osd_max_omap_entries_per_request = 10000 2024-08-18T15:17:06.270 INFO:teuthology.orchestra.run.smithi193.stdout:[global] 2024-08-18T15:17:06.270 INFO:teuthology.orchestra.run.smithi193.stdout: chdir = "" 2024-08-18T15:17:06.270 INFO:teuthology.orchestra.run.smithi193.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-08-18T15:17:06.270 INFO:teuthology.orchestra.run.smithi193.stdout: auth supported = cephx 2024-08-18T15:17:06.270 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:17:06.270 INFO:teuthology.orchestra.run.smithi193.stdout: filestore xattr use omap = true 2024-08-18T15:17:06.270 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:17:06.271 INFO:teuthology.orchestra.run.smithi193.stdout: mon clock drift allowed = 1.000 2024-08-18T15:17:06.271 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:17:06.271 INFO:teuthology.orchestra.run.smithi193.stdout: osd crush chooseleaf type = 0 2024-08-18T15:17:06.271 INFO:teuthology.orchestra.run.smithi193.stdout: auth debug = true 2024-08-18T15:17:06.271 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:17:06.271 INFO:teuthology.orchestra.run.smithi193.stdout: ms die on old message = true 2024-08-18T15:17:06.271 INFO:teuthology.orchestra.run.smithi193.stdout: ms die on bug = true 2024-08-18T15:17:06.271 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:17:06.271 INFO:teuthology.orchestra.run.smithi193.stdout: mon max pg per osd = 10000 # >= luminous 2024-08-18T15:17:06.271 INFO:teuthology.orchestra.run.smithi193.stdout: mon pg warn max object skew = 0 2024-08-18T15:17:06.272 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:17:06.272 INFO:teuthology.orchestra.run.smithi193.stdout: # disable pg_autoscaler by default for new pools 2024-08-18T15:17:06.272 INFO:teuthology.orchestra.run.smithi193.stdout: osd_pool_default_pg_autoscale_mode = off 2024-08-18T15:17:06.272 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:17:06.272 INFO:teuthology.orchestra.run.smithi193.stdout: osd pool default size = 2 2024-08-18T15:17:06.272 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:17:06.272 INFO:teuthology.orchestra.run.smithi193.stdout: mon osd allow primary affinity = true 2024-08-18T15:17:06.272 INFO:teuthology.orchestra.run.smithi193.stdout: mon osd allow pg remap = true 2024-08-18T15:17:06.272 INFO:teuthology.orchestra.run.smithi193.stdout: mon warn on legacy crush tunables = false 2024-08-18T15:17:06.272 INFO:teuthology.orchestra.run.smithi193.stdout: mon warn on crush straw calc version zero = false 2024-08-18T15:17:06.272 INFO:teuthology.orchestra.run.smithi193.stdout: mon warn on no sortbitwise = false 2024-08-18T15:17:06.273 INFO:teuthology.orchestra.run.smithi193.stdout: mon warn on osd down out interval zero = false 2024-08-18T15:17:06.273 INFO:teuthology.orchestra.run.smithi193.stdout: mon warn on too few osds = false 2024-08-18T15:17:06.273 INFO:teuthology.orchestra.run.smithi193.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-08-18T15:17:06.273 INFO:teuthology.orchestra.run.smithi193.stdout: mon_warn_on_pool_no_redundancy = false 2024-08-18T15:17:06.273 INFO:teuthology.orchestra.run.smithi193.stdout: mon_allow_pool_size_one = true 2024-08-18T15:17:06.273 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:17:06.273 INFO:teuthology.orchestra.run.smithi193.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-08-18T15:17:06.273 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:17:06.273 INFO:teuthology.orchestra.run.smithi193.stdout: osd default data pool replay window = 5 2024-08-18T15:17:06.273 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:17:06.273 INFO:teuthology.orchestra.run.smithi193.stdout: mon allow pool delete = true 2024-08-18T15:17:06.274 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:17:06.274 INFO:teuthology.orchestra.run.smithi193.stdout: mon cluster log file level = debug 2024-08-18T15:17:06.274 INFO:teuthology.orchestra.run.smithi193.stdout: debug asserts on shutdown = true 2024-08-18T15:17:06.274 INFO:teuthology.orchestra.run.smithi193.stdout: mon health detail to clog = false 2024-08-18T15:17:06.274 INFO:teuthology.orchestra.run.smithi193.stdout: mon host = 172.21.15.134 2024-08-18T15:17:06.274 INFO:teuthology.orchestra.run.smithi193.stdout: fsid = a11f7e7c-ddeb-4785-ae7a-25ca7716cfc1 2024-08-18T15:17:06.274 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:17:06.274 INFO:teuthology.orchestra.run.smithi193.stdout:[osd] 2024-08-18T15:17:06.274 INFO:teuthology.orchestra.run.smithi193.stdout: osd journal size = 100 2024-08-18T15:17:06.274 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:17:06.274 INFO:teuthology.orchestra.run.smithi193.stdout: osd scrub load threshold = 5.0 2024-08-18T15:17:06.275 INFO:teuthology.orchestra.run.smithi193.stdout: osd scrub max interval = 600 2024-08-18T15:17:06.275 INFO:teuthology.orchestra.run.smithi193.stdout: osd mclock profile = high_recovery_ops 2024-08-18T15:17:06.275 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:17:06.275 INFO:teuthology.orchestra.run.smithi193.stdout: osd recover clone overlap = true 2024-08-18T15:17:06.275 INFO:teuthology.orchestra.run.smithi193.stdout: osd recovery max chunk = 1048576 2024-08-18T15:17:06.275 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:17:06.275 INFO:teuthology.orchestra.run.smithi193.stdout: osd debug shutdown = true 2024-08-18T15:17:06.275 INFO:teuthology.orchestra.run.smithi193.stdout: osd debug op order = true 2024-08-18T15:17:06.275 INFO:teuthology.orchestra.run.smithi193.stdout: osd debug verify stray on activate = true 2024-08-18T15:17:06.276 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:17:06.276 INFO:teuthology.orchestra.run.smithi193.stdout: osd open classes on start = true 2024-08-18T15:17:06.276 INFO:teuthology.orchestra.run.smithi193.stdout: osd debug pg log writeout = true 2024-08-18T15:17:06.276 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:17:06.276 INFO:teuthology.orchestra.run.smithi193.stdout: osd deep scrub update digest min age = 30 2024-08-18T15:17:06.276 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:17:06.276 INFO:teuthology.orchestra.run.smithi193.stdout: osd map max advance = 10 2024-08-18T15:17:06.276 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:17:06.276 INFO:teuthology.orchestra.run.smithi193.stdout: journal zero on create = true 2024-08-18T15:17:06.276 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:17:06.276 INFO:teuthology.orchestra.run.smithi193.stdout: filestore ondisk finisher threads = 3 2024-08-18T15:17:06.276 INFO:teuthology.orchestra.run.smithi193.stdout: filestore apply finisher threads = 3 2024-08-18T15:17:06.277 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:17:06.277 INFO:teuthology.orchestra.run.smithi193.stdout: bdev debug aio = true 2024-08-18T15:17:06.277 INFO:teuthology.orchestra.run.smithi193.stdout: osd debug misdirected ops = true 2024-08-18T15:17:06.277 INFO:teuthology.orchestra.run.smithi193.stdout: debug ms = 1 2024-08-18T15:17:06.277 INFO:teuthology.orchestra.run.smithi193.stdout: debug osd = 20 2024-08-18T15:17:06.277 INFO:teuthology.orchestra.run.smithi193.stdout: rocksdb delete range threshold = 0 2024-08-18T15:17:06.277 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:17:06.277 INFO:teuthology.orchestra.run.smithi193.stdout:[mgr] 2024-08-18T15:17:06.277 INFO:teuthology.orchestra.run.smithi193.stdout: debug ms = 1 2024-08-18T15:17:06.277 INFO:teuthology.orchestra.run.smithi193.stdout: debug mgr = 20 2024-08-18T15:17:06.278 INFO:teuthology.orchestra.run.smithi193.stdout: debug mon = 20 2024-08-18T15:17:06.278 INFO:teuthology.orchestra.run.smithi193.stdout: debug auth = 20 2024-08-18T15:17:06.278 INFO:teuthology.orchestra.run.smithi193.stdout: mon reweight min pgs per osd = 4 2024-08-18T15:17:06.278 INFO:teuthology.orchestra.run.smithi193.stdout: mon reweight min bytes per osd = 10 2024-08-18T15:17:06.278 INFO:teuthology.orchestra.run.smithi193.stdout: mgr/telemetry/nag = false 2024-08-18T15:17:06.278 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:17:06.278 INFO:teuthology.orchestra.run.smithi193.stdout:[mon] 2024-08-18T15:17:06.278 INFO:teuthology.orchestra.run.smithi193.stdout: debug ms = 1 2024-08-18T15:17:06.278 INFO:teuthology.orchestra.run.smithi193.stdout: debug mon = 20 2024-08-18T15:17:06.278 INFO:teuthology.orchestra.run.smithi193.stdout: debug paxos = 20 2024-08-18T15:17:06.278 INFO:teuthology.orchestra.run.smithi193.stdout: debug auth = 20 2024-08-18T15:17:06.279 INFO:teuthology.orchestra.run.smithi193.stdout: mon data avail warn = 5 2024-08-18T15:17:06.279 INFO:teuthology.orchestra.run.smithi193.stdout: mon mgr mkfs grace = 240 2024-08-18T15:17:06.279 INFO:teuthology.orchestra.run.smithi193.stdout: mon reweight min pgs per osd = 4 2024-08-18T15:17:06.279 INFO:teuthology.orchestra.run.smithi193.stdout: mon osd reporter subtree level = osd 2024-08-18T15:17:06.279 INFO:teuthology.orchestra.run.smithi193.stdout: mon osd prime pg temp = true 2024-08-18T15:17:06.279 INFO:teuthology.orchestra.run.smithi193.stdout: mon reweight min bytes per osd = 10 2024-08-18T15:17:06.279 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:17:06.279 INFO:teuthology.orchestra.run.smithi193.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-08-18T15:17:06.279 INFO:teuthology.orchestra.run.smithi193.stdout: auth mon ticket ttl = 660 # 11m 2024-08-18T15:17:06.279 INFO:teuthology.orchestra.run.smithi193.stdout: auth service ticket ttl = 240 # 4m 2024-08-18T15:17:06.280 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:17:06.280 INFO:teuthology.orchestra.run.smithi193.stdout: # don't complain about insecure global_id in the test suite 2024-08-18T15:17:06.280 INFO:teuthology.orchestra.run.smithi193.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-08-18T15:17:06.280 INFO:teuthology.orchestra.run.smithi193.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-08-18T15:17:06.280 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:17:06.280 INFO:teuthology.orchestra.run.smithi193.stdout: # 1m isn't quite enough 2024-08-18T15:17:06.280 INFO:teuthology.orchestra.run.smithi193.stdout: mon_down_mkfs_grace = 2m 2024-08-18T15:17:06.280 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:17:06.280 INFO:teuthology.orchestra.run.smithi193.stdout: mon_warn_on_filestore_osds = false 2024-08-18T15:17:06.280 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:17:06.281 INFO:teuthology.orchestra.run.smithi193.stdout:[client] 2024-08-18T15:17:06.281 INFO:teuthology.orchestra.run.smithi193.stdout: rgw cache enabled = true 2024-08-18T15:17:06.281 INFO:teuthology.orchestra.run.smithi193.stdout: rgw enable ops log = true 2024-08-18T15:17:06.281 INFO:teuthology.orchestra.run.smithi193.stdout: rgw enable usage log = true 2024-08-18T15:17:06.281 INFO:teuthology.orchestra.run.smithi193.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-08-18T15:17:06.281 INFO:teuthology.orchestra.run.smithi193.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-08-18T15:17:06.281 INFO:teuthology.orchestra.run.smithi193.stdout: debug rgw = 20 2024-08-18T15:17:06.281 INFO:teuthology.orchestra.run.smithi193.stdout: debug rgw sync = 20 2024-08-18T15:17:06.281 INFO:teuthology.orchestra.run.smithi193.stdout: rgw crypt require ssl = False 2024-08-18T15:17:06.281 INFO:teuthology.orchestra.run.smithi193.stdout: rgw crypt s3 kms backend = testing 2024-08-18T15:17:06.282 INFO:teuthology.orchestra.run.smithi193.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2024-08-18T15:17:06.282 INFO:teuthology.orchestra.run.smithi193.stdout: rgw curl low speed time = 300 2024-08-18T15:17:06.282 INFO:teuthology.orchestra.run.smithi193.stdout: rgw data log num shards = 4 2024-08-18T15:17:06.282 INFO:teuthology.orchestra.run.smithi193.stdout: rgw md log max shards = 4 2024-08-18T15:17:06.282 INFO:teuthology.orchestra.run.smithi193.stdout: rgw sync data inject err probability = 0.1 2024-08-18T15:17:06.282 INFO:teuthology.orchestra.run.smithi193.stdout: rgw sync log trim interval = 0 2024-08-18T15:17:06.282 INFO:teuthology.orchestra.run.smithi193.stdout: rgw sync meta inject err probability = 0.1 2024-08-18T15:17:06.282 INFO:teuthology.orchestra.run.smithi193.stdout: rgw sync obj etag verify = True 2024-08-18T15:17:06.282 INFO:teuthology.orchestra.run.smithi193.stdout: setgroup = ceph 2024-08-18T15:17:06.282 INFO:teuthology.orchestra.run.smithi193.stdout: setuser = ceph 2024-08-18T15:17:06.282 INFO:teuthology.orchestra.run.smithi193.stdout:[mon.a] 2024-08-18T15:17:06.283 INFO:teuthology.orchestra.run.smithi193.stdout:[client.0] 2024-08-18T15:17:06.283 INFO:teuthology.orchestra.run.smithi193.stdout: rgw data notify interval msec = 0 2024-08-18T15:17:06.283 INFO:teuthology.orchestra.run.smithi193.stdout:[osd.0] 2024-08-18T15:17:06.283 INFO:teuthology.orchestra.run.smithi193.stdout: osd_max_omap_entries_per_request = 10 2024-08-18T15:17:06.283 INFO:teuthology.orchestra.run.smithi193.stdout:[osd.1] 2024-08-18T15:17:06.283 INFO:teuthology.orchestra.run.smithi193.stdout: osd_max_omap_entries_per_request = 1000 2024-08-18T15:17:06.283 INFO:teuthology.orchestra.run.smithi193.stdout:[osd.2] 2024-08-18T15:17:06.283 INFO:teuthology.orchestra.run.smithi193.stdout: osd_max_omap_entries_per_request = 10000 2024-08-18T15:17:06.285 INFO:tasks.ceph:Creating admin key on c1.mon.a... 2024-08-18T15:17:06.285 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c1.keyring 2024-08-18T15:17:06.380 INFO:tasks.ceph:Copying monmap to all nodes... 2024-08-18T15:17:06.380 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-08-18T15:17:06.380 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/etc/ceph/c1.keyring of=/dev/stdout 2024-08-18T15:17:06.395 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-08-18T15:17:06.396 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/home/ubuntu/cephtest/c1.monmap of=/dev/stdout 2024-08-18T15:17:06.450 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi134.front.sepia.ceph.com 2024-08-18T15:17:06.450 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-08-18T15:17:06.450 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/etc/ceph/c1.keyring 2024-08-18T15:17:06.450 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod 0644 /etc/ceph/c1.keyring 2024-08-18T15:17:06.524 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-08-18T15:17:06.524 DEBUG:teuthology.orchestra.run.smithi134:> dd of=/home/ubuntu/cephtest/c1.monmap 2024-08-18T15:17:06.581 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi193.front.sepia.ceph.com 2024-08-18T15:17:06.581 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-08-18T15:17:06.581 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/etc/ceph/c1.keyring 2024-08-18T15:17:06.581 DEBUG:teuthology.orchestra.run.smithi193:> sudo chmod 0644 /etc/ceph/c1.keyring 2024-08-18T15:17:06.620 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-08-18T15:17:06.620 DEBUG:teuthology.orchestra.run.smithi193:> dd of=/home/ubuntu/cephtest/c1.monmap 2024-08-18T15:17:06.677 INFO:tasks.ceph:Setting up mon nodes... 2024-08-18T15:17:06.677 INFO:tasks.ceph:Setting up mgr nodes... 2024-08-18T15:17:06.678 DEBUG:teuthology.orchestra.run.smithi134:> sudo mkdir -p /var/lib/ceph/mgr/c1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c1-x/keyring 2024-08-18T15:17:06.727 INFO:teuthology.orchestra.run.smithi134.stdout:creating /var/lib/ceph/mgr/c1-x/keyring 2024-08-18T15:17:06.730 INFO:tasks.ceph:Setting up mds nodes... 2024-08-18T15:17:06.730 INFO:tasks.ceph_client:Setting up client nodes... 2024-08-18T15:17:06.730 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c1.client.0.keyring && sudo chmod 0644 /etc/ceph/c1.client.0.keyring 2024-08-18T15:17:06.772 INFO:teuthology.orchestra.run.smithi134.stdout:creating /etc/ceph/c1.client.0.keyring 2024-08-18T15:17:06.784 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c1.client.1.keyring && sudo chmod 0644 /etc/ceph/c1.client.1.keyring 2024-08-18T15:17:06.865 INFO:teuthology.orchestra.run.smithi134.stdout:creating /etc/ceph/c1.client.1.keyring 2024-08-18T15:17:06.877 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-08-18T15:17:06.877 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi134.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2024-08-18T15:17:06.877 DEBUG:teuthology.orchestra.run.smithi134:> sudo mkdir -p /var/lib/ceph/osd/c1-0 2024-08-18T15:17:06.943 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2024-08-18T15:17:06.943 INFO:tasks.ceph:role: c1.osd.0 2024-08-18T15:17:06.943 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi134.front.sepia.ceph.com 2024-08-18T15:17:06.943 DEBUG:teuthology.orchestra.run.smithi134:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-08-18T15:17:07.014 INFO:teuthology.orchestra.run.smithi134.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-08-18T15:17:07.014 INFO:teuthology.orchestra.run.smithi134.stdout: = sectsz=512 attr=2, projid32bit=1 2024-08-18T15:17:07.014 INFO:teuthology.orchestra.run.smithi134.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-08-18T15:17:07.014 INFO:teuthology.orchestra.run.smithi134.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-08-18T15:17:07.014 INFO:teuthology.orchestra.run.smithi134.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-08-18T15:17:07.015 INFO:teuthology.orchestra.run.smithi134.stdout: = sunit=0 swidth=0 blks 2024-08-18T15:17:07.015 INFO:teuthology.orchestra.run.smithi134.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-08-18T15:17:07.015 INFO:teuthology.orchestra.run.smithi134.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-08-18T15:17:07.015 INFO:teuthology.orchestra.run.smithi134.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-08-18T15:17:07.015 INFO:teuthology.orchestra.run.smithi134.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-08-18T15:17:07.512 INFO:teuthology.orchestra.run.smithi134.stdout:Discarding blocks...Done. 2024-08-18T15:17:07.514 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi134.front.sepia.ceph.com -o noatime 2024-08-18T15:17:07.514 DEBUG:teuthology.orchestra.run.smithi134:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c1-0 2024-08-18T15:17:07.620 DEBUG:teuthology.orchestra.run.smithi134:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-0 2024-08-18T15:17:07.652 DEBUG:teuthology.orchestra.run.smithi134:> sudo mkdir -p /var/lib/ceph/osd/c1-1 2024-08-18T15:17:07.717 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2024-08-18T15:17:07.717 INFO:tasks.ceph:role: c1.osd.1 2024-08-18T15:17:07.717 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi134.front.sepia.ceph.com 2024-08-18T15:17:07.717 DEBUG:teuthology.orchestra.run.smithi134:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-08-18T15:17:07.790 INFO:teuthology.orchestra.run.smithi134.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-08-18T15:17:07.790 INFO:teuthology.orchestra.run.smithi134.stdout: = sectsz=512 attr=2, projid32bit=1 2024-08-18T15:17:07.790 INFO:teuthology.orchestra.run.smithi134.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-08-18T15:17:07.790 INFO:teuthology.orchestra.run.smithi134.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-08-18T15:17:07.790 INFO:teuthology.orchestra.run.smithi134.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-08-18T15:17:07.790 INFO:teuthology.orchestra.run.smithi134.stdout: = sunit=0 swidth=0 blks 2024-08-18T15:17:07.790 INFO:teuthology.orchestra.run.smithi134.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-08-18T15:17:07.790 INFO:teuthology.orchestra.run.smithi134.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-08-18T15:17:07.790 INFO:teuthology.orchestra.run.smithi134.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-08-18T15:17:07.791 INFO:teuthology.orchestra.run.smithi134.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-08-18T15:17:08.240 INFO:teuthology.orchestra.run.smithi134.stdout:Discarding blocks...Done. 2024-08-18T15:17:08.242 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi134.front.sepia.ceph.com -o noatime 2024-08-18T15:17:08.243 DEBUG:teuthology.orchestra.run.smithi134:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c1-1 2024-08-18T15:17:08.317 DEBUG:teuthology.orchestra.run.smithi134:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-1 2024-08-18T15:17:08.387 DEBUG:teuthology.orchestra.run.smithi134:> sudo mkdir -p /var/lib/ceph/osd/c1-2 2024-08-18T15:17:08.452 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2024-08-18T15:17:08.453 INFO:tasks.ceph:role: c1.osd.2 2024-08-18T15:17:08.453 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi134.front.sepia.ceph.com 2024-08-18T15:17:08.453 DEBUG:teuthology.orchestra.run.smithi134:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-08-18T15:17:08.525 INFO:teuthology.orchestra.run.smithi134.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-08-18T15:17:08.525 INFO:teuthology.orchestra.run.smithi134.stdout: = sectsz=512 attr=2, projid32bit=1 2024-08-18T15:17:08.525 INFO:teuthology.orchestra.run.smithi134.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-08-18T15:17:08.525 INFO:teuthology.orchestra.run.smithi134.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-08-18T15:17:08.525 INFO:teuthology.orchestra.run.smithi134.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-08-18T15:17:08.525 INFO:teuthology.orchestra.run.smithi134.stdout: = sunit=0 swidth=0 blks 2024-08-18T15:17:08.525 INFO:teuthology.orchestra.run.smithi134.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-08-18T15:17:08.526 INFO:teuthology.orchestra.run.smithi134.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-08-18T15:17:08.526 INFO:teuthology.orchestra.run.smithi134.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-08-18T15:17:08.526 INFO:teuthology.orchestra.run.smithi134.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-08-18T15:17:09.020 INFO:teuthology.orchestra.run.smithi134.stdout:Discarding blocks...Done. 2024-08-18T15:17:09.022 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi134.front.sepia.ceph.com -o noatime 2024-08-18T15:17:09.023 DEBUG:teuthology.orchestra.run.smithi134:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c1-2 2024-08-18T15:17:09.117 DEBUG:teuthology.orchestra.run.smithi134:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-2 2024-08-18T15:17:09.149 DEBUG:teuthology.orchestra.run.smithi134:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c1.monmap 2024-08-18T15:17:09.251 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:17:09.250+0000 7fdff1e16640 -1 auth: error reading file: /var/lib/ceph/osd/c1-0/keyring: can't open /var/lib/ceph/osd/c1-0/keyring: (2) No such file or directory 2024-08-18T15:17:09.251 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:17:09.250+0000 7fdff1e16640 -1 created new key in keyring /var/lib/ceph/osd/c1-0/keyring 2024-08-18T15:17:09.251 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:17:09.250+0000 7fdff1e16640 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2024-08-18T15:17:09.251 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:17:09.250+0000 7fdff1e16640 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2024-08-18T15:17:09.251 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:17:09.250+0000 7fdff1e16640 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2024-08-18T15:17:09.259 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:17:09.258+0000 7fdff1e16640 -1 bluestore(/var/lib/ceph/osd/c1-0) _read_fsid unparsable uuid 2024-08-18T15:17:11.348 DEBUG:teuthology.orchestra.run.smithi134:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-0 2024-08-18T15:17:11.377 DEBUG:teuthology.orchestra.run.smithi134:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c1.monmap 2024-08-18T15:17:11.468 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:17:11.467+0000 7f929600f640 -1 auth: error reading file: /var/lib/ceph/osd/c1-1/keyring: can't open /var/lib/ceph/osd/c1-1/keyring: (2) No such file or directory 2024-08-18T15:17:11.468 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:17:11.467+0000 7f929600f640 -1 created new key in keyring /var/lib/ceph/osd/c1-1/keyring 2024-08-18T15:17:11.468 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:17:11.467+0000 7f929600f640 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2024-08-18T15:17:11.468 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:17:11.467+0000 7f929600f640 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2024-08-18T15:17:11.468 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:17:11.467+0000 7f929600f640 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2024-08-18T15:17:11.469 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:17:11.468+0000 7f929600f640 -1 bluestore(/var/lib/ceph/osd/c1-1) _read_fsid unparsable uuid 2024-08-18T15:17:13.547 DEBUG:teuthology.orchestra.run.smithi134:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-1 2024-08-18T15:17:13.577 DEBUG:teuthology.orchestra.run.smithi134:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c1.monmap 2024-08-18T15:17:13.667 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:17:13.666+0000 7fe933418640 -1 auth: error reading file: /var/lib/ceph/osd/c1-2/keyring: can't open /var/lib/ceph/osd/c1-2/keyring: (2) No such file or directory 2024-08-18T15:17:13.667 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:17:13.666+0000 7fe933418640 -1 created new key in keyring /var/lib/ceph/osd/c1-2/keyring 2024-08-18T15:17:13.667 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:17:13.666+0000 7fe933418640 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2024-08-18T15:17:13.667 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:17:13.666+0000 7fe933418640 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2024-08-18T15:17:13.667 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:17:13.666+0000 7fe933418640 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2024-08-18T15:17:13.668 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:17:13.667+0000 7fe933418640 -1 bluestore(/var/lib/ceph/osd/c1-2) _read_fsid unparsable uuid 2024-08-18T15:17:15.761 DEBUG:teuthology.orchestra.run.smithi134:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-2 2024-08-18T15:17:15.790 INFO:tasks.ceph:Reading keys from all nodes... 2024-08-18T15:17:15.791 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-08-18T15:17:15.791 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/var/lib/ceph/mgr/c1-x/keyring of=/dev/stdout 2024-08-18T15:17:15.858 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-08-18T15:17:15.859 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/var/lib/ceph/osd/c1-0/keyring of=/dev/stdout 2024-08-18T15:17:15.926 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-08-18T15:17:15.926 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/var/lib/ceph/osd/c1-1/keyring of=/dev/stdout 2024-08-18T15:17:15.993 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-08-18T15:17:15.993 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/var/lib/ceph/osd/c1-2/keyring of=/dev/stdout 2024-08-18T15:17:16.060 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-08-18T15:17:16.060 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/etc/ceph/c1.client.0.keyring of=/dev/stdout 2024-08-18T15:17:16.116 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-08-18T15:17:16.116 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/etc/ceph/c1.client.1.keyring of=/dev/stdout 2024-08-18T15:17:16.172 INFO:tasks.ceph:Adding keys to all mons... 2024-08-18T15:17:16.172 DEBUG:teuthology.orchestra.run.smithi134:> sudo tee -a /etc/ceph/c1.keyring 2024-08-18T15:17:16.236 INFO:teuthology.orchestra.run.smithi134.stdout:[mgr.x] 2024-08-18T15:17:16.236 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQByEMJmuV1qKxAAREBA8gplneButR4UaW1QWA== 2024-08-18T15:17:16.236 INFO:teuthology.orchestra.run.smithi134.stdout:[osd.0] 2024-08-18T15:17:16.236 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQB1EMJme9r7DhAAZHrhlPxxUdXHS3zlsZBeJQ== 2024-08-18T15:17:16.236 INFO:teuthology.orchestra.run.smithi134.stdout:[osd.1] 2024-08-18T15:17:16.236 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQB3EMJmkxLrGxAAgxSeN8owAy8JkuM1hPU3lA== 2024-08-18T15:17:16.236 INFO:teuthology.orchestra.run.smithi134.stdout:[osd.2] 2024-08-18T15:17:16.236 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQB5EMJmVPLGJxAA5WOx6EXWNTOdHoy4PwtrwQ== 2024-08-18T15:17:16.236 INFO:teuthology.orchestra.run.smithi134.stdout:[client.0] 2024-08-18T15:17:16.236 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQByEMJm/0sNLhAA9iD+C2IcOAtZTLQT0ljafw== 2024-08-18T15:17:16.236 INFO:teuthology.orchestra.run.smithi134.stdout:[client.1] 2024-08-18T15:17:16.236 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQByEMJmAE2XMxAARy2H+28jOHvnT3dPM1IUsA== 2024-08-18T15:17:16.237 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-08-18T15:17:16.329 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-18T15:17:16.379 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-18T15:17:16.429 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-18T15:17:16.479 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-08-18T15:17:16.531 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-08-18T15:17:16.584 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-08-18T15:17:16.584 DEBUG:teuthology.orchestra.run.smithi134:> sudo mkdir -p /var/lib/ceph/mon/c1-a 2024-08-18T15:17:16.608 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c1 --mkfs -i a --monmap /home/ubuntu/cephtest/c1.monmap --keyring /etc/ceph/c1.keyring 2024-08-18T15:17:16.704 DEBUG:teuthology.orchestra.run.smithi134:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c1-a 2024-08-18T15:17:16.729 DEBUG:teuthology.orchestra.run.smithi134:> rm -- /home/ubuntu/cephtest/c1.monmap 2024-08-18T15:17:16.785 INFO:tasks.ceph:Starting mon daemons in cluster c1... 2024-08-18T15:17:16.785 INFO:tasks.ceph.c1.mon.a:Restarting daemon 2024-08-18T15:17:16.785 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c1 -i a 2024-08-18T15:17:16.827 INFO:tasks.ceph.c1.mon.a:Started 2024-08-18T15:17:16.827 INFO:tasks.ceph:Starting mgr daemons in cluster c1... 2024-08-18T15:17:16.827 INFO:tasks.ceph.c1.mgr.x:Restarting daemon 2024-08-18T15:17:16.827 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c1 -i x 2024-08-18T15:17:16.829 INFO:tasks.ceph.c1.mgr.x:Started 2024-08-18T15:17:16.829 INFO:tasks.ceph:Setting crush tunables to default 2024-08-18T15:17:16.829 DEBUG:teuthology.orchestra.run.smithi134:> sudo ceph --cluster c1 osd crush tunables default 2024-08-18T15:17:17.005 INFO:teuthology.orchestra.run.smithi134.stderr:adjusted tunables profile to default 2024-08-18T15:17:17.016 INFO:tasks.ceph:check_enable_crimson: False 2024-08-18T15:17:17.016 INFO:tasks.ceph:Starting osd daemons in cluster c1... 2024-08-18T15:17:17.017 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-08-18T15:17:17.017 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/var/lib/ceph/osd/c1-0/fsid of=/dev/stdout 2024-08-18T15:17:17.041 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-08-18T15:17:17.041 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/var/lib/ceph/osd/c1-1/fsid of=/dev/stdout 2024-08-18T15:17:17.106 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-08-18T15:17:17.107 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/var/lib/ceph/osd/c1-2/fsid of=/dev/stdout 2024-08-18T15:17:17.172 DEBUG:teuthology.orchestra.run.smithi134:> sudo ceph --cluster c1 osd new 6e8eac80-a6c5-4d97-bdca-81bea6a9bcdc 0 2024-08-18T15:17:17.204 INFO:tasks.ceph.c1.mgr.x.smithi134.stderr:2024-08-18T15:17:17.203+0000 7fef6f60d040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-08-18T15:17:17.374 INFO:tasks.ceph.c1.mgr.x.smithi134.stderr:2024-08-18T15:17:17.373+0000 7fef6f60d040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-08-18T15:17:17.383 INFO:teuthology.orchestra.run.smithi134.stdout:0 2024-08-18T15:17:17.394 DEBUG:teuthology.orchestra.run.smithi134:> sudo ceph --cluster c1 osd new 220ca1cb-cbb2-4902-94a4-79b08fb88bb5 1 2024-08-18T15:17:17.430 INFO:tasks.ceph.c1.mgr.x.smithi134.stderr:2024-08-18T15:17:17.430+0000 7fef6f60d040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-08-18T15:17:17.574 INFO:teuthology.orchestra.run.smithi134.stdout:1 2024-08-18T15:17:17.586 DEBUG:teuthology.orchestra.run.smithi134:> sudo ceph --cluster c1 osd new f2fced74-31e0-495f-95a0-c962ce80c8b0 2 2024-08-18T15:17:17.692 INFO:tasks.ceph.c1.mgr.x.smithi134.stderr:2024-08-18T15:17:17.692+0000 7fef6f60d040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-08-18T15:17:17.741 INFO:tasks.ceph.c1.mgr.x.smithi134.stderr:2024-08-18T15:17:17.740+0000 7fef6f60d040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-08-18T15:17:17.808 INFO:teuthology.orchestra.run.smithi134.stdout:2 2024-08-18T15:17:17.820 INFO:tasks.ceph.c1.osd.0:Restarting daemon 2024-08-18T15:17:17.820 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 0 2024-08-18T15:17:17.822 INFO:tasks.ceph.c1.osd.0:Started 2024-08-18T15:17:17.823 INFO:tasks.ceph.c1.osd.1:Restarting daemon 2024-08-18T15:17:17.823 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 1 2024-08-18T15:17:17.825 INFO:tasks.ceph.c1.osd.1:Started 2024-08-18T15:17:17.825 INFO:tasks.ceph.c1.osd.2:Restarting daemon 2024-08-18T15:17:17.825 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 2 2024-08-18T15:17:17.828 INFO:tasks.ceph.c1.osd.2:Started 2024-08-18T15:17:17.828 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2024-08-18T15:17:18.025 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:17:18.025 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":5,"fsid":"a11f7e7c-ddeb-4785-ae7a-25ca7716cfc1","created":"2024-08-18T15:17:16.928167+0000","modified":"2024-08-18T15:17:17.806884+0000","last_up_change":"0.000000","last_in_change":"2024-08-18T15:17:17.806884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6e8eac80-a6c5-4d97-bdca-81bea6a9bcdc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"220ca1cb-cbb2-4902-94a4-79b08fb88bb5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f2fced74-31e0-495f-95a0-c962ce80c8b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-18T15:17:18.036 INFO:tasks.ceph.ceph_manager.c1:[] 2024-08-18T15:17:18.037 INFO:tasks.ceph:Waiting for OSDs to come up 2024-08-18T15:17:18.339 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2024-08-18T15:17:18.440 INFO:tasks.ceph.c1.osd.0.smithi134.stderr:2024-08-18T15:17:18.439+0000 7fbc07e26640 -1 Falling back to public interface 2024-08-18T15:17:18.444 INFO:tasks.ceph.c1.osd.1.smithi134.stderr:2024-08-18T15:17:18.443+0000 7f58862ac640 -1 Falling back to public interface 2024-08-18T15:17:18.466 INFO:tasks.ceph.c1.osd.2.smithi134.stderr:2024-08-18T15:17:18.465+0000 7f720ff87640 -1 Falling back to public interface 2024-08-18T15:17:18.501 INFO:teuthology.misc.health.smithi134.stdout: 2024-08-18T15:17:18.501 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":5,"fsid":"a11f7e7c-ddeb-4785-ae7a-25ca7716cfc1","created":"2024-08-18T15:17:16.928167+0000","modified":"2024-08-18T15:17:17.806884+0000","last_up_change":"0.000000","last_in_change":"2024-08-18T15:17:17.806884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6e8eac80-a6c5-4d97-bdca-81bea6a9bcdc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"220ca1cb-cbb2-4902-94a4-79b08fb88bb5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f2fced74-31e0-495f-95a0-c962ce80c8b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-18T15:17:18.511 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-08-18T15:17:18.517 INFO:tasks.ceph.c1.mgr.x.smithi134.stderr:2024-08-18T15:17:18.516+0000 7fef6f60d040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-08-18T15:17:18.603 INFO:tasks.ceph.c1.mgr.x.smithi134.stderr:2024-08-18T15:17:18.602+0000 7fef6f60d040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-08-18T15:17:18.645 INFO:tasks.ceph.c1.mgr.x.smithi134.stderr:2024-08-18T15:17:18.644+0000 7fef6f60d040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-08-18T15:17:18.810 INFO:tasks.ceph.c1.mgr.x.smithi134.stderr:2024-08-18T15:17:18.809+0000 7fef6f60d040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-08-18T15:17:18.857 INFO:tasks.ceph.c1.mgr.x.smithi134.stderr:2024-08-18T15:17:18.856+0000 7fef6f60d040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-08-18T15:17:18.907 INFO:tasks.ceph.c1.mgr.x.smithi134.stderr:2024-08-18T15:17:18.907+0000 7fef6f60d040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-08-18T15:17:19.020 INFO:tasks.ceph.c1.mgr.x.smithi134.stderr:2024-08-18T15:17:19.020+0000 7fef6f60d040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-08-18T15:17:19.396 INFO:tasks.ceph.c1.mgr.x.smithi134.stderr:2024-08-18T15:17:19.395+0000 7fef6f60d040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-08-18T15:17:19.652 INFO:tasks.ceph.c1.mgr.x.smithi134.stderr:2024-08-18T15:17:19.651+0000 7fef6f60d040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-08-18T15:17:19.702 INFO:tasks.ceph.c1.mgr.x.smithi134.stderr:2024-08-18T15:17:19.701+0000 7fef6f60d040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-08-18T15:17:19.817 INFO:tasks.ceph.c1.mgr.x.smithi134.stderr:2024-08-18T15:17:19.816+0000 7fef6f60d040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-08-18T15:17:20.195 INFO:tasks.ceph.c1.mgr.x.smithi134.stderr:2024-08-18T15:17:20.194+0000 7fef6f60d040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-08-18T15:17:20.242 INFO:tasks.ceph.c1.mgr.x.smithi134.stderr:2024-08-18T15:17:20.241+0000 7fef6f60d040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-08-18T15:17:20.397 INFO:tasks.ceph.c1.mgr.x.smithi134.stderr:2024-08-18T15:17:20.397+0000 7fef6f60d040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-08-18T15:17:20.531 INFO:tasks.ceph.c1.mgr.x.smithi134.stderr:2024-08-18T15:17:20.530+0000 7fef6f60d040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-08-18T15:17:20.579 INFO:tasks.ceph.c1.mgr.x.smithi134.stderr:2024-08-18T15:17:20.578+0000 7fef6f60d040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-08-18T15:17:20.649 INFO:tasks.ceph.c1.mgr.x.smithi134.stderr:2024-08-18T15:17:20.649+0000 7fef6f60d040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-08-18T15:17:20.694 INFO:tasks.ceph.c1.mgr.x.smithi134.stderr:2024-08-18T15:17:20.694+0000 7fef6f60d040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-08-18T15:17:21.165 INFO:tasks.ceph.c1.mgr.x.smithi134.stderr:2024-08-18T15:17:21.164+0000 7fef6f60d040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-08-18T15:17:21.169 INFO:tasks.ceph.c1.osd.0.smithi134.stderr:2024-08-18T15:17:21.169+0000 7fbc07e26640 -1 osd.0 0 log_to_monitors true 2024-08-18T15:17:21.176 INFO:tasks.ceph.c1.osd.1.smithi134.stderr:2024-08-18T15:17:21.175+0000 7f58862ac640 -1 osd.1 0 log_to_monitors true 2024-08-18T15:17:21.195 INFO:tasks.ceph.c1.osd.2.smithi134.stderr:2024-08-18T15:17:21.194+0000 7f720ff87640 -1 osd.2 0 log_to_monitors true 2024-08-18T15:17:21.286 INFO:tasks.ceph.c1.mgr.x.smithi134.stderr:2024-08-18T15:17:21.285+0000 7fef6f60d040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-08-18T15:17:22.473 INFO:tasks.ceph.c1.mgr.x.smithi134.stderr:2024-08-18T15:17:22.473+0000 7fef10001640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-08-18T15:17:23.437 INFO:tasks.ceph.c1.osd.1.smithi134.stderr:2024-08-18T15:17:23.436+0000 7f5882239640 -1 osd.1 0 waiting for initial osdmap 2024-08-18T15:17:23.445 INFO:tasks.ceph.c1.osd.2.smithi134.stderr:2024-08-18T15:17:23.445+0000 7f720bf14640 -1 osd.2 0 waiting for initial osdmap 2024-08-18T15:17:23.454 INFO:tasks.ceph.c1.osd.1.smithi134.stderr:2024-08-18T15:17:23.454+0000 7f587a05a640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-18T15:17:23.457 INFO:tasks.ceph.c1.osd.2.smithi134.stderr:2024-08-18T15:17:23.457+0000 7f7203d35640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-18T15:17:23.463 INFO:tasks.ceph.c1.osd.0.smithi134.stderr:2024-08-18T15:17:23.463+0000 7fbc0380b640 -1 osd.0 0 waiting for initial osdmap 2024-08-18T15:17:23.474 INFO:tasks.ceph.c1.osd.0.smithi134.stderr:2024-08-18T15:17:23.474+0000 7fbbfb62c640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-18T15:17:24.814 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2024-08-18T15:17:25.096 INFO:teuthology.misc.health.smithi134.stdout: 2024-08-18T15:17:25.097 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":8,"fsid":"a11f7e7c-ddeb-4785-ae7a-25ca7716cfc1","created":"2024-08-18T15:17:16.928167+0000","modified":"2024-08-18T15:17:24.448999+0000","last_up_change":"2024-08-18T15:17:24.448999+0000","last_in_change":"2024-08-18T15:17:17.806884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6e8eac80-a6c5-4d97-bdca-81bea6a9bcdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6800","nonce":32960},{"type":"v1","addr":"172.21.15.134:6801","nonce":32960}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":32960},{"type":"v1","addr":"172.21.15.134:6803","nonce":32960}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":32960},{"type":"v1","addr":"172.21.15.134:6807","nonce":32960}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":32960},{"type":"v1","addr":"172.21.15.134:6805","nonce":32960}]},"public_addr":"172.21.15.134:6801/32960","cluster_addr":"172.21.15.134:6803/32960","heartbeat_back_addr":"172.21.15.134:6807/32960","heartbeat_front_addr":"172.21.15.134:6805/32960","state":["exists","up"]},{"osd":1,"uuid":"220ca1cb-cbb2-4902-94a4-79b08fb88bb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":32961},{"type":"v1","addr":"172.21.15.134:6809","nonce":32961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":32961},{"type":"v1","addr":"172.21.15.134:6811","nonce":32961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":32961},{"type":"v1","addr":"172.21.15.134:6815","nonce":32961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":32961},{"type":"v1","addr":"172.21.15.134:6813","nonce":32961}]},"public_addr":"172.21.15.134:6809/32961","cluster_addr":"172.21.15.134:6811/32961","heartbeat_back_addr":"172.21.15.134:6815/32961","heartbeat_front_addr":"172.21.15.134:6813/32961","state":["exists","up"]},{"osd":2,"uuid":"f2fced74-31e0-495f-95a0-c962ce80c8b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6816","nonce":32962},{"type":"v1","addr":"172.21.15.134:6817","nonce":32962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":32962},{"type":"v1","addr":"172.21.15.134:6819","nonce":32962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":32962},{"type":"v1","addr":"172.21.15.134:6823","nonce":32962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":32962},{"type":"v1","addr":"172.21.15.134:6821","nonce":32962}]},"public_addr":"172.21.15.134:6817/32962","cluster_addr":"172.21.15.134:6819/32962","heartbeat_back_addr":"172.21.15.134:6823/32962","heartbeat_front_addr":"172.21.15.134:6821/32962","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-18T15:17:25.108 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-08-18T15:17:25.108 INFO:tasks.ceph:Creating RBD pool 2024-08-18T15:17:25.108 DEBUG:teuthology.orchestra.run.smithi134:> sudo ceph --cluster c1 osd pool create rbd 8 2024-08-18T15:17:25.486 INFO:teuthology.orchestra.run.smithi134.stderr:pool 'rbd' created 2024-08-18T15:17:25.499 DEBUG:teuthology.orchestra.run.smithi134:> rbd --cluster c1 pool init rbd 2024-08-18T15:17:25.555 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:17:25.555 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:17:28.534 INFO:tasks.ceph:Starting mds daemons in cluster c1... 2024-08-18T15:17:28.534 INFO:tasks.ceph:Waiting until c1 daemons up and pgs clean... 2024-08-18T15:17:28.534 INFO:tasks.ceph.ceph_manager.c1:waiting for mgr available 2024-08-18T15:17:28.535 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 mgr dump --format=json 2024-08-18T15:17:28.535 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-08-18T15:17:28.869 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:17:28.879 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":5,"flags":0,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":32624},{"type":"v1","addr":"172.21.15.134:6825","nonce":32624}]},"active_addr":"172.21.15.134:6825/32624","active_change":"2024-08-18T15:17:21.287608+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.134:0","nonce":4093968244}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.134:0","nonce":492686216}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.134:0","nonce":4027884418}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.134:0","nonce":2760647730}]}]} 2024-08-18T15:17:28.881 INFO:tasks.ceph.ceph_manager.c1:mgr available! 2024-08-18T15:17:28.882 INFO:tasks.ceph.ceph_manager.c1:waiting for all up 2024-08-18T15:17:28.882 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2024-08-18T15:17:29.163 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:17:29.163 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":12,"fsid":"a11f7e7c-ddeb-4785-ae7a-25ca7716cfc1","created":"2024-08-18T15:17:16.928167+0000","modified":"2024-08-18T15:17:28.491804+0000","last_up_change":"2024-08-18T15:17:24.448999+0000","last_in_change":"2024-08-18T15:17:17.806884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-18T15:17:25.320778+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-08-18T15:17:25.408016+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":12,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6e8eac80-a6c5-4d97-bdca-81bea6a9bcdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6800","nonce":32960},{"type":"v1","addr":"172.21.15.134:6801","nonce":32960}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":32960},{"type":"v1","addr":"172.21.15.134:6803","nonce":32960}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":32960},{"type":"v1","addr":"172.21.15.134:6807","nonce":32960}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":32960},{"type":"v1","addr":"172.21.15.134:6805","nonce":32960}]},"public_addr":"172.21.15.134:6801/32960","cluster_addr":"172.21.15.134:6803/32960","heartbeat_back_addr":"172.21.15.134:6807/32960","heartbeat_front_addr":"172.21.15.134:6805/32960","state":["exists","up"]},{"osd":1,"uuid":"220ca1cb-cbb2-4902-94a4-79b08fb88bb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":32961},{"type":"v1","addr":"172.21.15.134:6809","nonce":32961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":32961},{"type":"v1","addr":"172.21.15.134:6811","nonce":32961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":32961},{"type":"v1","addr":"172.21.15.134:6815","nonce":32961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":32961},{"type":"v1","addr":"172.21.15.134:6813","nonce":32961}]},"public_addr":"172.21.15.134:6809/32961","cluster_addr":"172.21.15.134:6811/32961","heartbeat_back_addr":"172.21.15.134:6815/32961","heartbeat_front_addr":"172.21.15.134:6813/32961","state":["exists","up"]},{"osd":2,"uuid":"f2fced74-31e0-495f-95a0-c962ce80c8b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6816","nonce":32962},{"type":"v1","addr":"172.21.15.134:6817","nonce":32962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":32962},{"type":"v1","addr":"172.21.15.134:6819","nonce":32962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":32962},{"type":"v1","addr":"172.21.15.134:6823","nonce":32962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":32962},{"type":"v1","addr":"172.21.15.134:6821","nonce":32962}]},"public_addr":"172.21.15.134:6817/32962","cluster_addr":"172.21.15.134:6819/32962","heartbeat_back_addr":"172.21.15.134:6823/32962","heartbeat_front_addr":"172.21.15.134:6821/32962","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-18T15:17:22.127392+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-18T15:17:22.187421+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-18T15:17:22.176989+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-18T15:17:29.176 INFO:tasks.ceph.ceph_manager.c1:all up! 2024-08-18T15:17:29.177 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2024-08-18T15:17:29.457 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:17:29.458 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":12,"fsid":"a11f7e7c-ddeb-4785-ae7a-25ca7716cfc1","created":"2024-08-18T15:17:16.928167+0000","modified":"2024-08-18T15:17:28.491804+0000","last_up_change":"2024-08-18T15:17:24.448999+0000","last_in_change":"2024-08-18T15:17:17.806884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-18T15:17:25.320778+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-08-18T15:17:25.408016+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":12,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6e8eac80-a6c5-4d97-bdca-81bea6a9bcdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6800","nonce":32960},{"type":"v1","addr":"172.21.15.134:6801","nonce":32960}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":32960},{"type":"v1","addr":"172.21.15.134:6803","nonce":32960}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":32960},{"type":"v1","addr":"172.21.15.134:6807","nonce":32960}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":32960},{"type":"v1","addr":"172.21.15.134:6805","nonce":32960}]},"public_addr":"172.21.15.134:6801/32960","cluster_addr":"172.21.15.134:6803/32960","heartbeat_back_addr":"172.21.15.134:6807/32960","heartbeat_front_addr":"172.21.15.134:6805/32960","state":["exists","up"]},{"osd":1,"uuid":"220ca1cb-cbb2-4902-94a4-79b08fb88bb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":32961},{"type":"v1","addr":"172.21.15.134:6809","nonce":32961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":32961},{"type":"v1","addr":"172.21.15.134:6811","nonce":32961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":32961},{"type":"v1","addr":"172.21.15.134:6815","nonce":32961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":32961},{"type":"v1","addr":"172.21.15.134:6813","nonce":32961}]},"public_addr":"172.21.15.134:6809/32961","cluster_addr":"172.21.15.134:6811/32961","heartbeat_back_addr":"172.21.15.134:6815/32961","heartbeat_front_addr":"172.21.15.134:6813/32961","state":["exists","up"]},{"osd":2,"uuid":"f2fced74-31e0-495f-95a0-c962ce80c8b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6816","nonce":32962},{"type":"v1","addr":"172.21.15.134:6817","nonce":32962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":32962},{"type":"v1","addr":"172.21.15.134:6819","nonce":32962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":32962},{"type":"v1","addr":"172.21.15.134:6823","nonce":32962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":32962},{"type":"v1","addr":"172.21.15.134:6821","nonce":32962}]},"public_addr":"172.21.15.134:6817/32962","cluster_addr":"172.21.15.134:6819/32962","heartbeat_back_addr":"172.21.15.134:6823/32962","heartbeat_front_addr":"172.21.15.134:6821/32962","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-18T15:17:22.127392+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-18T15:17:22.187421+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-18T15:17:22.176989+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-18T15:17:29.469 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.0 flush_pg_stats 2024-08-18T15:17:29.470 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.1 flush_pg_stats 2024-08-18T15:17:29.470 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.2 flush_pg_stats 2024-08-18T15:17:29.606 INFO:teuthology.orchestra.run.smithi134.stdout:34359738372 2024-08-18T15:17:29.607 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.1 2024-08-18T15:17:29.626 INFO:teuthology.orchestra.run.smithi134.stdout:34359738373 2024-08-18T15:17:29.626 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2024-08-18T15:17:29.627 INFO:teuthology.orchestra.run.smithi134.stdout:34359738372 2024-08-18T15:17:29.627 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.2 2024-08-18T15:17:29.910 INFO:teuthology.orchestra.run.smithi134.stdout:34359738370 2024-08-18T15:17:29.923 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738372 got 34359738370 for osd.1 2024-08-18T15:17:29.961 INFO:teuthology.orchestra.run.smithi134.stdout:34359738370 2024-08-18T15:17:29.967 INFO:teuthology.orchestra.run.smithi134.stdout:34359738371 2024-08-18T15:17:29.973 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738372 got 34359738370 for osd.2 2024-08-18T15:17:29.979 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738373 got 34359738371 for osd.0 2024-08-18T15:17:30.924 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.1 2024-08-18T15:17:30.974 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.2 2024-08-18T15:17:30.980 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2024-08-18T15:17:31.297 INFO:teuthology.orchestra.run.smithi134.stdout:34359738370 2024-08-18T15:17:31.311 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738372 got 34359738370 for osd.2 2024-08-18T15:17:31.319 INFO:teuthology.orchestra.run.smithi134.stdout:34359738370 2024-08-18T15:17:31.332 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738372 got 34359738370 for osd.1 2024-08-18T15:17:31.372 INFO:teuthology.orchestra.run.smithi134.stdout:34359738371 2024-08-18T15:17:31.385 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738373 got 34359738371 for osd.0 2024-08-18T15:17:32.311 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.2 2024-08-18T15:17:32.333 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.1 2024-08-18T15:17:32.386 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2024-08-18T15:17:32.623 INFO:teuthology.orchestra.run.smithi134.stdout:34359738372 2024-08-18T15:17:32.637 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738372 got 34359738372 for osd.2 2024-08-18T15:17:32.637 DEBUG:teuthology.parallel:result is None 2024-08-18T15:17:32.657 INFO:teuthology.orchestra.run.smithi134.stdout:34359738372 2024-08-18T15:17:32.670 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738372 got 34359738372 for osd.1 2024-08-18T15:17:32.670 DEBUG:teuthology.parallel:result is None 2024-08-18T15:17:32.696 INFO:teuthology.orchestra.run.smithi134.stdout:34359738373 2024-08-18T15:17:32.709 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738373 got 34359738373 for osd.0 2024-08-18T15:17:32.709 DEBUG:teuthology.parallel:result is None 2024-08-18T15:17:32.710 INFO:tasks.ceph.ceph_manager.c1:waiting for clean 2024-08-18T15:17:32.710 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2024-08-18T15:17:33.027 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:17:33.027 INFO:teuthology.orchestra.run.smithi134.stderr:dumped all 2024-08-18T15:17:33.039 INFO:teuthology.orchestra.run.smithi134.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2024-08-18T15:17:31.909858+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":82556,"kb_used_data":1480,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":314490244,"statfs":{"total":322122547200,"available":322038009856,"internally_reserved":0,"allocated":1515520,"data_stored":1270744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4782,"internal_metadata":82963794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.614982"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-08-18T15:17:28.508096+0000","last_change":"2024-08-18T15:17:28.508096+0000","last_active":"2024-08-18T15:17:28.508096+0000","last_peered":"2024-08-18T15:17:28.508096+0000","last_clean":"2024-08-18T15:17:28.508096+0000","last_became_active":"2024-08-18T15:17:26.492543+0000","last_became_peered":"2024-08-18T15:17:26.492543+0000","last_unstale":"2024-08-18T15:17:28.508096+0000","last_undegraded":"2024-08-18T15:17:28.508096+0000","last_fullsized":"2024-08-18T15:17:28.508096+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-18T15:17:25.482110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-18T15:17:25.482110+0000","last_clean_scrub_stamp":"2024-08-18T15:17:25.482110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T00:31:28.306522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071700300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-08-18T15:17:28.507963+0000","last_change":"2024-08-18T15:17:28.507963+0000","last_active":"2024-08-18T15:17:28.507963+0000","last_peered":"2024-08-18T15:17:28.507963+0000","last_clean":"2024-08-18T15:17:28.507963+0000","last_became_active":"2024-08-18T15:17:26.492353+0000","last_became_peered":"2024-08-18T15:17:26.492353+0000","last_unstale":"2024-08-18T15:17:28.507963+0000","last_undegraded":"2024-08-18T15:17:28.507963+0000","last_fullsized":"2024-08-18T15:17:28.507963+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-18T15:17:25.482110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-18T15:17:25.482110+0000","last_clean_scrub_stamp":"2024-08-18T15:17:25.482110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T01:40:07.670501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081034599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-08-18T15:17:28.507583+0000","last_change":"2024-08-18T15:17:28.507583+0000","last_active":"2024-08-18T15:17:28.507583+0000","last_peered":"2024-08-18T15:17:28.507583+0000","last_clean":"2024-08-18T15:17:28.507583+0000","last_became_active":"2024-08-18T15:17:26.493929+0000","last_became_peered":"2024-08-18T15:17:26.493929+0000","last_unstale":"2024-08-18T15:17:28.507583+0000","last_undegraded":"2024-08-18T15:17:28.507583+0000","last_fullsized":"2024-08-18T15:17:28.507583+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-18T15:17:25.482110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-18T15:17:25.482110+0000","last_clean_scrub_stamp":"2024-08-18T15:17:25.482110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T15:46:52.738105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045364100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-08-18T15:17:28.507454+0000","last_change":"2024-08-18T15:17:28.507454+0000","last_active":"2024-08-18T15:17:28.507454+0000","last_peered":"2024-08-18T15:17:28.507454+0000","last_clean":"2024-08-18T15:17:28.507454+0000","last_became_active":"2024-08-18T15:17:26.493986+0000","last_became_peered":"2024-08-18T15:17:26.493986+0000","last_unstale":"2024-08-18T15:17:28.507454+0000","last_undegraded":"2024-08-18T15:17:28.507454+0000","last_fullsized":"2024-08-18T15:17:28.507454+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-18T15:17:25.482110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-18T15:17:25.482110+0000","last_clean_scrub_stamp":"2024-08-18T15:17:25.482110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T01:25:56.744096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034472300000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"12'2","reported_seq":17,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-08-18T15:17:28.506718+0000","last_change":"2024-08-18T15:17:28.505954+0000","last_active":"2024-08-18T15:17:28.506718+0000","last_peered":"2024-08-18T15:17:28.506718+0000","last_clean":"2024-08-18T15:17:28.506718+0000","last_became_active":"2024-08-18T15:17:26.491468+0000","last_became_peered":"2024-08-18T15:17:26.491468+0000","last_unstale":"2024-08-18T15:17:28.506718+0000","last_undegraded":"2024-08-18T15:17:28.506718+0000","last_fullsized":"2024-08-18T15:17:28.506718+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-18T15:17:25.482110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-18T15:17:25.482110+0000","last_clean_scrub_stamp":"2024-08-18T15:17:25.482110+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T19:52:29.924059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074101199999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-08-18T15:17:28.503626+0000","last_change":"2024-08-18T15:17:28.503626+0000","last_active":"2024-08-18T15:17:28.503626+0000","last_peered":"2024-08-18T15:17:28.503626+0000","last_clean":"2024-08-18T15:17:28.503626+0000","last_became_active":"2024-08-18T15:17:26.495283+0000","last_became_peered":"2024-08-18T15:17:26.495283+0000","last_unstale":"2024-08-18T15:17:28.503626+0000","last_undegraded":"2024-08-18T15:17:28.503626+0000","last_fullsized":"2024-08-18T15:17:28.503626+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-18T15:17:25.482110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-18T15:17:25.482110+0000","last_clean_scrub_stamp":"2024-08-18T15:17:25.482110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T19:52:29.924059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041338200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-08-18T15:17:28.503625+0000","last_change":"2024-08-18T15:17:28.503625+0000","last_active":"2024-08-18T15:17:28.503625+0000","last_peered":"2024-08-18T15:17:28.503625+0000","last_clean":"2024-08-18T15:17:28.503625+0000","last_became_active":"2024-08-18T15:17:26.492277+0000","last_became_peered":"2024-08-18T15:17:26.492277+0000","last_unstale":"2024-08-18T15:17:28.503625+0000","last_undegraded":"2024-08-18T15:17:28.503625+0000","last_fullsized":"2024-08-18T15:17:28.503625+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-18T15:17:25.482110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-18T15:17:25.482110+0000","last_clean_scrub_stamp":"2024-08-18T15:17:25.482110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T17:38:41.467723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037315200000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"10'1","reported_seq":16,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-08-18T15:17:28.507757+0000","last_change":"2024-08-18T15:17:28.507757+0000","last_active":"2024-08-18T15:17:28.507757+0000","last_peered":"2024-08-18T15:17:28.507757+0000","last_clean":"2024-08-18T15:17:28.507757+0000","last_became_active":"2024-08-18T15:17:26.493564+0000","last_became_peered":"2024-08-18T15:17:26.493564+0000","last_unstale":"2024-08-18T15:17:28.507757+0000","last_undegraded":"2024-08-18T15:17:28.507757+0000","last_fullsized":"2024-08-18T15:17:28.507757+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-18T15:17:25.482110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-18T15:17:25.482110+0000","last_clean_scrub_stamp":"2024-08-18T15:17:25.482110+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T23:43:35.734151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055015200000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"10'97","reported_seq":138,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-08-18T15:17:28.507143+0000","last_change":"2024-08-18T15:17:26.492706+0000","last_active":"2024-08-18T15:17:28.507143+0000","last_peered":"2024-08-18T15:17:28.507143+0000","last_clean":"2024-08-18T15:17:28.507143+0000","last_became_active":"2024-08-18T15:17:26.492423+0000","last_became_peered":"2024-08-18T15:17:26.492423+0000","last_unstale":"2024-08-18T15:17:28.507143+0000","last_undegraded":"2024-08-18T15:17:28.507143+0000","last_fullsized":"2024-08-18T15:17:28.507143+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-18T15:17:25.482110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-18T15:17:25.482110+0000","last_clean_scrub_stamp":"2024-08-18T15:17:25.482110+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T15:33:06.541239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27132,"kb_used_data":104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830468,"statfs":{"total":107374182400,"available":107346399232,"internally_reserved":0,"allocated":106496,"data_stored":29990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.314}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41699999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27712,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829888,"statfs":{"total":107374182400,"available":107345805312,"internally_reserved":0,"allocated":704512,"data_stored":620377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32400000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36199999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27712,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829888,"statfs":{"total":107374182400,"available":107345805312,"internally_reserved":0,"allocated":704512,"data_stored":620377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-18T15:17:33.040 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2024-08-18T15:17:33.320 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:17:33.320 INFO:teuthology.orchestra.run.smithi134.stderr:dumped all 2024-08-18T15:17:33.334 INFO:teuthology.orchestra.run.smithi134.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2024-08-18T15:17:33.295122+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":82556,"kb_used_data":1480,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":314490244,"statfs":{"total":322122547200,"available":322038009856,"internally_reserved":0,"allocated":1515520,"data_stored":1270744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4782,"internal_metadata":82963794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000246"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-08-18T15:17:28.508096+0000","last_change":"2024-08-18T15:17:28.508096+0000","last_active":"2024-08-18T15:17:28.508096+0000","last_peered":"2024-08-18T15:17:28.508096+0000","last_clean":"2024-08-18T15:17:28.508096+0000","last_became_active":"2024-08-18T15:17:26.492543+0000","last_became_peered":"2024-08-18T15:17:26.492543+0000","last_unstale":"2024-08-18T15:17:28.508096+0000","last_undegraded":"2024-08-18T15:17:28.508096+0000","last_fullsized":"2024-08-18T15:17:28.508096+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-18T15:17:25.482110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-18T15:17:25.482110+0000","last_clean_scrub_stamp":"2024-08-18T15:17:25.482110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T00:31:28.306522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071700300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-08-18T15:17:28.507963+0000","last_change":"2024-08-18T15:17:28.507963+0000","last_active":"2024-08-18T15:17:28.507963+0000","last_peered":"2024-08-18T15:17:28.507963+0000","last_clean":"2024-08-18T15:17:28.507963+0000","last_became_active":"2024-08-18T15:17:26.492353+0000","last_became_peered":"2024-08-18T15:17:26.492353+0000","last_unstale":"2024-08-18T15:17:28.507963+0000","last_undegraded":"2024-08-18T15:17:28.507963+0000","last_fullsized":"2024-08-18T15:17:28.507963+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-18T15:17:25.482110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-18T15:17:25.482110+0000","last_clean_scrub_stamp":"2024-08-18T15:17:25.482110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T01:40:07.670501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081034599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-08-18T15:17:28.507583+0000","last_change":"2024-08-18T15:17:28.507583+0000","last_active":"2024-08-18T15:17:28.507583+0000","last_peered":"2024-08-18T15:17:28.507583+0000","last_clean":"2024-08-18T15:17:28.507583+0000","last_became_active":"2024-08-18T15:17:26.493929+0000","last_became_peered":"2024-08-18T15:17:26.493929+0000","last_unstale":"2024-08-18T15:17:28.507583+0000","last_undegraded":"2024-08-18T15:17:28.507583+0000","last_fullsized":"2024-08-18T15:17:28.507583+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-18T15:17:25.482110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-18T15:17:25.482110+0000","last_clean_scrub_stamp":"2024-08-18T15:17:25.482110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T15:46:52.738105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045364100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-08-18T15:17:28.507454+0000","last_change":"2024-08-18T15:17:28.507454+0000","last_active":"2024-08-18T15:17:28.507454+0000","last_peered":"2024-08-18T15:17:28.507454+0000","last_clean":"2024-08-18T15:17:28.507454+0000","last_became_active":"2024-08-18T15:17:26.493986+0000","last_became_peered":"2024-08-18T15:17:26.493986+0000","last_unstale":"2024-08-18T15:17:28.507454+0000","last_undegraded":"2024-08-18T15:17:28.507454+0000","last_fullsized":"2024-08-18T15:17:28.507454+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-18T15:17:25.482110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-18T15:17:25.482110+0000","last_clean_scrub_stamp":"2024-08-18T15:17:25.482110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T01:25:56.744096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034472300000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"12'2","reported_seq":17,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-08-18T15:17:28.506718+0000","last_change":"2024-08-18T15:17:28.505954+0000","last_active":"2024-08-18T15:17:28.506718+0000","last_peered":"2024-08-18T15:17:28.506718+0000","last_clean":"2024-08-18T15:17:28.506718+0000","last_became_active":"2024-08-18T15:17:26.491468+0000","last_became_peered":"2024-08-18T15:17:26.491468+0000","last_unstale":"2024-08-18T15:17:28.506718+0000","last_undegraded":"2024-08-18T15:17:28.506718+0000","last_fullsized":"2024-08-18T15:17:28.506718+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-18T15:17:25.482110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-18T15:17:25.482110+0000","last_clean_scrub_stamp":"2024-08-18T15:17:25.482110+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T19:52:29.924059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074101199999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-08-18T15:17:28.503626+0000","last_change":"2024-08-18T15:17:28.503626+0000","last_active":"2024-08-18T15:17:28.503626+0000","last_peered":"2024-08-18T15:17:28.503626+0000","last_clean":"2024-08-18T15:17:28.503626+0000","last_became_active":"2024-08-18T15:17:26.495283+0000","last_became_peered":"2024-08-18T15:17:26.495283+0000","last_unstale":"2024-08-18T15:17:28.503626+0000","last_undegraded":"2024-08-18T15:17:28.503626+0000","last_fullsized":"2024-08-18T15:17:28.503626+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-18T15:17:25.482110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-18T15:17:25.482110+0000","last_clean_scrub_stamp":"2024-08-18T15:17:25.482110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T19:52:29.924059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041338200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-08-18T15:17:28.503625+0000","last_change":"2024-08-18T15:17:28.503625+0000","last_active":"2024-08-18T15:17:28.503625+0000","last_peered":"2024-08-18T15:17:28.503625+0000","last_clean":"2024-08-18T15:17:28.503625+0000","last_became_active":"2024-08-18T15:17:26.492277+0000","last_became_peered":"2024-08-18T15:17:26.492277+0000","last_unstale":"2024-08-18T15:17:28.503625+0000","last_undegraded":"2024-08-18T15:17:28.503625+0000","last_fullsized":"2024-08-18T15:17:28.503625+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-18T15:17:25.482110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-18T15:17:25.482110+0000","last_clean_scrub_stamp":"2024-08-18T15:17:25.482110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T17:38:41.467723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037315200000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"10'1","reported_seq":16,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-08-18T15:17:28.507757+0000","last_change":"2024-08-18T15:17:28.507757+0000","last_active":"2024-08-18T15:17:28.507757+0000","last_peered":"2024-08-18T15:17:28.507757+0000","last_clean":"2024-08-18T15:17:28.507757+0000","last_became_active":"2024-08-18T15:17:26.493564+0000","last_became_peered":"2024-08-18T15:17:26.493564+0000","last_unstale":"2024-08-18T15:17:28.507757+0000","last_undegraded":"2024-08-18T15:17:28.507757+0000","last_fullsized":"2024-08-18T15:17:28.507757+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-18T15:17:25.482110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-18T15:17:25.482110+0000","last_clean_scrub_stamp":"2024-08-18T15:17:25.482110+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T23:43:35.734151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055015200000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"10'97","reported_seq":138,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-08-18T15:17:28.507143+0000","last_change":"2024-08-18T15:17:26.492706+0000","last_active":"2024-08-18T15:17:28.507143+0000","last_peered":"2024-08-18T15:17:28.507143+0000","last_clean":"2024-08-18T15:17:28.507143+0000","last_became_active":"2024-08-18T15:17:26.492423+0000","last_became_peered":"2024-08-18T15:17:26.492423+0000","last_unstale":"2024-08-18T15:17:28.507143+0000","last_undegraded":"2024-08-18T15:17:28.507143+0000","last_fullsized":"2024-08-18T15:17:28.507143+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-18T15:17:25.482110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-18T15:17:25.482110+0000","last_clean_scrub_stamp":"2024-08-18T15:17:25.482110+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T15:33:06.541239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27132,"kb_used_data":104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830468,"statfs":{"total":107374182400,"available":107346399232,"internally_reserved":0,"allocated":106496,"data_stored":29990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.314}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41699999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27712,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829888,"statfs":{"total":107374182400,"available":107345805312,"internally_reserved":0,"allocated":704512,"data_stored":620377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32400000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36199999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27712,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829888,"statfs":{"total":107374182400,"available":107345805312,"internally_reserved":0,"allocated":704512,"data_stored":620377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-18T15:17:33.335 INFO:tasks.ceph.ceph_manager.c1:clean! 2024-08-18T15:17:33.335 INFO:tasks.ceph:Waiting until ceph cluster c1 is healthy... 2024-08-18T15:17:33.335 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy 2024-08-18T15:17:33.335 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 health --format=json 2024-08-18T15:17:33.652 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:17:33.652 INFO:teuthology.orchestra.run.smithi134.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-08-18T15:17:33.664 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy done 2024-08-18T15:17:33.664 INFO:teuthology.run_tasks:Running task ceph... 2024-08-18T15:17:33.673 INFO:tasks.ceph:Creating ceph cluster c2... 2024-08-18T15:17:33.674 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'cpu_profile': set(), 'cluster': 'c2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-08-18T15:17:33.674 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-08-16_22:40:05-rgw-reef-distro-default-smithi/7858171', 'branch': 'reef', 'description': 'rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zones supported-random-distro$/{centos_latest} tasks/test_multi}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7858171', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'teuthology-2024-08-16_22:40:05-rgw-reef-distro-default-smithi', 'no_nested_subset': False, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'reef'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': '03f4c3e85ce172201896147a7aef87ba256187f2', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '03f4c3e85ce172201896147a7aef87ba256187f2'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'enabled_features': ['resharding'], 'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zonegroup', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zone1'}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'test-zone2'}, {'endpoints': ['c1.client.1'], 'name': 'test-zone3'}]}]}, 'rgw-multisite-tests': {'args': ['tests.py']}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'workunit': {'branch': 'reef', 'sha1': '03f4c3e85ce172201896147a7aef87ba256187f2'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 930, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'seed': 4794, 'sha1': '03f4c3e85ce172201896147a7aef87ba256187f2', 'sleep_before_teardown': 0, 'subset': '0/1', 'suite': 'rgw', 'suite_branch': 'reef', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_03f4c3e85ce172201896147a7aef87ba256187f2/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': '03f4c3e85ce172201896147a7aef87ba256187f2', 'targets': {'smithi134.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIE9XgycsFVMPLcbi8grvCs2X7jnzF+W6OvDtQvdPSikD8DSyZ6GPlvctMpS+baiu/U3LWz8MtsURSjI/icSDh0=', 'smithi193.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCC+k2hQSoI+gvHGxnhhbNPw+ovFp8VjwRVMU4OOFllP2RKhpxQhCB1OIlvSYy7oNCv6WGBH8zahtKKyQE/oYyQ='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': '03f4c3e85ce172201896147a7aef87ba256187f2', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2', 'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': '03f4c3e85ce172201896147a7aef87ba256187f2', 'wait-for-scrub': False}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '67db276914d29505c9707d04f3a3b8fb9d8aee4b', 'timestamp': '2024-08-16_22:40:05', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-08-18T15:17:33.674 DEBUG:teuthology.orchestra.run.smithi134:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2024-08-18T15:17:33.677 DEBUG:teuthology.orchestra.run.smithi193:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2024-08-18T15:17:33.696 DEBUG:teuthology.orchestra.run.smithi134:> sudo install -d -m0777 -- /var/run/ceph 2024-08-18T15:17:33.733 DEBUG:teuthology.orchestra.run.smithi193:> sudo install -d -m0777 -- /var/run/ceph 2024-08-18T15:17:33.766 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-08-18T15:17:33.766 DEBUG:teuthology.orchestra.run.smithi193:> dd if=/scratch_devs of=/dev/stdout 2024-08-18T15:17:33.825 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-08-18T15:17:33.826 DEBUG:teuthology.orchestra.run.smithi193:> stat /dev/vg_nvme/lv_1 2024-08-18T15:17:33.883 INFO:teuthology.orchestra.run.smithi193.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-08-18T15:17:33.883 INFO:teuthology.orchestra.run.smithi193.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-18T15:17:33.883 INFO:teuthology.orchestra.run.smithi193.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-08-18T15:17:33.883 INFO:teuthology.orchestra.run.smithi193.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-18T15:17:33.883 INFO:teuthology.orchestra.run.smithi193.stdout:Context: system_u:object_r:device_t:s0 2024-08-18T15:17:33.883 INFO:teuthology.orchestra.run.smithi193.stdout:Access: 2024-08-18 15:17:00.429064862 +0000 2024-08-18T15:17:33.883 INFO:teuthology.orchestra.run.smithi193.stdout:Modify: 2024-08-18 15:14:09.610721057 +0000 2024-08-18T15:17:33.883 INFO:teuthology.orchestra.run.smithi193.stdout:Change: 2024-08-18 15:14:09.610721057 +0000 2024-08-18T15:17:33.883 INFO:teuthology.orchestra.run.smithi193.stdout: Birth: 2024-08-18 15:14:09.610721057 +0000 2024-08-18T15:17:33.884 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-08-18T15:17:33.949 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records in 2024-08-18T15:17:33.949 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records out 2024-08-18T15:17:33.949 INFO:teuthology.orchestra.run.smithi193.stderr:512 bytes copied, 0.000216908 s, 2.4 MB/s 2024-08-18T15:17:33.950 DEBUG:teuthology.orchestra.run.smithi193:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-08-18T15:17:34.008 DEBUG:teuthology.orchestra.run.smithi193:> stat /dev/vg_nvme/lv_2 2024-08-18T15:17:34.065 INFO:teuthology.orchestra.run.smithi193.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-08-18T15:17:34.102 INFO:teuthology.orchestra.run.smithi193.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-18T15:17:34.102 INFO:teuthology.orchestra.run.smithi193.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-08-18T15:17:34.102 INFO:teuthology.orchestra.run.smithi193.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-18T15:17:34.102 INFO:teuthology.orchestra.run.smithi193.stdout:Context: system_u:object_r:device_t:s0 2024-08-18T15:17:34.102 INFO:teuthology.orchestra.run.smithi193.stdout:Access: 2024-08-18 15:17:00.429064862 +0000 2024-08-18T15:17:34.102 INFO:teuthology.orchestra.run.smithi193.stdout:Modify: 2024-08-18 15:14:09.939716009 +0000 2024-08-18T15:17:34.102 INFO:teuthology.orchestra.run.smithi193.stdout:Change: 2024-08-18 15:14:09.939716009 +0000 2024-08-18T15:17:34.102 INFO:teuthology.orchestra.run.smithi193.stdout: Birth: 2024-08-18 15:14:09.939716009 +0000 2024-08-18T15:17:34.103 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-08-18T15:17:34.131 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records in 2024-08-18T15:17:34.132 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records out 2024-08-18T15:17:34.132 INFO:teuthology.orchestra.run.smithi193.stderr:512 bytes copied, 0.000233365 s, 2.2 MB/s 2024-08-18T15:17:34.133 DEBUG:teuthology.orchestra.run.smithi193:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-08-18T15:17:34.190 DEBUG:teuthology.orchestra.run.smithi193:> stat /dev/vg_nvme/lv_3 2024-08-18T15:17:34.246 INFO:teuthology.orchestra.run.smithi193.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-08-18T15:17:34.246 INFO:teuthology.orchestra.run.smithi193.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-18T15:17:34.246 INFO:teuthology.orchestra.run.smithi193.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-08-18T15:17:34.246 INFO:teuthology.orchestra.run.smithi193.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-18T15:17:34.246 INFO:teuthology.orchestra.run.smithi193.stdout:Context: system_u:object_r:device_t:s0 2024-08-18T15:17:34.246 INFO:teuthology.orchestra.run.smithi193.stdout:Access: 2024-08-18 15:17:00.430064847 +0000 2024-08-18T15:17:34.246 INFO:teuthology.orchestra.run.smithi193.stdout:Modify: 2024-08-18 15:14:10.261711067 +0000 2024-08-18T15:17:34.246 INFO:teuthology.orchestra.run.smithi193.stdout:Change: 2024-08-18 15:14:10.261711067 +0000 2024-08-18T15:17:34.246 INFO:teuthology.orchestra.run.smithi193.stdout: Birth: 2024-08-18 15:14:10.261711067 +0000 2024-08-18T15:17:34.247 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-08-18T15:17:34.310 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records in 2024-08-18T15:17:34.310 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records out 2024-08-18T15:17:34.310 INFO:teuthology.orchestra.run.smithi193.stderr:512 bytes copied, 0.000222045 s, 2.3 MB/s 2024-08-18T15:17:34.311 DEBUG:teuthology.orchestra.run.smithi193:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-08-18T15:17:34.368 DEBUG:teuthology.orchestra.run.smithi193:> stat /dev/vg_nvme/lv_4 2024-08-18T15:17:34.424 INFO:teuthology.orchestra.run.smithi193.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-08-18T15:17:34.424 INFO:teuthology.orchestra.run.smithi193.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-18T15:17:34.424 INFO:teuthology.orchestra.run.smithi193.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-08-18T15:17:34.424 INFO:teuthology.orchestra.run.smithi193.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-18T15:17:34.424 INFO:teuthology.orchestra.run.smithi193.stdout:Context: system_u:object_r:device_t:s0 2024-08-18T15:17:34.424 INFO:teuthology.orchestra.run.smithi193.stdout:Access: 2024-08-18 15:17:00.430064847 +0000 2024-08-18T15:17:34.424 INFO:teuthology.orchestra.run.smithi193.stdout:Modify: 2024-08-18 15:14:10.580706173 +0000 2024-08-18T15:17:34.425 INFO:teuthology.orchestra.run.smithi193.stdout:Change: 2024-08-18 15:14:10.580706173 +0000 2024-08-18T15:17:34.425 INFO:teuthology.orchestra.run.smithi193.stdout: Birth: 2024-08-18 15:14:10.580706173 +0000 2024-08-18T15:17:34.425 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-08-18T15:17:34.488 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records in 2024-08-18T15:17:34.489 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records out 2024-08-18T15:17:34.489 INFO:teuthology.orchestra.run.smithi193.stderr:512 bytes copied, 0.0002032 s, 2.5 MB/s 2024-08-18T15:17:34.490 DEBUG:teuthology.orchestra.run.smithi193:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-08-18T15:17:34.548 INFO:tasks.ceph:osd dev map: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2024-08-18T15:17:34.548 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi193.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2024-08-18T15:17:34.548 INFO:tasks.ceph:Generating config... 2024-08-18T15:17:34.549 INFO:tasks.ceph:[client] debug rgw = 20 2024-08-18T15:17:34.549 INFO:tasks.ceph:[client] debug rgw sync = 20 2024-08-18T15:17:34.549 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2024-08-18T15:17:34.549 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2024-08-18T15:17:34.549 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2024-08-18T15:17:34.550 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2024-08-18T15:17:34.550 INFO:tasks.ceph:[client] rgw data log num shards = 4 2024-08-18T15:17:34.550 INFO:tasks.ceph:[client] rgw md log max shards = 4 2024-08-18T15:17:34.550 INFO:tasks.ceph:[client] rgw sync data inject err probability = 0.1 2024-08-18T15:17:34.550 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2024-08-18T15:17:34.550 INFO:tasks.ceph:[client] rgw sync meta inject err probability = 0.1 2024-08-18T15:17:34.550 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2024-08-18T15:17:34.550 INFO:tasks.ceph:[client] setgroup = ceph 2024-08-18T15:17:34.550 INFO:tasks.ceph:[client] setuser = ceph 2024-08-18T15:17:34.550 INFO:tasks.ceph:[client.0] rgw data notify interval msec = 0 2024-08-18T15:17:34.550 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-08-18T15:17:34.550 INFO:tasks.ceph:[mgr] debug ms = 1 2024-08-18T15:17:34.550 INFO:tasks.ceph:[mon] debug mon = 20 2024-08-18T15:17:34.551 INFO:tasks.ceph:[mon] debug ms = 1 2024-08-18T15:17:34.551 INFO:tasks.ceph:[mon] debug paxos = 20 2024-08-18T15:17:34.551 INFO:tasks.ceph:[osd] debug ms = 1 2024-08-18T15:17:34.551 INFO:tasks.ceph:[osd] debug osd = 20 2024-08-18T15:17:34.551 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2024-08-18T15:17:34.551 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2024-08-18T15:17:34.551 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2024-08-18T15:17:34.551 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2024-08-18T15:17:34.551 INFO:tasks.ceph:Setting up c2.mon.a... 2024-08-18T15:17:34.551 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c2.keyring 2024-08-18T15:17:34.634 INFO:teuthology.orchestra.run.smithi193.stdout:creating /etc/ceph/c2.keyring 2024-08-18T15:17:34.637 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c2.keyring 2024-08-18T15:17:34.680 DEBUG:teuthology.orchestra.run.smithi193:> sudo chmod 0644 /etc/ceph/c2.keyring 2024-08-18T15:17:34.745 DEBUG:tasks.ceph:Ceph mon addresses: [('c2.mon.a', '172.21.15.193')] 2024-08-18T15:17:34.746 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.193'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'client.0': {'rgw data notify interval msec': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2024-08-18T15:17:34.746 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-08-18T15:17:34.746 DEBUG:teuthology.orchestra.run.smithi193:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-08-18T15:17:34.802 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.193 --print /home/ubuntu/cephtest/c2.monmap 2024-08-18T15:17:34.880 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:17:34.880 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:17:34.880 INFO:teuthology.orchestra.run.smithi193.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c2.monmap 2024-08-18T15:17:34.881 INFO:teuthology.orchestra.run.smithi193.stdout:monmaptool: generated fsid af7271d7-3f51-4045-9c51-62f690689cce 2024-08-18T15:17:34.881 INFO:teuthology.orchestra.run.smithi193.stdout:setting min_mon_release = pacific 2024-08-18T15:17:34.881 INFO:teuthology.orchestra.run.smithi193.stdout:epoch 0 2024-08-18T15:17:34.881 INFO:teuthology.orchestra.run.smithi193.stdout:fsid af7271d7-3f51-4045-9c51-62f690689cce 2024-08-18T15:17:34.881 INFO:teuthology.orchestra.run.smithi193.stdout:last_changed 2024-08-18T15:17:34.881107+0000 2024-08-18T15:17:34.881 INFO:teuthology.orchestra.run.smithi193.stdout:created 2024-08-18T15:17:34.881107+0000 2024-08-18T15:17:34.881 INFO:teuthology.orchestra.run.smithi193.stdout:min_mon_release 16 (pacific) 2024-08-18T15:17:34.881 INFO:teuthology.orchestra.run.smithi193.stdout:election_strategy: 1 2024-08-18T15:17:34.881 INFO:teuthology.orchestra.run.smithi193.stdout:0: [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] mon.a 2024-08-18T15:17:34.881 INFO:teuthology.orchestra.run.smithi193.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c2.monmap (1 monitors) 2024-08-18T15:17:34.882 DEBUG:teuthology.orchestra.run.smithi193:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-08-18T15:17:34.938 INFO:tasks.ceph:Writing /etc/ceph/c2.conf for FSID af7271d7-3f51-4045-9c51-62f690689cce... 2024-08-18T15:17:34.939 DEBUG:teuthology.orchestra.run.smithi134:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2024-08-18T15:17:34.941 DEBUG:teuthology.orchestra.run.smithi193:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2024-08-18T15:17:34.990 INFO:teuthology.orchestra.run.smithi134.stdout:[global] 2024-08-18T15:17:34.990 INFO:teuthology.orchestra.run.smithi134.stdout: chdir = "" 2024-08-18T15:17:34.990 INFO:teuthology.orchestra.run.smithi134.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-08-18T15:17:34.990 INFO:teuthology.orchestra.run.smithi134.stdout: auth supported = cephx 2024-08-18T15:17:34.990 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:17:34.990 INFO:teuthology.orchestra.run.smithi134.stdout: filestore xattr use omap = true 2024-08-18T15:17:34.990 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:17:34.990 INFO:teuthology.orchestra.run.smithi134.stdout: mon clock drift allowed = 1.000 2024-08-18T15:17:34.990 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:17:34.991 INFO:teuthology.orchestra.run.smithi134.stdout: osd crush chooseleaf type = 0 2024-08-18T15:17:34.991 INFO:teuthology.orchestra.run.smithi134.stdout: auth debug = true 2024-08-18T15:17:34.991 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:17:34.991 INFO:teuthology.orchestra.run.smithi134.stdout: ms die on old message = true 2024-08-18T15:17:34.991 INFO:teuthology.orchestra.run.smithi134.stdout: ms die on bug = true 2024-08-18T15:17:34.991 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:17:34.991 INFO:teuthology.orchestra.run.smithi134.stdout: mon max pg per osd = 10000 # >= luminous 2024-08-18T15:17:34.991 INFO:teuthology.orchestra.run.smithi134.stdout: mon pg warn max object skew = 0 2024-08-18T15:17:34.991 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:17:34.991 INFO:teuthology.orchestra.run.smithi134.stdout: # disable pg_autoscaler by default for new pools 2024-08-18T15:17:34.991 INFO:teuthology.orchestra.run.smithi134.stdout: osd_pool_default_pg_autoscale_mode = off 2024-08-18T15:17:34.991 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:17:34.992 INFO:teuthology.orchestra.run.smithi134.stdout: osd pool default size = 2 2024-08-18T15:17:34.992 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:17:34.992 INFO:teuthology.orchestra.run.smithi134.stdout: mon osd allow primary affinity = true 2024-08-18T15:17:34.992 INFO:teuthology.orchestra.run.smithi134.stdout: mon osd allow pg remap = true 2024-08-18T15:17:34.992 INFO:teuthology.orchestra.run.smithi134.stdout: mon warn on legacy crush tunables = false 2024-08-18T15:17:34.992 INFO:teuthology.orchestra.run.smithi134.stdout: mon warn on crush straw calc version zero = false 2024-08-18T15:17:34.992 INFO:teuthology.orchestra.run.smithi134.stdout: mon warn on no sortbitwise = false 2024-08-18T15:17:34.992 INFO:teuthology.orchestra.run.smithi134.stdout: mon warn on osd down out interval zero = false 2024-08-18T15:17:34.992 INFO:teuthology.orchestra.run.smithi134.stdout: mon warn on too few osds = false 2024-08-18T15:17:34.992 INFO:teuthology.orchestra.run.smithi134.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-08-18T15:17:34.992 INFO:teuthology.orchestra.run.smithi134.stdout: mon_warn_on_pool_no_redundancy = false 2024-08-18T15:17:34.992 INFO:teuthology.orchestra.run.smithi134.stdout: mon_allow_pool_size_one = true 2024-08-18T15:17:34.992 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:17:34.993 INFO:teuthology.orchestra.run.smithi134.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-08-18T15:17:34.993 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:17:34.993 INFO:teuthology.orchestra.run.smithi134.stdout: osd default data pool replay window = 5 2024-08-18T15:17:34.993 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:17:34.993 INFO:teuthology.orchestra.run.smithi134.stdout: mon allow pool delete = true 2024-08-18T15:17:34.993 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:17:34.993 INFO:teuthology.orchestra.run.smithi134.stdout: mon cluster log file level = debug 2024-08-18T15:17:34.993 INFO:teuthology.orchestra.run.smithi134.stdout: debug asserts on shutdown = true 2024-08-18T15:17:34.993 INFO:teuthology.orchestra.run.smithi134.stdout: mon health detail to clog = false 2024-08-18T15:17:34.993 INFO:teuthology.orchestra.run.smithi134.stdout: mon host = 172.21.15.193 2024-08-18T15:17:34.993 INFO:teuthology.orchestra.run.smithi134.stdout: fsid = af7271d7-3f51-4045-9c51-62f690689cce 2024-08-18T15:17:34.993 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:17:34.994 INFO:teuthology.orchestra.run.smithi134.stdout:[osd] 2024-08-18T15:17:34.994 INFO:teuthology.orchestra.run.smithi134.stdout: osd journal size = 100 2024-08-18T15:17:34.994 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:17:34.994 INFO:teuthology.orchestra.run.smithi134.stdout: osd scrub load threshold = 5.0 2024-08-18T15:17:34.994 INFO:teuthology.orchestra.run.smithi134.stdout: osd scrub max interval = 600 2024-08-18T15:17:34.994 INFO:teuthology.orchestra.run.smithi134.stdout: osd mclock profile = high_recovery_ops 2024-08-18T15:17:34.994 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:17:34.994 INFO:teuthology.orchestra.run.smithi134.stdout: osd recover clone overlap = true 2024-08-18T15:17:34.994 INFO:teuthology.orchestra.run.smithi134.stdout: osd recovery max chunk = 1048576 2024-08-18T15:17:34.994 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:17:34.994 INFO:teuthology.orchestra.run.smithi134.stdout: osd debug shutdown = true 2024-08-18T15:17:34.994 INFO:teuthology.orchestra.run.smithi134.stdout: osd debug op order = true 2024-08-18T15:17:34.994 INFO:teuthology.orchestra.run.smithi134.stdout: osd debug verify stray on activate = true 2024-08-18T15:17:34.994 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:17:34.995 INFO:teuthology.orchestra.run.smithi134.stdout: osd open classes on start = true 2024-08-18T15:17:34.995 INFO:teuthology.orchestra.run.smithi134.stdout: osd debug pg log writeout = true 2024-08-18T15:17:34.995 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:17:34.995 INFO:teuthology.orchestra.run.smithi134.stdout: osd deep scrub update digest min age = 30 2024-08-18T15:17:34.995 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:17:34.995 INFO:teuthology.orchestra.run.smithi134.stdout: osd map max advance = 10 2024-08-18T15:17:34.995 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:17:34.995 INFO:teuthology.orchestra.run.smithi134.stdout: journal zero on create = true 2024-08-18T15:17:34.995 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:17:34.995 INFO:teuthology.orchestra.run.smithi134.stdout: filestore ondisk finisher threads = 3 2024-08-18T15:17:34.995 INFO:teuthology.orchestra.run.smithi134.stdout: filestore apply finisher threads = 3 2024-08-18T15:17:34.995 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:17:34.995 INFO:teuthology.orchestra.run.smithi134.stdout: bdev debug aio = true 2024-08-18T15:17:34.996 INFO:teuthology.orchestra.run.smithi134.stdout: osd debug misdirected ops = true 2024-08-18T15:17:34.996 INFO:teuthology.orchestra.run.smithi134.stdout: debug ms = 1 2024-08-18T15:17:34.996 INFO:teuthology.orchestra.run.smithi134.stdout: debug osd = 20 2024-08-18T15:17:34.996 INFO:teuthology.orchestra.run.smithi134.stdout: rocksdb delete range threshold = 0 2024-08-18T15:17:34.996 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:17:34.996 INFO:teuthology.orchestra.run.smithi134.stdout:[mgr] 2024-08-18T15:17:34.996 INFO:teuthology.orchestra.run.smithi134.stdout: debug ms = 1 2024-08-18T15:17:34.996 INFO:teuthology.orchestra.run.smithi134.stdout: debug mgr = 20 2024-08-18T15:17:34.996 INFO:teuthology.orchestra.run.smithi134.stdout: debug mon = 20 2024-08-18T15:17:34.996 INFO:teuthology.orchestra.run.smithi134.stdout: debug auth = 20 2024-08-18T15:17:34.996 INFO:teuthology.orchestra.run.smithi134.stdout: mon reweight min pgs per osd = 4 2024-08-18T15:17:34.996 INFO:teuthology.orchestra.run.smithi134.stdout: mon reweight min bytes per osd = 10 2024-08-18T15:17:34.996 INFO:teuthology.orchestra.run.smithi134.stdout: mgr/telemetry/nag = false 2024-08-18T15:17:34.996 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:17:34.996 INFO:teuthology.orchestra.run.smithi134.stdout:[mon] 2024-08-18T15:17:34.997 INFO:teuthology.orchestra.run.smithi134.stdout: debug ms = 1 2024-08-18T15:17:34.997 INFO:teuthology.orchestra.run.smithi134.stdout: debug mon = 20 2024-08-18T15:17:34.997 INFO:teuthology.orchestra.run.smithi134.stdout: debug paxos = 20 2024-08-18T15:17:34.997 INFO:teuthology.orchestra.run.smithi134.stdout: debug auth = 20 2024-08-18T15:17:34.997 INFO:teuthology.orchestra.run.smithi134.stdout: mon data avail warn = 5 2024-08-18T15:17:34.997 INFO:teuthology.orchestra.run.smithi134.stdout: mon mgr mkfs grace = 240 2024-08-18T15:17:34.997 INFO:teuthology.orchestra.run.smithi134.stdout: mon reweight min pgs per osd = 4 2024-08-18T15:17:34.997 INFO:teuthology.orchestra.run.smithi134.stdout: mon osd reporter subtree level = osd 2024-08-18T15:17:34.997 INFO:teuthology.orchestra.run.smithi134.stdout: mon osd prime pg temp = true 2024-08-18T15:17:34.997 INFO:teuthology.orchestra.run.smithi134.stdout: mon reweight min bytes per osd = 10 2024-08-18T15:17:34.997 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:17:34.997 INFO:teuthology.orchestra.run.smithi134.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-08-18T15:17:34.997 INFO:teuthology.orchestra.run.smithi134.stdout: auth mon ticket ttl = 660 # 11m 2024-08-18T15:17:34.997 INFO:teuthology.orchestra.run.smithi134.stdout: auth service ticket ttl = 240 # 4m 2024-08-18T15:17:34.998 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:17:34.998 INFO:teuthology.orchestra.run.smithi134.stdout: # don't complain about insecure global_id in the test suite 2024-08-18T15:17:34.998 INFO:teuthology.orchestra.run.smithi134.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-08-18T15:17:34.998 INFO:teuthology.orchestra.run.smithi134.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-08-18T15:17:34.998 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:17:34.998 INFO:teuthology.orchestra.run.smithi134.stdout: # 1m isn't quite enough 2024-08-18T15:17:34.998 INFO:teuthology.orchestra.run.smithi134.stdout: mon_down_mkfs_grace = 2m 2024-08-18T15:17:34.998 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:17:34.998 INFO:teuthology.orchestra.run.smithi134.stdout: mon_warn_on_filestore_osds = false 2024-08-18T15:17:34.998 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:17:34.998 INFO:teuthology.orchestra.run.smithi134.stdout:[client] 2024-08-18T15:17:34.998 INFO:teuthology.orchestra.run.smithi134.stdout: rgw cache enabled = true 2024-08-18T15:17:34.998 INFO:teuthology.orchestra.run.smithi134.stdout: rgw enable ops log = true 2024-08-18T15:17:34.999 INFO:teuthology.orchestra.run.smithi134.stdout: rgw enable usage log = true 2024-08-18T15:17:34.999 INFO:teuthology.orchestra.run.smithi134.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-08-18T15:17:34.999 INFO:teuthology.orchestra.run.smithi134.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-08-18T15:17:34.999 INFO:teuthology.orchestra.run.smithi134.stdout: debug rgw = 20 2024-08-18T15:17:34.999 INFO:teuthology.orchestra.run.smithi134.stdout: debug rgw sync = 20 2024-08-18T15:17:34.999 INFO:teuthology.orchestra.run.smithi134.stdout: rgw crypt require ssl = False 2024-08-18T15:17:34.999 INFO:teuthology.orchestra.run.smithi134.stdout: rgw crypt s3 kms backend = testing 2024-08-18T15:17:34.999 INFO:teuthology.orchestra.run.smithi134.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2024-08-18T15:17:34.999 INFO:teuthology.orchestra.run.smithi134.stdout: rgw curl low speed time = 300 2024-08-18T15:17:34.999 INFO:teuthology.orchestra.run.smithi134.stdout: rgw data log num shards = 4 2024-08-18T15:17:34.999 INFO:teuthology.orchestra.run.smithi134.stdout: rgw md log max shards = 4 2024-08-18T15:17:34.999 INFO:teuthology.orchestra.run.smithi134.stdout: rgw sync data inject err probability = 0.1 2024-08-18T15:17:34.999 INFO:teuthology.orchestra.run.smithi134.stdout: rgw sync log trim interval = 0 2024-08-18T15:17:34.999 INFO:teuthology.orchestra.run.smithi134.stdout: rgw sync meta inject err probability = 0.1 2024-08-18T15:17:35.000 INFO:teuthology.orchestra.run.smithi134.stdout: rgw sync obj etag verify = True 2024-08-18T15:17:35.000 INFO:teuthology.orchestra.run.smithi134.stdout: setgroup = ceph 2024-08-18T15:17:35.000 INFO:teuthology.orchestra.run.smithi134.stdout: setuser = ceph 2024-08-18T15:17:35.000 INFO:teuthology.orchestra.run.smithi134.stdout:[mon.a] 2024-08-18T15:17:35.000 INFO:teuthology.orchestra.run.smithi134.stdout:[client.0] 2024-08-18T15:17:35.000 INFO:teuthology.orchestra.run.smithi134.stdout: rgw data notify interval msec = 0 2024-08-18T15:17:35.000 INFO:teuthology.orchestra.run.smithi134.stdout:[osd.0] 2024-08-18T15:17:35.000 INFO:teuthology.orchestra.run.smithi134.stdout: osd_max_omap_entries_per_request = 10 2024-08-18T15:17:35.000 INFO:teuthology.orchestra.run.smithi134.stdout:[osd.1] 2024-08-18T15:17:35.000 INFO:teuthology.orchestra.run.smithi134.stdout: osd_max_omap_entries_per_request = 1000 2024-08-18T15:17:35.000 INFO:teuthology.orchestra.run.smithi134.stdout:[osd.2] 2024-08-18T15:17:35.000 INFO:teuthology.orchestra.run.smithi134.stdout: osd_max_omap_entries_per_request = 10000 2024-08-18T15:17:35.020 INFO:teuthology.orchestra.run.smithi193.stdout:[global] 2024-08-18T15:17:35.020 INFO:teuthology.orchestra.run.smithi193.stdout: chdir = "" 2024-08-18T15:17:35.021 INFO:teuthology.orchestra.run.smithi193.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-08-18T15:17:35.021 INFO:teuthology.orchestra.run.smithi193.stdout: auth supported = cephx 2024-08-18T15:17:35.021 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:17:35.021 INFO:teuthology.orchestra.run.smithi193.stdout: filestore xattr use omap = true 2024-08-18T15:17:35.021 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:17:35.021 INFO:teuthology.orchestra.run.smithi193.stdout: mon clock drift allowed = 1.000 2024-08-18T15:17:35.021 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:17:35.021 INFO:teuthology.orchestra.run.smithi193.stdout: osd crush chooseleaf type = 0 2024-08-18T15:17:35.021 INFO:teuthology.orchestra.run.smithi193.stdout: auth debug = true 2024-08-18T15:17:35.021 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:17:35.021 INFO:teuthology.orchestra.run.smithi193.stdout: ms die on old message = true 2024-08-18T15:17:35.021 INFO:teuthology.orchestra.run.smithi193.stdout: ms die on bug = true 2024-08-18T15:17:35.021 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:17:35.021 INFO:teuthology.orchestra.run.smithi193.stdout: mon max pg per osd = 10000 # >= luminous 2024-08-18T15:17:35.022 INFO:teuthology.orchestra.run.smithi193.stdout: mon pg warn max object skew = 0 2024-08-18T15:17:35.022 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:17:35.022 INFO:teuthology.orchestra.run.smithi193.stdout: # disable pg_autoscaler by default for new pools 2024-08-18T15:17:35.022 INFO:teuthology.orchestra.run.smithi193.stdout: osd_pool_default_pg_autoscale_mode = off 2024-08-18T15:17:35.022 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:17:35.022 INFO:teuthology.orchestra.run.smithi193.stdout: osd pool default size = 2 2024-08-18T15:17:35.022 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:17:35.022 INFO:teuthology.orchestra.run.smithi193.stdout: mon osd allow primary affinity = true 2024-08-18T15:17:35.022 INFO:teuthology.orchestra.run.smithi193.stdout: mon osd allow pg remap = true 2024-08-18T15:17:35.022 INFO:teuthology.orchestra.run.smithi193.stdout: mon warn on legacy crush tunables = false 2024-08-18T15:17:35.022 INFO:teuthology.orchestra.run.smithi193.stdout: mon warn on crush straw calc version zero = false 2024-08-18T15:17:35.022 INFO:teuthology.orchestra.run.smithi193.stdout: mon warn on no sortbitwise = false 2024-08-18T15:17:35.022 INFO:teuthology.orchestra.run.smithi193.stdout: mon warn on osd down out interval zero = false 2024-08-18T15:17:35.022 INFO:teuthology.orchestra.run.smithi193.stdout: mon warn on too few osds = false 2024-08-18T15:17:35.022 INFO:teuthology.orchestra.run.smithi193.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-08-18T15:17:35.023 INFO:teuthology.orchestra.run.smithi193.stdout: mon_warn_on_pool_no_redundancy = false 2024-08-18T15:17:35.023 INFO:teuthology.orchestra.run.smithi193.stdout: mon_allow_pool_size_one = true 2024-08-18T15:17:35.023 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:17:35.023 INFO:teuthology.orchestra.run.smithi193.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-08-18T15:17:35.023 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:17:35.023 INFO:teuthology.orchestra.run.smithi193.stdout: osd default data pool replay window = 5 2024-08-18T15:17:35.023 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:17:35.023 INFO:teuthology.orchestra.run.smithi193.stdout: mon allow pool delete = true 2024-08-18T15:17:35.023 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:17:35.023 INFO:teuthology.orchestra.run.smithi193.stdout: mon cluster log file level = debug 2024-08-18T15:17:35.023 INFO:teuthology.orchestra.run.smithi193.stdout: debug asserts on shutdown = true 2024-08-18T15:17:35.023 INFO:teuthology.orchestra.run.smithi193.stdout: mon health detail to clog = false 2024-08-18T15:17:35.024 INFO:teuthology.orchestra.run.smithi193.stdout: mon host = 172.21.15.193 2024-08-18T15:17:35.024 INFO:teuthology.orchestra.run.smithi193.stdout: fsid = af7271d7-3f51-4045-9c51-62f690689cce 2024-08-18T15:17:35.024 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:17:35.024 INFO:teuthology.orchestra.run.smithi193.stdout:[osd] 2024-08-18T15:17:35.024 INFO:teuthology.orchestra.run.smithi193.stdout: osd journal size = 100 2024-08-18T15:17:35.024 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:17:35.024 INFO:teuthology.orchestra.run.smithi193.stdout: osd scrub load threshold = 5.0 2024-08-18T15:17:35.024 INFO:teuthology.orchestra.run.smithi193.stdout: osd scrub max interval = 600 2024-08-18T15:17:35.024 INFO:teuthology.orchestra.run.smithi193.stdout: osd mclock profile = high_recovery_ops 2024-08-18T15:17:35.024 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:17:35.024 INFO:teuthology.orchestra.run.smithi193.stdout: osd recover clone overlap = true 2024-08-18T15:17:35.024 INFO:teuthology.orchestra.run.smithi193.stdout: osd recovery max chunk = 1048576 2024-08-18T15:17:35.024 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:17:35.024 INFO:teuthology.orchestra.run.smithi193.stdout: osd debug shutdown = true 2024-08-18T15:17:35.025 INFO:teuthology.orchestra.run.smithi193.stdout: osd debug op order = true 2024-08-18T15:17:35.025 INFO:teuthology.orchestra.run.smithi193.stdout: osd debug verify stray on activate = true 2024-08-18T15:17:35.025 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:17:35.025 INFO:teuthology.orchestra.run.smithi193.stdout: osd open classes on start = true 2024-08-18T15:17:35.025 INFO:teuthology.orchestra.run.smithi193.stdout: osd debug pg log writeout = true 2024-08-18T15:17:35.025 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:17:35.025 INFO:teuthology.orchestra.run.smithi193.stdout: osd deep scrub update digest min age = 30 2024-08-18T15:17:35.025 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:17:35.025 INFO:teuthology.orchestra.run.smithi193.stdout: osd map max advance = 10 2024-08-18T15:17:35.025 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:17:35.025 INFO:teuthology.orchestra.run.smithi193.stdout: journal zero on create = true 2024-08-18T15:17:35.025 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:17:35.025 INFO:teuthology.orchestra.run.smithi193.stdout: filestore ondisk finisher threads = 3 2024-08-18T15:17:35.025 INFO:teuthology.orchestra.run.smithi193.stdout: filestore apply finisher threads = 3 2024-08-18T15:17:35.025 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:17:35.026 INFO:teuthology.orchestra.run.smithi193.stdout: bdev debug aio = true 2024-08-18T15:17:35.026 INFO:teuthology.orchestra.run.smithi193.stdout: osd debug misdirected ops = true 2024-08-18T15:17:35.026 INFO:teuthology.orchestra.run.smithi193.stdout: debug ms = 1 2024-08-18T15:17:35.026 INFO:teuthology.orchestra.run.smithi193.stdout: debug osd = 20 2024-08-18T15:17:35.026 INFO:teuthology.orchestra.run.smithi193.stdout: rocksdb delete range threshold = 0 2024-08-18T15:17:35.026 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:17:35.026 INFO:teuthology.orchestra.run.smithi193.stdout:[mgr] 2024-08-18T15:17:35.026 INFO:teuthology.orchestra.run.smithi193.stdout: debug ms = 1 2024-08-18T15:17:35.026 INFO:teuthology.orchestra.run.smithi193.stdout: debug mgr = 20 2024-08-18T15:17:35.026 INFO:teuthology.orchestra.run.smithi193.stdout: debug mon = 20 2024-08-18T15:17:35.026 INFO:teuthology.orchestra.run.smithi193.stdout: debug auth = 20 2024-08-18T15:17:35.026 INFO:teuthology.orchestra.run.smithi193.stdout: mon reweight min pgs per osd = 4 2024-08-18T15:17:35.026 INFO:teuthology.orchestra.run.smithi193.stdout: mon reweight min bytes per osd = 10 2024-08-18T15:17:35.026 INFO:teuthology.orchestra.run.smithi193.stdout: mgr/telemetry/nag = false 2024-08-18T15:17:35.026 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:17:35.027 INFO:teuthology.orchestra.run.smithi193.stdout:[mon] 2024-08-18T15:17:35.027 INFO:teuthology.orchestra.run.smithi193.stdout: debug ms = 1 2024-08-18T15:17:35.027 INFO:teuthology.orchestra.run.smithi193.stdout: debug mon = 20 2024-08-18T15:17:35.027 INFO:teuthology.orchestra.run.smithi193.stdout: debug paxos = 20 2024-08-18T15:17:35.027 INFO:teuthology.orchestra.run.smithi193.stdout: debug auth = 20 2024-08-18T15:17:35.027 INFO:teuthology.orchestra.run.smithi193.stdout: mon data avail warn = 5 2024-08-18T15:17:35.027 INFO:teuthology.orchestra.run.smithi193.stdout: mon mgr mkfs grace = 240 2024-08-18T15:17:35.027 INFO:teuthology.orchestra.run.smithi193.stdout: mon reweight min pgs per osd = 4 2024-08-18T15:17:35.027 INFO:teuthology.orchestra.run.smithi193.stdout: mon osd reporter subtree level = osd 2024-08-18T15:17:35.027 INFO:teuthology.orchestra.run.smithi193.stdout: mon osd prime pg temp = true 2024-08-18T15:17:35.027 INFO:teuthology.orchestra.run.smithi193.stdout: mon reweight min bytes per osd = 10 2024-08-18T15:17:35.027 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:17:35.027 INFO:teuthology.orchestra.run.smithi193.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-08-18T15:17:35.027 INFO:teuthology.orchestra.run.smithi193.stdout: auth mon ticket ttl = 660 # 11m 2024-08-18T15:17:35.027 INFO:teuthology.orchestra.run.smithi193.stdout: auth service ticket ttl = 240 # 4m 2024-08-18T15:17:35.028 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:17:35.028 INFO:teuthology.orchestra.run.smithi193.stdout: # don't complain about insecure global_id in the test suite 2024-08-18T15:17:35.028 INFO:teuthology.orchestra.run.smithi193.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-08-18T15:17:35.028 INFO:teuthology.orchestra.run.smithi193.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-08-18T15:17:35.028 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:17:35.028 INFO:teuthology.orchestra.run.smithi193.stdout: # 1m isn't quite enough 2024-08-18T15:17:35.028 INFO:teuthology.orchestra.run.smithi193.stdout: mon_down_mkfs_grace = 2m 2024-08-18T15:17:35.028 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:17:35.028 INFO:teuthology.orchestra.run.smithi193.stdout: mon_warn_on_filestore_osds = false 2024-08-18T15:17:35.028 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:17:35.028 INFO:teuthology.orchestra.run.smithi193.stdout:[client] 2024-08-18T15:17:35.028 INFO:teuthology.orchestra.run.smithi193.stdout: rgw cache enabled = true 2024-08-18T15:17:35.028 INFO:teuthology.orchestra.run.smithi193.stdout: rgw enable ops log = true 2024-08-18T15:17:35.029 INFO:teuthology.orchestra.run.smithi193.stdout: rgw enable usage log = true 2024-08-18T15:17:35.029 INFO:teuthology.orchestra.run.smithi193.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-08-18T15:17:35.029 INFO:teuthology.orchestra.run.smithi193.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-08-18T15:17:35.029 INFO:teuthology.orchestra.run.smithi193.stdout: debug rgw = 20 2024-08-18T15:17:35.029 INFO:teuthology.orchestra.run.smithi193.stdout: debug rgw sync = 20 2024-08-18T15:17:35.029 INFO:teuthology.orchestra.run.smithi193.stdout: rgw crypt require ssl = False 2024-08-18T15:17:35.029 INFO:teuthology.orchestra.run.smithi193.stdout: rgw crypt s3 kms backend = testing 2024-08-18T15:17:35.029 INFO:teuthology.orchestra.run.smithi193.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2024-08-18T15:17:35.029 INFO:teuthology.orchestra.run.smithi193.stdout: rgw curl low speed time = 300 2024-08-18T15:17:35.029 INFO:teuthology.orchestra.run.smithi193.stdout: rgw data log num shards = 4 2024-08-18T15:17:35.029 INFO:teuthology.orchestra.run.smithi193.stdout: rgw md log max shards = 4 2024-08-18T15:17:35.030 INFO:teuthology.orchestra.run.smithi193.stdout: rgw sync data inject err probability = 0.1 2024-08-18T15:17:35.030 INFO:teuthology.orchestra.run.smithi193.stdout: rgw sync log trim interval = 0 2024-08-18T15:17:35.030 INFO:teuthology.orchestra.run.smithi193.stdout: rgw sync meta inject err probability = 0.1 2024-08-18T15:17:35.030 INFO:teuthology.orchestra.run.smithi193.stdout: rgw sync obj etag verify = True 2024-08-18T15:17:35.030 INFO:teuthology.orchestra.run.smithi193.stdout: setgroup = ceph 2024-08-18T15:17:35.030 INFO:teuthology.orchestra.run.smithi193.stdout: setuser = ceph 2024-08-18T15:17:35.030 INFO:teuthology.orchestra.run.smithi193.stdout:[mon.a] 2024-08-18T15:17:35.030 INFO:teuthology.orchestra.run.smithi193.stdout:[client.0] 2024-08-18T15:17:35.030 INFO:teuthology.orchestra.run.smithi193.stdout: rgw data notify interval msec = 0 2024-08-18T15:17:35.030 INFO:teuthology.orchestra.run.smithi193.stdout:[osd.0] 2024-08-18T15:17:35.030 INFO:teuthology.orchestra.run.smithi193.stdout: osd_max_omap_entries_per_request = 10 2024-08-18T15:17:35.031 INFO:teuthology.orchestra.run.smithi193.stdout:[osd.1] 2024-08-18T15:17:35.031 INFO:teuthology.orchestra.run.smithi193.stdout: osd_max_omap_entries_per_request = 1000 2024-08-18T15:17:35.031 INFO:teuthology.orchestra.run.smithi193.stdout:[osd.2] 2024-08-18T15:17:35.031 INFO:teuthology.orchestra.run.smithi193.stdout: osd_max_omap_entries_per_request = 10000 2024-08-18T15:17:35.032 INFO:tasks.ceph:Creating admin key on c2.mon.a... 2024-08-18T15:17:35.032 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c2.keyring 2024-08-18T15:17:35.127 INFO:tasks.ceph:Copying monmap to all nodes... 2024-08-18T15:17:35.127 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-08-18T15:17:35.127 DEBUG:teuthology.orchestra.run.smithi193:> dd if=/etc/ceph/c2.keyring of=/dev/stdout 2024-08-18T15:17:35.142 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-08-18T15:17:35.142 DEBUG:teuthology.orchestra.run.smithi193:> dd if=/home/ubuntu/cephtest/c2.monmap of=/dev/stdout 2024-08-18T15:17:35.199 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi134.front.sepia.ceph.com 2024-08-18T15:17:35.199 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-08-18T15:17:35.199 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/etc/ceph/c2.keyring 2024-08-18T15:17:35.199 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod 0644 /etc/ceph/c2.keyring 2024-08-18T15:17:35.241 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-08-18T15:17:35.241 DEBUG:teuthology.orchestra.run.smithi134:> dd of=/home/ubuntu/cephtest/c2.monmap 2024-08-18T15:17:35.298 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi193.front.sepia.ceph.com 2024-08-18T15:17:35.298 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-08-18T15:17:35.299 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/etc/ceph/c2.keyring 2024-08-18T15:17:35.299 DEBUG:teuthology.orchestra.run.smithi193:> sudo chmod 0644 /etc/ceph/c2.keyring 2024-08-18T15:17:35.336 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-08-18T15:17:35.336 DEBUG:teuthology.orchestra.run.smithi193:> dd of=/home/ubuntu/cephtest/c2.monmap 2024-08-18T15:17:35.391 INFO:tasks.ceph:Setting up mon nodes... 2024-08-18T15:17:35.391 INFO:tasks.ceph:Setting up mgr nodes... 2024-08-18T15:17:35.392 DEBUG:teuthology.orchestra.run.smithi193:> sudo mkdir -p /var/lib/ceph/mgr/c2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c2-x/keyring 2024-08-18T15:17:35.482 INFO:teuthology.orchestra.run.smithi193.stdout:creating /var/lib/ceph/mgr/c2-x/keyring 2024-08-18T15:17:35.484 INFO:tasks.ceph:Setting up mds nodes... 2024-08-18T15:17:35.484 INFO:tasks.ceph_client:Setting up client nodes... 2024-08-18T15:17:35.485 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c2.client.0.keyring && sudo chmod 0644 /etc/ceph/c2.client.0.keyring 2024-08-18T15:17:35.525 INFO:teuthology.orchestra.run.smithi193.stdout:creating /etc/ceph/c2.client.0.keyring 2024-08-18T15:17:35.538 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c2.client.1.keyring && sudo chmod 0644 /etc/ceph/c2.client.1.keyring 2024-08-18T15:17:35.619 INFO:teuthology.orchestra.run.smithi193.stdout:creating /etc/ceph/c2.client.1.keyring 2024-08-18T15:17:35.631 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-08-18T15:17:35.632 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi134.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi193.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2024-08-18T15:17:35.632 DEBUG:teuthology.orchestra.run.smithi193:> sudo mkdir -p /var/lib/ceph/osd/c2-0 2024-08-18T15:17:35.696 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2024-08-18T15:17:35.696 INFO:tasks.ceph:role: c2.osd.0 2024-08-18T15:17:35.696 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi193.front.sepia.ceph.com 2024-08-18T15:17:35.697 DEBUG:teuthology.orchestra.run.smithi193:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-08-18T15:17:35.769 INFO:teuthology.orchestra.run.smithi193.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-08-18T15:17:35.769 INFO:teuthology.orchestra.run.smithi193.stdout: = sectsz=512 attr=2, projid32bit=1 2024-08-18T15:17:35.769 INFO:teuthology.orchestra.run.smithi193.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-08-18T15:17:35.769 INFO:teuthology.orchestra.run.smithi193.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-08-18T15:17:35.769 INFO:teuthology.orchestra.run.smithi193.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-08-18T15:17:35.769 INFO:teuthology.orchestra.run.smithi193.stdout: = sunit=0 swidth=0 blks 2024-08-18T15:17:35.769 INFO:teuthology.orchestra.run.smithi193.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-08-18T15:17:35.770 INFO:teuthology.orchestra.run.smithi193.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-08-18T15:17:35.770 INFO:teuthology.orchestra.run.smithi193.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-08-18T15:17:35.770 INFO:teuthology.orchestra.run.smithi193.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-08-18T15:17:36.270 INFO:teuthology.orchestra.run.smithi193.stdout:Discarding blocks...Done. 2024-08-18T15:17:36.272 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi193.front.sepia.ceph.com -o noatime 2024-08-18T15:17:36.273 DEBUG:teuthology.orchestra.run.smithi193:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c2-0 2024-08-18T15:17:36.393 DEBUG:teuthology.orchestra.run.smithi193:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-0 2024-08-18T15:17:36.424 DEBUG:teuthology.orchestra.run.smithi193:> sudo mkdir -p /var/lib/ceph/osd/c2-1 2024-08-18T15:17:36.490 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2024-08-18T15:17:36.490 INFO:tasks.ceph:role: c2.osd.1 2024-08-18T15:17:36.490 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi193.front.sepia.ceph.com 2024-08-18T15:17:36.490 DEBUG:teuthology.orchestra.run.smithi193:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-08-18T15:17:36.562 INFO:teuthology.orchestra.run.smithi193.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-08-18T15:17:36.563 INFO:teuthology.orchestra.run.smithi193.stdout: = sectsz=512 attr=2, projid32bit=1 2024-08-18T15:17:36.563 INFO:teuthology.orchestra.run.smithi193.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-08-18T15:17:36.563 INFO:teuthology.orchestra.run.smithi193.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-08-18T15:17:36.563 INFO:teuthology.orchestra.run.smithi193.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-08-18T15:17:36.563 INFO:teuthology.orchestra.run.smithi193.stdout: = sunit=0 swidth=0 blks 2024-08-18T15:17:36.563 INFO:teuthology.orchestra.run.smithi193.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-08-18T15:17:36.563 INFO:teuthology.orchestra.run.smithi193.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-08-18T15:17:36.563 INFO:teuthology.orchestra.run.smithi193.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-08-18T15:17:36.563 INFO:teuthology.orchestra.run.smithi193.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-08-18T15:17:37.034 INFO:teuthology.orchestra.run.smithi193.stdout:Discarding blocks...Done. 2024-08-18T15:17:37.036 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi193.front.sepia.ceph.com -o noatime 2024-08-18T15:17:37.037 DEBUG:teuthology.orchestra.run.smithi193:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c2-1 2024-08-18T15:17:37.119 DEBUG:teuthology.orchestra.run.smithi193:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-1 2024-08-18T15:17:37.189 DEBUG:teuthology.orchestra.run.smithi193:> sudo mkdir -p /var/lib/ceph/osd/c2-2 2024-08-18T15:17:37.254 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2024-08-18T15:17:37.254 INFO:tasks.ceph:role: c2.osd.2 2024-08-18T15:17:37.255 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi193.front.sepia.ceph.com 2024-08-18T15:17:37.255 DEBUG:teuthology.orchestra.run.smithi193:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-08-18T15:17:37.326 INFO:teuthology.orchestra.run.smithi193.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-08-18T15:17:37.326 INFO:teuthology.orchestra.run.smithi193.stdout: = sectsz=512 attr=2, projid32bit=1 2024-08-18T15:17:37.326 INFO:teuthology.orchestra.run.smithi193.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-08-18T15:17:37.326 INFO:teuthology.orchestra.run.smithi193.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-08-18T15:17:37.326 INFO:teuthology.orchestra.run.smithi193.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-08-18T15:17:37.326 INFO:teuthology.orchestra.run.smithi193.stdout: = sunit=0 swidth=0 blks 2024-08-18T15:17:37.326 INFO:teuthology.orchestra.run.smithi193.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-08-18T15:17:37.326 INFO:teuthology.orchestra.run.smithi193.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-08-18T15:17:37.326 INFO:teuthology.orchestra.run.smithi193.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-08-18T15:17:37.326 INFO:teuthology.orchestra.run.smithi193.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-08-18T15:17:37.847 INFO:teuthology.orchestra.run.smithi193.stdout:Discarding blocks...Done. 2024-08-18T15:17:37.849 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi193.front.sepia.ceph.com -o noatime 2024-08-18T15:17:37.849 DEBUG:teuthology.orchestra.run.smithi193:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c2-2 2024-08-18T15:17:37.987 DEBUG:teuthology.orchestra.run.smithi193:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-2 2024-08-18T15:17:38.019 DEBUG:teuthology.orchestra.run.smithi193:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c2.monmap 2024-08-18T15:17:38.124 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:17:38.124+0000 7fe377efb640 -1 auth: error reading file: /var/lib/ceph/osd/c2-0/keyring: can't open /var/lib/ceph/osd/c2-0/keyring: (2) No such file or directory 2024-08-18T15:17:38.127 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:17:38.127+0000 7fe377efb640 -1 created new key in keyring /var/lib/ceph/osd/c2-0/keyring 2024-08-18T15:17:38.127 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:17:38.127+0000 7fe377efb640 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2024-08-18T15:17:38.127 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:17:38.127+0000 7fe377efb640 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2024-08-18T15:17:38.128 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:17:38.127+0000 7fe377efb640 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2024-08-18T15:17:38.134 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:17:38.134+0000 7fe377efb640 -1 bluestore(/var/lib/ceph/osd/c2-0) _read_fsid unparsable uuid 2024-08-18T15:17:40.211 DEBUG:teuthology.orchestra.run.smithi193:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-0 2024-08-18T15:17:40.241 DEBUG:teuthology.orchestra.run.smithi193:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c2.monmap 2024-08-18T15:17:40.332 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:17:40.331+0000 7fc0c8a25640 -1 auth: error reading file: /var/lib/ceph/osd/c2-1/keyring: can't open /var/lib/ceph/osd/c2-1/keyring: (2) No such file or directory 2024-08-18T15:17:40.332 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:17:40.332+0000 7fc0c8a25640 -1 created new key in keyring /var/lib/ceph/osd/c2-1/keyring 2024-08-18T15:17:40.332 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:17:40.332+0000 7fc0c8a25640 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2024-08-18T15:17:40.333 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:17:40.332+0000 7fc0c8a25640 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2024-08-18T15:17:40.333 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:17:40.332+0000 7fc0c8a25640 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2024-08-18T15:17:40.333 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:17:40.332+0000 7fc0c8a25640 -1 bluestore(/var/lib/ceph/osd/c2-1) _read_fsid unparsable uuid 2024-08-18T15:17:42.413 DEBUG:teuthology.orchestra.run.smithi193:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-1 2024-08-18T15:17:42.443 DEBUG:teuthology.orchestra.run.smithi193:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c2.monmap 2024-08-18T15:17:42.535 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:17:42.532+0000 7f3b2de11640 -1 auth: error reading file: /var/lib/ceph/osd/c2-2/keyring: can't open /var/lib/ceph/osd/c2-2/keyring: (2) No such file or directory 2024-08-18T15:17:42.535 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:17:42.533+0000 7f3b2de11640 -1 created new key in keyring /var/lib/ceph/osd/c2-2/keyring 2024-08-18T15:17:42.535 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:17:42.533+0000 7f3b2de11640 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2024-08-18T15:17:42.535 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:17:42.533+0000 7f3b2de11640 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2024-08-18T15:17:42.535 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:17:42.533+0000 7f3b2de11640 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2024-08-18T15:17:42.535 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:17:42.533+0000 7f3b2de11640 -1 bluestore(/var/lib/ceph/osd/c2-2) _read_fsid unparsable uuid 2024-08-18T15:17:44.615 DEBUG:teuthology.orchestra.run.smithi193:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-2 2024-08-18T15:17:44.645 INFO:tasks.ceph:Reading keys from all nodes... 2024-08-18T15:17:44.645 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-08-18T15:17:44.645 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/var/lib/ceph/mgr/c2-x/keyring of=/dev/stdout 2024-08-18T15:17:44.714 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-08-18T15:17:44.714 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/var/lib/ceph/osd/c2-0/keyring of=/dev/stdout 2024-08-18T15:17:44.785 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-08-18T15:17:44.785 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/var/lib/ceph/osd/c2-1/keyring of=/dev/stdout 2024-08-18T15:17:44.851 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-08-18T15:17:44.852 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/var/lib/ceph/osd/c2-2/keyring of=/dev/stdout 2024-08-18T15:17:44.917 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-08-18T15:17:44.917 DEBUG:teuthology.orchestra.run.smithi193:> dd if=/etc/ceph/c2.client.0.keyring of=/dev/stdout 2024-08-18T15:17:44.973 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-08-18T15:17:44.974 DEBUG:teuthology.orchestra.run.smithi193:> dd if=/etc/ceph/c2.client.1.keyring of=/dev/stdout 2024-08-18T15:17:45.029 INFO:tasks.ceph:Adding keys to all mons... 2024-08-18T15:17:45.029 DEBUG:teuthology.orchestra.run.smithi193:> sudo tee -a /etc/ceph/c2.keyring 2024-08-18T15:17:45.093 INFO:teuthology.orchestra.run.smithi193.stdout:[mgr.x] 2024-08-18T15:17:45.093 INFO:teuthology.orchestra.run.smithi193.stdout: key = AQCPEMJmtCDCHBAA5svZo+x7MbxvyNU8ZVZ8Ug== 2024-08-18T15:17:45.093 INFO:teuthology.orchestra.run.smithi193.stdout:[osd.0] 2024-08-18T15:17:45.093 INFO:teuthology.orchestra.run.smithi193.stdout: key = AQCSEMJmcY93BxAA15jMO+4P3F3keAX7l+eYIA== 2024-08-18T15:17:45.093 INFO:teuthology.orchestra.run.smithi193.stdout:[osd.1] 2024-08-18T15:17:45.093 INFO:teuthology.orchestra.run.smithi193.stdout: key = AQCUEMJm2ODTExAAFzMSCvjA30xr1LnlsBTOzg== 2024-08-18T15:17:45.093 INFO:teuthology.orchestra.run.smithi193.stdout:[osd.2] 2024-08-18T15:17:45.093 INFO:teuthology.orchestra.run.smithi193.stdout: key = AQCWEMJmmAzPHxAAebMy+08lcpm/yBtnzTtMLg== 2024-08-18T15:17:45.093 INFO:teuthology.orchestra.run.smithi193.stdout:[client.0] 2024-08-18T15:17:45.093 INFO:teuthology.orchestra.run.smithi193.stdout: key = AQCPEMJmVGldHxAAL3+pC9Qx50JLvBXiy1qIgQ== 2024-08-18T15:17:45.094 INFO:teuthology.orchestra.run.smithi193.stdout:[client.1] 2024-08-18T15:17:45.094 INFO:teuthology.orchestra.run.smithi193.stdout: key = AQCPEMJm3y7vJBAAXBiJhLkKihnICOUekCbPUw== 2024-08-18T15:17:45.095 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-08-18T15:17:45.186 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-18T15:17:45.236 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-18T15:17:45.286 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-18T15:17:45.336 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-08-18T15:17:45.393 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-08-18T15:17:45.446 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-08-18T15:17:45.446 DEBUG:teuthology.orchestra.run.smithi193:> sudo mkdir -p /var/lib/ceph/mon/c2-a 2024-08-18T15:17:45.471 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c2 --mkfs -i a --monmap /home/ubuntu/cephtest/c2.monmap --keyring /etc/ceph/c2.keyring 2024-08-18T15:17:45.568 DEBUG:teuthology.orchestra.run.smithi193:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c2-a 2024-08-18T15:17:45.593 DEBUG:teuthology.orchestra.run.smithi193:> rm -- /home/ubuntu/cephtest/c2.monmap 2024-08-18T15:17:45.649 INFO:tasks.ceph:Starting mon daemons in cluster c2... 2024-08-18T15:17:45.649 INFO:tasks.ceph.c2.mon.a:Restarting daemon 2024-08-18T15:17:45.649 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c2 -i a 2024-08-18T15:17:45.692 INFO:tasks.ceph.c2.mon.a:Started 2024-08-18T15:17:45.692 INFO:tasks.ceph:Starting mgr daemons in cluster c2... 2024-08-18T15:17:45.692 INFO:tasks.ceph.c2.mgr.x:Restarting daemon 2024-08-18T15:17:45.692 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c2 -i x 2024-08-18T15:17:45.694 INFO:tasks.ceph.c2.mgr.x:Started 2024-08-18T15:17:45.694 INFO:tasks.ceph:Setting crush tunables to default 2024-08-18T15:17:45.694 DEBUG:teuthology.orchestra.run.smithi193:> sudo ceph --cluster c2 osd crush tunables default 2024-08-18T15:17:45.873 INFO:teuthology.orchestra.run.smithi193.stderr:adjusted tunables profile to default 2024-08-18T15:17:45.883 INFO:tasks.ceph:check_enable_crimson: False 2024-08-18T15:17:45.884 INFO:tasks.ceph:Starting osd daemons in cluster c2... 2024-08-18T15:17:45.884 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-08-18T15:17:45.884 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/var/lib/ceph/osd/c2-0/fsid of=/dev/stdout 2024-08-18T15:17:45.908 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-08-18T15:17:45.908 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/var/lib/ceph/osd/c2-1/fsid of=/dev/stdout 2024-08-18T15:17:45.973 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-08-18T15:17:45.974 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/var/lib/ceph/osd/c2-2/fsid of=/dev/stdout 2024-08-18T15:17:46.038 DEBUG:teuthology.orchestra.run.smithi193:> sudo ceph --cluster c2 osd new 0c8aafee-6ffe-4ac4-99de-b7cafefa9bb4 0 2024-08-18T15:17:46.076 INFO:tasks.ceph.c2.mgr.x.smithi193.stderr:2024-08-18T15:17:46.075+0000 7fd1d9856040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-08-18T15:17:46.254 INFO:teuthology.orchestra.run.smithi193.stdout:0 2024-08-18T15:17:46.258 INFO:tasks.ceph.c2.mgr.x.smithi193.stderr:2024-08-18T15:17:46.257+0000 7fd1d9856040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-08-18T15:17:46.265 DEBUG:teuthology.orchestra.run.smithi193:> sudo ceph --cluster c2 osd new 4daab6f9-7d9e-4f48-91ac-8fac4ff3e454 1 2024-08-18T15:17:46.314 INFO:tasks.ceph.c2.mgr.x.smithi193.stderr:2024-08-18T15:17:46.313+0000 7fd1d9856040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-08-18T15:17:46.444 INFO:teuthology.orchestra.run.smithi193.stdout:1 2024-08-18T15:17:46.455 DEBUG:teuthology.orchestra.run.smithi193:> sudo ceph --cluster c2 osd new 16749f2c-e694-4263-b058-a9da51dec43b 2 2024-08-18T15:17:46.578 INFO:tasks.ceph.c2.mgr.x.smithi193.stderr:2024-08-18T15:17:46.578+0000 7fd1d9856040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-08-18T15:17:46.626 INFO:tasks.ceph.c2.mgr.x.smithi193.stderr:2024-08-18T15:17:46.624+0000 7fd1d9856040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-08-18T15:17:46.634 INFO:teuthology.orchestra.run.smithi193.stdout:2 2024-08-18T15:17:46.646 INFO:tasks.ceph.c2.osd.0:Restarting daemon 2024-08-18T15:17:46.646 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 0 2024-08-18T15:17:46.649 INFO:tasks.ceph.c2.osd.0:Started 2024-08-18T15:17:46.649 INFO:tasks.ceph.c2.osd.1:Restarting daemon 2024-08-18T15:17:46.649 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 1 2024-08-18T15:17:46.651 INFO:tasks.ceph.c2.osd.1:Started 2024-08-18T15:17:46.651 INFO:tasks.ceph.c2.osd.2:Restarting daemon 2024-08-18T15:17:46.651 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 2 2024-08-18T15:17:46.653 INFO:tasks.ceph.c2.osd.2:Started 2024-08-18T15:17:46.653 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2024-08-18T15:17:46.845 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:17:46.845 INFO:teuthology.orchestra.run.smithi193.stdout:{"epoch":5,"fsid":"af7271d7-3f51-4045-9c51-62f690689cce","created":"2024-08-18T15:17:45.791736+0000","modified":"2024-08-18T15:17:46.633468+0000","last_up_change":"0.000000","last_in_change":"2024-08-18T15:17:46.633468+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"0c8aafee-6ffe-4ac4-99de-b7cafefa9bb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4daab6f9-7d9e-4f48-91ac-8fac4ff3e454","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"16749f2c-e694-4263-b058-a9da51dec43b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-18T15:17:46.856 INFO:tasks.ceph.ceph_manager.c2:[] 2024-08-18T15:17:46.856 INFO:tasks.ceph:Waiting for OSDs to come up 2024-08-18T15:17:47.158 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2024-08-18T15:17:47.248 INFO:tasks.ceph.c2.osd.1.smithi193.stderr:2024-08-18T15:17:47.247+0000 7f0592a11640 -1 Falling back to public interface 2024-08-18T15:17:47.256 INFO:tasks.ceph.c2.osd.2.smithi193.stderr:2024-08-18T15:17:47.255+0000 7f2b59199640 -1 Falling back to public interface 2024-08-18T15:17:47.269 INFO:tasks.ceph.c2.osd.0.smithi193.stderr:2024-08-18T15:17:47.269+0000 7f262661a640 -1 Falling back to public interface 2024-08-18T15:17:47.323 INFO:teuthology.misc.health.smithi193.stdout: 2024-08-18T15:17:47.323 INFO:teuthology.misc.health.smithi193.stdout:{"epoch":5,"fsid":"af7271d7-3f51-4045-9c51-62f690689cce","created":"2024-08-18T15:17:45.791736+0000","modified":"2024-08-18T15:17:46.633468+0000","last_up_change":"0.000000","last_in_change":"2024-08-18T15:17:46.633468+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"0c8aafee-6ffe-4ac4-99de-b7cafefa9bb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4daab6f9-7d9e-4f48-91ac-8fac4ff3e454","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"16749f2c-e694-4263-b058-a9da51dec43b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-18T15:17:47.333 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-08-18T15:17:47.400 INFO:tasks.ceph.c2.mgr.x.smithi193.stderr:2024-08-18T15:17:47.399+0000 7fd1d9856040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-08-18T15:17:47.484 INFO:tasks.ceph.c2.mgr.x.smithi193.stderr:2024-08-18T15:17:47.484+0000 7fd1d9856040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-08-18T15:17:47.526 INFO:tasks.ceph.c2.mgr.x.smithi193.stderr:2024-08-18T15:17:47.526+0000 7fd1d9856040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-08-18T15:17:47.688 INFO:tasks.ceph.c2.mgr.x.smithi193.stderr:2024-08-18T15:17:47.688+0000 7fd1d9856040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-08-18T15:17:47.734 INFO:tasks.ceph.c2.mgr.x.smithi193.stderr:2024-08-18T15:17:47.734+0000 7fd1d9856040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-08-18T15:17:47.786 INFO:tasks.ceph.c2.mgr.x.smithi193.stderr:2024-08-18T15:17:47.785+0000 7fd1d9856040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-08-18T15:17:47.898 INFO:tasks.ceph.c2.mgr.x.smithi193.stderr:2024-08-18T15:17:47.898+0000 7fd1d9856040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-08-18T15:17:48.272 INFO:tasks.ceph.c2.mgr.x.smithi193.stderr:2024-08-18T15:17:48.272+0000 7fd1d9856040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-08-18T15:17:48.525 INFO:tasks.ceph.c2.mgr.x.smithi193.stderr:2024-08-18T15:17:48.524+0000 7fd1d9856040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-08-18T15:17:48.575 INFO:tasks.ceph.c2.mgr.x.smithi193.stderr:2024-08-18T15:17:48.575+0000 7fd1d9856040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-08-18T15:17:48.690 INFO:tasks.ceph.c2.mgr.x.smithi193.stderr:2024-08-18T15:17:48.690+0000 7fd1d9856040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-08-18T15:17:49.071 INFO:tasks.ceph.c2.mgr.x.smithi193.stderr:2024-08-18T15:17:49.070+0000 7fd1d9856040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-08-18T15:17:49.117 INFO:tasks.ceph.c2.mgr.x.smithi193.stderr:2024-08-18T15:17:49.117+0000 7fd1d9856040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-08-18T15:17:49.271 INFO:tasks.ceph.c2.mgr.x.smithi193.stderr:2024-08-18T15:17:49.270+0000 7fd1d9856040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-08-18T15:17:49.405 INFO:tasks.ceph.c2.mgr.x.smithi193.stderr:2024-08-18T15:17:49.404+0000 7fd1d9856040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-08-18T15:17:49.455 INFO:tasks.ceph.c2.mgr.x.smithi193.stderr:2024-08-18T15:17:49.454+0000 7fd1d9856040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-08-18T15:17:49.522 INFO:tasks.ceph.c2.mgr.x.smithi193.stderr:2024-08-18T15:17:49.522+0000 7fd1d9856040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-08-18T15:17:49.567 INFO:tasks.ceph.c2.mgr.x.smithi193.stderr:2024-08-18T15:17:49.566+0000 7fd1d9856040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-08-18T15:17:49.989 INFO:tasks.ceph.c2.osd.1.smithi193.stderr:2024-08-18T15:17:49.988+0000 7f0592a11640 -1 osd.1 0 log_to_monitors true 2024-08-18T15:17:49.989 INFO:tasks.ceph.c2.osd.0.smithi193.stderr:2024-08-18T15:17:49.989+0000 7f262661a640 -1 osd.0 0 log_to_monitors true 2024-08-18T15:17:49.998 INFO:tasks.ceph.c2.osd.2.smithi193.stderr:2024-08-18T15:17:49.997+0000 7f2b59199640 -1 osd.2 0 log_to_monitors true 2024-08-18T15:17:50.048 INFO:tasks.ceph.c2.mgr.x.smithi193.stderr:2024-08-18T15:17:50.047+0000 7fd1d9856040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-08-18T15:17:50.161 INFO:tasks.ceph.c2.mgr.x.smithi193.stderr:2024-08-18T15:17:50.161+0000 7fd1d9856040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-08-18T15:17:51.293 INFO:tasks.ceph.c2.mgr.x.smithi193.stderr:2024-08-18T15:17:51.293+0000 7fd17a62d640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-08-18T15:17:52.263 INFO:tasks.ceph.c2.osd.1.smithi193.stderr:2024-08-18T15:17:52.263+0000 7f058e301640 -1 osd.1 0 waiting for initial osdmap 2024-08-18T15:17:52.277 INFO:tasks.ceph.c2.osd.1.smithi193.stderr:2024-08-18T15:17:52.277+0000 7f0586122640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-18T15:17:52.310 INFO:tasks.ceph.c2.osd.2.smithi193.stderr:2024-08-18T15:17:52.310+0000 7f2b55126640 -1 osd.2 0 waiting for initial osdmap 2024-08-18T15:17:52.314 INFO:tasks.ceph.c2.osd.0.smithi193.stderr:2024-08-18T15:17:52.313+0000 7f262207a640 -1 osd.0 0 waiting for initial osdmap 2024-08-18T15:17:52.319 INFO:tasks.ceph.c2.osd.2.smithi193.stderr:2024-08-18T15:17:52.319+0000 7f2b4cf47640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-18T15:17:52.325 INFO:tasks.ceph.c2.osd.0.smithi193.stderr:2024-08-18T15:17:52.324+0000 7f2619e9b640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-18T15:17:53.635 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2024-08-18T15:17:53.916 INFO:teuthology.misc.health.smithi193.stdout: 2024-08-18T15:17:53.917 INFO:teuthology.misc.health.smithi193.stdout:{"epoch":8,"fsid":"af7271d7-3f51-4045-9c51-62f690689cce","created":"2024-08-18T15:17:45.791736+0000","modified":"2024-08-18T15:17:53.268568+0000","last_up_change":"2024-08-18T15:17:53.268568+0000","last_in_change":"2024-08-18T15:17:46.633468+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"0c8aafee-6ffe-4ac4-99de-b7cafefa9bb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":32640},{"type":"v1","addr":"172.21.15.193:6817","nonce":32640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":32640},{"type":"v1","addr":"172.21.15.193:6819","nonce":32640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":32640},{"type":"v1","addr":"172.21.15.193:6823","nonce":32640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":32640},{"type":"v1","addr":"172.21.15.193:6821","nonce":32640}]},"public_addr":"172.21.15.193:6817/32640","cluster_addr":"172.21.15.193:6819/32640","heartbeat_back_addr":"172.21.15.193:6823/32640","heartbeat_front_addr":"172.21.15.193:6821/32640","state":["exists","up"]},{"osd":1,"uuid":"4daab6f9-7d9e-4f48-91ac-8fac4ff3e454","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":32638},{"type":"v1","addr":"172.21.15.193:6801","nonce":32638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":32638},{"type":"v1","addr":"172.21.15.193:6803","nonce":32638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":32638},{"type":"v1","addr":"172.21.15.193:6807","nonce":32638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":32638},{"type":"v1","addr":"172.21.15.193:6805","nonce":32638}]},"public_addr":"172.21.15.193:6801/32638","cluster_addr":"172.21.15.193:6803/32638","heartbeat_back_addr":"172.21.15.193:6807/32638","heartbeat_front_addr":"172.21.15.193:6805/32638","state":["exists","up"]},{"osd":2,"uuid":"16749f2c-e694-4263-b058-a9da51dec43b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":32639},{"type":"v1","addr":"172.21.15.193:6809","nonce":32639}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":32639},{"type":"v1","addr":"172.21.15.193:6811","nonce":32639}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":32639},{"type":"v1","addr":"172.21.15.193:6815","nonce":32639}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":32639},{"type":"v1","addr":"172.21.15.193:6813","nonce":32639}]},"public_addr":"172.21.15.193:6809/32639","cluster_addr":"172.21.15.193:6811/32639","heartbeat_back_addr":"172.21.15.193:6815/32639","heartbeat_front_addr":"172.21.15.193:6813/32639","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-18T15:17:53.927 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-08-18T15:17:53.928 INFO:tasks.ceph:Creating RBD pool 2024-08-18T15:17:53.928 DEBUG:teuthology.orchestra.run.smithi193:> sudo ceph --cluster c2 osd pool create rbd 8 2024-08-18T15:17:54.305 INFO:teuthology.orchestra.run.smithi193.stderr:pool 'rbd' created 2024-08-18T15:17:54.319 DEBUG:teuthology.orchestra.run.smithi193:> rbd --cluster c2 pool init rbd 2024-08-18T15:17:54.407 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:17:54.407 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:17:57.323 INFO:tasks.ceph:Starting mds daemons in cluster c2... 2024-08-18T15:17:57.324 INFO:tasks.ceph:Waiting until c2 daemons up and pgs clean... 2024-08-18T15:17:57.324 INFO:tasks.ceph.ceph_manager.c2:waiting for mgr available 2024-08-18T15:17:57.324 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 mgr dump --format=json 2024-08-18T15:17:57.325 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-08-18T15:17:57.660 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:17:57.671 INFO:teuthology.orchestra.run.smithi193.stdout:{"epoch":5,"flags":0,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":32302},{"type":"v1","addr":"172.21.15.193:6825","nonce":32302}]},"active_addr":"172.21.15.193:6825/32302","active_change":"2024-08-18T15:17:50.163284+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.193:0","nonce":799592788}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.193:0","nonce":2561518601}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.193:0","nonce":2984743993}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.193:0","nonce":2026609515}]}]} 2024-08-18T15:17:57.673 INFO:tasks.ceph.ceph_manager.c2:mgr available! 2024-08-18T15:17:57.673 INFO:tasks.ceph.ceph_manager.c2:waiting for all up 2024-08-18T15:17:57.673 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2024-08-18T15:17:57.950 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:17:57.950 INFO:teuthology.orchestra.run.smithi193.stdout:{"epoch":12,"fsid":"af7271d7-3f51-4045-9c51-62f690689cce","created":"2024-08-18T15:17:45.791736+0000","modified":"2024-08-18T15:17:57.309951+0000","last_up_change":"2024-08-18T15:17:53.268568+0000","last_in_change":"2024-08-18T15:17:46.633468+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-18T15:17:54.196761+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-08-18T15:17:54.225750+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":12,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0c8aafee-6ffe-4ac4-99de-b7cafefa9bb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":32640},{"type":"v1","addr":"172.21.15.193:6817","nonce":32640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":32640},{"type":"v1","addr":"172.21.15.193:6819","nonce":32640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":32640},{"type":"v1","addr":"172.21.15.193:6823","nonce":32640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":32640},{"type":"v1","addr":"172.21.15.193:6821","nonce":32640}]},"public_addr":"172.21.15.193:6817/32640","cluster_addr":"172.21.15.193:6819/32640","heartbeat_back_addr":"172.21.15.193:6823/32640","heartbeat_front_addr":"172.21.15.193:6821/32640","state":["exists","up"]},{"osd":1,"uuid":"4daab6f9-7d9e-4f48-91ac-8fac4ff3e454","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":32638},{"type":"v1","addr":"172.21.15.193:6801","nonce":32638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":32638},{"type":"v1","addr":"172.21.15.193:6803","nonce":32638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":32638},{"type":"v1","addr":"172.21.15.193:6807","nonce":32638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":32638},{"type":"v1","addr":"172.21.15.193:6805","nonce":32638}]},"public_addr":"172.21.15.193:6801/32638","cluster_addr":"172.21.15.193:6803/32638","heartbeat_back_addr":"172.21.15.193:6807/32638","heartbeat_front_addr":"172.21.15.193:6805/32638","state":["exists","up"]},{"osd":2,"uuid":"16749f2c-e694-4263-b058-a9da51dec43b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":32639},{"type":"v1","addr":"172.21.15.193:6809","nonce":32639}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":32639},{"type":"v1","addr":"172.21.15.193:6811","nonce":32639}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":32639},{"type":"v1","addr":"172.21.15.193:6815","nonce":32639}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":32639},{"type":"v1","addr":"172.21.15.193:6813","nonce":32639}]},"public_addr":"172.21.15.193:6809/32639","cluster_addr":"172.21.15.193:6811/32639","heartbeat_back_addr":"172.21.15.193:6815/32639","heartbeat_front_addr":"172.21.15.193:6813/32639","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-18T15:17:51.040628+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-18T15:17:50.991487+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-18T15:17:50.996027+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-18T15:17:57.963 INFO:tasks.ceph.ceph_manager.c2:all up! 2024-08-18T15:17:57.963 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2024-08-18T15:17:58.242 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:17:58.242 INFO:teuthology.orchestra.run.smithi193.stdout:{"epoch":12,"fsid":"af7271d7-3f51-4045-9c51-62f690689cce","created":"2024-08-18T15:17:45.791736+0000","modified":"2024-08-18T15:17:57.309951+0000","last_up_change":"2024-08-18T15:17:53.268568+0000","last_in_change":"2024-08-18T15:17:46.633468+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-18T15:17:54.196761+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-08-18T15:17:54.225750+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":12,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0c8aafee-6ffe-4ac4-99de-b7cafefa9bb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":32640},{"type":"v1","addr":"172.21.15.193:6817","nonce":32640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":32640},{"type":"v1","addr":"172.21.15.193:6819","nonce":32640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":32640},{"type":"v1","addr":"172.21.15.193:6823","nonce":32640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":32640},{"type":"v1","addr":"172.21.15.193:6821","nonce":32640}]},"public_addr":"172.21.15.193:6817/32640","cluster_addr":"172.21.15.193:6819/32640","heartbeat_back_addr":"172.21.15.193:6823/32640","heartbeat_front_addr":"172.21.15.193:6821/32640","state":["exists","up"]},{"osd":1,"uuid":"4daab6f9-7d9e-4f48-91ac-8fac4ff3e454","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":32638},{"type":"v1","addr":"172.21.15.193:6801","nonce":32638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":32638},{"type":"v1","addr":"172.21.15.193:6803","nonce":32638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":32638},{"type":"v1","addr":"172.21.15.193:6807","nonce":32638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":32638},{"type":"v1","addr":"172.21.15.193:6805","nonce":32638}]},"public_addr":"172.21.15.193:6801/32638","cluster_addr":"172.21.15.193:6803/32638","heartbeat_back_addr":"172.21.15.193:6807/32638","heartbeat_front_addr":"172.21.15.193:6805/32638","state":["exists","up"]},{"osd":2,"uuid":"16749f2c-e694-4263-b058-a9da51dec43b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":32639},{"type":"v1","addr":"172.21.15.193:6809","nonce":32639}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":32639},{"type":"v1","addr":"172.21.15.193:6811","nonce":32639}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":32639},{"type":"v1","addr":"172.21.15.193:6815","nonce":32639}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":32639},{"type":"v1","addr":"172.21.15.193:6813","nonce":32639}]},"public_addr":"172.21.15.193:6809/32639","cluster_addr":"172.21.15.193:6811/32639","heartbeat_back_addr":"172.21.15.193:6815/32639","heartbeat_front_addr":"172.21.15.193:6813/32639","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-18T15:17:51.040628+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-18T15:17:50.991487+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-18T15:17:50.996027+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-18T15:17:58.255 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.0 flush_pg_stats 2024-08-18T15:17:58.255 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.1 flush_pg_stats 2024-08-18T15:17:58.256 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.2 flush_pg_stats 2024-08-18T15:17:58.392 INFO:teuthology.orchestra.run.smithi193.stdout:34359738371 2024-08-18T15:17:58.392 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2024-08-18T15:17:58.409 INFO:teuthology.orchestra.run.smithi193.stdout:34359738372 2024-08-18T15:17:58.409 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2024-08-18T15:17:58.410 INFO:teuthology.orchestra.run.smithi193.stdout:34359738372 2024-08-18T15:17:58.411 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2024-08-18T15:17:58.700 INFO:teuthology.orchestra.run.smithi193.stdout:34359738369 2024-08-18T15:17:58.713 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738369 for osd.0 2024-08-18T15:17:58.769 INFO:teuthology.orchestra.run.smithi193.stdout:34359738370 2024-08-18T15:17:58.776 INFO:teuthology.orchestra.run.smithi193.stdout:34359738370 2024-08-18T15:17:58.782 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738372 got 34359738370 for osd.1 2024-08-18T15:17:58.789 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738372 got 34359738370 for osd.2 2024-08-18T15:17:59.715 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2024-08-18T15:17:59.784 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2024-08-18T15:17:59.789 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2024-08-18T15:18:00.045 INFO:teuthology.orchestra.run.smithi193.stdout:34359738369 2024-08-18T15:18:00.063 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738369 for osd.0 2024-08-18T15:18:00.096 INFO:teuthology.orchestra.run.smithi193.stdout:34359738370 2024-08-18T15:18:00.111 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738372 got 34359738370 for osd.1 2024-08-18T15:18:00.148 INFO:teuthology.orchestra.run.smithi193.stdout:34359738370 2024-08-18T15:18:00.160 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738372 got 34359738370 for osd.2 2024-08-18T15:18:01.064 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2024-08-18T15:18:01.112 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2024-08-18T15:18:01.162 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2024-08-18T15:18:01.390 INFO:teuthology.orchestra.run.smithi193.stdout:34359738371 2024-08-18T15:18:01.403 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738371 for osd.0 2024-08-18T15:18:01.404 DEBUG:teuthology.parallel:result is None 2024-08-18T15:18:01.470 INFO:teuthology.orchestra.run.smithi193.stdout:34359738372 2024-08-18T15:18:01.482 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738372 got 34359738372 for osd.1 2024-08-18T15:18:01.482 DEBUG:teuthology.parallel:result is None 2024-08-18T15:18:01.498 INFO:teuthology.orchestra.run.smithi193.stdout:34359738372 2024-08-18T15:18:01.511 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738372 got 34359738372 for osd.2 2024-08-18T15:18:01.511 DEBUG:teuthology.parallel:result is None 2024-08-18T15:18:01.511 INFO:tasks.ceph.ceph_manager.c2:waiting for clean 2024-08-18T15:18:01.511 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2024-08-18T15:18:01.828 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:18:01.829 INFO:teuthology.orchestra.run.smithi193.stderr:dumped all 2024-08-18T15:18:01.841 INFO:teuthology.orchestra.run.smithi193.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2024-08-18T15:18:00.775328+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":82680,"kb_used_data":1476,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":314490120,"statfs":{"total":322122547200,"available":322037882880,"internally_reserved":0,"allocated":1511424,"data_stored":1269633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4782,"internal_metadata":82963794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.604335"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-08-18T15:17:57.319874+0000","last_change":"2024-08-18T15:17:57.319874+0000","last_active":"2024-08-18T15:17:57.319874+0000","last_peered":"2024-08-18T15:17:57.319874+0000","last_clean":"2024-08-18T15:17:57.319874+0000","last_became_active":"2024-08-18T15:17:55.313421+0000","last_became_peered":"2024-08-18T15:17:55.313421+0000","last_unstale":"2024-08-18T15:17:57.319874+0000","last_undegraded":"2024-08-18T15:17:57.319874+0000","last_fullsized":"2024-08-18T15:17:57.319874+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-18T15:17:54.297862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-18T15:17:54.297862+0000","last_clean_scrub_stamp":"2024-08-18T15:17:54.297862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T21:16:29.231385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084604899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-08-18T15:17:57.319464+0000","last_change":"2024-08-18T15:17:57.319464+0000","last_active":"2024-08-18T15:17:57.319464+0000","last_peered":"2024-08-18T15:17:57.319464+0000","last_clean":"2024-08-18T15:17:57.319464+0000","last_became_active":"2024-08-18T15:17:55.314641+0000","last_became_peered":"2024-08-18T15:17:55.314641+0000","last_unstale":"2024-08-18T15:17:57.319464+0000","last_undegraded":"2024-08-18T15:17:57.319464+0000","last_fullsized":"2024-08-18T15:17:57.319464+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-18T15:17:54.297862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-18T15:17:54.297862+0000","last_clean_scrub_stamp":"2024-08-18T15:17:54.297862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T19:27:19.288997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064636900000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-08-18T15:17:57.319510+0000","last_change":"2024-08-18T15:17:57.319510+0000","last_active":"2024-08-18T15:17:57.319510+0000","last_peered":"2024-08-18T15:17:57.319510+0000","last_clean":"2024-08-18T15:17:57.319510+0000","last_became_active":"2024-08-18T15:17:55.313161+0000","last_became_peered":"2024-08-18T15:17:55.313161+0000","last_unstale":"2024-08-18T15:17:57.319510+0000","last_undegraded":"2024-08-18T15:17:57.319510+0000","last_fullsized":"2024-08-18T15:17:57.319510+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-18T15:17:54.297862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-18T15:17:54.297862+0000","last_clean_scrub_stamp":"2024-08-18T15:17:54.297862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T23:02:32.356681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000573176,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-08-18T15:17:57.319299+0000","last_change":"2024-08-18T15:17:57.319299+0000","last_active":"2024-08-18T15:17:57.319299+0000","last_peered":"2024-08-18T15:17:57.319299+0000","last_clean":"2024-08-18T15:17:57.319299+0000","last_became_active":"2024-08-18T15:17:55.314297+0000","last_became_peered":"2024-08-18T15:17:55.314297+0000","last_unstale":"2024-08-18T15:17:57.319299+0000","last_undegraded":"2024-08-18T15:17:57.319299+0000","last_fullsized":"2024-08-18T15:17:57.319299+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-18T15:17:54.297862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-18T15:17:54.297862+0000","last_clean_scrub_stamp":"2024-08-18T15:17:54.297862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T15:48:49.396230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052664900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"12'2","reported_seq":17,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-08-18T15:17:57.318303+0000","last_change":"2024-08-18T15:17:57.317711+0000","last_active":"2024-08-18T15:17:57.318303+0000","last_peered":"2024-08-18T15:17:57.318303+0000","last_clean":"2024-08-18T15:17:57.318303+0000","last_became_active":"2024-08-18T15:17:55.312239+0000","last_became_peered":"2024-08-18T15:17:55.312239+0000","last_unstale":"2024-08-18T15:17:57.318303+0000","last_undegraded":"2024-08-18T15:17:57.318303+0000","last_fullsized":"2024-08-18T15:17:57.318303+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-18T15:17:54.297862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-18T15:17:54.297862+0000","last_clean_scrub_stamp":"2024-08-18T15:17:54.297862+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T00:32:43.383902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081905400000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-08-18T15:17:57.314207+0000","last_change":"2024-08-18T15:17:57.314207+0000","last_active":"2024-08-18T15:17:57.314207+0000","last_peered":"2024-08-18T15:17:57.314207+0000","last_clean":"2024-08-18T15:17:57.314207+0000","last_became_active":"2024-08-18T15:17:55.313330+0000","last_became_peered":"2024-08-18T15:17:55.313330+0000","last_unstale":"2024-08-18T15:17:57.314207+0000","last_undegraded":"2024-08-18T15:17:57.314207+0000","last_fullsized":"2024-08-18T15:17:57.314207+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-18T15:17:54.297862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-18T15:17:54.297862+0000","last_clean_scrub_stamp":"2024-08-18T15:17:54.297862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T16:44:11.308468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000332417,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-08-18T15:17:57.314276+0000","last_change":"2024-08-18T15:17:57.314276+0000","last_active":"2024-08-18T15:17:57.314276+0000","last_peered":"2024-08-18T15:17:57.314276+0000","last_clean":"2024-08-18T15:17:57.314276+0000","last_became_active":"2024-08-18T15:17:55.313114+0000","last_became_peered":"2024-08-18T15:17:55.313114+0000","last_unstale":"2024-08-18T15:17:57.314276+0000","last_undegraded":"2024-08-18T15:17:57.314276+0000","last_fullsized":"2024-08-18T15:17:57.314276+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-18T15:17:54.297862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-18T15:17:54.297862+0000","last_clean_scrub_stamp":"2024-08-18T15:17:54.297862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T19:09:30.109642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035386899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"10'1","reported_seq":16,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-08-18T15:17:57.319215+0000","last_change":"2024-08-18T15:17:57.319215+0000","last_active":"2024-08-18T15:17:57.319215+0000","last_peered":"2024-08-18T15:17:57.319215+0000","last_clean":"2024-08-18T15:17:57.319215+0000","last_became_active":"2024-08-18T15:17:55.312608+0000","last_became_peered":"2024-08-18T15:17:55.312608+0000","last_unstale":"2024-08-18T15:17:57.319215+0000","last_undegraded":"2024-08-18T15:17:57.319215+0000","last_fullsized":"2024-08-18T15:17:57.319215+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-18T15:17:54.297862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-18T15:17:54.297862+0000","last_clean_scrub_stamp":"2024-08-18T15:17:54.297862+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T23:13:28.791986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049375799999999996,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"10'97","reported_seq":138,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-08-18T15:17:57.318809+0000","last_change":"2024-08-18T15:17:55.315756+0000","last_active":"2024-08-18T15:17:57.318809+0000","last_peered":"2024-08-18T15:17:57.318809+0000","last_clean":"2024-08-18T15:17:57.318809+0000","last_became_active":"2024-08-18T15:17:55.314818+0000","last_became_peered":"2024-08-18T15:17:55.314818+0000","last_unstale":"2024-08-18T15:17:57.318809+0000","last_undegraded":"2024-08-18T15:17:57.318809+0000","last_fullsized":"2024-08-18T15:17:57.318809+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-18T15:17:54.297862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-18T15:17:54.297862+0000","last_clean_scrub_stamp":"2024-08-18T15:17:54.297862+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T02:49:56.388676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27132,"kb_used_data":104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830468,"statfs":{"total":107374182400,"available":107346399232,"internally_reserved":0,"allocated":106496,"data_stored":29990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30299999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.374}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27712,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829888,"statfs":{"total":107374182400,"available":107345805312,"internally_reserved":0,"allocated":704512,"data_stored":620377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26200000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41599999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27836,"kb_used_data":684,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829764,"statfs":{"total":107374182400,"available":107345678336,"internally_reserved":0,"allocated":700416,"data_stored":619266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48099999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.432}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-18T15:18:01.842 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2024-08-18T15:18:02.118 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:18:02.119 INFO:teuthology.orchestra.run.smithi193.stderr:dumped all 2024-08-18T15:18:02.130 INFO:teuthology.orchestra.run.smithi193.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2024-08-18T15:18:00.775328+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":82680,"kb_used_data":1476,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":314490120,"statfs":{"total":322122547200,"available":322037882880,"internally_reserved":0,"allocated":1511424,"data_stored":1269633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4782,"internal_metadata":82963794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.604335"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-08-18T15:17:57.319874+0000","last_change":"2024-08-18T15:17:57.319874+0000","last_active":"2024-08-18T15:17:57.319874+0000","last_peered":"2024-08-18T15:17:57.319874+0000","last_clean":"2024-08-18T15:17:57.319874+0000","last_became_active":"2024-08-18T15:17:55.313421+0000","last_became_peered":"2024-08-18T15:17:55.313421+0000","last_unstale":"2024-08-18T15:17:57.319874+0000","last_undegraded":"2024-08-18T15:17:57.319874+0000","last_fullsized":"2024-08-18T15:17:57.319874+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-18T15:17:54.297862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-18T15:17:54.297862+0000","last_clean_scrub_stamp":"2024-08-18T15:17:54.297862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T21:16:29.231385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084604899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-08-18T15:17:57.319464+0000","last_change":"2024-08-18T15:17:57.319464+0000","last_active":"2024-08-18T15:17:57.319464+0000","last_peered":"2024-08-18T15:17:57.319464+0000","last_clean":"2024-08-18T15:17:57.319464+0000","last_became_active":"2024-08-18T15:17:55.314641+0000","last_became_peered":"2024-08-18T15:17:55.314641+0000","last_unstale":"2024-08-18T15:17:57.319464+0000","last_undegraded":"2024-08-18T15:17:57.319464+0000","last_fullsized":"2024-08-18T15:17:57.319464+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-18T15:17:54.297862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-18T15:17:54.297862+0000","last_clean_scrub_stamp":"2024-08-18T15:17:54.297862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T19:27:19.288997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064636900000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-08-18T15:17:57.319510+0000","last_change":"2024-08-18T15:17:57.319510+0000","last_active":"2024-08-18T15:17:57.319510+0000","last_peered":"2024-08-18T15:17:57.319510+0000","last_clean":"2024-08-18T15:17:57.319510+0000","last_became_active":"2024-08-18T15:17:55.313161+0000","last_became_peered":"2024-08-18T15:17:55.313161+0000","last_unstale":"2024-08-18T15:17:57.319510+0000","last_undegraded":"2024-08-18T15:17:57.319510+0000","last_fullsized":"2024-08-18T15:17:57.319510+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-18T15:17:54.297862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-18T15:17:54.297862+0000","last_clean_scrub_stamp":"2024-08-18T15:17:54.297862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T23:02:32.356681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000573176,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-08-18T15:17:57.319299+0000","last_change":"2024-08-18T15:17:57.319299+0000","last_active":"2024-08-18T15:17:57.319299+0000","last_peered":"2024-08-18T15:17:57.319299+0000","last_clean":"2024-08-18T15:17:57.319299+0000","last_became_active":"2024-08-18T15:17:55.314297+0000","last_became_peered":"2024-08-18T15:17:55.314297+0000","last_unstale":"2024-08-18T15:17:57.319299+0000","last_undegraded":"2024-08-18T15:17:57.319299+0000","last_fullsized":"2024-08-18T15:17:57.319299+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-18T15:17:54.297862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-18T15:17:54.297862+0000","last_clean_scrub_stamp":"2024-08-18T15:17:54.297862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T15:48:49.396230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052664900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"12'2","reported_seq":17,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-08-18T15:17:57.318303+0000","last_change":"2024-08-18T15:17:57.317711+0000","last_active":"2024-08-18T15:17:57.318303+0000","last_peered":"2024-08-18T15:17:57.318303+0000","last_clean":"2024-08-18T15:17:57.318303+0000","last_became_active":"2024-08-18T15:17:55.312239+0000","last_became_peered":"2024-08-18T15:17:55.312239+0000","last_unstale":"2024-08-18T15:17:57.318303+0000","last_undegraded":"2024-08-18T15:17:57.318303+0000","last_fullsized":"2024-08-18T15:17:57.318303+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-18T15:17:54.297862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-18T15:17:54.297862+0000","last_clean_scrub_stamp":"2024-08-18T15:17:54.297862+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T00:32:43.383902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081905400000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-08-18T15:17:57.314207+0000","last_change":"2024-08-18T15:17:57.314207+0000","last_active":"2024-08-18T15:17:57.314207+0000","last_peered":"2024-08-18T15:17:57.314207+0000","last_clean":"2024-08-18T15:17:57.314207+0000","last_became_active":"2024-08-18T15:17:55.313330+0000","last_became_peered":"2024-08-18T15:17:55.313330+0000","last_unstale":"2024-08-18T15:17:57.314207+0000","last_undegraded":"2024-08-18T15:17:57.314207+0000","last_fullsized":"2024-08-18T15:17:57.314207+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-18T15:17:54.297862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-18T15:17:54.297862+0000","last_clean_scrub_stamp":"2024-08-18T15:17:54.297862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T16:44:11.308468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000332417,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-08-18T15:17:57.314276+0000","last_change":"2024-08-18T15:17:57.314276+0000","last_active":"2024-08-18T15:17:57.314276+0000","last_peered":"2024-08-18T15:17:57.314276+0000","last_clean":"2024-08-18T15:17:57.314276+0000","last_became_active":"2024-08-18T15:17:55.313114+0000","last_became_peered":"2024-08-18T15:17:55.313114+0000","last_unstale":"2024-08-18T15:17:57.314276+0000","last_undegraded":"2024-08-18T15:17:57.314276+0000","last_fullsized":"2024-08-18T15:17:57.314276+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-18T15:17:54.297862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-18T15:17:54.297862+0000","last_clean_scrub_stamp":"2024-08-18T15:17:54.297862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T19:09:30.109642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035386899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"10'1","reported_seq":16,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-08-18T15:17:57.319215+0000","last_change":"2024-08-18T15:17:57.319215+0000","last_active":"2024-08-18T15:17:57.319215+0000","last_peered":"2024-08-18T15:17:57.319215+0000","last_clean":"2024-08-18T15:17:57.319215+0000","last_became_active":"2024-08-18T15:17:55.312608+0000","last_became_peered":"2024-08-18T15:17:55.312608+0000","last_unstale":"2024-08-18T15:17:57.319215+0000","last_undegraded":"2024-08-18T15:17:57.319215+0000","last_fullsized":"2024-08-18T15:17:57.319215+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-18T15:17:54.297862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-18T15:17:54.297862+0000","last_clean_scrub_stamp":"2024-08-18T15:17:54.297862+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T23:13:28.791986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049375799999999996,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"10'97","reported_seq":138,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-08-18T15:17:57.318809+0000","last_change":"2024-08-18T15:17:55.315756+0000","last_active":"2024-08-18T15:17:57.318809+0000","last_peered":"2024-08-18T15:17:57.318809+0000","last_clean":"2024-08-18T15:17:57.318809+0000","last_became_active":"2024-08-18T15:17:55.314818+0000","last_became_peered":"2024-08-18T15:17:55.314818+0000","last_unstale":"2024-08-18T15:17:57.318809+0000","last_undegraded":"2024-08-18T15:17:57.318809+0000","last_fullsized":"2024-08-18T15:17:57.318809+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-18T15:17:54.297862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-18T15:17:54.297862+0000","last_clean_scrub_stamp":"2024-08-18T15:17:54.297862+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T02:49:56.388676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27132,"kb_used_data":104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830468,"statfs":{"total":107374182400,"available":107346399232,"internally_reserved":0,"allocated":106496,"data_stored":29990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30299999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.374}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27712,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829888,"statfs":{"total":107374182400,"available":107345805312,"internally_reserved":0,"allocated":704512,"data_stored":620377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26200000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41599999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27836,"kb_used_data":684,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829764,"statfs":{"total":107374182400,"available":107345678336,"internally_reserved":0,"allocated":700416,"data_stored":619266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48099999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.432}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-18T15:18:02.131 INFO:tasks.ceph.ceph_manager.c2:clean! 2024-08-18T15:18:02.131 INFO:tasks.ceph:Waiting until ceph cluster c2 is healthy... 2024-08-18T15:18:02.132 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy 2024-08-18T15:18:02.132 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 health --format=json 2024-08-18T15:18:02.443 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:18:02.444 INFO:teuthology.orchestra.run.smithi193.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-08-18T15:18:02.456 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy done 2024-08-18T15:18:02.456 INFO:teuthology.run_tasks:Running task rgw... 2024-08-18T15:18:02.470 DEBUG:tasks.rgw:config is {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}} 2024-08-18T15:18:02.470 DEBUG:tasks.rgw:client list is dict_keys(['c1.client.0', 'c1.client.1', 'c2.client.0', 'c2.client.1']) 2024-08-18T15:18:02.470 INFO:tasks.rgw:Creating data pools 2024-08-18T15:18:02.471 DEBUG:tasks.rgw:Obtaining remote for client c1.client.0 2024-08-18T15:18:02.471 DEBUG:teuthology.orchestra.run.smithi134:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2024-08-18T15:18:02.938 INFO:teuthology.orchestra.run.smithi134.stderr:pool 'default.rgw.buckets.data' created 2024-08-18T15:18:02.956 DEBUG:teuthology.orchestra.run.smithi134:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2024-08-18T15:18:03.945 INFO:teuthology.orchestra.run.smithi134.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2024-08-18T15:18:03.975 DEBUG:teuthology.orchestra.run.smithi134:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2024-08-18T15:18:04.943 INFO:teuthology.orchestra.run.smithi134.stderr:pool 'default.rgw.buckets.index' created 2024-08-18T15:18:04.963 DEBUG:teuthology.orchestra.run.smithi134:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2024-08-18T15:18:05.947 INFO:teuthology.orchestra.run.smithi134.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2024-08-18T15:18:05.980 DEBUG:tasks.rgw:Obtaining remote for client c1.client.1 2024-08-18T15:18:05.980 DEBUG:teuthology.orchestra.run.smithi134:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2024-08-18T15:18:06.263 INFO:teuthology.orchestra.run.smithi134.stderr:pool 'default.rgw.buckets.data' already exists 2024-08-18T15:18:06.272 DEBUG:teuthology.orchestra.run.smithi134:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2024-08-18T15:18:06.946 INFO:teuthology.orchestra.run.smithi134.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2024-08-18T15:18:06.960 DEBUG:teuthology.orchestra.run.smithi134:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2024-08-18T15:18:07.244 INFO:teuthology.orchestra.run.smithi134.stderr:pool 'default.rgw.buckets.index' already exists 2024-08-18T15:18:07.257 DEBUG:teuthology.orchestra.run.smithi134:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2024-08-18T15:18:07.951 INFO:teuthology.orchestra.run.smithi134.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2024-08-18T15:18:07.964 DEBUG:tasks.rgw:Obtaining remote for client c2.client.0 2024-08-18T15:18:07.965 DEBUG:teuthology.orchestra.run.smithi193:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2024-08-18T15:18:08.788 INFO:teuthology.orchestra.run.smithi193.stderr:pool 'default.rgw.buckets.data' created 2024-08-18T15:18:08.810 DEBUG:teuthology.orchestra.run.smithi193:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2024-08-18T15:18:09.789 INFO:teuthology.orchestra.run.smithi193.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2024-08-18T15:18:09.819 DEBUG:teuthology.orchestra.run.smithi193:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2024-08-18T15:18:10.798 INFO:teuthology.orchestra.run.smithi193.stderr:pool 'default.rgw.buckets.index' created 2024-08-18T15:18:10.822 DEBUG:teuthology.orchestra.run.smithi193:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2024-08-18T15:18:11.803 INFO:teuthology.orchestra.run.smithi193.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2024-08-18T15:18:11.832 DEBUG:tasks.rgw:Obtaining remote for client c2.client.1 2024-08-18T15:18:11.832 DEBUG:teuthology.orchestra.run.smithi193:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2024-08-18T15:18:12.110 INFO:teuthology.orchestra.run.smithi193.stderr:pool 'default.rgw.buckets.data' already exists 2024-08-18T15:18:12.123 DEBUG:teuthology.orchestra.run.smithi193:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2024-08-18T15:18:12.803 INFO:teuthology.orchestra.run.smithi193.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2024-08-18T15:18:12.817 DEBUG:teuthology.orchestra.run.smithi193:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2024-08-18T15:18:13.096 INFO:teuthology.orchestra.run.smithi193.stderr:pool 'default.rgw.buckets.index' already exists 2024-08-18T15:18:13.109 DEBUG:teuthology.orchestra.run.smithi193:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2024-08-18T15:18:13.803 INFO:teuthology.orchestra.run.smithi193.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2024-08-18T15:18:13.818 DEBUG:tasks.rgw:Pools created 2024-08-18T15:18:13.821 INFO:tasks.rgw:Configuring compression type = random 2024-08-18T15:18:13.821 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'list'] 2024-08-18T15:18:13.821 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'list'] 2024-08-18T15:18:13.821 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user list 2024-08-18T15:18:13.918 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:18:13.919 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:18:13.931 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:13.930+0000 7fd42c60f880 20 rados->read ofs=0 len=0 2024-08-18T15:18:14.974 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:14.973+0000 7fd42c60f880 20 rados_obj.operate() r=-2 bl.length=0 2024-08-18T15:18:14.974 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:14.973+0000 7fd42c60f880 20 realm 2024-08-18T15:18:14.975 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:14.973+0000 7fd42c60f880 20 rados->read ofs=0 len=0 2024-08-18T15:18:14.976 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:14.974+0000 7fd42c60f880 20 rados_obj.operate() r=-2 bl.length=0 2024-08-18T15:18:14.976 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:14.974+0000 7fd42c60f880 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-08-18T15:18:14.976 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:14.974+0000 7fd42c60f880 20 rados->read ofs=0 len=0 2024-08-18T15:18:14.976 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:14.975+0000 7fd42c60f880 20 rados_obj.operate() r=-2 bl.length=0 2024-08-18T15:18:14.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:14.975+0000 7fd42c60f880 20 rados->read ofs=0 len=0 2024-08-18T15:18:14.985 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:14.984+0000 7fd42c60f880 20 rados_obj.operate() r=-2 bl.length=0 2024-08-18T15:18:14.985 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:14.984+0000 7fd42c60f880 20 rados->read ofs=0 len=0 2024-08-18T15:18:14.986 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:14.984+0000 7fd42c60f880 20 rados_obj.operate() r=-2 bl.length=0 2024-08-18T15:18:14.986 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:14.984+0000 7fd42c60f880 20 rados->read ofs=0 len=0 2024-08-18T15:18:14.986 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:14.984+0000 7fd42c60f880 20 rados_obj.operate() r=-2 bl.length=0 2024-08-18T15:18:14.986 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:14.984+0000 7fd42c60f880 10 Creating default zonegroup 2024-08-18T15:18:14.986 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:14.985+0000 7fd42c60f880 10 couldn't find old data placement pools config, setting up new ones for the zone 2024-08-18T15:18:14.994 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:14.994+0000 7fd42c60f880 10 WARNING: driver->list_zones() returned r=-2 2024-08-18T15:18:14.995 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:14.994+0000 7fd42c60f880 20 rados->read ofs=0 len=0 2024-08-18T15:18:14.995 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:14.994+0000 7fd42c60f880 20 rados_obj.operate() r=-2 bl.length=0 2024-08-18T15:18:14.999 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:14.998+0000 7fd42c60f880 20 rados->read ofs=0 len=0 2024-08-18T15:18:14.999 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:14.998+0000 7fd42c60f880 20 rados_obj.operate() r=-2 bl.length=0 2024-08-18T15:18:14.999 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:14.998+0000 7fd42c60f880 10 could not read realm id: (2) No such file or directory 2024-08-18T15:18:15.000 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:14.998+0000 7fd42c60f880 10 WARNING: failed to set zone as default, r=-22 2024-08-18T15:18:15.000 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:14.998+0000 7fd42c60f880 20 rados->read ofs=0 len=0 2024-08-18T15:18:15.000 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:14.998+0000 7fd42c60f880 20 rados_obj.operate() r=-2 bl.length=0 2024-08-18T15:18:15.003 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:15.003+0000 7fd42c60f880 20 rados->read ofs=0 len=0 2024-08-18T15:18:15.004 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:15.003+0000 7fd42c60f880 20 rados_obj.operate() r=0 bl.length=874 2024-08-18T15:18:15.004 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:15.003+0000 7fd42c60f880 20 rados->read ofs=0 len=0 2024-08-18T15:18:15.005 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:15.004+0000 7fd42c60f880 20 rados_obj.operate() r=0 bl.length=420 2024-08-18T15:18:15.005 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:15.004+0000 7fd42c60f880 10 Using default name default 2024-08-18T15:18:15.005 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:15.004+0000 7fd42c60f880 20 rados->read ofs=0 len=0 2024-08-18T15:18:15.005 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:15.004+0000 7fd42c60f880 20 rados_obj.operate() r=0 bl.length=46 2024-08-18T15:18:15.005 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:15.004+0000 7fd42c60f880 20 rados->read ofs=0 len=0 2024-08-18T15:18:15.005 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:15.004+0000 7fd42c60f880 20 rados_obj.operate() r=0 bl.length=874 2024-08-18T15:18:15.005 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:15.004+0000 7fd42c60f880 20 zone default found 2024-08-18T15:18:15.005 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:15.004+0000 7fd42c60f880 4 Realm: () 2024-08-18T15:18:15.006 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:15.004+0000 7fd42c60f880 4 ZoneGroup: default (4092a2de-f55e-4d20-bc95-323038e553e9) 2024-08-18T15:18:15.006 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:15.004+0000 7fd42c60f880 4 Zone: default (b7f3b7d5-73db-483f-bafe-9271ea0c228b) 2024-08-18T15:18:15.006 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:15.004+0000 7fd42c60f880 10 cannot find current period zonegroup using local zonegroup configuration 2024-08-18T15:18:15.006 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:15.004+0000 7fd42c60f880 20 zonegroup default 2024-08-18T15:18:15.006 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:15.004+0000 7fd42c60f880 20 rados->read ofs=0 len=0 2024-08-18T15:18:15.006 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:15.005+0000 7fd42c60f880 20 rados_obj.operate() r=-2 bl.length=0 2024-08-18T15:18:15.006 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:15.005+0000 7fd42c60f880 20 rados->read ofs=0 len=0 2024-08-18T15:18:16.990 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:16.990+0000 7fd42c60f880 20 rados_obj.operate() r=-2 bl.length=0 2024-08-18T15:18:16.991 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:16.990+0000 7fd42c60f880 20 rados->read ofs=0 len=0 2024-08-18T15:18:16.993 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:16.992+0000 7fd42c60f880 20 rados_obj.operate() r=-2 bl.length=0 2024-08-18T15:18:16.993 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:16.992+0000 7fd42c60f880 20 started sync module instance, tier type = 2024-08-18T15:18:16.993 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:16.992+0000 7fd42c60f880 20 started zone id=b7f3b7d5-73db-483f-bafe-9271ea0c228b (name=default) with tier type = 2024-08-18T15:18:19.035 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:19.034+0000 7fd42c60f880 20 add_watcher() i=0 2024-08-18T15:18:19.035 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:19.034+0000 7fd42c60f880 20 add_watcher() i=1 2024-08-18T15:18:19.035 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:19.034+0000 7fd42c60f880 20 add_watcher() i=2 2024-08-18T15:18:19.035 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:19.034+0000 7fd42c60f880 20 add_watcher() i=3 2024-08-18T15:18:19.035 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:19.034+0000 7fd42c60f880 20 add_watcher() i=4 2024-08-18T15:18:19.035 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:19.034+0000 7fd42c60f880 20 add_watcher() i=5 2024-08-18T15:18:19.035 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:19.034+0000 7fd42c60f880 20 add_watcher() i=6 2024-08-18T15:18:19.037 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:19.037+0000 7fd42c60f880 20 add_watcher() i=7 2024-08-18T15:18:19.037 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:19.037+0000 7fd42c60f880 2 all 8 watchers are set, enabling cache 2024-08-18T15:18:19.038 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:19.037+0000 7fd42c60f880 5 tl::expected, obj_version>, boost::system::error_code> logback_generations::read(const DoutPrefixProvider*, optional_yield):410: oid=data_loggenerations_metadata not found 2024-08-18T15:18:19.038 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:19.037+0000 7fd42c60f880 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.0 2024-08-18T15:18:19.038 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:19.037+0000 7fd42c60f880 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-08-18T15:18:19.039 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:19.037+0000 7fd42c60f880 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.0 does not exist 2024-08-18T15:18:19.039 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:19.037+0000 7fd42c60f880 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.1 2024-08-18T15:18:19.039 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:19.037+0000 7fd42c60f880 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-08-18T15:18:19.039 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:19.038+0000 7fd42c60f880 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.1 does not exist 2024-08-18T15:18:19.039 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:19.038+0000 7fd42c60f880 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.2 2024-08-18T15:18:19.039 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:19.038+0000 7fd42c60f880 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-08-18T15:18:19.039 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:19.038+0000 7fd42c60f880 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.2 does not exist 2024-08-18T15:18:19.039 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:19.038+0000 7fd42c60f880 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.3 2024-08-18T15:18:19.039 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:19.038+0000 7fd42c60f880 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-08-18T15:18:19.039 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:19.038+0000 7fd42c60f880 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.3 does not exist 2024-08-18T15:18:19.039 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:19.038+0000 7fd42c60f880 20 static int rgw::cls::fifo::FIFO::create(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):1210 entering 2024-08-18T15:18:19.040 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:19.039+0000 7fd42c60f880 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-08-18T15:18:19.048 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:19.047+0000 7fd40faa3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-08-18T15:18:19.048 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:19.047+0000 7fd42c60f880 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-08-18T15:18:19.048 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:19.047+0000 7fd42c60f880 20 check_secure_mon_conn(): mode 1 is insecure 2024-08-18T15:18:21.997 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:21.996+0000 7fd42c60f880 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2024-08-18T15:18:21.997 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:21.996+0000 7fd42c60f880 5 note: GC not initialized 2024-08-18T15:18:21.998 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:21.997+0000 7fd42c60f880 20 init_complete bucket index max shards: 11 2024-08-18T15:18:21.998 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:21.997+0000 7fd42c60f880 10 rgw notify: Started notification manager with: 1 workers 2024-08-18T15:18:21.998 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:21.997+0000 7fd40b29a640 20 reqs_thread_entry: start 2024-08-18T15:18:21.998 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:21.997+0000 7fd409296640 20 rgw notify: INFO: next queues processing will happen at: Sun Aug 18 15:18:52 2024 2024-08-18T15:18:21.998 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-08-18T15:18:21.998 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:21.997+0000 7fd408a95640 20 reqs_thread_entry: start 2024-08-18T15:18:22.020 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:22.020+0000 7fd40faa3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-08-18T15:18:22.023 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:22.022+0000 7fd42c60f880 20 remove_watcher() i=0 2024-08-18T15:18:22.023 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:22.022+0000 7fd42c60f880 2 removed watcher, disabling cache 2024-08-18T15:18:22.024 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:22.023+0000 7fd42c60f880 20 remove_watcher() i=1 2024-08-18T15:18:22.025 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:22.024+0000 7fd42c60f880 20 remove_watcher() i=2 2024-08-18T15:18:22.026 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:22.025+0000 7fd42c60f880 20 remove_watcher() i=3 2024-08-18T15:18:22.027 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:22.026+0000 7fd42c60f880 20 remove_watcher() i=4 2024-08-18T15:18:22.028 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:22.027+0000 7fd42c60f880 20 remove_watcher() i=5 2024-08-18T15:18:22.029 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:22.028+0000 7fd42c60f880 20 remove_watcher() i=6 2024-08-18T15:18:22.030 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:22.029+0000 7fd42c60f880 20 remove_watcher() i=7 2024-08-18T15:18:22.044 INFO:teuthology.orchestra.run.smithi134.stdout:[] 2024-08-18T15:18:22.049 DEBUG:tasks.util.rgw: json result: [] 2024-08-18T15:18:22.050 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-08-18T15:18:22.050 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-08-18T15:18:22.050 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2024-08-18T15:18:22.102 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:18:22.102 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:18:22.113 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:22.112+0000 7fe8a9015880 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-08-18T15:18:22.113 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:22.112+0000 7fe8a9015880 20 check_secure_mon_conn(): mode 1 is insecure 2024-08-18T15:18:22.115 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:22.115+0000 7fe888d39640 20 reqs_thread_entry: start 2024-08-18T15:18:22.122 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T15:18:22.123 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "b7f3b7d5-73db-483f-bafe-9271ea0c228b", 2024-08-18T15:18:22.123 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "default", 2024-08-18T15:18:22.123 INFO:teuthology.orchestra.run.smithi134.stdout: "domain_root": "default.rgw.meta:root", 2024-08-18T15:18:22.123 INFO:teuthology.orchestra.run.smithi134.stdout: "control_pool": "default.rgw.control", 2024-08-18T15:18:22.123 INFO:teuthology.orchestra.run.smithi134.stdout: "gc_pool": "default.rgw.log:gc", 2024-08-18T15:18:22.123 INFO:teuthology.orchestra.run.smithi134.stdout: "lc_pool": "default.rgw.log:lc", 2024-08-18T15:18:22.123 INFO:teuthology.orchestra.run.smithi134.stdout: "log_pool": "default.rgw.log", 2024-08-18T15:18:22.123 INFO:teuthology.orchestra.run.smithi134.stdout: "intent_log_pool": "default.rgw.log:intent", 2024-08-18T15:18:22.123 INFO:teuthology.orchestra.run.smithi134.stdout: "usage_log_pool": "default.rgw.log:usage", 2024-08-18T15:18:22.123 INFO:teuthology.orchestra.run.smithi134.stdout: "roles_pool": "default.rgw.meta:roles", 2024-08-18T15:18:22.123 INFO:teuthology.orchestra.run.smithi134.stdout: "reshard_pool": "default.rgw.log:reshard", 2024-08-18T15:18:22.123 INFO:teuthology.orchestra.run.smithi134.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2024-08-18T15:18:22.123 INFO:teuthology.orchestra.run.smithi134.stdout: "user_email_pool": "default.rgw.meta:users.email", 2024-08-18T15:18:22.124 INFO:teuthology.orchestra.run.smithi134.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2024-08-18T15:18:22.124 INFO:teuthology.orchestra.run.smithi134.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2024-08-18T15:18:22.124 INFO:teuthology.orchestra.run.smithi134.stdout: "otp_pool": "default.rgw.otp", 2024-08-18T15:18:22.124 INFO:teuthology.orchestra.run.smithi134.stdout: "system_key": { 2024-08-18T15:18:22.124 INFO:teuthology.orchestra.run.smithi134.stdout: "access_key": "", 2024-08-18T15:18:22.124 INFO:teuthology.orchestra.run.smithi134.stdout: "secret_key": "" 2024-08-18T15:18:22.124 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:18:22.124 INFO:teuthology.orchestra.run.smithi134.stdout: "placement_pools": [ 2024-08-18T15:18:22.124 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:18:22.124 INFO:teuthology.orchestra.run.smithi134.stdout: "key": "default-placement", 2024-08-18T15:18:22.124 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:18:22.124 INFO:teuthology.orchestra.run.smithi134.stdout: "index_pool": "default.rgw.buckets.index", 2024-08-18T15:18:22.124 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_classes": { 2024-08-18T15:18:22.124 INFO:teuthology.orchestra.run.smithi134.stdout: "STANDARD": { 2024-08-18T15:18:22.124 INFO:teuthology.orchestra.run.smithi134.stdout: "data_pool": "default.rgw.buckets.data", 2024-08-18T15:18:22.125 INFO:teuthology.orchestra.run.smithi134.stdout: "compression_type": "random" 2024-08-18T15:18:22.125 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:18:22.125 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:18:22.125 INFO:teuthology.orchestra.run.smithi134.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2024-08-18T15:18:22.125 INFO:teuthology.orchestra.run.smithi134.stdout: "index_type": 0, 2024-08-18T15:18:22.125 INFO:teuthology.orchestra.run.smithi134.stdout: "inline_data": true 2024-08-18T15:18:22.125 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:18:22.125 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:18:22.125 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T15:18:22.125 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_id": "", 2024-08-18T15:18:22.125 INFO:teuthology.orchestra.run.smithi134.stdout: "notif_pool": "default.rgw.log:notif" 2024-08-18T15:18:22.125 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T15:18:22.126 DEBUG:tasks.util.rgw: json result: {'id': 'b7f3b7d5-73db-483f-bafe-9271ea0c228b', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2024-08-18T15:18:22.126 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['user', 'list'] 2024-08-18T15:18:22.126 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'user', 'list'] 2024-08-18T15:18:22.127 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 user list 2024-08-18T15:18:22.169 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:18:22.169 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:18:22.182 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:22.179+0000 7f06176bb880 20 rados->read ofs=0 len=0 2024-08-18T15:18:22.182 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:22.179+0000 7f06176bb880 20 rados_obj.operate() r=-2 bl.length=0 2024-08-18T15:18:22.182 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:22.179+0000 7f06176bb880 20 realm 2024-08-18T15:18:22.182 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:22.179+0000 7f06176bb880 20 rados->read ofs=0 len=0 2024-08-18T15:18:22.182 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:22.180+0000 7f06176bb880 20 rados_obj.operate() r=-2 bl.length=0 2024-08-18T15:18:22.182 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:22.180+0000 7f06176bb880 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-08-18T15:18:22.182 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:22.180+0000 7f06176bb880 20 rados->read ofs=0 len=0 2024-08-18T15:18:22.183 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:22.180+0000 7f06176bb880 20 rados_obj.operate() r=-2 bl.length=0 2024-08-18T15:18:22.183 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:22.180+0000 7f06176bb880 20 rados->read ofs=0 len=0 2024-08-18T15:18:22.183 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:22.180+0000 7f06176bb880 20 rados_obj.operate() r=0 bl.length=46 2024-08-18T15:18:22.183 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:22.180+0000 7f06176bb880 20 rados->read ofs=0 len=0 2024-08-18T15:18:22.183 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:22.181+0000 7f06176bb880 20 rados_obj.operate() r=0 bl.length=890 2024-08-18T15:18:22.183 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:22.181+0000 7f06176bb880 20 searching for the correct realm 2024-08-18T15:18:22.189 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:22.187+0000 7f06176bb880 20 RGWRados::pool_iterate: got zonegroup_info.4092a2de-f55e-4d20-bc95-323038e553e9 2024-08-18T15:18:22.190 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:22.187+0000 7f06176bb880 20 RGWRados::pool_iterate: got zone_info.b7f3b7d5-73db-483f-bafe-9271ea0c228b 2024-08-18T15:18:22.190 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:22.187+0000 7f06176bb880 20 RGWRados::pool_iterate: got zone_names.default 2024-08-18T15:18:22.190 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:22.187+0000 7f06176bb880 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-08-18T15:18:22.190 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:22.187+0000 7f06176bb880 20 rados->read ofs=0 len=0 2024-08-18T15:18:22.190 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:22.187+0000 7f06176bb880 20 rados_obj.operate() r=-2 bl.length=0 2024-08-18T15:18:22.190 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:22.187+0000 7f06176bb880 20 rados->read ofs=0 len=0 2024-08-18T15:18:22.190 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:22.188+0000 7f06176bb880 20 rados_obj.operate() r=0 bl.length=46 2024-08-18T15:18:22.190 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:22.188+0000 7f06176bb880 20 rados->read ofs=0 len=0 2024-08-18T15:18:22.190 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:22.188+0000 7f06176bb880 20 rados_obj.operate() r=0 bl.length=420 2024-08-18T15:18:22.191 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:22.188+0000 7f06176bb880 20 zone default found 2024-08-18T15:18:22.191 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:22.188+0000 7f06176bb880 4 Realm: () 2024-08-18T15:18:22.191 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:22.188+0000 7f06176bb880 4 ZoneGroup: default (4092a2de-f55e-4d20-bc95-323038e553e9) 2024-08-18T15:18:22.191 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:22.188+0000 7f06176bb880 4 Zone: default (b7f3b7d5-73db-483f-bafe-9271ea0c228b) 2024-08-18T15:18:22.191 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:22.188+0000 7f06176bb880 10 cannot find current period zonegroup using local zonegroup configuration 2024-08-18T15:18:22.191 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:22.188+0000 7f06176bb880 20 zonegroup default 2024-08-18T15:18:22.191 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:22.188+0000 7f06176bb880 20 rados->read ofs=0 len=0 2024-08-18T15:18:22.191 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:22.188+0000 7f06176bb880 20 rados_obj.operate() r=-2 bl.length=0 2024-08-18T15:18:22.191 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:22.188+0000 7f06176bb880 20 rados->read ofs=0 len=0 2024-08-18T15:18:22.191 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:22.188+0000 7f06176bb880 20 rados_obj.operate() r=-2 bl.length=0 2024-08-18T15:18:22.191 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:22.188+0000 7f06176bb880 20 rados->read ofs=0 len=0 2024-08-18T15:18:22.192 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:22.189+0000 7f06176bb880 20 rados_obj.operate() r=-2 bl.length=0 2024-08-18T15:18:22.192 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:22.189+0000 7f06176bb880 20 started sync module instance, tier type = 2024-08-18T15:18:22.192 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:22.189+0000 7f06176bb880 20 started zone id=b7f3b7d5-73db-483f-bafe-9271ea0c228b (name=default) with tier type = 2024-08-18T15:18:22.207 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:22.206+0000 7f06176bb880 20 add_watcher() i=0 2024-08-18T15:18:22.207 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:22.206+0000 7f06176bb880 20 add_watcher() i=1 2024-08-18T15:18:22.207 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:22.206+0000 7f06176bb880 20 add_watcher() i=2 2024-08-18T15:18:22.207 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:22.206+0000 7f06176bb880 20 add_watcher() i=3 2024-08-18T15:18:22.207 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:22.206+0000 7f06176bb880 20 add_watcher() i=4 2024-08-18T15:18:22.207 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:22.206+0000 7f06176bb880 20 add_watcher() i=5 2024-08-18T15:18:22.207 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:22.206+0000 7f06176bb880 20 add_watcher() i=6 2024-08-18T15:18:22.209 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:22.209+0000 7f06176bb880 20 add_watcher() i=7 2024-08-18T15:18:22.209 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:22.209+0000 7f06176bb880 2 all 8 watchers are set, enabling cache 2024-08-18T15:18:22.212 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:22.212+0000 7f05fce2e640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-08-18T15:18:22.212 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:22.212+0000 7f06176bb880 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-08-18T15:18:22.212 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:22.212+0000 7f06176bb880 20 check_secure_mon_conn(): mode 1 is insecure 2024-08-18T15:18:22.216 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:22.215+0000 7f06176bb880 5 note: GC not initialized 2024-08-18T15:18:22.216 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:22.216+0000 7f05f8625640 20 reqs_thread_entry: start 2024-08-18T15:18:22.216 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:22.216+0000 7f06176bb880 20 init_complete bucket index max shards: 11 2024-08-18T15:18:22.217 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:22.216+0000 7f06176bb880 10 rgw notify: Started notification manager with: 1 workers 2024-08-18T15:18:22.217 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:22.216+0000 7f05f6621640 20 rgw notify: INFO: next queues processing will happen at: Sun Aug 18 15:18:52 2024 2024-08-18T15:18:22.217 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-08-18T15:18:22.217 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:22.216+0000 7f05f5dff640 20 reqs_thread_entry: start 2024-08-18T15:18:22.223 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:22.223+0000 7f05fce2e640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-08-18T15:18:22.227 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:22.227+0000 7f06176bb880 20 remove_watcher() i=0 2024-08-18T15:18:22.227 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:22.227+0000 7f06176bb880 2 removed watcher, disabling cache 2024-08-18T15:18:22.228 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:22.227+0000 7f06176bb880 20 remove_watcher() i=1 2024-08-18T15:18:22.228 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:22.228+0000 7f06176bb880 20 remove_watcher() i=2 2024-08-18T15:18:22.229 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:22.229+0000 7f06176bb880 20 remove_watcher() i=3 2024-08-18T15:18:22.233 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:22.232+0000 7f06176bb880 20 remove_watcher() i=4 2024-08-18T15:18:22.234 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:22.233+0000 7f06176bb880 20 remove_watcher() i=5 2024-08-18T15:18:22.234 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:22.234+0000 7f06176bb880 20 remove_watcher() i=6 2024-08-18T15:18:22.237 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:22.236+0000 7f06176bb880 20 remove_watcher() i=7 2024-08-18T15:18:22.239 INFO:teuthology.orchestra.run.smithi134.stdout:[] 2024-08-18T15:18:22.241 DEBUG:tasks.util.rgw: json result: [] 2024-08-18T15:18:22.241 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-08-18T15:18:22.241 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-08-18T15:18:22.241 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2024-08-18T15:18:22.329 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:18:22.329 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:18:22.340 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:22.339+0000 7f6d68c86880 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-08-18T15:18:22.340 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:22.339+0000 7f6d68c86880 20 check_secure_mon_conn(): mode 1 is insecure 2024-08-18T15:18:22.341 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:22.341+0000 7f6d4abf2640 20 reqs_thread_entry: start 2024-08-18T15:18:22.348 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T15:18:22.348 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "b7f3b7d5-73db-483f-bafe-9271ea0c228b", 2024-08-18T15:18:22.348 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "default", 2024-08-18T15:18:22.348 INFO:teuthology.orchestra.run.smithi134.stdout: "domain_root": "default.rgw.meta:root", 2024-08-18T15:18:22.348 INFO:teuthology.orchestra.run.smithi134.stdout: "control_pool": "default.rgw.control", 2024-08-18T15:18:22.348 INFO:teuthology.orchestra.run.smithi134.stdout: "gc_pool": "default.rgw.log:gc", 2024-08-18T15:18:22.348 INFO:teuthology.orchestra.run.smithi134.stdout: "lc_pool": "default.rgw.log:lc", 2024-08-18T15:18:22.348 INFO:teuthology.orchestra.run.smithi134.stdout: "log_pool": "default.rgw.log", 2024-08-18T15:18:22.349 INFO:teuthology.orchestra.run.smithi134.stdout: "intent_log_pool": "default.rgw.log:intent", 2024-08-18T15:18:22.349 INFO:teuthology.orchestra.run.smithi134.stdout: "usage_log_pool": "default.rgw.log:usage", 2024-08-18T15:18:22.349 INFO:teuthology.orchestra.run.smithi134.stdout: "roles_pool": "default.rgw.meta:roles", 2024-08-18T15:18:22.349 INFO:teuthology.orchestra.run.smithi134.stdout: "reshard_pool": "default.rgw.log:reshard", 2024-08-18T15:18:22.349 INFO:teuthology.orchestra.run.smithi134.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2024-08-18T15:18:22.349 INFO:teuthology.orchestra.run.smithi134.stdout: "user_email_pool": "default.rgw.meta:users.email", 2024-08-18T15:18:22.349 INFO:teuthology.orchestra.run.smithi134.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2024-08-18T15:18:22.349 INFO:teuthology.orchestra.run.smithi134.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2024-08-18T15:18:22.349 INFO:teuthology.orchestra.run.smithi134.stdout: "otp_pool": "default.rgw.otp", 2024-08-18T15:18:22.349 INFO:teuthology.orchestra.run.smithi134.stdout: "system_key": { 2024-08-18T15:18:22.349 INFO:teuthology.orchestra.run.smithi134.stdout: "access_key": "", 2024-08-18T15:18:22.349 INFO:teuthology.orchestra.run.smithi134.stdout: "secret_key": "" 2024-08-18T15:18:22.350 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:18:22.350 INFO:teuthology.orchestra.run.smithi134.stdout: "placement_pools": [ 2024-08-18T15:18:22.350 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:18:22.350 INFO:teuthology.orchestra.run.smithi134.stdout: "key": "default-placement", 2024-08-18T15:18:22.350 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:18:22.350 INFO:teuthology.orchestra.run.smithi134.stdout: "index_pool": "default.rgw.buckets.index", 2024-08-18T15:18:22.350 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_classes": { 2024-08-18T15:18:22.350 INFO:teuthology.orchestra.run.smithi134.stdout: "STANDARD": { 2024-08-18T15:18:22.350 INFO:teuthology.orchestra.run.smithi134.stdout: "data_pool": "default.rgw.buckets.data", 2024-08-18T15:18:22.350 INFO:teuthology.orchestra.run.smithi134.stdout: "compression_type": "random" 2024-08-18T15:18:22.350 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:18:22.350 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:18:22.350 INFO:teuthology.orchestra.run.smithi134.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2024-08-18T15:18:22.351 INFO:teuthology.orchestra.run.smithi134.stdout: "index_type": 0, 2024-08-18T15:18:22.351 INFO:teuthology.orchestra.run.smithi134.stdout: "inline_data": true 2024-08-18T15:18:22.351 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:18:22.351 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:18:22.351 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T15:18:22.351 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_id": "", 2024-08-18T15:18:22.351 INFO:teuthology.orchestra.run.smithi134.stdout: "notif_pool": "default.rgw.log:notif" 2024-08-18T15:18:22.351 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T15:18:22.352 DEBUG:tasks.util.rgw: json result: {'id': 'b7f3b7d5-73db-483f-bafe-9271ea0c228b', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2024-08-18T15:18:22.352 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['user', 'list'] 2024-08-18T15:18:22.352 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'user', 'list'] 2024-08-18T15:18:22.352 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 user list 2024-08-18T15:18:22.453 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:18:22.453 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:18:22.465 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:22.464+0000 7f540160f880 20 rados->read ofs=0 len=0 2024-08-18T15:18:23.833 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:23.832+0000 7f540160f880 20 rados_obj.operate() r=-2 bl.length=0 2024-08-18T15:18:23.833 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:23.832+0000 7f540160f880 20 realm 2024-08-18T15:18:23.833 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:23.832+0000 7f540160f880 20 rados->read ofs=0 len=0 2024-08-18T15:18:23.834 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:23.833+0000 7f540160f880 20 rados_obj.operate() r=-2 bl.length=0 2024-08-18T15:18:23.834 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:23.833+0000 7f540160f880 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-08-18T15:18:23.834 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:23.833+0000 7f540160f880 20 rados->read ofs=0 len=0 2024-08-18T15:18:23.834 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:23.833+0000 7f540160f880 20 rados_obj.operate() r=-2 bl.length=0 2024-08-18T15:18:23.834 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:23.833+0000 7f540160f880 20 rados->read ofs=0 len=0 2024-08-18T15:18:23.838 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:23.838+0000 7f540160f880 20 rados_obj.operate() r=-2 bl.length=0 2024-08-18T15:18:23.838 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:23.838+0000 7f540160f880 20 rados->read ofs=0 len=0 2024-08-18T15:18:23.840 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:23.840+0000 7f540160f880 20 rados_obj.operate() r=-2 bl.length=0 2024-08-18T15:18:23.840 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:23.840+0000 7f540160f880 20 rados->read ofs=0 len=0 2024-08-18T15:18:23.841 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:23.841+0000 7f540160f880 20 rados_obj.operate() r=-2 bl.length=0 2024-08-18T15:18:23.841 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:23.841+0000 7f540160f880 10 Creating default zonegroup 2024-08-18T15:18:23.841 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:23.841+0000 7f540160f880 10 couldn't find old data placement pools config, setting up new ones for the zone 2024-08-18T15:18:23.851 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:23.851+0000 7f540160f880 10 WARNING: driver->list_zones() returned r=-2 2024-08-18T15:18:23.852 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:23.851+0000 7f540160f880 20 rados->read ofs=0 len=0 2024-08-18T15:18:23.852 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:23.851+0000 7f540160f880 20 rados_obj.operate() r=-2 bl.length=0 2024-08-18T15:18:23.854 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:23.853+0000 7f540160f880 20 rados->read ofs=0 len=0 2024-08-18T15:18:23.854 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:23.854+0000 7f540160f880 20 rados_obj.operate() r=-2 bl.length=0 2024-08-18T15:18:23.855 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:23.854+0000 7f540160f880 10 could not read realm id: (2) No such file or directory 2024-08-18T15:18:23.855 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:23.854+0000 7f540160f880 10 WARNING: failed to set zone as default, r=-22 2024-08-18T15:18:23.855 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:23.854+0000 7f540160f880 20 rados->read ofs=0 len=0 2024-08-18T15:18:23.855 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:23.854+0000 7f540160f880 20 rados_obj.operate() r=-2 bl.length=0 2024-08-18T15:18:23.856 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:23.856+0000 7f540160f880 20 rados->read ofs=0 len=0 2024-08-18T15:18:23.858 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:23.857+0000 7f540160f880 20 rados_obj.operate() r=0 bl.length=874 2024-08-18T15:18:23.858 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:23.857+0000 7f540160f880 20 rados->read ofs=0 len=0 2024-08-18T15:18:23.858 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:23.857+0000 7f540160f880 20 rados_obj.operate() r=0 bl.length=420 2024-08-18T15:18:23.858 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:23.857+0000 7f540160f880 10 Using default name default 2024-08-18T15:18:23.858 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:23.857+0000 7f540160f880 20 rados->read ofs=0 len=0 2024-08-18T15:18:23.858 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:23.857+0000 7f540160f880 20 rados_obj.operate() r=0 bl.length=46 2024-08-18T15:18:23.858 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:23.857+0000 7f540160f880 20 rados->read ofs=0 len=0 2024-08-18T15:18:23.859 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:23.858+0000 7f540160f880 20 rados_obj.operate() r=0 bl.length=874 2024-08-18T15:18:23.859 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:23.858+0000 7f540160f880 20 zone default found 2024-08-18T15:18:23.859 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:23.858+0000 7f540160f880 4 Realm: () 2024-08-18T15:18:23.859 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:23.858+0000 7f540160f880 4 ZoneGroup: default (c7bfbc3f-d539-4577-b159-e21c88bd1d56) 2024-08-18T15:18:23.859 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:23.858+0000 7f540160f880 4 Zone: default (62e1fe33-bc2f-4d3d-bcaa-573c30da6868) 2024-08-18T15:18:23.859 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:23.858+0000 7f540160f880 10 cannot find current period zonegroup using local zonegroup configuration 2024-08-18T15:18:23.860 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:23.858+0000 7f540160f880 20 zonegroup default 2024-08-18T15:18:23.860 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:23.858+0000 7f540160f880 20 rados->read ofs=0 len=0 2024-08-18T15:18:23.860 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:23.858+0000 7f540160f880 20 rados_obj.operate() r=-2 bl.length=0 2024-08-18T15:18:23.860 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:23.858+0000 7f540160f880 20 rados->read ofs=0 len=0 2024-08-18T15:18:25.844 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:25.843+0000 7f540160f880 20 rados_obj.operate() r=-2 bl.length=0 2024-08-18T15:18:25.844 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:25.843+0000 7f540160f880 20 rados->read ofs=0 len=0 2024-08-18T15:18:25.847 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:25.846+0000 7f540160f880 20 rados_obj.operate() r=-2 bl.length=0 2024-08-18T15:18:25.847 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:25.846+0000 7f540160f880 20 started sync module instance, tier type = 2024-08-18T15:18:25.847 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:25.847+0000 7f540160f880 20 started zone id=62e1fe33-bc2f-4d3d-bcaa-573c30da6868 (name=default) with tier type = 2024-08-18T15:18:27.886 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:27.885+0000 7f540160f880 20 add_watcher() i=0 2024-08-18T15:18:27.886 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:27.885+0000 7f540160f880 20 add_watcher() i=1 2024-08-18T15:18:27.886 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:27.885+0000 7f540160f880 20 add_watcher() i=2 2024-08-18T15:18:27.886 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:27.885+0000 7f540160f880 20 add_watcher() i=3 2024-08-18T15:18:27.886 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:27.885+0000 7f540160f880 20 add_watcher() i=4 2024-08-18T15:18:27.886 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:27.885+0000 7f540160f880 20 add_watcher() i=5 2024-08-18T15:18:27.886 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:27.885+0000 7f540160f880 20 add_watcher() i=6 2024-08-18T15:18:27.888 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:27.887+0000 7f540160f880 20 add_watcher() i=7 2024-08-18T15:18:27.888 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:27.887+0000 7f540160f880 2 all 8 watchers are set, enabling cache 2024-08-18T15:18:27.889 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:27.888+0000 7f540160f880 5 tl::expected, obj_version>, boost::system::error_code> logback_generations::read(const DoutPrefixProvider*, optional_yield):410: oid=data_loggenerations_metadata not found 2024-08-18T15:18:27.889 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:27.888+0000 7f540160f880 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.0 2024-08-18T15:18:27.889 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:27.888+0000 7f540160f880 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-08-18T15:18:27.889 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:27.888+0000 7f540160f880 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.0 does not exist 2024-08-18T15:18:27.889 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:27.888+0000 7f540160f880 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.1 2024-08-18T15:18:27.889 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:27.888+0000 7f540160f880 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-08-18T15:18:27.889 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:27.888+0000 7f540160f880 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.1 does not exist 2024-08-18T15:18:27.889 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:27.888+0000 7f540160f880 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.2 2024-08-18T15:18:27.889 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:27.888+0000 7f540160f880 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-08-18T15:18:27.889 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:27.888+0000 7f540160f880 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.2 does not exist 2024-08-18T15:18:27.889 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:27.888+0000 7f540160f880 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.3 2024-08-18T15:18:27.889 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:27.888+0000 7f540160f880 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-08-18T15:18:27.890 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:27.889+0000 7f540160f880 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.3 does not exist 2024-08-18T15:18:27.890 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:27.889+0000 7f540160f880 20 static int rgw::cls::fifo::FIFO::create(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):1210 entering 2024-08-18T15:18:27.893 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:27.893+0000 7f540160f880 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-08-18T15:18:27.898 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:27.897+0000 7f53e4ac2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-08-18T15:18:27.898 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:27.897+0000 7f540160f880 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-08-18T15:18:27.898 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:27.897+0000 7f540160f880 20 check_secure_mon_conn(): mode 1 is insecure 2024-08-18T15:18:30.867 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:30.866+0000 7f540160f880 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2024-08-18T15:18:30.867 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:30.866+0000 7f540160f880 5 note: GC not initialized 2024-08-18T15:18:30.867 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:30.866+0000 7f53e02b9640 20 reqs_thread_entry: start 2024-08-18T15:18:30.867 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:30.867+0000 7f540160f880 20 init_complete bucket index max shards: 11 2024-08-18T15:18:30.868 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:30.867+0000 7f540160f880 10 rgw notify: Started notification manager with: 1 workers 2024-08-18T15:18:30.868 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:30.867+0000 7f53de2b5640 20 rgw notify: INFO: next queues processing will happen at: Sun Aug 18 15:19:01 2024 2024-08-18T15:18:30.868 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-08-18T15:18:30.868 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:30.867+0000 7f53dda93640 20 reqs_thread_entry: start 2024-08-18T15:18:30.886 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:30.886+0000 7f53e4ac2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-08-18T15:18:30.893 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:30.892+0000 7f540160f880 20 remove_watcher() i=0 2024-08-18T15:18:30.893 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:30.892+0000 7f540160f880 2 removed watcher, disabling cache 2024-08-18T15:18:30.894 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:30.893+0000 7f540160f880 20 remove_watcher() i=1 2024-08-18T15:18:30.895 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:30.895+0000 7f540160f880 20 remove_watcher() i=2 2024-08-18T15:18:30.896 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:30.896+0000 7f540160f880 20 remove_watcher() i=3 2024-08-18T15:18:30.898 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:30.898+0000 7f540160f880 20 remove_watcher() i=4 2024-08-18T15:18:30.900 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:30.899+0000 7f540160f880 20 remove_watcher() i=5 2024-08-18T15:18:30.901 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:30.901+0000 7f540160f880 20 remove_watcher() i=6 2024-08-18T15:18:30.902 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:30.902+0000 7f540160f880 20 remove_watcher() i=7 2024-08-18T15:18:30.906 INFO:teuthology.orchestra.run.smithi193.stdout:[] 2024-08-18T15:18:30.950 DEBUG:tasks.util.rgw: json result: [] 2024-08-18T15:18:30.950 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-08-18T15:18:30.950 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-08-18T15:18:30.950 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2024-08-18T15:18:31.002 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:18:31.003 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:18:31.013 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:31.012+0000 7f6a54b42880 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-08-18T15:18:31.013 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:31.012+0000 7f6a54b42880 20 check_secure_mon_conn(): mode 1 is insecure 2024-08-18T15:18:31.016 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:31.015+0000 7f6a34d0f640 20 reqs_thread_entry: start 2024-08-18T15:18:31.023 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:18:31.023 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "62e1fe33-bc2f-4d3d-bcaa-573c30da6868", 2024-08-18T15:18:31.023 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "default", 2024-08-18T15:18:31.023 INFO:teuthology.orchestra.run.smithi193.stdout: "domain_root": "default.rgw.meta:root", 2024-08-18T15:18:31.023 INFO:teuthology.orchestra.run.smithi193.stdout: "control_pool": "default.rgw.control", 2024-08-18T15:18:31.023 INFO:teuthology.orchestra.run.smithi193.stdout: "gc_pool": "default.rgw.log:gc", 2024-08-18T15:18:31.023 INFO:teuthology.orchestra.run.smithi193.stdout: "lc_pool": "default.rgw.log:lc", 2024-08-18T15:18:31.023 INFO:teuthology.orchestra.run.smithi193.stdout: "log_pool": "default.rgw.log", 2024-08-18T15:18:31.023 INFO:teuthology.orchestra.run.smithi193.stdout: "intent_log_pool": "default.rgw.log:intent", 2024-08-18T15:18:31.024 INFO:teuthology.orchestra.run.smithi193.stdout: "usage_log_pool": "default.rgw.log:usage", 2024-08-18T15:18:31.024 INFO:teuthology.orchestra.run.smithi193.stdout: "roles_pool": "default.rgw.meta:roles", 2024-08-18T15:18:31.024 INFO:teuthology.orchestra.run.smithi193.stdout: "reshard_pool": "default.rgw.log:reshard", 2024-08-18T15:18:31.024 INFO:teuthology.orchestra.run.smithi193.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2024-08-18T15:18:31.024 INFO:teuthology.orchestra.run.smithi193.stdout: "user_email_pool": "default.rgw.meta:users.email", 2024-08-18T15:18:31.024 INFO:teuthology.orchestra.run.smithi193.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2024-08-18T15:18:31.024 INFO:teuthology.orchestra.run.smithi193.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2024-08-18T15:18:31.024 INFO:teuthology.orchestra.run.smithi193.stdout: "otp_pool": "default.rgw.otp", 2024-08-18T15:18:31.024 INFO:teuthology.orchestra.run.smithi193.stdout: "system_key": { 2024-08-18T15:18:31.024 INFO:teuthology.orchestra.run.smithi193.stdout: "access_key": "", 2024-08-18T15:18:31.024 INFO:teuthology.orchestra.run.smithi193.stdout: "secret_key": "" 2024-08-18T15:18:31.024 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:18:31.025 INFO:teuthology.orchestra.run.smithi193.stdout: "placement_pools": [ 2024-08-18T15:18:31.025 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:18:31.025 INFO:teuthology.orchestra.run.smithi193.stdout: "key": "default-placement", 2024-08-18T15:18:31.025 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:18:31.025 INFO:teuthology.orchestra.run.smithi193.stdout: "index_pool": "default.rgw.buckets.index", 2024-08-18T15:18:31.025 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_classes": { 2024-08-18T15:18:31.025 INFO:teuthology.orchestra.run.smithi193.stdout: "STANDARD": { 2024-08-18T15:18:31.025 INFO:teuthology.orchestra.run.smithi193.stdout: "data_pool": "default.rgw.buckets.data", 2024-08-18T15:18:31.025 INFO:teuthology.orchestra.run.smithi193.stdout: "compression_type": "random" 2024-08-18T15:18:31.025 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:18:31.025 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:18:31.025 INFO:teuthology.orchestra.run.smithi193.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2024-08-18T15:18:31.026 INFO:teuthology.orchestra.run.smithi193.stdout: "index_type": 0, 2024-08-18T15:18:31.026 INFO:teuthology.orchestra.run.smithi193.stdout: "inline_data": true 2024-08-18T15:18:31.026 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:18:31.026 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:18:31.026 INFO:teuthology.orchestra.run.smithi193.stdout: ], 2024-08-18T15:18:31.026 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_id": "", 2024-08-18T15:18:31.026 INFO:teuthology.orchestra.run.smithi193.stdout: "notif_pool": "default.rgw.log:notif" 2024-08-18T15:18:31.026 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:18:31.032 DEBUG:tasks.util.rgw: json result: {'id': '62e1fe33-bc2f-4d3d-bcaa-573c30da6868', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2024-08-18T15:18:31.033 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['user', 'list'] 2024-08-18T15:18:31.033 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'user', 'list'] 2024-08-18T15:18:31.033 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 user list 2024-08-18T15:18:31.076 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:18:31.076 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:18:31.086 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:31.085+0000 7f61aca17880 20 rados->read ofs=0 len=0 2024-08-18T15:18:31.087 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:31.086+0000 7f61aca17880 20 rados_obj.operate() r=-2 bl.length=0 2024-08-18T15:18:31.087 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:31.086+0000 7f61aca17880 20 realm 2024-08-18T15:18:31.087 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:31.086+0000 7f61aca17880 20 rados->read ofs=0 len=0 2024-08-18T15:18:31.087 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:31.086+0000 7f61aca17880 20 rados_obj.operate() r=-2 bl.length=0 2024-08-18T15:18:31.087 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:31.086+0000 7f61aca17880 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-08-18T15:18:31.087 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:31.086+0000 7f61aca17880 20 rados->read ofs=0 len=0 2024-08-18T15:18:31.087 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:31.086+0000 7f61aca17880 20 rados_obj.operate() r=-2 bl.length=0 2024-08-18T15:18:31.087 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:31.086+0000 7f61aca17880 20 rados->read ofs=0 len=0 2024-08-18T15:18:31.087 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:31.087+0000 7f61aca17880 20 rados_obj.operate() r=0 bl.length=46 2024-08-18T15:18:31.088 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:31.087+0000 7f61aca17880 20 rados->read ofs=0 len=0 2024-08-18T15:18:31.088 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:31.087+0000 7f61aca17880 20 rados_obj.operate() r=0 bl.length=890 2024-08-18T15:18:31.088 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:31.087+0000 7f61aca17880 20 searching for the correct realm 2024-08-18T15:18:31.095 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:31.094+0000 7f61aca17880 20 RGWRados::pool_iterate: got zone_info.62e1fe33-bc2f-4d3d-bcaa-573c30da6868 2024-08-18T15:18:31.095 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:31.094+0000 7f61aca17880 20 RGWRados::pool_iterate: got zone_names.default 2024-08-18T15:18:31.095 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:31.094+0000 7f61aca17880 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-08-18T15:18:31.096 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:31.094+0000 7f61aca17880 20 RGWRados::pool_iterate: got zonegroup_info.c7bfbc3f-d539-4577-b159-e21c88bd1d56 2024-08-18T15:18:31.096 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:31.094+0000 7f61aca17880 20 rados->read ofs=0 len=0 2024-08-18T15:18:31.096 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:31.094+0000 7f61aca17880 20 rados_obj.operate() r=-2 bl.length=0 2024-08-18T15:18:31.096 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:31.094+0000 7f61aca17880 20 rados->read ofs=0 len=0 2024-08-18T15:18:31.096 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:31.094+0000 7f61aca17880 20 rados_obj.operate() r=0 bl.length=46 2024-08-18T15:18:31.096 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:31.094+0000 7f61aca17880 20 rados->read ofs=0 len=0 2024-08-18T15:18:31.096 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:31.094+0000 7f61aca17880 20 rados_obj.operate() r=0 bl.length=420 2024-08-18T15:18:31.096 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:31.094+0000 7f61aca17880 20 zone default found 2024-08-18T15:18:31.096 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:31.094+0000 7f61aca17880 4 Realm: () 2024-08-18T15:18:31.096 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:31.094+0000 7f61aca17880 4 ZoneGroup: default (c7bfbc3f-d539-4577-b159-e21c88bd1d56) 2024-08-18T15:18:31.096 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:31.094+0000 7f61aca17880 4 Zone: default (62e1fe33-bc2f-4d3d-bcaa-573c30da6868) 2024-08-18T15:18:31.096 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:31.094+0000 7f61aca17880 10 cannot find current period zonegroup using local zonegroup configuration 2024-08-18T15:18:31.096 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:31.094+0000 7f61aca17880 20 zonegroup default 2024-08-18T15:18:31.097 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:31.094+0000 7f61aca17880 20 rados->read ofs=0 len=0 2024-08-18T15:18:31.097 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:31.095+0000 7f61aca17880 20 rados_obj.operate() r=-2 bl.length=0 2024-08-18T15:18:31.097 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:31.095+0000 7f61aca17880 20 rados->read ofs=0 len=0 2024-08-18T15:18:31.097 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:31.095+0000 7f61aca17880 20 rados_obj.operate() r=-2 bl.length=0 2024-08-18T15:18:31.097 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:31.095+0000 7f61aca17880 20 rados->read ofs=0 len=0 2024-08-18T15:18:31.097 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:31.095+0000 7f61aca17880 20 rados_obj.operate() r=-2 bl.length=0 2024-08-18T15:18:31.097 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:31.095+0000 7f61aca17880 20 started sync module instance, tier type = 2024-08-18T15:18:31.097 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:31.095+0000 7f61aca17880 20 started zone id=62e1fe33-bc2f-4d3d-bcaa-573c30da6868 (name=default) with tier type = 2024-08-18T15:18:31.121 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:31.120+0000 7f61aca17880 20 add_watcher() i=0 2024-08-18T15:18:31.121 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:31.120+0000 7f61aca17880 20 add_watcher() i=1 2024-08-18T15:18:31.121 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:31.120+0000 7f61aca17880 20 add_watcher() i=2 2024-08-18T15:18:31.121 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:31.120+0000 7f61aca17880 20 add_watcher() i=3 2024-08-18T15:18:31.121 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:31.120+0000 7f61aca17880 20 add_watcher() i=4 2024-08-18T15:18:31.121 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:31.120+0000 7f61aca17880 20 add_watcher() i=5 2024-08-18T15:18:31.126 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:31.126+0000 7f61aca17880 20 add_watcher() i=6 2024-08-18T15:18:31.126 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:31.126+0000 7f61aca17880 20 add_watcher() i=7 2024-08-18T15:18:31.126 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:31.126+0000 7f61aca17880 2 all 8 watchers are set, enabling cache 2024-08-18T15:18:31.127 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:31.127+0000 7f618fec9640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-08-18T15:18:31.128 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:31.127+0000 7f61aca17880 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-08-18T15:18:31.128 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:31.127+0000 7f61aca17880 20 check_secure_mon_conn(): mode 1 is insecure 2024-08-18T15:18:31.131 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:31.131+0000 7f61aca17880 5 note: GC not initialized 2024-08-18T15:18:31.131 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:31.131+0000 7f618b6c0640 20 reqs_thread_entry: start 2024-08-18T15:18:31.132 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:31.131+0000 7f61aca17880 20 init_complete bucket index max shards: 11 2024-08-18T15:18:31.132 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:31.131+0000 7f61aca17880 10 rgw notify: Started notification manager with: 1 workers 2024-08-18T15:18:31.132 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:31.131+0000 7f61896bc640 20 rgw notify: INFO: next queues processing will happen at: Sun Aug 18 15:19:01 2024 2024-08-18T15:18:31.132 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-08-18T15:18:31.132 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:31.131+0000 7f6188e9a640 20 reqs_thread_entry: start 2024-08-18T15:18:31.138 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:31.138+0000 7f618fec9640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-08-18T15:18:31.140 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:31.139+0000 7f61aca17880 20 remove_watcher() i=0 2024-08-18T15:18:31.140 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:31.139+0000 7f61aca17880 2 removed watcher, disabling cache 2024-08-18T15:18:31.142 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:31.142+0000 7f61aca17880 20 remove_watcher() i=1 2024-08-18T15:18:31.143 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:31.143+0000 7f61aca17880 20 remove_watcher() i=2 2024-08-18T15:18:31.146 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:31.145+0000 7f61aca17880 20 remove_watcher() i=3 2024-08-18T15:18:31.148 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:31.148+0000 7f61aca17880 20 remove_watcher() i=4 2024-08-18T15:18:31.149 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:31.149+0000 7f61aca17880 20 remove_watcher() i=5 2024-08-18T15:18:31.152 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:31.151+0000 7f61aca17880 20 remove_watcher() i=6 2024-08-18T15:18:31.154 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:31.154+0000 7f61aca17880 20 remove_watcher() i=7 2024-08-18T15:18:31.157 INFO:teuthology.orchestra.run.smithi193.stdout:[] 2024-08-18T15:18:31.160 DEBUG:tasks.util.rgw: json result: [] 2024-08-18T15:18:31.160 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-08-18T15:18:31.160 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-08-18T15:18:31.160 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2024-08-18T15:18:31.252 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:18:31.252 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:18:31.262 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:31.261+0000 7f1427524880 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-08-18T15:18:31.262 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:31.261+0000 7f1427524880 20 check_secure_mon_conn(): mode 1 is insecure 2024-08-18T15:18:31.263 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:31.263+0000 7f1407594640 20 reqs_thread_entry: start 2024-08-18T15:18:31.271 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:18:31.271 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "62e1fe33-bc2f-4d3d-bcaa-573c30da6868", 2024-08-18T15:18:31.272 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "default", 2024-08-18T15:18:31.272 INFO:teuthology.orchestra.run.smithi193.stdout: "domain_root": "default.rgw.meta:root", 2024-08-18T15:18:31.272 INFO:teuthology.orchestra.run.smithi193.stdout: "control_pool": "default.rgw.control", 2024-08-18T15:18:31.272 INFO:teuthology.orchestra.run.smithi193.stdout: "gc_pool": "default.rgw.log:gc", 2024-08-18T15:18:31.272 INFO:teuthology.orchestra.run.smithi193.stdout: "lc_pool": "default.rgw.log:lc", 2024-08-18T15:18:31.272 INFO:teuthology.orchestra.run.smithi193.stdout: "log_pool": "default.rgw.log", 2024-08-18T15:18:31.272 INFO:teuthology.orchestra.run.smithi193.stdout: "intent_log_pool": "default.rgw.log:intent", 2024-08-18T15:18:31.273 INFO:teuthology.orchestra.run.smithi193.stdout: "usage_log_pool": "default.rgw.log:usage", 2024-08-18T15:18:31.273 INFO:teuthology.orchestra.run.smithi193.stdout: "roles_pool": "default.rgw.meta:roles", 2024-08-18T15:18:31.273 INFO:teuthology.orchestra.run.smithi193.stdout: "reshard_pool": "default.rgw.log:reshard", 2024-08-18T15:18:31.273 INFO:teuthology.orchestra.run.smithi193.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2024-08-18T15:18:31.273 INFO:teuthology.orchestra.run.smithi193.stdout: "user_email_pool": "default.rgw.meta:users.email", 2024-08-18T15:18:31.273 INFO:teuthology.orchestra.run.smithi193.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2024-08-18T15:18:31.273 INFO:teuthology.orchestra.run.smithi193.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2024-08-18T15:18:31.274 INFO:teuthology.orchestra.run.smithi193.stdout: "otp_pool": "default.rgw.otp", 2024-08-18T15:18:31.274 INFO:teuthology.orchestra.run.smithi193.stdout: "system_key": { 2024-08-18T15:18:31.274 INFO:teuthology.orchestra.run.smithi193.stdout: "access_key": "", 2024-08-18T15:18:31.274 INFO:teuthology.orchestra.run.smithi193.stdout: "secret_key": "" 2024-08-18T15:18:31.274 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:18:31.274 INFO:teuthology.orchestra.run.smithi193.stdout: "placement_pools": [ 2024-08-18T15:18:31.275 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:18:31.275 INFO:teuthology.orchestra.run.smithi193.stdout: "key": "default-placement", 2024-08-18T15:18:31.275 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:18:31.275 INFO:teuthology.orchestra.run.smithi193.stdout: "index_pool": "default.rgw.buckets.index", 2024-08-18T15:18:31.275 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_classes": { 2024-08-18T15:18:31.275 INFO:teuthology.orchestra.run.smithi193.stdout: "STANDARD": { 2024-08-18T15:18:31.275 INFO:teuthology.orchestra.run.smithi193.stdout: "data_pool": "default.rgw.buckets.data", 2024-08-18T15:18:31.275 INFO:teuthology.orchestra.run.smithi193.stdout: "compression_type": "random" 2024-08-18T15:18:31.276 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:18:31.276 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:18:31.276 INFO:teuthology.orchestra.run.smithi193.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2024-08-18T15:18:31.276 INFO:teuthology.orchestra.run.smithi193.stdout: "index_type": 0, 2024-08-18T15:18:31.276 INFO:teuthology.orchestra.run.smithi193.stdout: "inline_data": true 2024-08-18T15:18:31.276 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:18:31.276 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:18:31.277 INFO:teuthology.orchestra.run.smithi193.stdout: ], 2024-08-18T15:18:31.277 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_id": "", 2024-08-18T15:18:31.277 INFO:teuthology.orchestra.run.smithi193.stdout: "notif_pool": "default.rgw.log:notif" 2024-08-18T15:18:31.277 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:18:31.279 DEBUG:tasks.util.rgw: json result: {'id': '62e1fe33-bc2f-4d3d-bcaa-573c30da6868', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2024-08-18T15:18:31.279 INFO:tasks.rgw:Starting rgw... 2024-08-18T15:18:31.279 INFO:tasks.rgw:rgw c1.client.0 config is {'port': 8000} 2024-08-18T15:18:31.279 INFO:tasks.rgw:Using beast as radosgw frontend 2024-08-18T15:18:31.280 DEBUG:teuthology.orchestra.run.smithi134:> sudo echo -n http://smithi134.front.sepia.ceph.com:8000 | sudo tee /home/ubuntu/cephtest/url_file 2024-08-18T15:18:31.312 INFO:teuthology.orchestra.run.smithi134.stdout:http://smithi134.front.sepia.ceph.com:8000 2024-08-18T15:18:31.313 DEBUG:teuthology.orchestra.run.smithi134:> sudo chown ceph /home/ubuntu/cephtest/url_file 2024-08-18T15:18:31.391 INFO:tasks.rgw.c1.client.0:Restarting daemon 2024-08-18T15:18:31.392 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2024-08-18T15:18:31.440 INFO:tasks.rgw.c1.client.0:Started 2024-08-18T15:18:31.440 INFO:tasks.rgw:rgw c1.client.1 config is {'port': 8001} 2024-08-18T15:18:31.440 INFO:tasks.rgw:Using beast as radosgw frontend 2024-08-18T15:18:31.441 DEBUG:teuthology.orchestra.run.smithi134:> sudo echo -n http://smithi134.front.sepia.ceph.com:8001 | sudo tee /home/ubuntu/cephtest/url_file 2024-08-18T15:18:31.472 INFO:teuthology.orchestra.run.smithi134.stdout:http://smithi134.front.sepia.ceph.com:8001 2024-08-18T15:18:31.473 DEBUG:teuthology.orchestra.run.smithi134:> sudo chown ceph /home/ubuntu/cephtest/url_file 2024-08-18T15:18:31.545 INFO:tasks.rgw.c1.client.1:Restarting daemon 2024-08-18T15:18:31.545 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2024-08-18T15:18:31.590 INFO:tasks.rgw.c1.client.1:Started 2024-08-18T15:18:31.591 INFO:tasks.rgw:rgw c2.client.0 config is {'port': 8000} 2024-08-18T15:18:31.591 INFO:tasks.rgw:Using beast as radosgw frontend 2024-08-18T15:18:31.591 DEBUG:teuthology.orchestra.run.smithi193:> sudo echo -n http://smithi193.front.sepia.ceph.com:8000 | sudo tee /home/ubuntu/cephtest/url_file 2024-08-18T15:18:31.619 INFO:teuthology.orchestra.run.smithi193.stdout:http://smithi193.front.sepia.ceph.com:8000 2024-08-18T15:18:31.619 DEBUG:teuthology.orchestra.run.smithi193:> sudo chown ceph /home/ubuntu/cephtest/url_file 2024-08-18T15:18:31.685 INFO:tasks.rgw.c2.client.0:Restarting daemon 2024-08-18T15:18:31.685 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2024-08-18T15:18:31.728 INFO:tasks.rgw.c2.client.0:Started 2024-08-18T15:18:31.728 INFO:tasks.rgw:rgw c2.client.1 config is {'port': 8001} 2024-08-18T15:18:31.728 INFO:tasks.rgw:Using beast as radosgw frontend 2024-08-18T15:18:31.728 DEBUG:teuthology.orchestra.run.smithi193:> sudo echo -n http://smithi193.front.sepia.ceph.com:8001 | sudo tee /home/ubuntu/cephtest/url_file 2024-08-18T15:18:31.754 INFO:teuthology.orchestra.run.smithi193.stdout:http://smithi193.front.sepia.ceph.com:8001 2024-08-18T15:18:31.754 DEBUG:teuthology.orchestra.run.smithi193:> sudo chown ceph /home/ubuntu/cephtest/url_file 2024-08-18T15:18:31.824 INFO:tasks.rgw.c2.client.1:Restarting daemon 2024-08-18T15:18:31.825 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2024-08-18T15:18:31.869 INFO:tasks.rgw.c2.client.1:Started 2024-08-18T15:18:31.869 INFO:tasks.rgw:Polling c1.client.0 until it starts accepting connections on http://smithi134.front.sepia.ceph.com:8000/ 2024-08-18T15:18:31.869 DEBUG:teuthology.orchestra.run.smithi134:> curl http://smithi134.front.sepia.ceph.com:8000/ 2024-08-18T15:18:31.898 INFO:teuthology.orchestra.run.smithi134.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-08-18T15:18:31.898 INFO:teuthology.orchestra.run.smithi134.stderr: Dload Upload Total Spent Left Speed 2024-08-18T15:18:31.900 INFO:teuthology.orchestra.run.smithi134.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-08-18T15:18:31.900 INFO:teuthology.orchestra.run.smithi134.stdout:anonymous 2024-08-18T15:18:31.900 INFO:tasks.rgw:Polling c1.client.1 until it starts accepting connections on http://smithi134.front.sepia.ceph.com:8001/ 2024-08-18T15:18:31.900 DEBUG:teuthology.orchestra.run.smithi134:> curl http://smithi134.front.sepia.ceph.com:8001/ 2024-08-18T15:18:31.962 INFO:teuthology.orchestra.run.smithi134.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-08-18T15:18:31.962 INFO:teuthology.orchestra.run.smithi134.stderr: Dload Upload Total Spent Left Speed 2024-08-18T15:18:31.964 INFO:teuthology.orchestra.run.smithi134.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-08-18T15:18:31.964 INFO:teuthology.orchestra.run.smithi134.stdout:anonymous 2024-08-18T15:18:31.964 INFO:tasks.rgw:Polling c2.client.0 until it starts accepting connections on http://smithi193.front.sepia.ceph.com:8000/ 2024-08-18T15:18:31.964 DEBUG:teuthology.orchestra.run.smithi193:> curl http://smithi193.front.sepia.ceph.com:8000/ 2024-08-18T15:18:32.019 INFO:teuthology.orchestra.run.smithi193.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-08-18T15:18:32.019 INFO:teuthology.orchestra.run.smithi193.stderr: Dload Upload Total Spent Left Speed 2024-08-18T15:18:32.019 INFO:teuthology.orchestra.run.smithi193.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 26750 0 --:--:-- --:--:-- --:--:-- 26750 2024-08-18T15:18:32.021 INFO:teuthology.orchestra.run.smithi193.stdout:anonymous 2024-08-18T15:18:32.021 INFO:tasks.rgw:Polling c2.client.1 until it starts accepting connections on http://smithi193.front.sepia.ceph.com:8001/ 2024-08-18T15:18:32.021 DEBUG:teuthology.orchestra.run.smithi193:> curl http://smithi193.front.sepia.ceph.com:8001/ 2024-08-18T15:18:32.085 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-08-18T15:18:32.085 INFO:teuthology.orchestra.run.smithi193.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-08-18T15:18:32.085 INFO:teuthology.orchestra.run.smithi193.stderr: Dload Upload Total Spent Left Speed 2024-08-18T15:18:32.085 INFO:teuthology.orchestra.run.smithi193.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2024-08-18T15:18:32.085 INFO:teuthology.orchestra.run.smithi193.stderr:curl: (7) Failed to connect to smithi193.front.sepia.ceph.com port 8001: Connection refused 2024-08-18T15:18:33.087 DEBUG:teuthology.orchestra.run.smithi193:> curl http://smithi193.front.sepia.ceph.com:8001/ 2024-08-18T15:18:33.106 INFO:teuthology.orchestra.run.smithi193.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-08-18T15:18:33.106 INFO:teuthology.orchestra.run.smithi193.stderr: Dload Upload Total Spent Left Speed 2024-08-18T15:18:33.108 INFO:teuthology.orchestra.run.smithi193.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-08-18T15:18:33.108 INFO:teuthology.orchestra.run.smithi193.stdout:anonymous 2024-08-18T15:18:33.108 INFO:teuthology.run_tasks:Running task rgw-multisite... 2024-08-18T15:18:35.023 INFO:tasks.rgw_multisite:creating realm.. 2024-08-18T15:18:35.024 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:18:35.024 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:18:35.024 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 realm create --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T15:18:35.074 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:18:35.074 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:18:35.109 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T15:18:35.109 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "2827c6f0-bf0d-4836-944c-9e7fcf22457b", 2024-08-18T15:18:35.110 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "test-realm", 2024-08-18T15:18:35.110 INFO:teuthology.orchestra.run.smithi134.stdout: "current_period": "453536fc-4ae4-4b27-be1c-ba0601521b0c", 2024-08-18T15:18:35.110 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 1 2024-08-18T15:18:35.110 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T15:18:35.111 INFO:tasks.rgw_multisite:creating master zonegroup.. 2024-08-18T15:18:35.111 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'set', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:18:35.111 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'set', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:18:35.112 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup set --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T15:18:35.158 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:18:35.158 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:18:35.172 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:35.172+0000 7fc88c4b9880 -1 old format 2024-08-18T15:18:35.182 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T15:18:35.182 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "test-zonegroup", 2024-08-18T15:18:35.182 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "test-zonegroup", 2024-08-18T15:18:35.183 INFO:teuthology.orchestra.run.smithi134.stdout: "api_name": "", 2024-08-18T15:18:35.183 INFO:teuthology.orchestra.run.smithi134.stdout: "is_master": true, 2024-08-18T15:18:35.183 INFO:teuthology.orchestra.run.smithi134.stdout: "endpoints": [ 2024-08-18T15:18:35.183 INFO:teuthology.orchestra.run.smithi134.stdout: "http://smithi134.front.sepia.ceph.com:8000" 2024-08-18T15:18:35.183 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T15:18:35.183 INFO:teuthology.orchestra.run.smithi134.stdout: "hostnames": [], 2024-08-18T15:18:35.183 INFO:teuthology.orchestra.run.smithi134.stdout: "hostnames_s3website": [], 2024-08-18T15:18:35.183 INFO:teuthology.orchestra.run.smithi134.stdout: "master_zone": "", 2024-08-18T15:18:35.183 INFO:teuthology.orchestra.run.smithi134.stdout: "zones": [], 2024-08-18T15:18:35.183 INFO:teuthology.orchestra.run.smithi134.stdout: "placement_targets": [ 2024-08-18T15:18:35.183 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:18:35.183 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "default-placement", 2024-08-18T15:18:35.183 INFO:teuthology.orchestra.run.smithi134.stdout: "tags": [], 2024-08-18T15:18:35.184 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_classes": [] 2024-08-18T15:18:35.184 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:18:35.184 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T15:18:35.184 INFO:teuthology.orchestra.run.smithi134.stdout: "default_placement": "default-placement", 2024-08-18T15:18:35.184 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_id": "2827c6f0-bf0d-4836-944c-9e7fcf22457b", 2024-08-18T15:18:35.184 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_policy": { 2024-08-18T15:18:35.184 INFO:teuthology.orchestra.run.smithi134.stdout: "groups": [] 2024-08-18T15:18:35.184 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:18:35.184 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled_features": [ 2024-08-18T15:18:35.184 INFO:teuthology.orchestra.run.smithi134.stdout: "resharding" 2024-08-18T15:18:35.184 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T15:18:35.184 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T15:18:35.185 INFO:tasks.rgw_multisite:creating master zone.. 2024-08-18T15:18:35.185 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'MRX9HX7RYBDXO1UM', '--secret', 'hGUsvhUdpGZAYWjQwu3StBaYXJPhuCEj', '--endpoints', 'http://smithi134.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:18:35.186 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'MRX9HX7RYBDXO1UM', '--secret', 'hGUsvhUdpGZAYWjQwu3StBaYXJPhuCEj', '--endpoints', 'http://smithi134.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:18:35.186 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --master --access-key MRX9HX7RYBDXO1UM --secret hGUsvhUdpGZAYWjQwu3StBaYXJPhuCEj --endpoints http://smithi134.front.sepia.ceph.com:8000 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T15:18:35.279 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:18:35.279 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:18:35.294 INFO:teuthology.orchestra.run.smithi134.stderr:NOTICE: set zone's realm_id=2827c6f0-bf0d-4836-944c-9e7fcf22457b 2024-08-18T15:18:35.315 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T15:18:35.315 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:18:35.315 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "test-zone1", 2024-08-18T15:18:35.315 INFO:teuthology.orchestra.run.smithi134.stdout: "domain_root": "test-zone1.rgw.meta:root", 2024-08-18T15:18:35.316 INFO:teuthology.orchestra.run.smithi134.stdout: "control_pool": "test-zone1.rgw.control", 2024-08-18T15:18:35.316 INFO:teuthology.orchestra.run.smithi134.stdout: "gc_pool": "test-zone1.rgw.log:gc", 2024-08-18T15:18:35.316 INFO:teuthology.orchestra.run.smithi134.stdout: "lc_pool": "test-zone1.rgw.log:lc", 2024-08-18T15:18:35.316 INFO:teuthology.orchestra.run.smithi134.stdout: "log_pool": "test-zone1.rgw.log", 2024-08-18T15:18:35.316 INFO:teuthology.orchestra.run.smithi134.stdout: "intent_log_pool": "test-zone1.rgw.log:intent", 2024-08-18T15:18:35.316 INFO:teuthology.orchestra.run.smithi134.stdout: "usage_log_pool": "test-zone1.rgw.log:usage", 2024-08-18T15:18:35.316 INFO:teuthology.orchestra.run.smithi134.stdout: "roles_pool": "test-zone1.rgw.meta:roles", 2024-08-18T15:18:35.316 INFO:teuthology.orchestra.run.smithi134.stdout: "reshard_pool": "test-zone1.rgw.log:reshard", 2024-08-18T15:18:35.316 INFO:teuthology.orchestra.run.smithi134.stdout: "user_keys_pool": "test-zone1.rgw.meta:users.keys", 2024-08-18T15:18:35.316 INFO:teuthology.orchestra.run.smithi134.stdout: "user_email_pool": "test-zone1.rgw.meta:users.email", 2024-08-18T15:18:35.316 INFO:teuthology.orchestra.run.smithi134.stdout: "user_swift_pool": "test-zone1.rgw.meta:users.swift", 2024-08-18T15:18:35.316 INFO:teuthology.orchestra.run.smithi134.stdout: "user_uid_pool": "test-zone1.rgw.meta:users.uid", 2024-08-18T15:18:35.317 INFO:teuthology.orchestra.run.smithi134.stdout: "otp_pool": "test-zone1.rgw.otp", 2024-08-18T15:18:35.317 INFO:teuthology.orchestra.run.smithi134.stdout: "system_key": { 2024-08-18T15:18:35.317 INFO:teuthology.orchestra.run.smithi134.stdout: "access_key": "MRX9HX7RYBDXO1UM", 2024-08-18T15:18:35.317 INFO:teuthology.orchestra.run.smithi134.stdout: "secret_key": "hGUsvhUdpGZAYWjQwu3StBaYXJPhuCEj" 2024-08-18T15:18:35.317 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:18:35.317 INFO:teuthology.orchestra.run.smithi134.stdout: "placement_pools": [ 2024-08-18T15:18:35.317 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:18:35.317 INFO:teuthology.orchestra.run.smithi134.stdout: "key": "default-placement", 2024-08-18T15:18:35.317 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:18:35.317 INFO:teuthology.orchestra.run.smithi134.stdout: "index_pool": "test-zone1.rgw.buckets.index", 2024-08-18T15:18:35.317 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_classes": { 2024-08-18T15:18:35.318 INFO:teuthology.orchestra.run.smithi134.stdout: "STANDARD": { 2024-08-18T15:18:35.318 INFO:teuthology.orchestra.run.smithi134.stdout: "data_pool": "test-zone1.rgw.buckets.data" 2024-08-18T15:18:35.318 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:18:35.318 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:18:35.318 INFO:teuthology.orchestra.run.smithi134.stdout: "data_extra_pool": "test-zone1.rgw.buckets.non-ec", 2024-08-18T15:18:35.318 INFO:teuthology.orchestra.run.smithi134.stdout: "index_type": 0, 2024-08-18T15:18:35.318 INFO:teuthology.orchestra.run.smithi134.stdout: "inline_data": true 2024-08-18T15:18:35.318 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:18:35.318 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:18:35.318 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T15:18:35.318 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_id": "2827c6f0-bf0d-4836-944c-9e7fcf22457b", 2024-08-18T15:18:35.318 INFO:teuthology.orchestra.run.smithi134.stdout: "notif_pool": "test-zone1.rgw.log:notif" 2024-08-18T15:18:35.319 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T15:18:35.320 DEBUG:teuthology.orchestra.run.smithi134:> sudo ceph osd pool create test-zone1.rgw.buckets.data 64 64 --cluster c1 2024-08-18T15:18:36.021 INFO:teuthology.orchestra.run.smithi134.stderr:pool 'test-zone1.rgw.buckets.data' created 2024-08-18T15:18:36.100 DEBUG:teuthology.orchestra.run.smithi134:> sudo ceph osd pool application enable test-zone1.rgw.buckets.data rgw --cluster c1 2024-08-18T15:18:37.025 INFO:teuthology.orchestra.run.smithi134.stderr:enabled application 'rgw' on pool 'test-zone1.rgw.buckets.data' 2024-08-18T15:18:37.060 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:18:37.060 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:18:37.061 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T15:18:37.113 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:18:37.113 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:18:37.132 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T15:18:37.132 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:18:37.132 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "test-zone1", 2024-08-18T15:18:37.133 INFO:teuthology.orchestra.run.smithi134.stdout: "domain_root": "test-zone1.rgw.meta:root", 2024-08-18T15:18:37.133 INFO:teuthology.orchestra.run.smithi134.stdout: "control_pool": "test-zone1.rgw.control", 2024-08-18T15:18:37.133 INFO:teuthology.orchestra.run.smithi134.stdout: "gc_pool": "test-zone1.rgw.log:gc", 2024-08-18T15:18:37.133 INFO:teuthology.orchestra.run.smithi134.stdout: "lc_pool": "test-zone1.rgw.log:lc", 2024-08-18T15:18:37.133 INFO:teuthology.orchestra.run.smithi134.stdout: "log_pool": "test-zone1.rgw.log", 2024-08-18T15:18:37.133 INFO:teuthology.orchestra.run.smithi134.stdout: "intent_log_pool": "test-zone1.rgw.log:intent", 2024-08-18T15:18:37.133 INFO:teuthology.orchestra.run.smithi134.stdout: "usage_log_pool": "test-zone1.rgw.log:usage", 2024-08-18T15:18:37.133 INFO:teuthology.orchestra.run.smithi134.stdout: "roles_pool": "test-zone1.rgw.meta:roles", 2024-08-18T15:18:37.133 INFO:teuthology.orchestra.run.smithi134.stdout: "reshard_pool": "test-zone1.rgw.log:reshard", 2024-08-18T15:18:37.133 INFO:teuthology.orchestra.run.smithi134.stdout: "user_keys_pool": "test-zone1.rgw.meta:users.keys", 2024-08-18T15:18:37.133 INFO:teuthology.orchestra.run.smithi134.stdout: "user_email_pool": "test-zone1.rgw.meta:users.email", 2024-08-18T15:18:37.133 INFO:teuthology.orchestra.run.smithi134.stdout: "user_swift_pool": "test-zone1.rgw.meta:users.swift", 2024-08-18T15:18:37.133 INFO:teuthology.orchestra.run.smithi134.stdout: "user_uid_pool": "test-zone1.rgw.meta:users.uid", 2024-08-18T15:18:37.134 INFO:teuthology.orchestra.run.smithi134.stdout: "otp_pool": "test-zone1.rgw.otp", 2024-08-18T15:18:37.134 INFO:teuthology.orchestra.run.smithi134.stdout: "system_key": { 2024-08-18T15:18:37.134 INFO:teuthology.orchestra.run.smithi134.stdout: "access_key": "MRX9HX7RYBDXO1UM", 2024-08-18T15:18:37.134 INFO:teuthology.orchestra.run.smithi134.stdout: "secret_key": "hGUsvhUdpGZAYWjQwu3StBaYXJPhuCEj" 2024-08-18T15:18:37.134 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:18:37.134 INFO:teuthology.orchestra.run.smithi134.stdout: "placement_pools": [ 2024-08-18T15:18:37.134 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:18:37.134 INFO:teuthology.orchestra.run.smithi134.stdout: "key": "default-placement", 2024-08-18T15:18:37.134 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:18:37.134 INFO:teuthology.orchestra.run.smithi134.stdout: "index_pool": "test-zone1.rgw.buckets.index", 2024-08-18T15:18:37.134 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_classes": { 2024-08-18T15:18:37.134 INFO:teuthology.orchestra.run.smithi134.stdout: "STANDARD": { 2024-08-18T15:18:37.134 INFO:teuthology.orchestra.run.smithi134.stdout: "data_pool": "test-zone1.rgw.buckets.data", 2024-08-18T15:18:37.135 INFO:teuthology.orchestra.run.smithi134.stdout: "compression_type": "random" 2024-08-18T15:18:37.135 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:18:37.135 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:18:37.135 INFO:teuthology.orchestra.run.smithi134.stdout: "data_extra_pool": "test-zone1.rgw.buckets.non-ec", 2024-08-18T15:18:37.135 INFO:teuthology.orchestra.run.smithi134.stdout: "index_type": 0, 2024-08-18T15:18:37.135 INFO:teuthology.orchestra.run.smithi134.stdout: "inline_data": true 2024-08-18T15:18:37.135 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:18:37.135 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:18:37.135 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T15:18:37.135 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_id": "2827c6f0-bf0d-4836-944c-9e7fcf22457b", 2024-08-18T15:18:37.135 INFO:teuthology.orchestra.run.smithi134.stdout: "notif_pool": "test-zone1.rgw.log:notif" 2024-08-18T15:18:37.135 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T15:18:37.136 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:18:37.136 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:18:37.137 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T15:18:37.183 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:18:37.183 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:18:37.196 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:18:37.195+0000 7fe007215880 0 period (453536fc-4ae4-4b27-be1c-ba0601521b0c does not have zone 96bf8502-2f39-4d53-8196-3378c28da3b4 configured 2024-08-18T15:18:44.148 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T15:18:44.149 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "b0d74758-1871-4454-bc0f-e3e4b6c30105", 2024-08-18T15:18:44.149 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 1, 2024-08-18T15:18:44.149 INFO:teuthology.orchestra.run.smithi134.stdout: "predecessor_uuid": "453536fc-4ae4-4b27-be1c-ba0601521b0c", 2024-08-18T15:18:44.149 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": [], 2024-08-18T15:18:44.149 INFO:teuthology.orchestra.run.smithi134.stdout: "period_map": { 2024-08-18T15:18:44.149 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "b0d74758-1871-4454-bc0f-e3e4b6c30105", 2024-08-18T15:18:44.149 INFO:teuthology.orchestra.run.smithi134.stdout: "zonegroups": [ 2024-08-18T15:18:44.149 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:18:44.149 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "test-zonegroup", 2024-08-18T15:18:44.149 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "test-zonegroup", 2024-08-18T15:18:44.149 INFO:teuthology.orchestra.run.smithi134.stdout: "api_name": "", 2024-08-18T15:18:44.149 INFO:teuthology.orchestra.run.smithi134.stdout: "is_master": true, 2024-08-18T15:18:44.150 INFO:teuthology.orchestra.run.smithi134.stdout: "endpoints": [ 2024-08-18T15:18:44.150 INFO:teuthology.orchestra.run.smithi134.stdout: "http://smithi134.front.sepia.ceph.com:8000" 2024-08-18T15:18:44.150 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T15:18:44.150 INFO:teuthology.orchestra.run.smithi134.stdout: "hostnames": [], 2024-08-18T15:18:44.150 INFO:teuthology.orchestra.run.smithi134.stdout: "hostnames_s3website": [], 2024-08-18T15:18:44.150 INFO:teuthology.orchestra.run.smithi134.stdout: "master_zone": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:18:44.150 INFO:teuthology.orchestra.run.smithi134.stdout: "zones": [ 2024-08-18T15:18:44.150 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:18:44.150 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:18:44.150 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "test-zone1", 2024-08-18T15:18:44.150 INFO:teuthology.orchestra.run.smithi134.stdout: "endpoints": [ 2024-08-18T15:18:44.150 INFO:teuthology.orchestra.run.smithi134.stdout: "http://smithi134.front.sepia.ceph.com:8000" 2024-08-18T15:18:44.150 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T15:18:44.151 INFO:teuthology.orchestra.run.smithi134.stdout: "log_meta": false, 2024-08-18T15:18:44.151 INFO:teuthology.orchestra.run.smithi134.stdout: "log_data": false, 2024-08-18T15:18:44.151 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_index_max_shards": 11, 2024-08-18T15:18:44.151 INFO:teuthology.orchestra.run.smithi134.stdout: "read_only": false, 2024-08-18T15:18:44.151 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_type": "", 2024-08-18T15:18:44.151 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from_all": true, 2024-08-18T15:18:44.151 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from": [], 2024-08-18T15:18:44.151 INFO:teuthology.orchestra.run.smithi134.stdout: "redirect_zone": "", 2024-08-18T15:18:44.151 INFO:teuthology.orchestra.run.smithi134.stdout: "supported_features": [ 2024-08-18T15:18:44.151 INFO:teuthology.orchestra.run.smithi134.stdout: "compress-encrypted", 2024-08-18T15:18:44.151 INFO:teuthology.orchestra.run.smithi134.stdout: "resharding" 2024-08-18T15:18:44.151 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T15:18:44.151 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:18:44.151 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T15:18:44.152 INFO:teuthology.orchestra.run.smithi134.stdout: "placement_targets": [ 2024-08-18T15:18:44.152 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:18:44.152 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "default-placement", 2024-08-18T15:18:44.152 INFO:teuthology.orchestra.run.smithi134.stdout: "tags": [], 2024-08-18T15:18:44.152 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_classes": [ 2024-08-18T15:18:44.152 INFO:teuthology.orchestra.run.smithi134.stdout: "STANDARD" 2024-08-18T15:18:44.152 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T15:18:44.152 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:18:44.152 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T15:18:44.153 INFO:teuthology.orchestra.run.smithi134.stdout: "default_placement": "default-placement", 2024-08-18T15:18:44.153 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_id": "2827c6f0-bf0d-4836-944c-9e7fcf22457b", 2024-08-18T15:18:44.153 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_policy": { 2024-08-18T15:18:44.153 INFO:teuthology.orchestra.run.smithi134.stdout: "groups": [] 2024-08-18T15:18:44.153 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:18:44.153 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled_features": [ 2024-08-18T15:18:44.154 INFO:teuthology.orchestra.run.smithi134.stdout: "resharding" 2024-08-18T15:18:44.154 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T15:18:44.154 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:18:44.154 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T15:18:44.154 INFO:teuthology.orchestra.run.smithi134.stdout: "short_zone_ids": [ 2024-08-18T15:18:44.154 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:18:44.154 INFO:teuthology.orchestra.run.smithi134.stdout: "key": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:18:44.154 INFO:teuthology.orchestra.run.smithi134.stdout: "val": 1721399886 2024-08-18T15:18:44.154 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:18:44.154 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T15:18:44.154 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:18:44.154 INFO:teuthology.orchestra.run.smithi134.stdout: "master_zonegroup": "test-zonegroup", 2024-08-18T15:18:44.155 INFO:teuthology.orchestra.run.smithi134.stdout: "master_zone": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:18:44.155 INFO:teuthology.orchestra.run.smithi134.stdout: "period_config": { 2024-08-18T15:18:44.155 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_quota": { 2024-08-18T15:18:44.155 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false, 2024-08-18T15:18:44.155 INFO:teuthology.orchestra.run.smithi134.stdout: "check_on_raw": false, 2024-08-18T15:18:44.155 INFO:teuthology.orchestra.run.smithi134.stdout: "max_size": -1, 2024-08-18T15:18:44.155 INFO:teuthology.orchestra.run.smithi134.stdout: "max_size_kb": 0, 2024-08-18T15:18:44.155 INFO:teuthology.orchestra.run.smithi134.stdout: "max_objects": -1 2024-08-18T15:18:44.155 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:18:44.155 INFO:teuthology.orchestra.run.smithi134.stdout: "user_quota": { 2024-08-18T15:18:44.156 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false, 2024-08-18T15:18:44.156 INFO:teuthology.orchestra.run.smithi134.stdout: "check_on_raw": false, 2024-08-18T15:18:44.156 INFO:teuthology.orchestra.run.smithi134.stdout: "max_size": -1, 2024-08-18T15:18:44.156 INFO:teuthology.orchestra.run.smithi134.stdout: "max_size_kb": 0, 2024-08-18T15:18:44.156 INFO:teuthology.orchestra.run.smithi134.stdout: "max_objects": -1 2024-08-18T15:18:44.156 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:18:44.156 INFO:teuthology.orchestra.run.smithi134.stdout: "user_ratelimit": { 2024-08-18T15:18:44.156 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_ops": 0, 2024-08-18T15:18:44.156 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_ops": 0, 2024-08-18T15:18:44.156 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_bytes": 0, 2024-08-18T15:18:44.156 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_bytes": 0, 2024-08-18T15:18:44.157 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false 2024-08-18T15:18:44.157 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:18:44.157 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_ratelimit": { 2024-08-18T15:18:44.157 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_ops": 0, 2024-08-18T15:18:44.157 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_ops": 0, 2024-08-18T15:18:44.157 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_bytes": 0, 2024-08-18T15:18:44.157 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_bytes": 0, 2024-08-18T15:18:44.157 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false 2024-08-18T15:18:44.157 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:18:44.157 INFO:teuthology.orchestra.run.smithi134.stdout: "anonymous_ratelimit": { 2024-08-18T15:18:44.158 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_ops": 0, 2024-08-18T15:18:44.158 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_ops": 0, 2024-08-18T15:18:44.158 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_bytes": 0, 2024-08-18T15:18:44.158 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_bytes": 0, 2024-08-18T15:18:44.158 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false 2024-08-18T15:18:44.158 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:18:44.158 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:18:44.158 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_id": "2827c6f0-bf0d-4836-944c-9e7fcf22457b", 2024-08-18T15:18:44.158 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_name": "test-realm", 2024-08-18T15:18:44.159 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:18:44.159 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T15:18:44.160 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2024-08-18T15:18:44.160 INFO:teuthology.orchestra.run:waiting for 300 2024-08-18T15:18:44.160 INFO:tasks.rgw.c1.client.0.smithi134.stdout:2024-08-18T15:18:44.160+0000 7fe8a63ca640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground (PID: 37684) UID: 0 2024-08-18T15:18:44.161 INFO:tasks.rgw.c1.client.0.smithi134.stdout:2024-08-18T15:18:44.160+0000 7fe8ac42f880 -1 shutting down 2024-08-18T15:18:47.040 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2024-08-18T15:18:50.261 INFO:tasks.rgw.c1.client.0:Stopped 2024-08-18T15:18:50.261 INFO:tasks.rgw.c1.client.0:Restarting daemon 2024-08-18T15:18:50.261 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2024-08-18T15:18:50.264 INFO:tasks.rgw.c1.client.0:Started 2024-08-18T15:18:50.264 DEBUG:teuthology.orchestra.run.smithi134:> curl http://smithi134.front.sepia.ceph.com:8000 2024-08-18T15:18:50.287 INFO:teuthology.orchestra.run.smithi134.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-08-18T15:18:50.287 INFO:teuthology.orchestra.run.smithi134.stderr: Dload Upload Total Spent Left Speed 2024-08-18T15:18:50.287 INFO:teuthology.orchestra.run.smithi134.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2024-08-18T15:18:50.288 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-08-18T15:18:50.288 INFO:teuthology.orchestra.run.smithi134.stderr:curl: (7) Failed to connect to smithi134.front.sepia.ceph.com port 8000: Connection refused 2024-08-18T15:18:51.289 DEBUG:teuthology.orchestra.run.smithi134:> curl http://smithi134.front.sepia.ceph.com:8000 2024-08-18T15:18:51.309 INFO:teuthology.orchestra.run.smithi134.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-08-18T15:18:51.310 INFO:teuthology.orchestra.run.smithi134.stderr: Dload Upload Total Spent Left Speed 2024-08-18T15:18:51.310 INFO:teuthology.orchestra.run.smithi134.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-08-18T15:18:51.311 INFO:teuthology.orchestra.run.smithi134.stdout:anonymous 2024-08-18T15:18:51.311 INFO:tasks.rgw_multisite:creating admin user.. 2024-08-18T15:18:51.311 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'MRX9HX7RYBDXO1UM', '--secret', 'hGUsvhUdpGZAYWjQwu3StBaYXJPhuCEj', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:18:51.312 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'MRX9HX7RYBDXO1UM', '--secret', 'hGUsvhUdpGZAYWjQwu3StBaYXJPhuCEj', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:18:51.312 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid realm-admin --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --display-name 'Realm Admin' --system --access-key MRX9HX7RYBDXO1UM --secret hGUsvhUdpGZAYWjQwu3StBaYXJPhuCEj --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T15:18:51.406 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:18:51.406 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:18:51.440 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T15:18:51.440 INFO:teuthology.orchestra.run.smithi134.stdout: "user_id": "realm-admin", 2024-08-18T15:18:51.440 INFO:teuthology.orchestra.run.smithi134.stdout: "display_name": "Realm Admin", 2024-08-18T15:18:51.440 INFO:teuthology.orchestra.run.smithi134.stdout: "email": "", 2024-08-18T15:18:51.440 INFO:teuthology.orchestra.run.smithi134.stdout: "suspended": 0, 2024-08-18T15:18:51.441 INFO:teuthology.orchestra.run.smithi134.stdout: "max_buckets": 1000, 2024-08-18T15:18:51.441 INFO:teuthology.orchestra.run.smithi134.stdout: "subusers": [], 2024-08-18T15:18:51.441 INFO:teuthology.orchestra.run.smithi134.stdout: "keys": [ 2024-08-18T15:18:51.441 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:18:51.441 INFO:teuthology.orchestra.run.smithi134.stdout: "user": "realm-admin", 2024-08-18T15:18:51.441 INFO:teuthology.orchestra.run.smithi134.stdout: "access_key": "MRX9HX7RYBDXO1UM", 2024-08-18T15:18:51.441 INFO:teuthology.orchestra.run.smithi134.stdout: "secret_key": "hGUsvhUdpGZAYWjQwu3StBaYXJPhuCEj" 2024-08-18T15:18:51.441 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:18:51.441 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T15:18:51.441 INFO:teuthology.orchestra.run.smithi134.stdout: "swift_keys": [], 2024-08-18T15:18:51.441 INFO:teuthology.orchestra.run.smithi134.stdout: "caps": [], 2024-08-18T15:18:51.441 INFO:teuthology.orchestra.run.smithi134.stdout: "op_mask": "read, write, delete", 2024-08-18T15:18:51.441 INFO:teuthology.orchestra.run.smithi134.stdout: "system": true, 2024-08-18T15:18:51.442 INFO:teuthology.orchestra.run.smithi134.stdout: "default_placement": "", 2024-08-18T15:18:51.442 INFO:teuthology.orchestra.run.smithi134.stdout: "default_storage_class": "", 2024-08-18T15:18:51.442 INFO:teuthology.orchestra.run.smithi134.stdout: "placement_tags": [], 2024-08-18T15:18:51.442 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_quota": { 2024-08-18T15:18:51.442 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false, 2024-08-18T15:18:51.442 INFO:teuthology.orchestra.run.smithi134.stdout: "check_on_raw": false, 2024-08-18T15:18:51.442 INFO:teuthology.orchestra.run.smithi134.stdout: "max_size": -1, 2024-08-18T15:18:51.442 INFO:teuthology.orchestra.run.smithi134.stdout: "max_size_kb": 0, 2024-08-18T15:18:51.442 INFO:teuthology.orchestra.run.smithi134.stdout: "max_objects": -1 2024-08-18T15:18:51.442 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:18:51.442 INFO:teuthology.orchestra.run.smithi134.stdout: "user_quota": { 2024-08-18T15:18:51.442 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false, 2024-08-18T15:18:51.443 INFO:teuthology.orchestra.run.smithi134.stdout: "check_on_raw": false, 2024-08-18T15:18:51.443 INFO:teuthology.orchestra.run.smithi134.stdout: "max_size": -1, 2024-08-18T15:18:51.443 INFO:teuthology.orchestra.run.smithi134.stdout: "max_size_kb": 0, 2024-08-18T15:18:51.443 INFO:teuthology.orchestra.run.smithi134.stdout: "max_objects": -1 2024-08-18T15:18:51.443 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:18:51.443 INFO:teuthology.orchestra.run.smithi134.stdout: "temp_url_keys": [], 2024-08-18T15:18:51.443 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "rgw", 2024-08-18T15:18:51.443 INFO:teuthology.orchestra.run.smithi134.stdout: "mfa_ids": [] 2024-08-18T15:18:51.443 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T15:18:51.443 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:18:51.453 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2024-08-18T15:18:51.453 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi134.front.sepia.ceph.com:8000', '--access-key', 'MRX9HX7RYBDXO1UM', '--secret', 'hGUsvhUdpGZAYWjQwu3StBaYXJPhuCEj', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:18:51.453 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi134.front.sepia.ceph.com:8000', '--access-key', 'MRX9HX7RYBDXO1UM', '--secret', 'hGUsvhUdpGZAYWjQwu3StBaYXJPhuCEj', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:18:51.453 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi134.front.sepia.ceph.com:8000 --access-key MRX9HX7RYBDXO1UM --secret hGUsvhUdpGZAYWjQwu3StBaYXJPhuCEj --cluster c2 --debug-rgw 0 --debug-ms 0 2024-08-18T15:18:51.512 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:18:51.513 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:18:51.548 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:18:51.549 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "2827c6f0-bf0d-4836-944c-9e7fcf22457b", 2024-08-18T15:18:51.549 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "test-realm", 2024-08-18T15:18:51.549 INFO:teuthology.orchestra.run.smithi193.stdout: "current_period": "b0d74758-1871-4454-bc0f-e3e4b6c30105", 2024-08-18T15:18:51.549 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 2 2024-08-18T15:18:51.549 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:18:51.550 INFO:tasks.rgw_multisite:creating zone.. 2024-08-18T15:18:51.550 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--access-key', 'MRX9HX7RYBDXO1UM', '--secret', 'hGUsvhUdpGZAYWjQwu3StBaYXJPhuCEj', '--endpoints', 'http://smithi193.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:18:51.550 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--access-key', 'MRX9HX7RYBDXO1UM', '--secret', 'hGUsvhUdpGZAYWjQwu3StBaYXJPhuCEj', '--endpoints', 'http://smithi193.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:18:51.551 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --access-key MRX9HX7RYBDXO1UM --secret hGUsvhUdpGZAYWjQwu3StBaYXJPhuCEj --endpoints http://smithi193.front.sepia.ceph.com:8000 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-08-18T15:18:51.606 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:18:51.606 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:18:51.621 INFO:teuthology.orchestra.run.smithi193.stderr:NOTICE: set zone's realm_id=2827c6f0-bf0d-4836-944c-9e7fcf22457b 2024-08-18T15:18:51.637 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:18:51.637 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "10090ccf-aad6-416d-9775-df8b7b27345a", 2024-08-18T15:18:51.637 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "test-zone2", 2024-08-18T15:18:51.638 INFO:teuthology.orchestra.run.smithi193.stdout: "domain_root": "test-zone2.rgw.meta:root", 2024-08-18T15:18:51.638 INFO:teuthology.orchestra.run.smithi193.stdout: "control_pool": "test-zone2.rgw.control", 2024-08-18T15:18:51.638 INFO:teuthology.orchestra.run.smithi193.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2024-08-18T15:18:51.638 INFO:teuthology.orchestra.run.smithi193.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2024-08-18T15:18:51.638 INFO:teuthology.orchestra.run.smithi193.stdout: "log_pool": "test-zone2.rgw.log", 2024-08-18T15:18:51.638 INFO:teuthology.orchestra.run.smithi193.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2024-08-18T15:18:51.638 INFO:teuthology.orchestra.run.smithi193.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2024-08-18T15:18:51.638 INFO:teuthology.orchestra.run.smithi193.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2024-08-18T15:18:51.638 INFO:teuthology.orchestra.run.smithi193.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2024-08-18T15:18:51.638 INFO:teuthology.orchestra.run.smithi193.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2024-08-18T15:18:51.638 INFO:teuthology.orchestra.run.smithi193.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2024-08-18T15:18:51.638 INFO:teuthology.orchestra.run.smithi193.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2024-08-18T15:18:51.639 INFO:teuthology.orchestra.run.smithi193.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2024-08-18T15:18:51.639 INFO:teuthology.orchestra.run.smithi193.stdout: "otp_pool": "test-zone2.rgw.otp", 2024-08-18T15:18:51.639 INFO:teuthology.orchestra.run.smithi193.stdout: "system_key": { 2024-08-18T15:18:51.639 INFO:teuthology.orchestra.run.smithi193.stdout: "access_key": "MRX9HX7RYBDXO1UM", 2024-08-18T15:18:51.639 INFO:teuthology.orchestra.run.smithi193.stdout: "secret_key": "hGUsvhUdpGZAYWjQwu3StBaYXJPhuCEj" 2024-08-18T15:18:51.639 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:18:51.639 INFO:teuthology.orchestra.run.smithi193.stdout: "placement_pools": [ 2024-08-18T15:18:51.639 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:18:51.639 INFO:teuthology.orchestra.run.smithi193.stdout: "key": "default-placement", 2024-08-18T15:18:51.639 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:18:51.639 INFO:teuthology.orchestra.run.smithi193.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2024-08-18T15:18:51.639 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_classes": { 2024-08-18T15:18:51.639 INFO:teuthology.orchestra.run.smithi193.stdout: "STANDARD": { 2024-08-18T15:18:51.640 INFO:teuthology.orchestra.run.smithi193.stdout: "data_pool": "test-zone2.rgw.buckets.data" 2024-08-18T15:18:51.640 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:18:51.640 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:18:51.640 INFO:teuthology.orchestra.run.smithi193.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2024-08-18T15:18:51.640 INFO:teuthology.orchestra.run.smithi193.stdout: "index_type": 0, 2024-08-18T15:18:51.640 INFO:teuthology.orchestra.run.smithi193.stdout: "inline_data": true 2024-08-18T15:18:51.640 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:18:51.640 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:18:51.640 INFO:teuthology.orchestra.run.smithi193.stdout: ], 2024-08-18T15:18:51.640 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_id": "2827c6f0-bf0d-4836-944c-9e7fcf22457b", 2024-08-18T15:18:51.640 INFO:teuthology.orchestra.run.smithi193.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2024-08-18T15:18:51.640 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:18:51.641 DEBUG:teuthology.orchestra.run.smithi193:> sudo ceph osd pool create test-zone2.rgw.buckets.data 64 64 --cluster c2 2024-08-18T15:18:52.747 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 has been restored 2024-08-18T15:18:52.895 INFO:teuthology.orchestra.run.smithi193.stderr:pool 'test-zone2.rgw.buckets.data' created 2024-08-18T15:18:52.919 DEBUG:teuthology.orchestra.run.smithi193:> sudo ceph osd pool application enable test-zone2.rgw.buckets.data rgw --cluster c2 2024-08-18T15:18:53.895 INFO:teuthology.orchestra.run.smithi193.stderr:enabled application 'rgw' on pool 'test-zone2.rgw.buckets.data' 2024-08-18T15:18:53.933 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:18:53.933 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:18:53.933 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2024-08-18T15:18:53.985 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:18:53.985 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:18:54.004 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:18:54.005 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "10090ccf-aad6-416d-9775-df8b7b27345a", 2024-08-18T15:18:54.005 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "test-zone2", 2024-08-18T15:18:54.005 INFO:teuthology.orchestra.run.smithi193.stdout: "domain_root": "test-zone2.rgw.meta:root", 2024-08-18T15:18:54.005 INFO:teuthology.orchestra.run.smithi193.stdout: "control_pool": "test-zone2.rgw.control", 2024-08-18T15:18:54.005 INFO:teuthology.orchestra.run.smithi193.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2024-08-18T15:18:54.005 INFO:teuthology.orchestra.run.smithi193.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2024-08-18T15:18:54.005 INFO:teuthology.orchestra.run.smithi193.stdout: "log_pool": "test-zone2.rgw.log", 2024-08-18T15:18:54.005 INFO:teuthology.orchestra.run.smithi193.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2024-08-18T15:18:54.005 INFO:teuthology.orchestra.run.smithi193.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2024-08-18T15:18:54.005 INFO:teuthology.orchestra.run.smithi193.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2024-08-18T15:18:54.005 INFO:teuthology.orchestra.run.smithi193.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2024-08-18T15:18:54.005 INFO:teuthology.orchestra.run.smithi193.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2024-08-18T15:18:54.005 INFO:teuthology.orchestra.run.smithi193.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2024-08-18T15:18:54.006 INFO:teuthology.orchestra.run.smithi193.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2024-08-18T15:18:54.006 INFO:teuthology.orchestra.run.smithi193.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2024-08-18T15:18:54.006 INFO:teuthology.orchestra.run.smithi193.stdout: "otp_pool": "test-zone2.rgw.otp", 2024-08-18T15:18:54.006 INFO:teuthology.orchestra.run.smithi193.stdout: "system_key": { 2024-08-18T15:18:54.006 INFO:teuthology.orchestra.run.smithi193.stdout: "access_key": "MRX9HX7RYBDXO1UM", 2024-08-18T15:18:54.006 INFO:teuthology.orchestra.run.smithi193.stdout: "secret_key": "hGUsvhUdpGZAYWjQwu3StBaYXJPhuCEj" 2024-08-18T15:18:54.006 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:18:54.006 INFO:teuthology.orchestra.run.smithi193.stdout: "placement_pools": [ 2024-08-18T15:18:54.006 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:18:54.006 INFO:teuthology.orchestra.run.smithi193.stdout: "key": "default-placement", 2024-08-18T15:18:54.006 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:18:54.006 INFO:teuthology.orchestra.run.smithi193.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2024-08-18T15:18:54.007 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_classes": { 2024-08-18T15:18:54.007 INFO:teuthology.orchestra.run.smithi193.stdout: "STANDARD": { 2024-08-18T15:18:54.007 INFO:teuthology.orchestra.run.smithi193.stdout: "data_pool": "test-zone2.rgw.buckets.data", 2024-08-18T15:18:54.007 INFO:teuthology.orchestra.run.smithi193.stdout: "compression_type": "random" 2024-08-18T15:18:54.007 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:18:54.007 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:18:54.007 INFO:teuthology.orchestra.run.smithi193.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2024-08-18T15:18:54.007 INFO:teuthology.orchestra.run.smithi193.stdout: "index_type": 0, 2024-08-18T15:18:54.007 INFO:teuthology.orchestra.run.smithi193.stdout: "inline_data": true 2024-08-18T15:18:54.007 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:18:54.007 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:18:54.007 INFO:teuthology.orchestra.run.smithi193.stdout: ], 2024-08-18T15:18:54.007 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_id": "2827c6f0-bf0d-4836-944c-9e7fcf22457b", 2024-08-18T15:18:54.008 INFO:teuthology.orchestra.run.smithi193.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2024-08-18T15:18:54.008 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:18:54.009 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:18:54.009 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:18:54.009 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2024-08-18T15:18:54.059 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:18:54.059 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:18:54.068 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:18:54.068+0000 7f5daa43f880 0 period (b0d74758-1871-4454-bc0f-e3e4b6c30105 does not have zone 10090ccf-aad6-416d-9775-df8b7b27345a configured 2024-08-18T15:19:00.965 INFO:teuthology.orchestra.run.smithi193.stderr:Sending period to new master zone 96bf8502-2f39-4d53-8196-3378c28da3b4 2024-08-18T15:19:00.992 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:19:00.992 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "b0d74758-1871-4454-bc0f-e3e4b6c30105", 2024-08-18T15:19:00.992 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 2, 2024-08-18T15:19:00.992 INFO:teuthology.orchestra.run.smithi193.stdout: "predecessor_uuid": "453536fc-4ae4-4b27-be1c-ba0601521b0c", 2024-08-18T15:19:00.992 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": [], 2024-08-18T15:19:00.992 INFO:teuthology.orchestra.run.smithi193.stdout: "period_map": { 2024-08-18T15:19:00.992 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "b0d74758-1871-4454-bc0f-e3e4b6c30105", 2024-08-18T15:19:00.993 INFO:teuthology.orchestra.run.smithi193.stdout: "zonegroups": [ 2024-08-18T15:19:00.993 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:19:00.993 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "test-zonegroup", 2024-08-18T15:19:00.993 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "test-zonegroup", 2024-08-18T15:19:00.993 INFO:teuthology.orchestra.run.smithi193.stdout: "api_name": "", 2024-08-18T15:19:00.993 INFO:teuthology.orchestra.run.smithi193.stdout: "is_master": true, 2024-08-18T15:19:00.993 INFO:teuthology.orchestra.run.smithi193.stdout: "endpoints": [ 2024-08-18T15:19:00.993 INFO:teuthology.orchestra.run.smithi193.stdout: "http://smithi134.front.sepia.ceph.com:8000" 2024-08-18T15:19:00.993 INFO:teuthology.orchestra.run.smithi193.stdout: ], 2024-08-18T15:19:00.993 INFO:teuthology.orchestra.run.smithi193.stdout: "hostnames": [], 2024-08-18T15:19:00.993 INFO:teuthology.orchestra.run.smithi193.stdout: "hostnames_s3website": [], 2024-08-18T15:19:00.994 INFO:teuthology.orchestra.run.smithi193.stdout: "master_zone": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:19:00.994 INFO:teuthology.orchestra.run.smithi193.stdout: "zones": [ 2024-08-18T15:19:00.994 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:19:00.994 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "10090ccf-aad6-416d-9775-df8b7b27345a", 2024-08-18T15:19:00.994 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "test-zone2", 2024-08-18T15:19:00.994 INFO:teuthology.orchestra.run.smithi193.stdout: "endpoints": [ 2024-08-18T15:19:00.994 INFO:teuthology.orchestra.run.smithi193.stdout: "http://smithi193.front.sepia.ceph.com:8000" 2024-08-18T15:19:00.994 INFO:teuthology.orchestra.run.smithi193.stdout: ], 2024-08-18T15:19:00.994 INFO:teuthology.orchestra.run.smithi193.stdout: "log_meta": false, 2024-08-18T15:19:00.994 INFO:teuthology.orchestra.run.smithi193.stdout: "log_data": true, 2024-08-18T15:19:00.994 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket_index_max_shards": 11, 2024-08-18T15:19:00.995 INFO:teuthology.orchestra.run.smithi193.stdout: "read_only": false, 2024-08-18T15:19:00.995 INFO:teuthology.orchestra.run.smithi193.stdout: "tier_type": "", 2024-08-18T15:19:00.998 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_from_all": true, 2024-08-18T15:19:00.999 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_from": [], 2024-08-18T15:19:00.999 INFO:teuthology.orchestra.run.smithi193.stdout: "redirect_zone": "", 2024-08-18T15:19:00.999 INFO:teuthology.orchestra.run.smithi193.stdout: "supported_features": [ 2024-08-18T15:19:00.999 INFO:teuthology.orchestra.run.smithi193.stdout: "compress-encrypted", 2024-08-18T15:19:00.999 INFO:teuthology.orchestra.run.smithi193.stdout: "resharding" 2024-08-18T15:19:00.999 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:19:00.999 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:19:00.999 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:19:00.999 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:19:00.999 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "test-zone1", 2024-08-18T15:19:00.999 INFO:teuthology.orchestra.run.smithi193.stdout: "endpoints": [ 2024-08-18T15:19:01.000 INFO:teuthology.orchestra.run.smithi193.stdout: "http://smithi134.front.sepia.ceph.com:8000" 2024-08-18T15:19:01.000 INFO:teuthology.orchestra.run.smithi193.stdout: ], 2024-08-18T15:19:01.000 INFO:teuthology.orchestra.run.smithi193.stdout: "log_meta": false, 2024-08-18T15:19:01.000 INFO:teuthology.orchestra.run.smithi193.stdout: "log_data": true, 2024-08-18T15:19:01.000 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket_index_max_shards": 11, 2024-08-18T15:19:01.000 INFO:teuthology.orchestra.run.smithi193.stdout: "read_only": false, 2024-08-18T15:19:01.000 INFO:teuthology.orchestra.run.smithi193.stdout: "tier_type": "", 2024-08-18T15:19:01.000 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_from_all": true, 2024-08-18T15:19:01.000 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_from": [], 2024-08-18T15:19:01.000 INFO:teuthology.orchestra.run.smithi193.stdout: "redirect_zone": "", 2024-08-18T15:19:01.000 INFO:teuthology.orchestra.run.smithi193.stdout: "supported_features": [ 2024-08-18T15:19:01.001 INFO:teuthology.orchestra.run.smithi193.stdout: "compress-encrypted", 2024-08-18T15:19:01.001 INFO:teuthology.orchestra.run.smithi193.stdout: "resharding" 2024-08-18T15:19:01.001 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:19:01.001 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:19:01.001 INFO:teuthology.orchestra.run.smithi193.stdout: ], 2024-08-18T15:19:01.001 INFO:teuthology.orchestra.run.smithi193.stdout: "placement_targets": [ 2024-08-18T15:19:01.001 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:19:01.001 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "default-placement", 2024-08-18T15:19:01.001 INFO:teuthology.orchestra.run.smithi193.stdout: "tags": [], 2024-08-18T15:19:01.001 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_classes": [ 2024-08-18T15:19:01.001 INFO:teuthology.orchestra.run.smithi193.stdout: "STANDARD" 2024-08-18T15:19:01.002 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:19:01.002 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:19:01.002 INFO:teuthology.orchestra.run.smithi193.stdout: ], 2024-08-18T15:19:01.002 INFO:teuthology.orchestra.run.smithi193.stdout: "default_placement": "default-placement", 2024-08-18T15:19:01.002 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_id": "2827c6f0-bf0d-4836-944c-9e7fcf22457b", 2024-08-18T15:19:01.002 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_policy": { 2024-08-18T15:19:01.002 INFO:teuthology.orchestra.run.smithi193.stdout: "groups": [] 2024-08-18T15:19:01.002 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:19:01.002 INFO:teuthology.orchestra.run.smithi193.stdout: "enabled_features": [ 2024-08-18T15:19:01.002 INFO:teuthology.orchestra.run.smithi193.stdout: "resharding" 2024-08-18T15:19:01.002 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:19:01.003 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:19:01.003 INFO:teuthology.orchestra.run.smithi193.stdout: ], 2024-08-18T15:19:01.003 INFO:teuthology.orchestra.run.smithi193.stdout: "short_zone_ids": [ 2024-08-18T15:19:01.003 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:19:01.003 INFO:teuthology.orchestra.run.smithi193.stdout: "key": "10090ccf-aad6-416d-9775-df8b7b27345a", 2024-08-18T15:19:01.003 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 874289624 2024-08-18T15:19:01.003 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:19:01.003 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:19:01.003 INFO:teuthology.orchestra.run.smithi193.stdout: "key": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:19:01.003 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 1721399886 2024-08-18T15:19:01.003 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:19:01.003 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:19:01.004 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:19:01.004 INFO:teuthology.orchestra.run.smithi193.stdout: "master_zonegroup": "test-zonegroup", 2024-08-18T15:19:01.004 INFO:teuthology.orchestra.run.smithi193.stdout: "master_zone": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:19:01.004 INFO:teuthology.orchestra.run.smithi193.stdout: "period_config": { 2024-08-18T15:19:01.004 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket_quota": { 2024-08-18T15:19:01.004 INFO:teuthology.orchestra.run.smithi193.stdout: "enabled": false, 2024-08-18T15:19:01.004 INFO:teuthology.orchestra.run.smithi193.stdout: "check_on_raw": false, 2024-08-18T15:19:01.004 INFO:teuthology.orchestra.run.smithi193.stdout: "max_size": -1, 2024-08-18T15:19:01.004 INFO:teuthology.orchestra.run.smithi193.stdout: "max_size_kb": 0, 2024-08-18T15:19:01.004 INFO:teuthology.orchestra.run.smithi193.stdout: "max_objects": -1 2024-08-18T15:19:01.004 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:19:01.005 INFO:teuthology.orchestra.run.smithi193.stdout: "user_quota": { 2024-08-18T15:19:01.005 INFO:teuthology.orchestra.run.smithi193.stdout: "enabled": false, 2024-08-18T15:19:01.005 INFO:teuthology.orchestra.run.smithi193.stdout: "check_on_raw": false, 2024-08-18T15:19:01.005 INFO:teuthology.orchestra.run.smithi193.stdout: "max_size": -1, 2024-08-18T15:19:01.005 INFO:teuthology.orchestra.run.smithi193.stdout: "max_size_kb": 0, 2024-08-18T15:19:01.005 INFO:teuthology.orchestra.run.smithi193.stdout: "max_objects": -1 2024-08-18T15:19:01.005 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:19:01.005 INFO:teuthology.orchestra.run.smithi193.stdout: "user_ratelimit": { 2024-08-18T15:19:01.005 INFO:teuthology.orchestra.run.smithi193.stdout: "max_read_ops": 0, 2024-08-18T15:19:01.005 INFO:teuthology.orchestra.run.smithi193.stdout: "max_write_ops": 0, 2024-08-18T15:19:01.005 INFO:teuthology.orchestra.run.smithi193.stdout: "max_read_bytes": 0, 2024-08-18T15:19:01.005 INFO:teuthology.orchestra.run.smithi193.stdout: "max_write_bytes": 0, 2024-08-18T15:19:01.006 INFO:teuthology.orchestra.run.smithi193.stdout: "enabled": false 2024-08-18T15:19:01.006 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:19:01.006 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket_ratelimit": { 2024-08-18T15:19:01.006 INFO:teuthology.orchestra.run.smithi193.stdout: "max_read_ops": 0, 2024-08-18T15:19:01.006 INFO:teuthology.orchestra.run.smithi193.stdout: "max_write_ops": 0, 2024-08-18T15:19:01.008 INFO:teuthology.orchestra.run.smithi193.stdout: "max_read_bytes": 0, 2024-08-18T15:19:01.008 INFO:teuthology.orchestra.run.smithi193.stdout: "max_write_bytes": 0, 2024-08-18T15:19:01.008 INFO:teuthology.orchestra.run.smithi193.stdout: "enabled": false 2024-08-18T15:19:01.008 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:19:01.008 INFO:teuthology.orchestra.run.smithi193.stdout: "anonymous_ratelimit": { 2024-08-18T15:19:01.009 INFO:teuthology.orchestra.run.smithi193.stdout: "max_read_ops": 0, 2024-08-18T15:19:01.009 INFO:teuthology.orchestra.run.smithi193.stdout: "max_write_ops": 0, 2024-08-18T15:19:01.009 INFO:teuthology.orchestra.run.smithi193.stdout: "max_read_bytes": 0, 2024-08-18T15:19:01.009 INFO:teuthology.orchestra.run.smithi193.stdout: "max_write_bytes": 0, 2024-08-18T15:19:01.009 INFO:teuthology.orchestra.run.smithi193.stdout: "enabled": false 2024-08-18T15:19:01.009 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:19:01.009 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:19:01.009 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_id": "2827c6f0-bf0d-4836-944c-9e7fcf22457b", 2024-08-18T15:19:01.009 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_name": "test-realm", 2024-08-18T15:19:01.009 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:19:01.009 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:19:01.011 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2024-08-18T15:19:01.011 INFO:teuthology.orchestra.run:waiting for 300 2024-08-18T15:19:01.011 INFO:tasks.rgw.c2.client.0.smithi193.stdout:2024-08-18T15:19:01.011+0000 7fd4f3efc640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground (PID: 37340) UID: 0 2024-08-18T15:19:01.012 INFO:tasks.rgw.c2.client.0.smithi193.stdout:2024-08-18T15:19:01.011+0000 7fd4fa40e880 -1 shutting down 2024-08-18T15:19:05.032 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2024-08-18T15:19:07.112 INFO:tasks.rgw.c2.client.0:Stopped 2024-08-18T15:19:07.112 INFO:tasks.rgw.c2.client.0:Restarting daemon 2024-08-18T15:19:07.112 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2024-08-18T15:19:07.115 INFO:tasks.rgw.c2.client.0:Started 2024-08-18T15:19:07.115 DEBUG:teuthology.orchestra.run.smithi193:> curl http://smithi193.front.sepia.ceph.com:8000 2024-08-18T15:19:07.137 INFO:teuthology.orchestra.run.smithi193.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-08-18T15:19:07.138 INFO:teuthology.orchestra.run.smithi193.stderr: Dload Upload Total Spent Left Speed 2024-08-18T15:19:07.138 INFO:teuthology.orchestra.run.smithi193.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2024-08-18T15:19:07.139 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-08-18T15:19:07.139 INFO:teuthology.orchestra.run.smithi193.stderr:curl: (7) Failed to connect to smithi193.front.sepia.ceph.com port 8000: Connection refused 2024-08-18T15:19:08.141 DEBUG:teuthology.orchestra.run.smithi193:> curl http://smithi193.front.sepia.ceph.com:8000 2024-08-18T15:19:08.161 INFO:teuthology.orchestra.run.smithi193.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-08-18T15:19:08.162 INFO:teuthology.orchestra.run.smithi193.stderr: Dload Upload Total Spent Left Speed 2024-08-18T15:19:08.163 INFO:teuthology.orchestra.run.smithi193.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-08-18T15:19:08.163 INFO:teuthology.orchestra.run.smithi193.stdout:anonymous 2024-08-18T15:19:08.163 INFO:tasks.rgw_multisite:creating zone.. 2024-08-18T15:19:08.164 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'MRX9HX7RYBDXO1UM', '--secret', 'hGUsvhUdpGZAYWjQwu3StBaYXJPhuCEj', '--endpoints', 'http://smithi134.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:19:08.164 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'MRX9HX7RYBDXO1UM', '--secret', 'hGUsvhUdpGZAYWjQwu3StBaYXJPhuCEj', '--endpoints', 'http://smithi134.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:19:08.164 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key MRX9HX7RYBDXO1UM --secret hGUsvhUdpGZAYWjQwu3StBaYXJPhuCEj --endpoints http://smithi134.front.sepia.ceph.com:8001 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T15:19:08.221 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:19:08.221 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:19:08.237 INFO:teuthology.orchestra.run.smithi134.stderr:NOTICE: set zone's realm_id=2827c6f0-bf0d-4836-944c-9e7fcf22457b 2024-08-18T15:19:08.252 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T15:19:08.252 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T15:19:08.252 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "test-zone3", 2024-08-18T15:19:08.252 INFO:teuthology.orchestra.run.smithi134.stdout: "domain_root": "test-zone3.rgw.meta:root", 2024-08-18T15:19:08.252 INFO:teuthology.orchestra.run.smithi134.stdout: "control_pool": "test-zone3.rgw.control", 2024-08-18T15:19:08.253 INFO:teuthology.orchestra.run.smithi134.stdout: "gc_pool": "test-zone3.rgw.log:gc", 2024-08-18T15:19:08.253 INFO:teuthology.orchestra.run.smithi134.stdout: "lc_pool": "test-zone3.rgw.log:lc", 2024-08-18T15:19:08.253 INFO:teuthology.orchestra.run.smithi134.stdout: "log_pool": "test-zone3.rgw.log", 2024-08-18T15:19:08.253 INFO:teuthology.orchestra.run.smithi134.stdout: "intent_log_pool": "test-zone3.rgw.log:intent", 2024-08-18T15:19:08.253 INFO:teuthology.orchestra.run.smithi134.stdout: "usage_log_pool": "test-zone3.rgw.log:usage", 2024-08-18T15:19:08.253 INFO:teuthology.orchestra.run.smithi134.stdout: "roles_pool": "test-zone3.rgw.meta:roles", 2024-08-18T15:19:08.253 INFO:teuthology.orchestra.run.smithi134.stdout: "reshard_pool": "test-zone3.rgw.log:reshard", 2024-08-18T15:19:08.253 INFO:teuthology.orchestra.run.smithi134.stdout: "user_keys_pool": "test-zone3.rgw.meta:users.keys", 2024-08-18T15:19:08.253 INFO:teuthology.orchestra.run.smithi134.stdout: "user_email_pool": "test-zone3.rgw.meta:users.email", 2024-08-18T15:19:08.253 INFO:teuthology.orchestra.run.smithi134.stdout: "user_swift_pool": "test-zone3.rgw.meta:users.swift", 2024-08-18T15:19:08.253 INFO:teuthology.orchestra.run.smithi134.stdout: "user_uid_pool": "test-zone3.rgw.meta:users.uid", 2024-08-18T15:19:08.253 INFO:teuthology.orchestra.run.smithi134.stdout: "otp_pool": "test-zone3.rgw.otp", 2024-08-18T15:19:08.253 INFO:teuthology.orchestra.run.smithi134.stdout: "system_key": { 2024-08-18T15:19:08.254 INFO:teuthology.orchestra.run.smithi134.stdout: "access_key": "MRX9HX7RYBDXO1UM", 2024-08-18T15:19:08.254 INFO:teuthology.orchestra.run.smithi134.stdout: "secret_key": "hGUsvhUdpGZAYWjQwu3StBaYXJPhuCEj" 2024-08-18T15:19:08.254 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:19:08.254 INFO:teuthology.orchestra.run.smithi134.stdout: "placement_pools": [ 2024-08-18T15:19:08.254 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:19:08.254 INFO:teuthology.orchestra.run.smithi134.stdout: "key": "default-placement", 2024-08-18T15:19:08.254 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:19:08.254 INFO:teuthology.orchestra.run.smithi134.stdout: "index_pool": "test-zone3.rgw.buckets.index", 2024-08-18T15:19:08.254 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_classes": { 2024-08-18T15:19:08.254 INFO:teuthology.orchestra.run.smithi134.stdout: "STANDARD": { 2024-08-18T15:19:08.254 INFO:teuthology.orchestra.run.smithi134.stdout: "data_pool": "test-zone3.rgw.buckets.data" 2024-08-18T15:19:08.254 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:19:08.254 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:19:08.255 INFO:teuthology.orchestra.run.smithi134.stdout: "data_extra_pool": "test-zone3.rgw.buckets.non-ec", 2024-08-18T15:19:08.255 INFO:teuthology.orchestra.run.smithi134.stdout: "index_type": 0, 2024-08-18T15:19:08.255 INFO:teuthology.orchestra.run.smithi134.stdout: "inline_data": true 2024-08-18T15:19:08.255 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:19:08.255 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:19:08.255 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T15:19:08.255 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_id": "2827c6f0-bf0d-4836-944c-9e7fcf22457b", 2024-08-18T15:19:08.255 INFO:teuthology.orchestra.run.smithi134.stdout: "notif_pool": "test-zone3.rgw.log:notif" 2024-08-18T15:19:08.255 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T15:19:08.256 DEBUG:teuthology.orchestra.run.smithi134:> sudo ceph osd pool create test-zone3.rgw.buckets.data 64 64 --cluster c1 2024-08-18T15:19:09.094 INFO:teuthology.orchestra.run.smithi134.stderr:pool 'test-zone3.rgw.buckets.data' created 2024-08-18T15:19:09.123 DEBUG:teuthology.orchestra.run.smithi134:> sudo ceph osd pool application enable test-zone3.rgw.buckets.data rgw --cluster c1 2024-08-18T15:19:10.101 INFO:teuthology.orchestra.run.smithi134.stderr:enabled application 'rgw' on pool 'test-zone3.rgw.buckets.data' 2024-08-18T15:19:10.137 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:19:10.137 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:19:10.137 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T15:19:10.189 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:19:10.189 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:19:10.209 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T15:19:10.209 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T15:19:10.209 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "test-zone3", 2024-08-18T15:19:10.209 INFO:teuthology.orchestra.run.smithi134.stdout: "domain_root": "test-zone3.rgw.meta:root", 2024-08-18T15:19:10.209 INFO:teuthology.orchestra.run.smithi134.stdout: "control_pool": "test-zone3.rgw.control", 2024-08-18T15:19:10.209 INFO:teuthology.orchestra.run.smithi134.stdout: "gc_pool": "test-zone3.rgw.log:gc", 2024-08-18T15:19:10.209 INFO:teuthology.orchestra.run.smithi134.stdout: "lc_pool": "test-zone3.rgw.log:lc", 2024-08-18T15:19:10.209 INFO:teuthology.orchestra.run.smithi134.stdout: "log_pool": "test-zone3.rgw.log", 2024-08-18T15:19:10.209 INFO:teuthology.orchestra.run.smithi134.stdout: "intent_log_pool": "test-zone3.rgw.log:intent", 2024-08-18T15:19:10.209 INFO:teuthology.orchestra.run.smithi134.stdout: "usage_log_pool": "test-zone3.rgw.log:usage", 2024-08-18T15:19:10.210 INFO:teuthology.orchestra.run.smithi134.stdout: "roles_pool": "test-zone3.rgw.meta:roles", 2024-08-18T15:19:10.210 INFO:teuthology.orchestra.run.smithi134.stdout: "reshard_pool": "test-zone3.rgw.log:reshard", 2024-08-18T15:19:10.210 INFO:teuthology.orchestra.run.smithi134.stdout: "user_keys_pool": "test-zone3.rgw.meta:users.keys", 2024-08-18T15:19:10.210 INFO:teuthology.orchestra.run.smithi134.stdout: "user_email_pool": "test-zone3.rgw.meta:users.email", 2024-08-18T15:19:10.210 INFO:teuthology.orchestra.run.smithi134.stdout: "user_swift_pool": "test-zone3.rgw.meta:users.swift", 2024-08-18T15:19:10.210 INFO:teuthology.orchestra.run.smithi134.stdout: "user_uid_pool": "test-zone3.rgw.meta:users.uid", 2024-08-18T15:19:10.210 INFO:teuthology.orchestra.run.smithi134.stdout: "otp_pool": "test-zone3.rgw.otp", 2024-08-18T15:19:10.210 INFO:teuthology.orchestra.run.smithi134.stdout: "system_key": { 2024-08-18T15:19:10.210 INFO:teuthology.orchestra.run.smithi134.stdout: "access_key": "MRX9HX7RYBDXO1UM", 2024-08-18T15:19:10.210 INFO:teuthology.orchestra.run.smithi134.stdout: "secret_key": "hGUsvhUdpGZAYWjQwu3StBaYXJPhuCEj" 2024-08-18T15:19:10.210 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:19:10.210 INFO:teuthology.orchestra.run.smithi134.stdout: "placement_pools": [ 2024-08-18T15:19:10.211 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:19:10.211 INFO:teuthology.orchestra.run.smithi134.stdout: "key": "default-placement", 2024-08-18T15:19:10.211 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:19:10.211 INFO:teuthology.orchestra.run.smithi134.stdout: "index_pool": "test-zone3.rgw.buckets.index", 2024-08-18T15:19:10.211 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_classes": { 2024-08-18T15:19:10.211 INFO:teuthology.orchestra.run.smithi134.stdout: "STANDARD": { 2024-08-18T15:19:10.211 INFO:teuthology.orchestra.run.smithi134.stdout: "data_pool": "test-zone3.rgw.buckets.data", 2024-08-18T15:19:10.211 INFO:teuthology.orchestra.run.smithi134.stdout: "compression_type": "random" 2024-08-18T15:19:10.211 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:19:10.211 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:19:10.211 INFO:teuthology.orchestra.run.smithi134.stdout: "data_extra_pool": "test-zone3.rgw.buckets.non-ec", 2024-08-18T15:19:10.211 INFO:teuthology.orchestra.run.smithi134.stdout: "index_type": 0, 2024-08-18T15:19:10.211 INFO:teuthology.orchestra.run.smithi134.stdout: "inline_data": true 2024-08-18T15:19:10.212 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:19:10.212 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:19:10.212 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T15:19:10.212 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_id": "2827c6f0-bf0d-4836-944c-9e7fcf22457b", 2024-08-18T15:19:10.212 INFO:teuthology.orchestra.run.smithi134.stdout: "notif_pool": "test-zone3.rgw.log:notif" 2024-08-18T15:19:10.212 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T15:19:10.213 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:19:10.213 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:19:10.213 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T15:19:10.260 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:19:10.260 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:19:10.272 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:19:10.272+0000 7f6e2cf26880 0 period (b0d74758-1871-4454-bc0f-e3e4b6c30105 does not have zone 018ea73b-7528-48da-aafa-0118123784f7 configured 2024-08-18T15:19:10.739 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2024-08-18T15:19:17.168 INFO:teuthology.orchestra.run.smithi134.stderr:Sending period to new master zone 96bf8502-2f39-4d53-8196-3378c28da3b4 2024-08-18T15:19:17.194 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T15:19:17.195 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "b0d74758-1871-4454-bc0f-e3e4b6c30105", 2024-08-18T15:19:17.195 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 3, 2024-08-18T15:19:17.195 INFO:teuthology.orchestra.run.smithi134.stdout: "predecessor_uuid": "453536fc-4ae4-4b27-be1c-ba0601521b0c", 2024-08-18T15:19:17.195 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": [], 2024-08-18T15:19:17.195 INFO:teuthology.orchestra.run.smithi134.stdout: "period_map": { 2024-08-18T15:19:17.195 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "b0d74758-1871-4454-bc0f-e3e4b6c30105", 2024-08-18T15:19:17.195 INFO:teuthology.orchestra.run.smithi134.stdout: "zonegroups": [ 2024-08-18T15:19:17.195 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:19:17.195 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "test-zonegroup", 2024-08-18T15:19:17.195 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "test-zonegroup", 2024-08-18T15:19:17.195 INFO:teuthology.orchestra.run.smithi134.stdout: "api_name": "", 2024-08-18T15:19:17.195 INFO:teuthology.orchestra.run.smithi134.stdout: "is_master": true, 2024-08-18T15:19:17.196 INFO:teuthology.orchestra.run.smithi134.stdout: "endpoints": [ 2024-08-18T15:19:17.196 INFO:teuthology.orchestra.run.smithi134.stdout: "http://smithi134.front.sepia.ceph.com:8000" 2024-08-18T15:19:17.196 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T15:19:17.196 INFO:teuthology.orchestra.run.smithi134.stdout: "hostnames": [], 2024-08-18T15:19:17.196 INFO:teuthology.orchestra.run.smithi134.stdout: "hostnames_s3website": [], 2024-08-18T15:19:17.196 INFO:teuthology.orchestra.run.smithi134.stdout: "master_zone": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:19:17.196 INFO:teuthology.orchestra.run.smithi134.stdout: "zones": [ 2024-08-18T15:19:17.196 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:19:17.196 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T15:19:17.196 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "test-zone3", 2024-08-18T15:19:17.196 INFO:teuthology.orchestra.run.smithi134.stdout: "endpoints": [ 2024-08-18T15:19:17.196 INFO:teuthology.orchestra.run.smithi134.stdout: "http://smithi134.front.sepia.ceph.com:8001" 2024-08-18T15:19:17.197 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T15:19:17.197 INFO:teuthology.orchestra.run.smithi134.stdout: "log_meta": false, 2024-08-18T15:19:17.197 INFO:teuthology.orchestra.run.smithi134.stdout: "log_data": true, 2024-08-18T15:19:17.197 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_index_max_shards": 11, 2024-08-18T15:19:17.197 INFO:teuthology.orchestra.run.smithi134.stdout: "read_only": false, 2024-08-18T15:19:17.197 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_type": "", 2024-08-18T15:19:17.197 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from_all": true, 2024-08-18T15:19:17.197 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from": [], 2024-08-18T15:19:17.197 INFO:teuthology.orchestra.run.smithi134.stdout: "redirect_zone": "", 2024-08-18T15:19:17.197 INFO:teuthology.orchestra.run.smithi134.stdout: "supported_features": [ 2024-08-18T15:19:17.197 INFO:teuthology.orchestra.run.smithi134.stdout: "compress-encrypted", 2024-08-18T15:19:17.197 INFO:teuthology.orchestra.run.smithi134.stdout: "resharding" 2024-08-18T15:19:17.197 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T15:19:17.197 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:19:17.198 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:19:17.198 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "10090ccf-aad6-416d-9775-df8b7b27345a", 2024-08-18T15:19:17.198 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "test-zone2", 2024-08-18T15:19:17.198 INFO:teuthology.orchestra.run.smithi134.stdout: "endpoints": [ 2024-08-18T15:19:17.198 INFO:teuthology.orchestra.run.smithi134.stdout: "http://smithi193.front.sepia.ceph.com:8000" 2024-08-18T15:19:17.198 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T15:19:17.198 INFO:teuthology.orchestra.run.smithi134.stdout: "log_meta": false, 2024-08-18T15:19:17.198 INFO:teuthology.orchestra.run.smithi134.stdout: "log_data": true, 2024-08-18T15:19:17.198 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_index_max_shards": 11, 2024-08-18T15:19:17.198 INFO:teuthology.orchestra.run.smithi134.stdout: "read_only": false, 2024-08-18T15:19:17.198 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_type": "", 2024-08-18T15:19:17.198 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from_all": true, 2024-08-18T15:19:17.198 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from": [], 2024-08-18T15:19:17.199 INFO:teuthology.orchestra.run.smithi134.stdout: "redirect_zone": "", 2024-08-18T15:19:17.199 INFO:teuthology.orchestra.run.smithi134.stdout: "supported_features": [ 2024-08-18T15:19:17.199 INFO:teuthology.orchestra.run.smithi134.stdout: "compress-encrypted", 2024-08-18T15:19:17.199 INFO:teuthology.orchestra.run.smithi134.stdout: "resharding" 2024-08-18T15:19:17.199 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T15:19:17.199 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:19:17.199 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:19:17.199 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:19:17.199 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "test-zone1", 2024-08-18T15:19:17.199 INFO:teuthology.orchestra.run.smithi134.stdout: "endpoints": [ 2024-08-18T15:19:17.199 INFO:teuthology.orchestra.run.smithi134.stdout: "http://smithi134.front.sepia.ceph.com:8000" 2024-08-18T15:19:17.199 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T15:19:17.199 INFO:teuthology.orchestra.run.smithi134.stdout: "log_meta": false, 2024-08-18T15:19:17.199 INFO:teuthology.orchestra.run.smithi134.stdout: "log_data": true, 2024-08-18T15:19:17.200 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_index_max_shards": 11, 2024-08-18T15:19:17.200 INFO:teuthology.orchestra.run.smithi134.stdout: "read_only": false, 2024-08-18T15:19:17.200 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_type": "", 2024-08-18T15:19:17.200 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from_all": true, 2024-08-18T15:19:17.200 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from": [], 2024-08-18T15:19:17.200 INFO:teuthology.orchestra.run.smithi134.stdout: "redirect_zone": "", 2024-08-18T15:19:17.200 INFO:teuthology.orchestra.run.smithi134.stdout: "supported_features": [ 2024-08-18T15:19:17.200 INFO:teuthology.orchestra.run.smithi134.stdout: "compress-encrypted", 2024-08-18T15:19:17.200 INFO:teuthology.orchestra.run.smithi134.stdout: "resharding" 2024-08-18T15:19:17.200 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T15:19:17.200 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:19:17.200 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T15:19:17.201 INFO:teuthology.orchestra.run.smithi134.stdout: "placement_targets": [ 2024-08-18T15:19:17.201 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:19:17.201 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "default-placement", 2024-08-18T15:19:17.201 INFO:teuthology.orchestra.run.smithi134.stdout: "tags": [], 2024-08-18T15:19:17.201 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_classes": [ 2024-08-18T15:19:17.201 INFO:teuthology.orchestra.run.smithi134.stdout: "STANDARD" 2024-08-18T15:19:17.201 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T15:19:17.201 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:19:17.201 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T15:19:17.201 INFO:teuthology.orchestra.run.smithi134.stdout: "default_placement": "default-placement", 2024-08-18T15:19:17.202 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_id": "2827c6f0-bf0d-4836-944c-9e7fcf22457b", 2024-08-18T15:19:17.202 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_policy": { 2024-08-18T15:19:17.202 INFO:teuthology.orchestra.run.smithi134.stdout: "groups": [] 2024-08-18T15:19:17.202 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:19:17.202 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled_features": [ 2024-08-18T15:19:17.202 INFO:teuthology.orchestra.run.smithi134.stdout: "resharding" 2024-08-18T15:19:17.202 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T15:19:17.202 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:19:17.202 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T15:19:17.202 INFO:teuthology.orchestra.run.smithi134.stdout: "short_zone_ids": [ 2024-08-18T15:19:17.203 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:19:17.203 INFO:teuthology.orchestra.run.smithi134.stdout: "key": "018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T15:19:17.203 INFO:teuthology.orchestra.run.smithi134.stdout: "val": 462036791 2024-08-18T15:19:17.203 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:19:17.203 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:19:17.203 INFO:teuthology.orchestra.run.smithi134.stdout: "key": "10090ccf-aad6-416d-9775-df8b7b27345a", 2024-08-18T15:19:17.203 INFO:teuthology.orchestra.run.smithi134.stdout: "val": 874289624 2024-08-18T15:19:17.203 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:19:17.205 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:19:17.205 INFO:teuthology.orchestra.run.smithi134.stdout: "key": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:19:17.205 INFO:teuthology.orchestra.run.smithi134.stdout: "val": 1721399886 2024-08-18T15:19:17.205 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:19:17.206 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T15:19:17.206 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:19:17.206 INFO:teuthology.orchestra.run.smithi134.stdout: "master_zonegroup": "test-zonegroup", 2024-08-18T15:19:17.206 INFO:teuthology.orchestra.run.smithi134.stdout: "master_zone": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:19:17.206 INFO:teuthology.orchestra.run.smithi134.stdout: "period_config": { 2024-08-18T15:19:17.206 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_quota": { 2024-08-18T15:19:17.206 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false, 2024-08-18T15:19:17.206 INFO:teuthology.orchestra.run.smithi134.stdout: "check_on_raw": false, 2024-08-18T15:19:17.207 INFO:teuthology.orchestra.run.smithi134.stdout: "max_size": -1, 2024-08-18T15:19:17.207 INFO:teuthology.orchestra.run.smithi134.stdout: "max_size_kb": 0, 2024-08-18T15:19:17.207 INFO:teuthology.orchestra.run.smithi134.stdout: "max_objects": -1 2024-08-18T15:19:17.207 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:19:17.207 INFO:teuthology.orchestra.run.smithi134.stdout: "user_quota": { 2024-08-18T15:19:17.207 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false, 2024-08-18T15:19:17.207 INFO:teuthology.orchestra.run.smithi134.stdout: "check_on_raw": false, 2024-08-18T15:19:17.207 INFO:teuthology.orchestra.run.smithi134.stdout: "max_size": -1, 2024-08-18T15:19:17.207 INFO:teuthology.orchestra.run.smithi134.stdout: "max_size_kb": 0, 2024-08-18T15:19:17.207 INFO:teuthology.orchestra.run.smithi134.stdout: "max_objects": -1 2024-08-18T15:19:17.207 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:19:17.208 INFO:teuthology.orchestra.run.smithi134.stdout: "user_ratelimit": { 2024-08-18T15:19:17.208 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_ops": 0, 2024-08-18T15:19:17.208 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_ops": 0, 2024-08-18T15:19:17.208 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_bytes": 0, 2024-08-18T15:19:17.208 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_bytes": 0, 2024-08-18T15:19:17.208 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false 2024-08-18T15:19:17.208 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:19:17.208 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_ratelimit": { 2024-08-18T15:19:17.208 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_ops": 0, 2024-08-18T15:19:17.209 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_ops": 0, 2024-08-18T15:19:17.209 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_bytes": 0, 2024-08-18T15:19:17.209 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_bytes": 0, 2024-08-18T15:19:17.209 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false 2024-08-18T15:19:17.209 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:19:17.209 INFO:teuthology.orchestra.run.smithi134.stdout: "anonymous_ratelimit": { 2024-08-18T15:19:17.209 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_ops": 0, 2024-08-18T15:19:17.209 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_ops": 0, 2024-08-18T15:19:17.209 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_bytes": 0, 2024-08-18T15:19:17.210 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_bytes": 0, 2024-08-18T15:19:17.210 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false 2024-08-18T15:19:17.210 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:19:17.210 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:19:17.210 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_id": "2827c6f0-bf0d-4836-944c-9e7fcf22457b", 2024-08-18T15:19:17.210 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_name": "test-realm", 2024-08-18T15:19:17.210 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:19:17.210 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T15:19:17.212 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2024-08-18T15:19:17.212 INFO:teuthology.orchestra.run:waiting for 300 2024-08-18T15:19:17.213 INFO:tasks.rgw.c1.client.1.smithi134.stdout:2024-08-18T15:19:17.212+0000 7fc47efea640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground (PID: 37824) UID: 0 2024-08-18T15:19:17.213 INFO:tasks.rgw.c1.client.1.smithi134.stdout:2024-08-18T15:19:17.212+0000 7fc48540c880 -1 shutting down 2024-08-18T15:19:21.174 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2024-08-18T15:19:23.314 INFO:tasks.rgw.c1.client.1:Stopped 2024-08-18T15:19:23.314 INFO:tasks.rgw.c1.client.1:Restarting daemon 2024-08-18T15:19:23.315 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2024-08-18T15:19:23.317 INFO:tasks.rgw.c1.client.1:Started 2024-08-18T15:19:23.317 DEBUG:teuthology.orchestra.run.smithi134:> curl http://smithi134.front.sepia.ceph.com:8001 2024-08-18T15:19:23.340 INFO:teuthology.orchestra.run.smithi134.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-08-18T15:19:23.340 INFO:teuthology.orchestra.run.smithi134.stderr: Dload Upload Total Spent Left Speed 2024-08-18T15:19:23.340 INFO:teuthology.orchestra.run.smithi134.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2024-08-18T15:19:23.341 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-08-18T15:19:23.341 INFO:teuthology.orchestra.run.smithi134.stderr:curl: (7) Failed to connect to smithi134.front.sepia.ceph.com port 8001: Connection refused 2024-08-18T15:19:24.342 DEBUG:teuthology.orchestra.run.smithi134:> curl http://smithi134.front.sepia.ceph.com:8001 2024-08-18T15:19:24.362 INFO:teuthology.orchestra.run.smithi134.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-08-18T15:19:24.362 INFO:teuthology.orchestra.run.smithi134.stderr: Dload Upload Total Spent Left Speed 2024-08-18T15:19:24.363 INFO:teuthology.orchestra.run.smithi134.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-08-18T15:19:24.363 INFO:teuthology.orchestra.run.smithi134.stdout:anonymous 2024-08-18T15:19:24.364 INFO:tasks.rgw_multisite:rgw multisite configuration completed 2024-08-18T15:19:24.364 INFO:teuthology.run_tasks:Running task rgw-multisite-tests... 2024-08-18T15:19:24.376 INFO:tasks.rgw_multisite_tests:importing tests from /home/teuthworker/src/git.ceph.com_ceph_03f4c3e85ce172201896147a7aef87ba256187f2/qa/../src/test/rgw/rgw_multi 2024-08-18T15:19:24.421 INFO:tasks.rgw_multisite_tests:creating test user.. 2024-08-18T15:19:24.421 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--caps', 'roles=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:19:24.422 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--caps', 'roles=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:19:24.422 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid rgw-multisite-test-user --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --display-name 'Multisite Test User' --gen-access-key --gen-secret --caps 'roles=*' --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T15:19:24.474 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:19:24.474 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:19:24.508 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T15:19:24.508 INFO:teuthology.orchestra.run.smithi134.stdout: "user_id": "rgw-multisite-test-user", 2024-08-18T15:19:24.508 INFO:teuthology.orchestra.run.smithi134.stdout: "display_name": "Multisite Test User", 2024-08-18T15:19:24.508 INFO:teuthology.orchestra.run.smithi134.stdout: "email": "", 2024-08-18T15:19:24.508 INFO:teuthology.orchestra.run.smithi134.stdout: "suspended": 0, 2024-08-18T15:19:24.508 INFO:teuthology.orchestra.run.smithi134.stdout: "max_buckets": 1000, 2024-08-18T15:19:24.508 INFO:teuthology.orchestra.run.smithi134.stdout: "subusers": [], 2024-08-18T15:19:24.508 INFO:teuthology.orchestra.run.smithi134.stdout: "keys": [ 2024-08-18T15:19:24.508 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:19:24.508 INFO:teuthology.orchestra.run.smithi134.stdout: "user": "rgw-multisite-test-user", 2024-08-18T15:19:24.508 INFO:teuthology.orchestra.run.smithi134.stdout: "access_key": "9MCA4XK1WTKHXCPQPFCD", 2024-08-18T15:19:24.509 INFO:teuthology.orchestra.run.smithi134.stdout: "secret_key": "iHP97Ju3B7A6GAsvql2EJonosYTZOTdTEUoCJmqx" 2024-08-18T15:19:24.509 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:19:24.509 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T15:19:24.509 INFO:teuthology.orchestra.run.smithi134.stdout: "swift_keys": [], 2024-08-18T15:19:24.509 INFO:teuthology.orchestra.run.smithi134.stdout: "caps": [ 2024-08-18T15:19:24.509 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:19:24.509 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "roles", 2024-08-18T15:19:24.509 INFO:teuthology.orchestra.run.smithi134.stdout: "perm": "*" 2024-08-18T15:19:24.509 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:19:24.509 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T15:19:24.509 INFO:teuthology.orchestra.run.smithi134.stdout: "op_mask": "read, write, delete", 2024-08-18T15:19:24.509 INFO:teuthology.orchestra.run.smithi134.stdout: "default_placement": "", 2024-08-18T15:19:24.509 INFO:teuthology.orchestra.run.smithi134.stdout: "default_storage_class": "", 2024-08-18T15:19:24.510 INFO:teuthology.orchestra.run.smithi134.stdout: "placement_tags": [], 2024-08-18T15:19:24.510 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_quota": { 2024-08-18T15:19:24.510 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false, 2024-08-18T15:19:24.510 INFO:teuthology.orchestra.run.smithi134.stdout: "check_on_raw": false, 2024-08-18T15:19:24.510 INFO:teuthology.orchestra.run.smithi134.stdout: "max_size": -1, 2024-08-18T15:19:24.510 INFO:teuthology.orchestra.run.smithi134.stdout: "max_size_kb": 0, 2024-08-18T15:19:24.510 INFO:teuthology.orchestra.run.smithi134.stdout: "max_objects": -1 2024-08-18T15:19:24.510 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:19:24.510 INFO:teuthology.orchestra.run.smithi134.stdout: "user_quota": { 2024-08-18T15:19:24.510 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false, 2024-08-18T15:19:24.510 INFO:teuthology.orchestra.run.smithi134.stdout: "check_on_raw": false, 2024-08-18T15:19:24.510 INFO:teuthology.orchestra.run.smithi134.stdout: "max_size": -1, 2024-08-18T15:19:24.510 INFO:teuthology.orchestra.run.smithi134.stdout: "max_size_kb": 0, 2024-08-18T15:19:24.511 INFO:teuthology.orchestra.run.smithi134.stdout: "max_objects": -1 2024-08-18T15:19:24.511 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:19:24.511 INFO:teuthology.orchestra.run.smithi134.stdout: "temp_url_keys": [], 2024-08-18T15:19:24.511 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "rgw", 2024-08-18T15:19:24.511 INFO:teuthology.orchestra.run.smithi134.stdout: "mfa_ids": [] 2024-08-18T15:19:24.511 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T15:19:24.511 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:19:24.520 INFO:rgw_multi.tests:meta checkpoint 2024-08-18T15:19:24.520 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:19:24.520 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:19:24.520 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:19:24.566 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:19:24.566 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:19:24.595 INFO:teuthology.orchestra.run.smithi134.stderr:No --period given, using current period=b0d74758-1871-4454-bc0f-e3e4b6c30105 2024-08-18T15:19:24.607 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T15:19:24.607 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:19:24.607 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:19:24.607 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:19:24.506194Z" 2024-08-18T15:19:24.607 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:19:24.607 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:19:24.607 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T15:19:24.607 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T15:19:24.608 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:19:24.608 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:19:24.608 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T15:19:24.608 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T15:19:24.608 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:19:24.608 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:19:24.608 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T15:19:24.608 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T15:19:24.608 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:19:24.608 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T15:19:24.609 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-08-18T15:19:24.609 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:19:24.609 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:19:24.610 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:19:24.663 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:19:24.664 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:19:24.698 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:19:24.696+0000 7f1c12538880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:19:24.698 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:19:24.697+0000 7f1c12538880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:19:24.709 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:19:24.709 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:19:24.709 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:19:24.709 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:19:24.709 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:19:24.709 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "b0d74758-1871-4454-bc0f-e3e4b6c30105", 2024-08-18T15:19:24.709 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:19:24.710 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:19:24.710 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:19:24.710 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:19:24.710 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:19:24.710 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:19:24.710 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:19:24.710 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:19:24.710 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:19:24.710 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:19:24.710 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:19:24.710 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:19:24.710 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:19:24.711 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:19:24.711 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:19:24.711 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:19:24.711 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:19:24.711 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:19:24.711 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:19:24.711 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:19:24.711 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:19:24.711 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:19:24.711 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:19:24.711 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:19:24.711 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:19:24.711 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:19:24.712 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:19:24.712 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:19:24.712 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:19:24.712 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:19:24.712 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:19:24.712 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:19:24.712 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:19:24.712 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:19:24.712 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:19:24.712 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:19:24.712 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:19:24.712 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:19:24.713 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:19:24.713 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:19:24.713 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:19:24.713 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:19:24.713 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:19:24.713 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:19:24.713 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:19:24.713 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:19:24.713 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:19:24.713 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:19:24.714 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:19:24.714 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:19:24.714 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:19:24.714 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:19:24.714 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:19:24.714 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:19:24.714 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:19:24.714 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:19:24.714 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:19:24.714 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:19:24Z" 2024-08-18T15:19:24.714 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:19:24.715 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1723994364.506194_23.1 target= 2024-08-18T15:19:26.880 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2024-08-18T15:19:29.717 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:19:29.717 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:19:29.717 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:19:29.772 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:19:29.773 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:19:29.805 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:19:29.805+0000 7f2b47f26880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:19:29.806 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:19:29.806+0000 7f2b47f26880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:19:29.820 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:19:29.820 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:19:29.820 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:19:29.820 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:19:29.820 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:19:29.820 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "b0d74758-1871-4454-bc0f-e3e4b6c30105", 2024-08-18T15:19:29.820 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:19:29.820 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:19:29.820 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:19:29.820 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:19:29.820 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:19:29.820 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:19:29.821 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:19:29.821 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:19:29.821 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:19:29.821 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:19:29.821 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:19:29.821 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:19:29.821 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:19:29.821 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:19:29.821 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:19:29.821 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:19:29.821 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:19:29.821 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:19:29.821 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:19:29.821 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:19:29.822 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:19:29.822 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:19:29.822 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:19:29.822 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:19:29.822 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:19:29.822 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:19:29.822 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:19:29.822 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:19:29.822 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:19:29.822 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:19:29.822 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:19:29.822 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:19:29.822 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:19:29.823 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:19:29.823 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:19:29.823 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:19:29.823 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:19:29.823 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:19:29.823 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:19:29.823 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:19:29.823 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:19:29.823 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:19:29.823 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:19:29.823 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:19:29.823 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:19:29.823 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:19:29.823 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:19:29.824 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:19:29.824 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:19:29.824 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:19:29.824 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:19:29.824 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:19:29.824 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:19:29.824 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:19:29.824 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:19:29.824 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:19:29.824 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:19:29.825 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:19:29Z" 2024-08-18T15:19:29.825 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:19:29.826 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1723994364.506194_23.1 target= 2024-08-18T15:19:34.826 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:19:34.826 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:19:34.827 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:19:34.881 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:19:34.881 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:19:34.912 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:19:34.912+0000 7f2496f20880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:19:34.913 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:19:34.913+0000 7f2496f20880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:19:34.925 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:19:34.925 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:19:34.925 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:19:34.925 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:19:34.925 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:19:34.925 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "b0d74758-1871-4454-bc0f-e3e4b6c30105", 2024-08-18T15:19:34.925 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:19:34.925 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:19:34.926 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:19:34.926 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:19:34.926 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:19:34.926 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:19:34.926 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:19:34.926 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:19:34.926 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:19:34.926 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:19:34.926 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:19:34.926 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:19:34.927 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:19:34.927 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:19:34.927 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:19:34.927 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:19:34.927 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:19:34.927 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:19:34.927 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:19:34.927 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:19:34.927 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:19:34.927 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:19:34.928 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:19:34.928 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:19:34.928 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:19:34.928 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:19:34.928 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:19:34.928 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:19:34.928 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:19:34.928 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:19:34.928 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:19:34.928 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:19:34.928 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:19:34.928 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:19:34.928 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:19:34.928 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:19:34.929 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:19:34.929 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:19:34.929 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:19:34.929 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:19:34.929 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:19:34.929 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:19:34.929 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:19:34.929 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:19:34.929 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:19:34.929 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:19:34.929 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:19:34.929 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:19:34.929 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:19:34.929 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:19:34.930 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:19:34.930 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:19:34.930 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:19:34.930 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:19:34.930 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:19:34.930 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:19:34.930 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:19:34.930 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:19:34Z" 2024-08-18T15:19:34.930 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:19:34.931 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1723994364.506194_23.1 target= 2024-08-18T15:19:39.932 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:19:39.933 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:19:39.933 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:19:39.985 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:19:39.985 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:19:40.014 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:19:40.014+0000 7f2776a26880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:19:40.015 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:19:40.015+0000 7f2776a26880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:19:40.027 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:19:40.027 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:19:40.027 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:19:40.027 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:19:40.027 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:19:40.027 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "b0d74758-1871-4454-bc0f-e3e4b6c30105", 2024-08-18T15:19:40.027 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:19:40.027 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:19:40.027 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:19:40.028 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:19:40.028 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:19:40.028 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:19:40.028 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:19:40.028 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:19:40.028 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:19:40.028 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:19:40.028 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:19:40.028 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:19:40.028 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:19:40.028 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:19:40.028 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:19:40.028 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:19:40.028 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:19:40.029 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:19:40.029 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:19:40.029 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:19:40.029 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:19:40.029 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:19:40.029 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:19:40.029 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:19:40.029 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:19:40.029 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:19:40.029 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:19:40.029 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:19:40.029 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:19:40.029 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:19:40.030 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:19:40.030 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:19:40.030 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:19:40.030 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:19:40.030 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:19:40.030 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:19:40.030 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:19:40.031 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:19:40.031 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:19:40.031 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:19:40.031 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:19:40.032 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:19:40.032 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:19:40.032 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:19:40.032 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:19:40.032 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:19:40.032 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:19:40.032 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:19:40.032 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:19:40.032 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:19:40.032 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:19:40.032 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:19:40.033 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:19:40.033 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:19:40.033 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:19:40.033 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:19:40.033 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:19:40.033 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:19:40Z" 2024-08-18T15:19:40.033 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:19:40.034 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1723994364.506194_23.1 target= 2024-08-18T15:19:45.034 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:19:45.035 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:19:45.035 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:19:45.090 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:19:45.090 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:19:45.121 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:19:45.120+0000 7f33eca0f880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:19:45.122 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:19:45.121+0000 7f33eca0f880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:19:45.134 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:19:45.135 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:19:45.135 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:19:45.135 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:19:45.135 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:19:45.135 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "b0d74758-1871-4454-bc0f-e3e4b6c30105", 2024-08-18T15:19:45.135 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:19:45.135 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:19:45.135 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:19:45.135 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:19:45.135 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:19:45.135 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:19:45.136 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:19:45.138 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:19:45.138 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:19:45.138 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:19:45.138 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:19:45.138 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:19:45.138 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:19:45.138 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:19:45.138 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:19:45.138 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:19:45.138 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:19:45.138 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:19:45.139 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:19:45.139 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:19:45.139 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:19:45.140 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:19:45.140 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:19:45.140 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:19:45.140 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:19:45.140 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:19:45.140 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:19:45.140 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:19:45.140 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:19:45.140 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:19:45.140 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:19:45.141 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:19:45.141 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:19:45.141 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:19:45.141 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:19:45.141 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:19:45.141 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:19:45.141 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:19:45.141 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:19:45.141 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:19:45.141 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:19:45.141 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:19:45.141 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:19:45.142 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:19:45.142 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:19:45.142 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:19:45.142 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:19:45.142 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:19:45.142 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:19:45.142 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:19:45.142 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:19:45.142 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:19:45.142 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:19:45.142 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:19:45.143 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:19:45.143 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:19:45.143 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:19:45.143 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:19:45Z" 2024-08-18T15:19:45.143 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:19:45.144 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1723994364.506194_23.1 target= 2024-08-18T15:19:50.146 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:19:50.146 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:19:50.146 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:19:50.210 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:19:50.210 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:19:50.241 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:19:50.241+0000 7f217a412880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:19:50.242 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:19:50.242+0000 7f217a412880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:19:50.254 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:19:50.254 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:19:50.254 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:19:50.254 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:19:50.254 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:19:50.254 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "b0d74758-1871-4454-bc0f-e3e4b6c30105", 2024-08-18T15:19:50.254 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:19:50.255 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:19:50.255 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:19:50.255 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:19:50.255 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:19:50.255 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:19:50.255 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:19:50.255 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:19:50.255 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:19:50.255 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:19:50.255 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:19:50.255 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:19:50.255 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:19:50.255 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:19:50.255 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:19:50.256 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:19:50.256 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:19:50.256 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:19:50.256 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:19:50.256 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:19:50.256 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:19:50.256 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:19:50.256 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:19:50.256 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:19:50.256 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:19:50.256 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:19:50.256 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:19:50.256 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:19:50.256 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:19:50.257 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:19:50.257 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:19:50.257 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:19:50.257 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:19:50.257 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:19:50.257 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:19:50.257 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:19:50.257 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:19:50.257 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:19:50.257 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:19:50.257 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:19:50.257 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:19:50.257 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:19:50.257 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:19:50.258 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:19:50.258 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:19:50.258 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:19:50.258 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:19:50.258 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:19:50.258 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:19:50.258 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:19:50.258 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:19:50.258 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:19:50.258 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:19:50.258 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:19:50.258 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:19:50.258 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:19:50.259 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:19:50.259 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:19:50Z" 2024-08-18T15:19:50.259 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:19:50.260 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1723994364.506194_23.1 target= 2024-08-18T15:19:55.260 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:19:55.261 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:19:55.261 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:19:55.316 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:19:55.316 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:19:55.348 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:19:55.347+0000 7feba0531880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:19:55.348 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:19:55.348+0000 7feba0531880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:19:55.361 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:19:55.361 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:19:55.361 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:19:55.361 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:19:55.361 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:19:55.361 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "b0d74758-1871-4454-bc0f-e3e4b6c30105", 2024-08-18T15:19:55.361 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:19:55.361 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:19:55.362 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:19:55.362 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:19:55.362 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:19:55.362 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:19:55.362 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:19:55.362 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:19:55.362 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:19:55.362 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:19:55.362 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:19:55.362 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:19:55.362 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:19:55.362 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:19:55.363 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:19:55.363 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:19:55.363 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:19:55.363 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:19:55.363 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:19:55.363 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:19:55.363 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:19:55.363 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:19:55.363 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:19:55.363 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:19:55.363 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:19:55.363 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:19:55.363 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:19:55.363 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:19:55.364 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:19:55.364 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:19:55.364 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:19:55.364 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:19:55.364 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:19:55.364 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:19:55.364 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:19:55.364 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:19:55.364 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:19:55.365 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:19:55.365 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:19:55.365 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:19:55.365 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:19:55.365 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:19:55.365 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:19:55.365 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:19:55.365 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:19:55.365 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:19:55.365 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:19:55.366 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:19:55.366 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:19:55.366 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:19:55.366 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:19:55.366 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:19:55.366 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:19:55.366 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:19:55.366 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:19:55.366 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:19:55.367 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:19:55.367 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:19:55Z" 2024-08-18T15:19:55.367 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:19:55.368 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1723994364.506194_23.1 target= 2024-08-18T15:20:00.368 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:20:00.368 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:20:00.369 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:20:00.424 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:20:00.424 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:20:00.455 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:20:00.455+0000 7fb803142880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:20:00.456 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:20:00.456+0000 7fb803142880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:20:00.468 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:20:00.468 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:20:00.468 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:20:00.468 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:20:00.468 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:20:00.468 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "b0d74758-1871-4454-bc0f-e3e4b6c30105", 2024-08-18T15:20:00.468 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:20:00.468 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:20:00.468 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:20:00.469 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:20:00.469 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:20:00.469 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:20:00.469 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:20:00.469 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:20:00.469 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:20:00.469 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:20:00.469 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:20:00.469 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:20:00.469 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:20:00.469 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:20:00.469 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:20:00.469 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:20:00.469 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:20:00.470 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:20:00.470 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:20:00.470 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:20:00.470 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:20:00.470 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:20:00.470 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:20:00.470 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:20:00.470 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:20:00.470 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:20:00.470 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:20:00.470 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:20:00.470 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:20:00.470 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:20:00.471 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:20:00.471 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:20:00.471 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:20:00.471 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:20:00.471 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:20:00.471 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:20:00.471 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:20:00.471 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:20:00.471 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:20:00.471 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:20:00.471 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:20:00.472 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:20:00.472 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:20:00.472 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:20:00.472 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:20:00.472 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:20:00.472 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:20:00.473 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:20:00.473 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:20:00.473 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:20:00.473 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:20:00.473 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:20:00.473 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:20:00.473 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:20:00.473 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:20:00.473 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:20:00.473 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:20:00.473 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:20:00Z" 2024-08-18T15:20:00.474 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:20:00.475 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1723994364.506194_23.1 target= 2024-08-18T15:20:05.476 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:20:05.476 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:20:05.476 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:20:05.531 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:20:05.531 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:20:05.563 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:20:05.563+0000 7f04f9b20880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:20:05.564 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:20:05.564+0000 7f04f9b20880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:20:05.577 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:20:05.577 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:20:05.577 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:20:05.577 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:20:05.577 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:20:05.577 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "b0d74758-1871-4454-bc0f-e3e4b6c30105", 2024-08-18T15:20:05.578 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:20:05.578 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:20:05.578 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:20:05.578 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:20:05.578 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:20:05.578 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:20:05.578 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:20:05.578 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:20:05.578 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:20:05.578 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:20:05.578 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:20:05.578 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:20:05.579 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:20:05.579 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:20:05.579 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:20:05.579 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:20:05.579 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:20:05.579 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:20:05.579 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:20:05.579 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:20:05.579 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:20:05.579 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:20:05.579 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:20:05.579 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:20:05.579 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:20:05.579 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:20:05.580 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:20:05.580 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:20:05.580 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:20:05.580 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:20:05.580 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:20:05.580 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:20:05.580 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:20:05.580 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:20:05.580 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:20:05.580 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:20:05.580 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:20:05.580 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:20:05.580 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:20:05.581 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:20:05.581 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:20:05.581 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:20:05.581 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:20:05.581 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:20:05.581 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:20:05.581 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:20:05.581 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:20:05.581 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:20:05.581 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:20:05.581 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:20:05.581 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:20:05.581 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:20:05.582 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:20:05.582 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:20:05.582 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:20:05.582 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:20:05.582 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:20:05.582 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:20:05Z" 2024-08-18T15:20:05.582 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:20:05.583 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1723994364.506194_23.1 target= 2024-08-18T15:20:10.584 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:20:10.585 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:20:10.585 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:20:10.640 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:20:10.640 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:20:10.673 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:20:10.673+0000 7f7fc0219880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:20:10.674 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:20:10.674+0000 7f7fc0219880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:20:10.686 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:20:10.686 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:20:10.686 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:20:10.686 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:20:10.686 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:20:10.686 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "b0d74758-1871-4454-bc0f-e3e4b6c30105", 2024-08-18T15:20:10.686 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:20:10.686 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:20:10.687 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:20:10.687 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:20:10.687 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:20:10.687 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:20:10.687 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:20:10.687 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:20:10.687 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:20:10.687 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:20:10.687 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:20:10.687 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:20:10.687 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:20:10.688 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:20:10.688 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:20:10.688 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:20:10.688 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:20:10.688 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:20:10.688 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:20:10.688 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:20:10.688 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:20:10.688 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:20:10.688 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:20:10.688 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:20:10.688 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:20:10.688 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:20:10.689 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:20:10.689 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:20:10.689 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:20:10.689 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:20:10.689 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:20:10.689 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:20:10.689 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:20:10.689 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:20:10.689 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:20:10.689 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:20:10.689 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:20:10.689 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:20:10.689 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:20:10.690 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:20:10.690 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:20:10.690 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:20:10.690 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:20:10.690 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:20:10.690 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:20:10.690 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:20:10.690 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:20:10.690 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:20:10.690 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:20:10.690 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:20:10.690 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:20:10.691 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:20:10.691 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:20:10.691 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:20:10.691 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:20:10.691 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:20:10.691 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:20:10.691 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:20:10Z" 2024-08-18T15:20:10.691 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:20:10.692 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1723994364.506194_23.1 target= 2024-08-18T15:20:15.693 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:20:15.693 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:20:15.693 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:20:15.749 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:20:15.749 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:20:15.780 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:20:15.779+0000 7f2a8d529880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:20:15.781 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:20:15.780+0000 7f2a8d529880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:20:15.794 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:20:15.794 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:20:15.794 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:20:15.794 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:20:15.794 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:20:15.794 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "b0d74758-1871-4454-bc0f-e3e4b6c30105", 2024-08-18T15:20:15.795 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:20:15.795 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:20:15.795 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:20:15.795 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:20:15.795 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:20:15.795 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:20:15.795 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:20:15.795 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:20:15.795 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:20:15.795 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:20:15.795 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:20:15.795 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:20:15.795 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:20:15.796 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:20:15.796 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:20:15.796 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:20:15.796 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:20:15.796 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:20:15.796 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:20:15.796 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:20:15.796 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:20:15.796 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:20:15.796 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:20:15.796 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:20:15.796 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:20:15.796 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:20:15.796 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:20:15.797 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:20:15.797 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:20:15.797 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:20:15.797 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:20:15.797 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:20:15.797 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:20:15.797 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:20:15.797 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:20:15.797 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:20:15.797 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:20:15.797 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:20:15.797 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:20:15.797 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:20:15.798 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:20:15.798 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:20:15.798 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:20:15.798 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:20:15.798 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:20:15.798 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:20:15.798 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:20:15.798 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:20:15.798 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:20:15.798 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:20:15.798 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:20:15.798 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:20:15.798 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:20:15.799 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:20:15.799 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:20:15.799 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:20:15.799 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:20:15.799 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:20:15Z" 2024-08-18T15:20:15.799 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:20:15.800 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1723994364.506194_23.1 target= 2024-08-18T15:20:20.800 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:20:20.801 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:20:20.801 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:20:20.856 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:20:20.856 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:20:20.887 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:20:20.886+0000 7f9255811880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:20:20.888 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:20:20.887+0000 7f9255811880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:20:20.901 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:20:20.901 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:20:20.901 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:20:20.901 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:20:20.901 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:20:20.901 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "b0d74758-1871-4454-bc0f-e3e4b6c30105", 2024-08-18T15:20:20.901 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:20:20.901 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:20:20.901 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:20:20.901 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:20:20.901 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:20:20.902 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:20:20.902 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:20:20.902 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:20:20.902 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:20:20.902 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:20:20.902 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:20:20.902 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:20:20.902 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:20:20.902 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:20:20.902 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:20:20.902 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:20:20.902 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:20:20.903 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:20:20.903 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:20:20.903 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:20:20.903 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:20:20.903 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:20:20.903 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:20:20.903 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:20:20.903 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:20:20.903 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:20:20.903 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:20:20.903 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:20:20.903 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:20:20.903 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:20:20.903 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:20:20.904 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:20:20.904 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:20:20.904 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:20:20.904 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:20:20.904 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:20:20.904 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:20:20.904 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:20:20.904 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:20:20.904 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:20:20.904 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:20:20.904 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:20:20.904 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:20:20.905 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:20:20.905 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:20:20.905 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:20:20.905 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:20:20.905 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:20:20.905 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:20:20.905 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:20:20.905 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:20:20.905 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:20:20.905 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:20:20.905 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:20:20.905 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:20:20.905 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:20:20.906 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:20:20.906 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:20:20Z" 2024-08-18T15:20:20.906 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:20:20.907 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1723994364.506194_23.1 target= 2024-08-18T15:20:25.908 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:20:25.908 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:20:25.909 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:20:25.962 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:20:25.962 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:20:25.992 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:20:25.991+0000 7f20eb00f880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:20:25.993 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:20:25.993+0000 7f20eb00f880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:20:26.005 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:20:26.005 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:20:26.005 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:20:26.005 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:20:26.005 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:20:26.005 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "b0d74758-1871-4454-bc0f-e3e4b6c30105", 2024-08-18T15:20:26.005 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:20:26.005 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:20:26.005 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:20:26.005 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:20:26.005 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:20:26.005 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:20:26.006 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:20:26.006 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:20:26.006 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:20:26.006 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:20:26.006 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:20:26.006 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:20:26.006 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:20:26.006 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:20:26.006 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:20:26.006 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:20:26.006 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:20:26.006 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:20:26.007 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:20:26.007 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:20:26.007 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:20:26.007 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:20:26.007 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:20:26.007 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:20:26.007 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:20:26.007 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:20:26.007 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:20:26.007 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:20:26.007 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:20:26.007 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:20:26.007 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:20:26.008 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:20:26.008 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:20:26.008 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:20:26.008 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:20:26.008 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:20:26.008 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:20:26.008 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:20:26.008 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:20:26.008 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:20:26.008 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:20:26.008 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:20:26.008 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:20:26.008 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:20:26.009 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:20:26.009 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:20:26.009 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:20:26.009 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:20:26.009 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:20:26.009 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:20:26.009 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:20:26.009 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:20:26.009 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:20:26.009 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:20:26.009 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:20:26.009 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:20:26.009 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:20:26.010 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:20:25Z" 2024-08-18T15:20:26.010 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:20:26.011 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1723994364.506194_23.1 target= 2024-08-18T15:20:31.012 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:20:31.012 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:20:31.013 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:20:31.069 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:20:31.069 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:20:31.099 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:20:31.099+0000 7fdf35c0e880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:20:31.100 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:20:31.100+0000 7fdf35c0e880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:20:31.112 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:20:31.112 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:20:31.112 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:20:31.113 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:20:31.113 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:20:31.113 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "b0d74758-1871-4454-bc0f-e3e4b6c30105", 2024-08-18T15:20:31.113 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:20:31.113 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:20:31.113 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:20:31.113 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:20:31.113 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:20:31.113 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:20:31.113 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:20:31.113 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:20:31.113 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:20:31.113 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:20:31.113 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:20:31.114 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:20:31.114 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:20:31.114 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:20:31.114 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:20:31.114 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:20:31.114 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:20:31.114 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:20:31.114 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:20:31.114 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:20:31.114 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:20:31.114 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:20:31.114 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:20:31.115 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:20:31.115 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:20:31.115 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:20:31.115 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:20:31.115 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:20:31.115 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:20:31.115 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:20:31.115 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:20:31.115 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:20:31.115 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:20:31.115 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:20:31.115 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:20:31.115 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:20:31.115 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:20:31.116 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:20:31.116 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:20:31.116 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:20:31.116 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:20:31.116 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:20:31.116 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:20:31.116 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:20:31.116 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:20:31.116 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:20:31.116 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:20:31.116 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:20:31.116 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:20:31.116 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:20:31.117 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:20:31.117 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:20:31.117 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:20:31.117 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:20:31.117 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:20:31.117 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:20:31.117 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:20:31.117 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:20:31Z" 2024-08-18T15:20:31.117 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:20:31.118 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1723994364.506194_23.1 target= 2024-08-18T15:20:36.119 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:20:36.120 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:20:36.120 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:20:36.174 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:20:36.174 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:20:36.205 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:20:36.205+0000 7fab54c98880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:20:36.206 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:20:36.206+0000 7fab54c98880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:20:36.219 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:20:36.220 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:20:36.220 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:20:36.220 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:20:36.220 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:20:36.220 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "b0d74758-1871-4454-bc0f-e3e4b6c30105", 2024-08-18T15:20:36.220 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:20:36.220 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:20:36.220 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:20:36.220 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:20:36.220 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:20:36.220 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:20:36.221 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:20:36.221 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:20:36.221 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:20:36.221 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:20:36.221 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:20:36.221 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:20:36.221 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:20:36.221 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:20:36.221 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:20:36.221 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:20:36.221 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:20:36.221 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:20:36.222 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:20:36.222 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:20:36.222 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:20:36.222 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:20:36.222 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:20:36.222 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:20:36.222 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:20:36.222 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:20:36.222 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:20:36.222 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:20:36.222 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:20:36.222 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:20:36.223 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:20:36.223 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:20:36.223 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:20:36.223 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:20:36.223 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:20:36.223 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:20:36.223 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:20:36.223 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:20:36.223 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:20:36.223 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:20:36.223 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:20:36.223 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:20:36.223 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:20:36.223 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:20:36.224 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:20:36.224 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:20:36.224 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:20:36.224 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:20:36.224 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:20:36.224 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:20:36.224 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:20:36.224 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:20:36.224 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:20:36.224 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:20:36.224 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:20:36.224 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:20:36.224 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:20:36.224 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:20:36Z" 2024-08-18T15:20:36.224 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:20:36.225 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1723994364.506194_23.1 target= 2024-08-18T15:20:41.226 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:20:41.227 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:20:41.227 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:20:41.282 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:20:41.282 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:20:41.313 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:20:41.313+0000 7f628af19880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:20:41.314 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:20:41.314+0000 7f628af19880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:20:41.328 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:20:41.328 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:20:41.328 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:20:41.328 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:20:41.328 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:20:41.328 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "b0d74758-1871-4454-bc0f-e3e4b6c30105", 2024-08-18T15:20:41.328 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:20:41.328 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:20:41.328 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:20:41.328 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:20:41.329 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:20:41.329 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:20:41.329 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:20:41.329 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:20:41.329 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:20:41.329 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:20:41.329 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:20:41.329 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:20:41.329 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:20:41.329 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:20:41.329 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:20:41.329 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:20:41.329 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:20:41.330 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:20:41.330 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:20:41.330 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:20:41.330 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:20:41.330 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:20:41.330 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:20:41.330 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:20:41.330 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:20:41.330 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:20:41.330 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:20:41.330 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:20:41.330 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:20:41.330 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:20:41.331 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:20:41.331 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:20:41.331 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:20:41.331 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:20:41.331 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:20:41.331 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:20:41.331 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:20:41.331 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:20:41.331 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:20:41.331 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:20:41.331 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:20:41.331 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:20:41.331 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:20:41.332 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:20:41.332 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:20:41.332 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:20:41.332 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:20:41.332 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:20:41.332 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:20:41.332 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:20:41.332 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:20:41.332 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:20:41.332 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:20:41.332 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:20:41.332 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:20:41.332 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:20:41.332 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:20:41.333 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:20:41Z" 2024-08-18T15:20:41.333 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:20:41.333 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1723994364.506194_23.1 target= 2024-08-18T15:20:46.335 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:20:46.335 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:20:46.335 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:20:46.391 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:20:46.391 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:20:46.422 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:20:46.421+0000 7f7af8132880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:20:46.422 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:20:46.422+0000 7f7af8132880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:20:46.434 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:20:46.434 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:20:46.435 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:20:46.435 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:20:46.435 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:20:46.435 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "b0d74758-1871-4454-bc0f-e3e4b6c30105", 2024-08-18T15:20:46.435 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:20:46.435 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:20:46.435 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:20:46.435 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:20:46.435 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:20:46.435 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:20:46.435 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:20:46.435 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:20:46.436 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:20:46.436 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:20:46.436 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:20:46.436 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:20:46.436 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:20:46.436 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:20:46.436 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:20:46.436 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:20:46.436 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:20:46.436 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:20:46.436 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:20:46.436 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:20:46.436 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:20:46.437 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:20:46.437 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:20:46.437 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:20:46.437 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:20:46.437 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:20:46.437 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:20:46.437 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:20:46.437 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:20:46.437 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:20:46.437 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:20:46.437 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:20:46.437 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:20:46.437 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:20:46.437 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:20:46.438 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:20:46.438 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:20:46.438 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:20:46.438 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:20:46.438 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:20:46.438 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:20:46.438 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:20:46.438 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:20:46.438 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:20:46.438 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:20:46.438 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:20:46.438 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:20:46.439 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:20:46.439 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:20:46.439 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:20:46.439 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:20:46.439 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:20:46.439 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:20:46.439 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:20:46.439 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:20:46.439 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:20:46.439 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:20:46.439 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:20:46Z" 2024-08-18T15:20:46.439 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:20:46.440 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1723994364.506194_23.1 target= 2024-08-18T15:20:51.441 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:20:51.441 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:20:51.442 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:20:51.497 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:20:51.497 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:20:51.529 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:20:51.529+0000 7f6d3d336880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:20:51.530 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:20:51.530+0000 7f6d3d336880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:20:51.542 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:20:51.542 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:20:51.542 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:20:51.543 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:20:51.543 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:20:51.543 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "b0d74758-1871-4454-bc0f-e3e4b6c30105", 2024-08-18T15:20:51.543 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:20:51.543 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:20:51.543 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:20:51.543 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:20:51.543 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:20:51.543 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:20:51.543 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:20:51.543 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:20:51.543 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:20:51.543 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:20:51.544 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:20:51.544 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:20:51.544 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:20:51.544 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:20:51.544 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:20:51.544 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:20:51.544 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:20:51.544 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:20:51.544 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:20:51.544 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:20:51.544 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:20:51.544 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:20:51.544 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:20:51.544 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:20:51.545 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:20:51.545 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:20:51.545 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:20:51.545 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:20:51.545 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:20:51.545 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:20:51.545 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:20:51.545 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:20:51.545 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:20:51.545 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:20:51.545 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:20:51.545 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:20:51.545 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:20:51.546 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:20:51.546 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:20:51.546 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:20:51.546 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:20:51.546 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:20:51.546 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:20:51.546 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:20:51.546 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:20:51.546 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:20:51.546 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:20:51.546 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:20:51.546 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:20:51.546 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:20:51.546 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:20:51.547 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:20:51.547 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:20:51.547 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:20:51.547 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:20:51.547 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:20:51.547 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:20:51.547 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:20:51Z" 2024-08-18T15:20:51.547 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:20:51.548 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1723994364.506194_23.1 target= 2024-08-18T15:20:56.549 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:20:56.549 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:20:56.549 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:20:56.606 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:20:56.606 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:20:56.637 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:20:56.637+0000 7fc432e15880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:20:56.638 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:20:56.638+0000 7fc432e15880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:20:56.650 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:20:56.650 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:20:56.650 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:20:56.650 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:20:56.651 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:20:56.651 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "b0d74758-1871-4454-bc0f-e3e4b6c30105", 2024-08-18T15:20:56.651 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:20:56.651 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:20:56.651 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:20:56.651 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:20:56.651 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:20:56.651 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:20:56.651 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:20:56.651 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:20:56.651 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:20:56.651 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:20:56.651 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:20:56.652 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:20:56.652 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:20:56.652 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:20:56.652 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:20:56.652 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:20:56.652 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:20:56.652 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:20:56.652 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:20:56.652 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:20:56.652 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:20:56.652 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:20:56.652 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:20:56.652 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:20:56.652 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:20:56.653 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:20:56.653 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:20:56.653 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:20:56.653 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:20:56.653 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:20:56.653 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:20:56.653 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:20:56.653 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:20:56.653 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:20:56.653 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:20:56.653 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:20:56.653 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:20:56.653 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:20:56.653 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:20:56.653 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:20:56.653 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:20:56.654 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:20:56.654 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:20:56.654 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:20:56.654 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:20:56.654 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:20:56.654 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:20:56.654 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:20:56.654 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:20:56.654 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:20:56.654 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:20:56.654 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:20:56.654 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:20:56.654 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:20:56.655 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:20:56.655 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:20:56.655 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:20:56.655 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:20:56Z" 2024-08-18T15:20:56.655 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:20:56.656 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1723994364.506194_23.1 target= 2024-08-18T15:21:01.656 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:21:01.657 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:21:01.657 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:21:01.712 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:21:01.713 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:21:01.744 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:21:01.744+0000 7fb52f724880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:21:01.745 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:21:01.745+0000 7fb52f724880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:21:01.759 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:21:01.759 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:21:01.759 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:21:01.759 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:21:01.759 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:21:01.759 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "b0d74758-1871-4454-bc0f-e3e4b6c30105", 2024-08-18T15:21:01.759 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:21:01.759 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:01.759 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:21:01.759 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:21:01.759 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:21:01.759 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:21:01.760 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:21:01.760 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:21:01.760 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:21:01.760 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:21:01.760 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:21:01.760 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:21:01.760 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:21:01.760 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:21:01.760 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:01.760 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:21:01.760 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:21:01.760 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:21:01.760 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:21:01.761 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:21:01.761 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:21:01.761 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:21:01.761 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:21:01.761 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:21:01.761 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:21:01.761 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:21:01.761 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:01.761 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:21:01.761 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:21:01.761 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:21:01.761 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:21:01.761 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:21:01.761 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:21:01.762 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:21:01.762 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:21:01.762 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:21:01.762 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:21:01.762 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:21:01.762 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:01.762 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:21:01.762 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:21:01.762 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:21:01.762 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:21:01.762 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:21:01.762 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:21:01.762 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:21:01.762 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:21:01.763 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:21:01.763 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:21:01.763 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:21:01.763 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:21:01.763 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:21:01.763 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:01.763 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:21:01.763 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:21:01.763 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:21:01.763 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:01.763 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:21:01Z" 2024-08-18T15:21:01.763 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:21:01.764 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1723994364.506194_23.1 target= 2024-08-18T15:21:06.765 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:21:06.765 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:21:06.765 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:21:06.821 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:21:06.821 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:21:06.852 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:21:06.851+0000 7fc2e453a880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:21:06.853 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:21:06.852+0000 7fc2e453a880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:21:06.865 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:21:06.865 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:21:06.866 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:21:06.866 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:21:06.866 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:21:06.866 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "b0d74758-1871-4454-bc0f-e3e4b6c30105", 2024-08-18T15:21:06.866 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:21:06.866 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:06.866 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:21:06.866 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:21:06.866 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:21:06.866 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:21:06.866 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:21:06.866 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:21:06.866 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:21:06.867 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:21:06.867 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:21:06.867 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:21:06.867 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:21:06.867 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:21:06.867 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:06.867 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:21:06.867 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:21:06.867 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:21:06.867 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:21:06.867 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:21:06.867 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:21:06.868 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:21:06.868 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:21:06.868 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:21:06.868 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:21:06.868 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:21:06.868 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:06.868 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:21:06.868 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:21:06.868 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:21:06.868 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:21:06.868 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:21:06.868 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:21:06.868 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:21:06.869 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:21:06.869 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:21:06.869 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:21:06.869 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:21:06.869 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:06.869 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:21:06.869 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:21:06.869 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:21:06.869 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:21:06.869 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:21:06.869 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:21:06.870 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:21:06.870 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:21:06.870 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:21:06.870 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:21:06.870 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:21:06.870 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:21:06.870 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:21:06.870 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:06.870 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:21:06.870 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:21:06.870 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:21:06.870 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:06.871 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:21:06Z" 2024-08-18T15:21:06.871 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:21:06.872 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1723994364.506194_23.1 target= 2024-08-18T15:21:11.872 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:21:11.872 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:21:11.873 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:21:11.930 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:21:11.930 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:21:11.960 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:21:11.960+0000 7f5166690880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:21:11.961 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:21:11.961+0000 7f5166690880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:21:11.973 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:21:11.973 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:21:11.973 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:21:11.973 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:21:11.974 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:21:11.974 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "b0d74758-1871-4454-bc0f-e3e4b6c30105", 2024-08-18T15:21:11.974 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:21:11.974 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:11.974 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:21:11.974 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:21:11.974 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:21:11.974 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:21:11.974 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:21:11.974 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:21:11.975 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:21:11.975 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:21:11.975 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:21:11.975 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:21:11.975 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:21:11.975 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:21:11.975 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:11.975 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:21:11.975 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:21:11.975 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:21:11.975 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:21:11.975 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:21:11.975 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:21:11.976 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:21:11.976 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:21:11.976 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:21:11.976 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:21:11.976 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:21:11.976 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:11.976 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:21:11.976 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:21:11.976 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:21:11.976 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:21:11.976 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:21:11.976 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:21:11.976 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:21:11.977 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:21:11.977 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:21:11.977 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:21:11.977 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:21:11.977 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:11.977 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:21:11.977 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:21:11.977 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:21:11.977 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:21:11.977 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:21:11.977 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:21:11.977 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:21:11.977 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:21:11.977 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:21:11.978 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:21:11.978 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:21:11.978 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:21:11.978 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:21:11.978 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:11.978 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:21:11.978 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:21:11.978 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:21:11.978 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:11.978 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:21:11Z" 2024-08-18T15:21:11.978 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:21:11.979 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1723994364.506194_23.1 target= 2024-08-18T15:21:16.980 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:21:16.980 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:21:16.981 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:21:17.036 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:21:17.036 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:21:17.069 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:21:17.069+0000 7f757b332880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:21:17.070 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:21:17.070+0000 7f757b332880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:21:17.082 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:21:17.082 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:21:17.082 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:21:17.082 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:21:17.083 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:21:17.083 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "b0d74758-1871-4454-bc0f-e3e4b6c30105", 2024-08-18T15:21:17.083 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:21:17.083 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:17.083 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:21:17.083 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:21:17.083 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:21:17.083 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:21:17.083 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:21:17.083 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:21:17.083 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:21:17.083 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:21:17.083 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:21:17.083 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:21:17.084 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:21:17.084 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:21:17.084 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:17.084 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:21:17.084 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:21:17.084 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:21:17.084 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:21:17.084 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:21:17.084 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:21:17.084 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:21:17.084 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:21:17.084 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:21:17.084 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:21:17.085 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:21:17.085 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:17.085 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:21:17.085 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:21:17.085 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:21:17.085 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:21:17.085 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:21:17.085 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:21:17.085 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:21:17.085 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:21:17.085 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:21:17.085 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:21:17.085 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:21:17.086 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:17.086 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:21:17.086 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:21:17.086 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:21:17.086 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:21:17.086 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:21:17.086 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:21:17.086 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:21:17.086 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:21:17.086 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:21:17.086 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:21:17.086 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:21:17.086 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:21:17.087 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:21:17.087 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:17.087 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:21:17.087 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:21:17.087 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:21:17.087 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:17.087 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:21:17Z" 2024-08-18T15:21:17.087 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:21:17.088 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1723994364.506194_23.1 target= 2024-08-18T15:21:22.089 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:21:22.090 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:21:22.090 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:21:22.145 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:21:22.145 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:21:22.176 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:21:22.176+0000 7fe1dfa12880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:21:22.177 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:21:22.177+0000 7fe1dfa12880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:21:22.189 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:21:22.189 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:21:22.189 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:21:22.189 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:21:22.189 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:21:22.190 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "b0d74758-1871-4454-bc0f-e3e4b6c30105", 2024-08-18T15:21:22.190 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:21:22.190 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:22.190 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:21:22.190 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:21:22.190 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:21:22.190 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:21:22.190 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:21:22.190 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:21:22.190 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:21:22.190 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:21:22.190 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:21:22.191 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:21:22.191 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:21:22.191 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:21:22.191 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:22.191 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:21:22.191 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:21:22.191 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:21:22.191 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:21:22.191 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:21:22.191 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:21:22.191 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:21:22.191 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:21:22.192 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:21:22.192 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:21:22.192 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:21:22.192 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:22.192 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:21:22.192 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:21:22.192 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:21:22.192 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:21:22.192 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:21:22.192 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:21:22.192 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:21:22.192 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:21:22.192 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:21:22.193 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:21:22.193 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:21:22.193 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:22.193 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:21:22.193 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:21:22.193 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:21:22.193 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:21:22.193 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:21:22.194 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:21:22.194 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:21:22.194 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:21:22.194 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:21:22.194 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:21:22.194 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:21:22.195 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:21:22.195 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:21:22.195 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:22.195 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:21:22.195 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:21:22.196 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:21:22.196 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:22.196 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:21:22Z" 2024-08-18T15:21:22.196 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:21:22.198 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-08-18T15:21:22.198 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-08-18T15:21:22.198 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:21:22.198 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:21:22.198 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:21:22.254 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:21:22.254 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:21:22.285 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:21:22.285+0000 7f60fc520880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:21:22.286 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:21:22.286+0000 7f60fc520880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:21:22.298 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T15:21:22.298 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T15:21:22.298 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T15:21:22.298 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T15:21:22.298 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T15:21:22.298 INFO:teuthology.orchestra.run.smithi134.stdout: "period": "b0d74758-1871-4454-bc0f-e3e4b6c30105", 2024-08-18T15:21:22.298 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:21:22.298 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:21:22.298 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T15:21:22.299 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:21:22.299 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T15:21:22.299 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:21:22.299 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:21:22.299 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:21:22.299 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:21:22.299 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 1, 2024-08-18T15:21:22.299 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:21:22.299 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:21:22.299 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:21:22.299 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:21:22.299 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:21:22.299 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:21:22.300 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T15:21:22.300 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:21:22.300 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:21:22.300 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T15:21:22.300 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:21:22.300 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:21:22.300 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:21:22.300 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T15:21:22.300 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:21:22.300 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:21:22.300 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:21:22.300 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:21:22.300 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T15:21:22.301 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:21:22.301 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:21:22.301 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T15:21:22.301 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:21:22.301 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:21:22.301 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:21:22.301 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T15:21:22.301 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:21:22.301 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:21:22.301 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:21:22.301 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:21:22.301 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T15:21:22.301 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:21:22.301 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:21:22.302 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T15:21:22.302 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:21:22.302 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:21:22.302 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:21:22.302 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T15:21:22.302 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:21:22.302 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:21:22.302 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:21:22.302 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T15:21:22.302 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:21:22.302 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T15:21:22.302 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 1, 2024-08-18T15:21:22.302 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 1 2024-08-18T15:21:22.302 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:21:22.303 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T15:21:22Z" 2024-08-18T15:21:22.303 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T15:21:22.303 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-08-18T15:21:22.304 INFO:rgw_multi.tests:meta checkpoint 2024-08-18T15:21:22.304 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:21:22.304 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:21:22.304 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:21:22.351 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:21:22.351 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:21:22.380 INFO:teuthology.orchestra.run.smithi134.stderr:No --period given, using current period=b0d74758-1871-4454-bc0f-e3e4b6c30105 2024-08-18T15:21:22.391 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T15:21:22.391 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:21:22.392 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:21:22.392 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:19:24.506194Z" 2024-08-18T15:21:22.392 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:21:22.392 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:21:22.392 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T15:21:22.392 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T15:21:22.392 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:21:22.392 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:21:22.392 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T15:21:22.392 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T15:21:22.392 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:21:22.392 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:21:22.392 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T15:21:22.393 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T15:21:22.393 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:21:22.393 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T15:21:22.394 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-08-18T15:21:22.394 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:21:22.394 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:21:22.394 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:21:22.446 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:21:22.446 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:21:22.476 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:21:22.476+0000 7f3169c15880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:21:22.477 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:21:22.477+0000 7f3169c15880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:21:22.489 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:21:22.489 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:21:22.489 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:21:22.489 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:21:22.489 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:21:22.489 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "b0d74758-1871-4454-bc0f-e3e4b6c30105", 2024-08-18T15:21:22.489 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:21:22.490 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:22.490 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:21:22.490 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:21:22.490 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:21:22.490 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:21:22.490 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:21:22.490 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:21:22.490 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:21:22.490 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:21:22.490 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:21:22.490 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:21:22.490 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:21:22.490 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:21:22.491 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:22.491 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:21:22.491 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:21:22.491 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:21:22.491 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:21:22.491 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:21:22.491 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:21:22.491 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:21:22.491 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:21:22.491 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:21:22.491 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:21:22.491 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:21:22.492 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:22.492 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:21:22.492 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:21:22.492 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:21:22.492 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:21:22.492 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:21:22.492 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:21:22.492 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:21:22.492 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:21:22.492 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:21:22.492 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:21:22.492 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:21:22.492 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:22.493 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:21:22.493 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:21:22.493 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:21:22.493 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:21:22.493 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:21:22.493 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:21:22.493 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:21:22.493 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:21:22.493 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:21:22.493 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:21:22.493 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:21:22.493 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:21:22.493 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:21:22.494 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:22.494 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:21:22.494 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:21:22.494 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:21:22.494 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:22.494 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:21:22Z" 2024-08-18T15:21:22.494 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:21:22.495 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-08-18T15:21:22.495 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-08-18T15:21:22.495 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:21:22.495 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:21:22.495 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:21:22.542 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:21:22.543 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:21:22.579 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:21:22.578+0000 7fa5c6084880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:21:22.579 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:21:22.579+0000 7fa5c6084880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:21:22.590 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T15:21:22.590 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T15:21:22.590 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T15:21:22.590 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T15:21:22.590 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T15:21:22.590 INFO:teuthology.orchestra.run.smithi134.stdout: "period": "b0d74758-1871-4454-bc0f-e3e4b6c30105", 2024-08-18T15:21:22.590 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:21:22.591 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:21:22.591 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T15:21:22.591 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:21:22.591 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T15:21:22.591 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:21:22.591 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:21:22.591 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:21:22.591 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:21:22.591 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 1, 2024-08-18T15:21:22.591 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:21:22.591 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:21:22.591 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:21:22.591 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:21:22.592 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:21:22.592 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:21:22.592 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T15:21:22.592 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:21:22.592 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:21:22.592 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T15:21:22.592 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:21:22.592 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:21:22.592 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:21:22.592 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T15:21:22.592 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:21:22.592 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:21:22.592 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:21:22.593 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:21:22.593 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T15:21:22.593 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:21:22.593 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:21:22.593 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T15:21:22.593 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:21:22.593 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:21:22.593 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:21:22.593 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T15:21:22.593 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:21:22.593 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:21:22.593 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:21:22.593 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:21:22.593 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T15:21:22.594 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:21:22.594 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:21:22.594 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T15:21:22.594 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:21:22.594 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:21:22.594 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:21:22.594 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T15:21:22.594 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:21:22.594 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:21:22.594 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:21:22.594 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T15:21:22.594 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:21:22.594 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T15:21:22.594 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 1, 2024-08-18T15:21:22.595 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 1 2024-08-18T15:21:22.595 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:21:22.595 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T15:21:22Z" 2024-08-18T15:21:22.595 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T15:21:22.596 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-08-18T15:21:22.596 INFO:tasks.rgw_multisite_tests:running rgw multisite tests on '/home/teuthworker/src/git.ceph.com_ceph_03f4c3e85ce172201896147a7aef87ba256187f2/qa/../src/test/rgw/rgw_multi' with args=['tests.py'] 2024-08-18T15:21:22.845 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=kibgmb-1 2024-08-18T15:21:25.519 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=kibgmb-2 2024-08-18T15:21:28.080 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=kibgmb-3 2024-08-18T15:21:28.110 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:21:28.110 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:21:28.111 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:21:28.162 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:21:28.163 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:21:28.194 INFO:teuthology.orchestra.run.smithi134.stderr:No --period given, using current period=b0d74758-1871-4454-bc0f-e3e4b6c30105 2024-08-18T15:21:28.207 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T15:21:28.207 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:21:28.207 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:21:28.207 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:19:24.506194Z" 2024-08-18T15:21:28.207 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:21:28.207 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:21:28.207 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T15:21:28.207 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T15:21:28.207 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:21:28.207 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:21:28.208 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994488.100145_60.1", 2024-08-18T15:21:28.208 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:21:28.100145Z" 2024-08-18T15:21:28.208 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:21:28.208 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:21:28.208 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T15:21:28.208 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T15:21:28.208 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:21:28.208 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T15:21:28.209 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-08-18T15:21:28.210 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:21:28.210 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:21:28.210 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:21:28.268 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:21:28.268 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:21:28.305 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:21:28.304+0000 7f59b0d36880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:21:28.307 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:21:28.306+0000 7f59b0d36880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:21:28.319 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:21:28.319 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:21:28.319 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:21:28.319 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:21:28.320 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:21:28.320 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "b0d74758-1871-4454-bc0f-e3e4b6c30105", 2024-08-18T15:21:28.320 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:21:28.320 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:28.320 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:21:28.320 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:21:28.320 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:21:28.320 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:21:28.320 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:21:28.320 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:21:28.320 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:21:28.320 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:21:28.321 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:21:28.321 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:21:28.321 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:21:28.321 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:21:28.321 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:28.321 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:21:28.321 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:21:28.321 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:21:28.321 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:21:28.321 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:21:28.321 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:21:28.321 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:21:28.321 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:21:28.322 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:21:28.322 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:21:28.322 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:21:28.322 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:28.322 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:21:28.322 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:21:28.322 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:21:28.322 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:21:28.322 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994485.499738_49.1", 2024-08-18T15:21:28.322 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:21:28.322 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:21:28.323 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:21:28.323 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:25.499738Z", 2024-08-18T15:21:28.323 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:21:28.323 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:21:28.323 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:28.323 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:21:28.323 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:21:28.323 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:21:28.323 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:21:28.323 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:21:28.323 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:21:28.323 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:21:28.323 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:21:28.324 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:21:28.324 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:21:28.324 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:21:28.324 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:21:28.324 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:21:28.324 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:28.324 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:21:28.324 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:21:28.324 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:21:28.324 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:28.324 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:21:28Z" 2024-08-18T15:21:28.325 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:21:28.326 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1723994488.100145_60.1 target=1_1723994485.499738_49.1 2024-08-18T15:21:33.326 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:21:33.326 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:21:33.327 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:21:33.383 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:21:33.383 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:21:33.414 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:21:33.414+0000 7f7c03cbb880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:21:33.415 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:21:33.415+0000 7f7c03cbb880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:21:33.427 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:21:33.428 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:21:33.428 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:21:33.428 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:21:33.428 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:21:33.428 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "b0d74758-1871-4454-bc0f-e3e4b6c30105", 2024-08-18T15:21:33.428 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:21:33.428 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:33.428 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:21:33.429 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:21:33.429 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:21:33.429 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:21:33.429 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:21:33.429 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:21:33.429 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:21:33.429 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:21:33.429 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:21:33.429 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:21:33.429 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:21:33.429 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:21:33.429 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:33.430 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:21:33.430 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:21:33.430 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:21:33.430 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:21:33.430 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:21:33.430 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:21:33.430 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:21:33.430 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:21:33.430 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:21:33.430 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:21:33.430 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:21:33.430 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:33.431 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:21:33.431 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:21:33.431 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:21:33.431 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:21:33.431 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994488.100145_60.1", 2024-08-18T15:21:33.431 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:21:33.431 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:21:33.431 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:21:33.431 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:28.100145Z", 2024-08-18T15:21:33.431 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:21:33.431 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:21:33.431 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:33.431 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:21:33.432 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:21:33.432 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:21:33.432 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:21:33.432 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:21:33.432 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:21:33.432 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:21:33.432 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:21:33.432 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:21:33.432 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:21:33.432 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:21:33.432 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:21:33.432 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:21:33.432 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:33.433 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:21:33.433 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:21:33.433 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:21:33.433 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:33.433 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:21:33Z" 2024-08-18T15:21:33.433 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:21:33.434 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-08-18T15:21:33.434 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-08-18T15:21:33.434 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:21:33.434 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:21:33.435 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:21:33.489 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:21:33.489 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:21:33.522 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:21:33.521+0000 7fb094328880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:21:33.523 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:21:33.522+0000 7fb094328880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:21:33.535 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T15:21:33.535 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T15:21:33.535 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T15:21:33.535 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T15:21:33.535 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T15:21:33.535 INFO:teuthology.orchestra.run.smithi134.stdout: "period": "b0d74758-1871-4454-bc0f-e3e4b6c30105", 2024-08-18T15:21:33.535 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:21:33.535 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:21:33.536 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T15:21:33.536 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:21:33.536 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T15:21:33.536 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:21:33.536 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:21:33.536 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:21:33.536 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:21:33.536 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 1, 2024-08-18T15:21:33.536 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:21:33.536 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:21:33.536 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:21:33.536 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:21:33.536 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:21:33.537 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:21:33.537 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T15:21:33.537 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:21:33.537 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:21:33.537 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T15:21:33.537 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:21:33.537 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:21:33.537 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:21:33.537 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T15:21:33.537 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:21:33.537 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:21:33.537 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:21:33.537 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:21:33.538 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T15:21:33.538 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:21:33.538 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:21:33.538 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994488.100145_60.1", 2024-08-18T15:21:33.538 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:21:33.538 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:21:33.538 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:21:33.538 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:28.100145Z", 2024-08-18T15:21:33.538 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:21:33.538 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:21:33.538 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:21:33.538 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:21:33.539 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T15:21:33.539 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:21:33.539 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:21:33.539 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T15:21:33.539 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:21:33.539 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:21:33.539 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:21:33.539 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T15:21:33.539 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:21:33.539 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:21:33.539 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:21:33.539 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T15:21:33.539 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:21:33.539 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T15:21:33.539 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 1, 2024-08-18T15:21:33.540 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 1 2024-08-18T15:21:33.540 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:21:33.540 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T15:21:33Z" 2024-08-18T15:21:33.540 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T15:21:33.541 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-08-18T15:21:33.558 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_create ... ok 2024-08-18T15:21:33.559 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=kibgmb-4 2024-08-18T15:21:33.572 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=kibgmb-5 2024-08-18T15:21:33.598 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=kibgmb-6 2024-08-18T15:21:33.631 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:21:33.631 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:21:33.631 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:21:33.682 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:21:33.682 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:21:33.727 INFO:teuthology.orchestra.run.smithi134.stderr:No --period given, using current period=b0d74758-1871-4454-bc0f-e3e4b6c30105 2024-08-18T15:21:33.738 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T15:21:33.739 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:21:33.739 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:21:33.739 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:19:24.506194Z" 2024-08-18T15:21:33.739 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:21:33.739 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:21:33.739 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T15:21:33.739 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T15:21:33.739 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:21:33.739 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:21:33.739 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994493.616159_72.1", 2024-08-18T15:21:33.739 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:21:33.616159Z" 2024-08-18T15:21:33.740 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:21:33.740 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:21:33.740 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T15:21:33.740 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T15:21:33.740 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:21:33.740 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T15:21:33.741 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-08-18T15:21:33.741 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:21:33.741 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:21:33.741 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:21:33.791 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:21:33.791 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:21:33.821 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:21:33.820+0000 7f663e80f880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:21:33.821 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:21:33.821+0000 7f663e80f880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:21:33.832 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:21:33.833 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:21:33.833 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:21:33.833 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:21:33.833 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:21:33.833 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "b0d74758-1871-4454-bc0f-e3e4b6c30105", 2024-08-18T15:21:33.833 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:21:33.833 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:33.833 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:21:33.833 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:21:33.833 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:21:33.833 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:21:33.833 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:21:33.834 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:21:33.834 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:21:33.834 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:21:33.834 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:21:33.834 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:21:33.834 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:21:33.834 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:21:33.834 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:33.834 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:21:33.834 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:21:33.834 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:21:33.834 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:21:33.834 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:21:33.835 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:21:33.835 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:21:33.835 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:21:33.835 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:21:33.835 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:21:33.835 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:21:33.835 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:33.835 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:21:33.835 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:21:33.835 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:21:33.835 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:21:33.835 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994493.616159_72.1", 2024-08-18T15:21:33.835 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:21:33.836 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:21:33.836 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:21:33.836 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:33.616159Z", 2024-08-18T15:21:33.836 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:21:33.836 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:21:33.836 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:33.836 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:21:33.836 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:21:33.836 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:21:33.836 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:21:33.836 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:21:33.836 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:21:33.836 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:21:33.836 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:21:33.837 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:21:33.837 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:21:33.837 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:21:33.837 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:21:33.837 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:21:33.837 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:33.837 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:21:33.837 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:21:33.837 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:21:33.837 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:33.837 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:21:33Z" 2024-08-18T15:21:33.837 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:21:33.838 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-08-18T15:21:33.838 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-08-18T15:21:33.839 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:21:33.839 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:21:33.839 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:21:33.886 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:21:33.886 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:21:33.913 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:21:33.912+0000 7fb6f5b20880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:21:33.914 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:21:33.913+0000 7fb6f5b20880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:21:33.927 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T15:21:33.927 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T15:21:33.927 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T15:21:33.927 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T15:21:33.928 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T15:21:33.928 INFO:teuthology.orchestra.run.smithi134.stdout: "period": "b0d74758-1871-4454-bc0f-e3e4b6c30105", 2024-08-18T15:21:33.928 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:21:33.928 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:21:33.928 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T15:21:33.928 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:21:33.928 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T15:21:33.928 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:21:33.928 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:21:33.928 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:21:33.928 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:21:33.928 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 1, 2024-08-18T15:21:33.928 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:21:33.929 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:21:33.929 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:21:33.929 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:21:33.929 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:21:33.929 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:21:33.929 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T15:21:33.929 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:21:33.929 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:21:33.929 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T15:21:33.929 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:21:33.929 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:21:33.929 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:21:33.929 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T15:21:33.929 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:21:33.930 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:21:33.930 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:21:33.930 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:21:33.930 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T15:21:33.930 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:21:33.930 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:21:33.930 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994493.616159_72.1", 2024-08-18T15:21:33.930 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:21:33.930 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:21:33.930 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:21:33.930 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:33.616159Z", 2024-08-18T15:21:33.930 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:21:33.930 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:21:33.931 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:21:33.931 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:21:33.931 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T15:21:33.931 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:21:33.931 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:21:33.931 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T15:21:33.931 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:21:33.931 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:21:33.931 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:21:33.931 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T15:21:33.931 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:21:33.931 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:21:33.931 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:21:33.932 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T15:21:33.932 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:21:33.932 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T15:21:33.932 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 1, 2024-08-18T15:21:33.932 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 1 2024-08-18T15:21:33.932 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:21:33.932 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T15:21:33Z" 2024-08-18T15:21:33.932 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T15:21:33.933 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-08-18T15:21:34.162 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:21:34.162 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:21:34.162 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:21:34.208 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:21:34.208 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:21:34.241 INFO:teuthology.orchestra.run.smithi134.stderr:No --period given, using current period=b0d74758-1871-4454-bc0f-e3e4b6c30105 2024-08-18T15:21:34.253 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T15:21:34.253 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:21:34.254 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:21:34.254 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:19:24.506194Z" 2024-08-18T15:21:34.254 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:21:34.254 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:21:34.254 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T15:21:34.254 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T15:21:34.254 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:21:34.254 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:21:34.254 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994493.616159_72.1", 2024-08-18T15:21:34.254 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:21:33.616159Z" 2024-08-18T15:21:34.254 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:21:34.254 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:21:34.254 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T15:21:34.255 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T15:21:34.255 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:21:34.255 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T15:21:34.256 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-08-18T15:21:34.256 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:21:34.256 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:21:34.256 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:21:34.305 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:21:34.306 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:21:34.335 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:21:34.334+0000 7ff0dba86880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:21:34.335 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:21:34.335+0000 7ff0dba86880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:21:34.346 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:21:34.346 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:21:34.347 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:21:34.347 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:21:34.347 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:21:34.347 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "b0d74758-1871-4454-bc0f-e3e4b6c30105", 2024-08-18T15:21:34.347 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:21:34.347 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:34.347 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:21:34.347 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:21:34.347 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:21:34.347 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:21:34.347 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:21:34.347 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:21:34.347 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:21:34.348 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:21:34.348 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:21:34.348 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:21:34.348 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:21:34.348 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:21:34.348 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:34.348 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:21:34.348 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:21:34.348 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:21:34.348 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:21:34.348 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:21:34.348 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:21:34.348 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:21:34.349 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:21:34.349 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:21:34.349 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:21:34.349 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:21:34.349 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:34.349 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:21:34.349 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:21:34.349 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:21:34.349 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:21:34.349 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994493.616159_72.1", 2024-08-18T15:21:34.349 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:21:34.349 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:21:34.349 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:21:34.350 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:33.616159Z", 2024-08-18T15:21:34.350 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:21:34.350 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:21:34.350 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:34.350 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:21:34.350 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:21:34.350 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:21:34.350 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:21:34.350 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:21:34.350 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:21:34.350 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:21:34.350 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:21:34.351 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:21:34.351 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:21:34.351 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:21:34.351 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:21:34.351 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:21:34.351 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:34.351 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:21:34.351 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:21:34.351 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:21:34.351 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:34.351 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:21:34Z" 2024-08-18T15:21:34.351 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:21:34.353 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-08-18T15:21:34.353 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-08-18T15:21:34.353 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:21:34.353 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:21:34.353 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:21:34.400 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:21:34.400 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:21:34.428 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:21:34.427+0000 7f3e1f522880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:21:34.428 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:21:34.428+0000 7f3e1f522880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:21:34.439 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T15:21:34.440 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T15:21:34.440 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T15:21:34.440 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T15:21:34.440 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T15:21:34.440 INFO:teuthology.orchestra.run.smithi134.stdout: "period": "b0d74758-1871-4454-bc0f-e3e4b6c30105", 2024-08-18T15:21:34.440 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:21:34.440 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:21:34.440 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T15:21:34.440 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:21:34.440 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T15:21:34.440 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:21:34.440 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:21:34.441 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:21:34.441 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:21:34.441 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 1, 2024-08-18T15:21:34.441 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:21:34.441 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:21:34.441 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:21:34.441 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:21:34.441 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:21:34.441 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:21:34.441 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T15:21:34.441 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:21:34.441 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:21:34.442 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T15:21:34.442 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:21:34.442 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:21:34.442 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:21:34.442 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T15:21:34.442 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:21:34.442 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:21:34.442 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:21:34.442 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:21:34.442 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T15:21:34.442 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:21:34.442 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:21:34.442 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994493.616159_72.1", 2024-08-18T15:21:34.443 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:21:34.443 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:21:34.443 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:21:34.443 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:33.616159Z", 2024-08-18T15:21:34.443 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:21:34.443 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:21:34.443 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:21:34.443 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:21:34.443 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T15:21:34.443 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:21:34.444 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:21:34.444 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T15:21:34.444 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:21:34.444 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:21:34.444 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:21:34.444 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T15:21:34.444 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:21:34.444 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:21:34.444 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:21:34.444 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T15:21:34.444 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:21:34.444 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T15:21:34.444 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 1, 2024-08-18T15:21:34.445 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 1 2024-08-18T15:21:34.445 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:21:34.445 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T15:21:34Z" 2024-08-18T15:21:34.445 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T15:21:34.446 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-08-18T15:21:34.462 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_recreate ... ok 2024-08-18T15:21:34.462 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=kibgmb-7 2024-08-18T15:21:34.476 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=kibgmb-8 2024-08-18T15:21:34.498 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=kibgmb-9 2024-08-18T15:21:34.538 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:21:34.538 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:21:34.539 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:21:34.586 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:21:34.586 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:21:34.614 INFO:teuthology.orchestra.run.smithi134.stderr:No --period given, using current period=b0d74758-1871-4454-bc0f-e3e4b6c30105 2024-08-18T15:21:34.626 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T15:21:34.626 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:21:34.626 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:21:34.626 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:19:24.506194Z" 2024-08-18T15:21:34.627 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:21:34.627 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:21:34.627 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T15:21:34.627 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T15:21:34.627 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:21:34.627 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:21:34.627 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994494.523006_84.1", 2024-08-18T15:21:34.627 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:21:34.523006Z" 2024-08-18T15:21:34.627 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:21:34.627 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:21:34.627 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T15:21:34.627 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T15:21:34.627 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:21:34.628 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T15:21:34.628 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-08-18T15:21:34.629 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:21:34.629 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:21:34.629 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:21:34.681 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:21:34.681 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:21:34.718 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:21:34.718+0000 7ff641136880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:21:34.719 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:21:34.719+0000 7ff641136880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:21:34.731 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:21:34.731 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:21:34.731 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:21:34.732 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:21:34.732 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:21:34.732 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "b0d74758-1871-4454-bc0f-e3e4b6c30105", 2024-08-18T15:21:34.732 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:21:34.732 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:34.732 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:21:34.732 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:21:34.732 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:21:34.732 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:21:34.732 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:21:34.732 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:21:34.733 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:21:34.733 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:21:34.733 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:21:34.733 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:21:34.733 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:21:34.733 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:21:34.733 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:34.733 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:21:34.733 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:21:34.733 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:21:34.733 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:21:34.734 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:21:34.734 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:21:34.734 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:21:34.734 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:21:34.734 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:21:34.734 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:21:34.734 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:21:34.734 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:34.734 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:21:34.734 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:21:34.734 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:21:34.735 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:21:34.735 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.523006_84.1", 2024-08-18T15:21:34.735 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:21:34.735 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:21:34.735 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:21:34.735 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.523006Z", 2024-08-18T15:21:34.735 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:21:34.735 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:21:34.735 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:34.735 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:21:34.735 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:21:34.736 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:21:34.736 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:21:34.736 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:21:34.736 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:21:34.736 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:21:34.736 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:21:34.736 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:21:34.736 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:21:34.736 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:21:34.736 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:21:34.736 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:21:34.737 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:34.737 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:21:34.737 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:21:34.737 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:21:34.737 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:34.737 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:21:34Z" 2024-08-18T15:21:34.737 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:21:34.738 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-08-18T15:21:34.738 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-08-18T15:21:34.738 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:21:34.738 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:21:34.739 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:21:34.786 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:21:34.787 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:21:34.817 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:21:34.817+0000 7fa86060f880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:21:34.818 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:21:34.818+0000 7fa86060f880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:21:34.829 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T15:21:34.829 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T15:21:34.829 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T15:21:34.829 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T15:21:34.829 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T15:21:34.830 INFO:teuthology.orchestra.run.smithi134.stdout: "period": "b0d74758-1871-4454-bc0f-e3e4b6c30105", 2024-08-18T15:21:34.830 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:21:34.830 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:21:34.830 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T15:21:34.830 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:21:34.830 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T15:21:34.830 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:21:34.830 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:21:34.830 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:21:34.830 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:21:34.830 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 1, 2024-08-18T15:21:34.831 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:21:34.831 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:21:34.831 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:21:34.831 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:21:34.831 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:21:34.831 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:21:34.831 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T15:21:34.831 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:21:34.831 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:21:34.831 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T15:21:34.831 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:21:34.831 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:21:34.831 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:21:34.831 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T15:21:34.831 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:21:34.832 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:21:34.832 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:21:34.832 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:21:34.832 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T15:21:34.832 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:21:34.832 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:21:34.832 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994494.523006_84.1", 2024-08-18T15:21:34.832 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:21:34.832 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:21:34.832 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:21:34.832 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:34.523006Z", 2024-08-18T15:21:34.832 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:21:34.832 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:21:34.832 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:21:34.833 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:21:34.833 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T15:21:34.833 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:21:34.833 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:21:34.833 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T15:21:34.833 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:21:34.833 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:21:34.833 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:21:34.833 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T15:21:34.833 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:21:34.833 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:21:34.833 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:21:34.833 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T15:21:34.833 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:21:34.833 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T15:21:34.834 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 1, 2024-08-18T15:21:34.834 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 1 2024-08-18T15:21:34.834 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:21:34.834 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T15:21:34Z" 2024-08-18T15:21:34.834 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T15:21:34.835 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-08-18T15:21:34.975 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:21:34.975 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:21:34.975 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:21:35.022 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:21:35.022 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:21:35.050 INFO:teuthology.orchestra.run.smithi134.stderr:No --period given, using current period=b0d74758-1871-4454-bc0f-e3e4b6c30105 2024-08-18T15:21:35.070 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T15:21:35.071 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:21:35.071 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:21:35.071 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:19:24.506194Z" 2024-08-18T15:21:35.071 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:21:35.071 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:21:35.071 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T15:21:35.071 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T15:21:35.071 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:21:35.071 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:21:35.071 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:21:35.071 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:21:34.952005Z" 2024-08-18T15:21:35.071 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:21:35.072 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:21:35.072 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T15:21:35.072 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T15:21:35.072 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:21:35.072 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T15:21:35.073 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-08-18T15:21:35.073 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:21:35.073 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:21:35.073 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:21:35.121 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:21:35.121 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:21:35.149 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:21:35.148+0000 7f3f64138880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:21:35.149 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:21:35.149+0000 7f3f64138880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:21:35.161 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:21:35.161 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:21:35.161 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:21:35.161 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:21:35.161 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:21:35.162 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "b0d74758-1871-4454-bc0f-e3e4b6c30105", 2024-08-18T15:21:35.162 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:21:35.162 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:35.162 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:21:35.162 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:21:35.162 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:21:35.162 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:21:35.162 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:21:35.162 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:21:35.162 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:21:35.162 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:21:35.162 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:21:35.163 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:21:35.163 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:21:35.163 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:21:35.163 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:35.163 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:21:35.163 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:21:35.163 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:21:35.163 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:21:35.163 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:21:35.163 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:21:35.163 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:21:35.163 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:21:35.163 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:21:35.164 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:21:35.164 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:21:35.164 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:35.164 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:21:35.164 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:21:35.164 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:21:35.164 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:21:35.164 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.523006_84.1", 2024-08-18T15:21:35.164 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:21:35.164 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:21:35.164 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:21:35.164 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.523006Z", 2024-08-18T15:21:35.164 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:21:35.165 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:21:35.165 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:35.165 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:21:35.165 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:21:35.165 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:21:35.165 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:21:35.165 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:21:35.165 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:21:35.165 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:21:35.165 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:21:35.165 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:21:35.165 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:21:35.165 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:21:35.166 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:21:35.166 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:21:35.166 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:35.166 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:21:35.166 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:21:35.166 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:21:35.166 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:35.166 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:21:35Z" 2024-08-18T15:21:35.166 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:21:35.167 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1723994494.952005_96.1 target=1_1723994494.523006_84.1 2024-08-18T15:21:40.168 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:21:40.169 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:21:40.169 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:21:40.224 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:21:40.224 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:21:40.255 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:21:40.255+0000 7f5b80f38880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:21:40.256 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:21:40.256+0000 7f5b80f38880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:21:40.269 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:21:40.269 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:21:40.269 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:21:40.269 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:21:40.269 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:21:40.269 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "b0d74758-1871-4454-bc0f-e3e4b6c30105", 2024-08-18T15:21:40.269 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:21:40.270 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:40.270 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:21:40.270 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:21:40.270 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:21:40.270 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:21:40.270 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:21:40.270 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:21:40.270 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:21:40.270 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:21:40.270 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:21:40.270 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:21:40.270 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:21:40.271 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:21:40.271 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:40.271 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:21:40.271 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:21:40.271 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:21:40.271 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:21:40.271 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:21:40.271 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:21:40.271 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:21:40.271 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:21:40.271 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:21:40.271 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:21:40.271 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:21:40.271 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:40.272 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:21:40.272 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:21:40.272 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:21:40.272 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:21:40.272 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:21:40.272 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:21:40.272 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:21:40.272 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:21:40.272 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:21:40.272 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:21:40.272 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:21:40.272 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:40.272 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:21:40.273 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:21:40.273 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:21:40.273 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:21:40.273 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:21:40.273 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:21:40.273 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:21:40.273 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:21:40.273 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:21:40.273 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:21:40.273 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:21:40.273 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:21:40.273 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:21:40.273 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:40.273 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:21:40.273 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:21:40.274 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:21:40.274 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:40.274 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:21:40Z" 2024-08-18T15:21:40.274 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:21:40.275 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-08-18T15:21:40.275 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-08-18T15:21:40.275 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:21:40.275 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:21:40.275 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:21:40.328 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:21:40.328 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:21:40.368 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:21:40.367+0000 7f682161f880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:21:40.369 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:21:40.368+0000 7f682161f880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:21:40.383 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T15:21:40.384 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T15:21:40.384 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T15:21:40.384 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T15:21:40.384 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T15:21:40.384 INFO:teuthology.orchestra.run.smithi134.stdout: "period": "b0d74758-1871-4454-bc0f-e3e4b6c30105", 2024-08-18T15:21:40.384 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:21:40.384 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:21:40.384 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T15:21:40.384 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:21:40.384 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T15:21:40.384 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:21:40.384 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:21:40.384 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:21:40.385 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:21:40.385 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 1, 2024-08-18T15:21:40.385 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:21:40.385 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:21:40.385 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:21:40.385 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:21:40.385 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:21:40.385 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:21:40.385 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T15:21:40.385 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:21:40.385 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:21:40.385 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T15:21:40.385 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:21:40.385 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:21:40.386 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:21:40.386 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T15:21:40.386 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:21:40.386 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:21:40.386 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:21:40.386 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:21:40.386 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T15:21:40.386 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:21:40.386 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:21:40.386 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994494.861305_85.1", 2024-08-18T15:21:40.386 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:21:40.386 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:21:40.387 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:21:40.387 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:34.861305Z", 2024-08-18T15:21:40.387 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:21:40.387 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:21:40.387 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:21:40.387 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:21:40.387 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T15:21:40.387 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:21:40.387 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:21:40.387 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T15:21:40.387 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:21:40.387 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:21:40.387 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:21:40.387 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T15:21:40.388 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:21:40.388 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:21:40.388 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:21:40.388 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T15:21:40.388 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:21:40.388 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T15:21:40.388 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 1, 2024-08-18T15:21:40.388 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 1 2024-08-18T15:21:40.388 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:21:40.388 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T15:21:40Z" 2024-08-18T15:21:40.388 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T15:21:40.391 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1723994494.952005_96.1 target=1_1723994494.861305_85.1 2024-08-18T15:21:45.391 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:21:45.392 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:21:45.392 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:21:45.447 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:21:45.447 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:21:45.478 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:21:45.477+0000 7fb5ba336880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:21:45.479 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:21:45.478+0000 7fb5ba336880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:21:45.491 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T15:21:45.491 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T15:21:45.491 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T15:21:45.491 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T15:21:45.491 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T15:21:45.491 INFO:teuthology.orchestra.run.smithi134.stdout: "period": "b0d74758-1871-4454-bc0f-e3e4b6c30105", 2024-08-18T15:21:45.491 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:21:45.491 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:21:45.492 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T15:21:45.492 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:21:45.492 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T15:21:45.492 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:21:45.492 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:21:45.492 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:21:45.492 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:21:45.492 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 1, 2024-08-18T15:21:45.492 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:21:45.492 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:21:45.492 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:21:45.492 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:21:45.492 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:21:45.493 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:21:45.493 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T15:21:45.493 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:21:45.493 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:21:45.493 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T15:21:45.493 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:21:45.493 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:21:45.493 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:21:45.493 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T15:21:45.493 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:21:45.493 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:21:45.493 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:21:45.493 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:21:45.493 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T15:21:45.494 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:21:45.494 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:21:45.494 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:21:45.494 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:21:45.494 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:21:45.494 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:21:45.494 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:21:45.494 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:21:45.494 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:21:45.494 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:21:45.494 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:21:45.494 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T15:21:45.495 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:21:45.495 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:21:45.495 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T15:21:45.495 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:21:45.495 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:21:45.495 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:21:45.495 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T15:21:45.495 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:21:45.495 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:21:45.495 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:21:45.495 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T15:21:45.495 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:21:45.495 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T15:21:45.496 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 1, 2024-08-18T15:21:45.496 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 1 2024-08-18T15:21:45.496 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:21:45.496 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T15:21:45Z" 2024-08-18T15:21:45.496 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T15:21:45.497 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-08-18T15:21:45.507 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_remove ... ok 2024-08-18T15:21:45.507 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=kibgmb-10 2024-08-18T15:21:45.527 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=kibgmb-11 2024-08-18T15:21:45.553 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=kibgmb-12 2024-08-18T15:21:45.847 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:21:45.848 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:21:45.848 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:21:45.894 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:21:45.895 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:21:45.923 INFO:teuthology.orchestra.run.smithi134.stderr:No --period given, using current period=b0d74758-1871-4454-bc0f-e3e4b6c30105 2024-08-18T15:21:45.934 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T15:21:45.934 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:21:45.934 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:21:45.934 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:19:24.506194Z" 2024-08-18T15:21:45.935 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:21:45.935 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:21:45.935 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T15:21:45.935 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T15:21:45.935 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:21:45.935 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:21:45.935 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:21:45.935 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:21:34.952005Z" 2024-08-18T15:21:45.935 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:21:45.935 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:21:45.935 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994505.566000_62.1", 2024-08-18T15:21:45.935 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:21:45.566000Z" 2024-08-18T15:21:45.935 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:21:45.935 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T15:21:45.937 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-08-18T15:21:45.937 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:21:45.937 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:21:45.937 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:21:45.987 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:21:45.987 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:21:46.015 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:21:46.015+0000 7f0f55731880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:21:46.016 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:21:46.016+0000 7f0f55731880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:21:46.028 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:21:46.028 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:21:46.028 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:21:46.028 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:21:46.028 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:21:46.028 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "b0d74758-1871-4454-bc0f-e3e4b6c30105", 2024-08-18T15:21:46.028 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:21:46.029 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:46.029 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:21:46.029 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:21:46.029 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:21:46.029 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:21:46.029 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:21:46.029 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:21:46.029 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:21:46.029 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:21:46.029 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:21:46.029 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:21:46.029 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:21:46.029 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:21:46.030 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:46.030 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:21:46.030 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:21:46.030 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:21:46.030 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:21:46.030 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:21:46.030 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:21:46.030 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:21:46.030 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:21:46.030 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:21:46.030 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:21:46.030 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:21:46.030 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:46.031 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:21:46.031 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:21:46.031 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:21:46.031 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:21:46.031 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:21:46.031 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:21:46.031 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:21:46.031 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:21:46.031 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:21:46.031 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:21:46.031 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:21:46.031 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:46.031 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:21:46.032 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:21:46.032 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:21:46.032 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:21:46.032 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:21:46.032 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:21:46.032 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:21:46.032 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:21:46.032 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:21:46.032 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:21:46.032 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:21:46.032 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:21:46.032 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:21:46.032 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:46.032 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:21:46.033 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:21:46.033 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:21:46.033 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:46.033 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:21:46Z" 2024-08-18T15:21:46.033 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:21:46.034 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1723994505.566000_62.1 target= 2024-08-18T15:21:51.035 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:21:51.035 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:21:51.035 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:21:51.091 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:21:51.092 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:21:51.123 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:21:51.122+0000 7fe1dfb34880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:21:51.124 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:21:51.123+0000 7fe1dfb34880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:21:51.136 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:21:51.136 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:21:51.136 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:21:51.136 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:21:51.136 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:21:51.136 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "b0d74758-1871-4454-bc0f-e3e4b6c30105", 2024-08-18T15:21:51.136 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:21:51.137 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:51.137 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:21:51.137 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:21:51.137 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:21:51.137 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:21:51.137 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:21:51.137 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:21:51.137 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:21:51.137 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:21:51.137 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:21:51.137 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:21:51.137 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:21:51.137 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:21:51.137 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:51.138 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:21:51.138 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:21:51.138 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:21:51.138 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:21:51.138 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:21:51.138 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:21:51.138 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:21:51.138 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:21:51.138 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:21:51.138 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:21:51.138 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:21:51.138 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:51.139 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:21:51.139 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:21:51.139 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:21:51.139 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:21:51.139 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:21:51.139 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:21:51.139 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:21:51.139 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:21:51.139 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:21:51.139 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:21:51.140 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:21:51.140 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:51.140 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:21:51.140 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:21:51.140 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:21:51.140 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:21:51.140 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994505.566000_62.1", 2024-08-18T15:21:51.140 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:21:51.140 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:21:51.140 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:21:51.140 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:45.566000Z", 2024-08-18T15:21:51.140 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:21:51.141 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:21:51.141 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:21:51.141 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:21:51.141 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:51.141 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:21:51.141 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:21:51.141 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:21:51.141 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:21:51.141 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:21:51Z" 2024-08-18T15:21:51.141 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:21:51.142 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-08-18T15:21:51.142 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-08-18T15:21:51.142 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:21:51.143 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:21:51.143 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:21:51.198 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:21:51.198 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:21:51.231 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:21:51.230+0000 7fa7a6542880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:21:51.232 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:21:51.231+0000 7fa7a6542880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:21:51.245 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T15:21:51.245 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T15:21:51.245 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T15:21:51.246 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T15:21:51.246 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T15:21:51.246 INFO:teuthology.orchestra.run.smithi134.stdout: "period": "b0d74758-1871-4454-bc0f-e3e4b6c30105", 2024-08-18T15:21:51.246 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:21:51.246 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:21:51.246 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T15:21:51.246 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:21:51.246 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T15:21:51.246 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:21:51.246 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:21:51.246 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:21:51.246 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:21:51.246 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 1, 2024-08-18T15:21:51.247 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:21:51.247 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:21:51.247 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:21:51.247 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:21:51.247 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:21:51.248 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:21:51.248 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T15:21:51.248 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:21:51.248 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:21:51.249 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T15:21:51.249 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:21:51.249 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:21:51.249 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:21:51.249 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T15:21:51.249 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:21:51.249 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:21:51.249 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:21:51.249 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:21:51.249 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T15:21:51.249 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:21:51.249 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:21:51.250 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:21:51.250 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:21:51.250 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:21:51.250 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:21:51.250 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:21:51.250 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:21:51.250 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:21:51.250 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:21:51.250 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:21:51.250 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T15:21:51.250 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:21:51.250 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:21:51.250 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994505.566000_62.1", 2024-08-18T15:21:51.251 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:21:51.251 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:21:51.251 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:21:51.251 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:45.566000Z", 2024-08-18T15:21:51.251 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:21:51.251 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:21:51.251 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:21:51.251 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T15:21:51.251 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:21:51.251 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T15:21:51.251 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 1, 2024-08-18T15:21:51.251 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 1 2024-08-18T15:21:51.251 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:21:51.252 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T15:21:51Z" 2024-08-18T15:21:51.252 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T15:21:51.253 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-08-18T15:21:51.253 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:21:51.253 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:21:51.253 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-10 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T15:21:51.313 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:21:51.313 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:21:51.345 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:21:51.344+0000 7f4701daf880 1 waiting to reach incremental sync.. 2024-08-18T15:21:56.346 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:21:56.345+0000 7f4701daf880 1 waiting to reach incremental sync.. 2024-08-18T15:22:01.347 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:22:01.347+0000 7f4701daf880 1 bucket sync caught up with source: 2024-08-18T15:22:01.348 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, 00000000001.6.6, , 00000000001.4.6, , , , 00000000001.8.6, 00000000001.5.6, , ] 2024-08-18T15:22:01.348 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, 00000000001.6.6, , 00000000001.4.6, , , , 00000000001.8.6, 00000000001.5.6, , ] 2024-08-18T15:22:01.348 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:22:01.347+0000 7f4701daf880 0 bucket checkpoint complete 2024-08-18T15:22:01.363 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-10 zones={test-zone2, test-zone1} 2024-08-18T15:22:01.369 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:22:01.373 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-08-18T15:22:01.373 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-08-18T15:22:01.373 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-08-18T15:22:01.373 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-08-18T15:22:01.373 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:22:01.376 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-08-18T15:22:01.376 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-08-18T15:22:01.376 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-08-18T15:22:01.376 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-08-18T15:22:01.381 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-08-18T15:22:01.432 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-08-18T15:22:01.444 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-08-18T15:22:01.452 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-08-18T15:22:01.459 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-08-18T15:22:01.514 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-08-18T15:22:01.523 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-08-18T15:22:01.575 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-08-18T15:22:01.631 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-08-18T15:22:01.642 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-08-18T15:22:01.693 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-08-18T15:22:01.749 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-08-18T15:22:01.802 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-10 zones={test-zone2, test-zone1} 2024-08-18T15:22:01.802 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:22:01.802 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:22:01.803 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-10 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T15:22:01.863 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:22:01.863 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:22:01.899 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:22:01.898+0000 7fb7e1f36880 1 waiting to reach incremental sync.. 2024-08-18T15:22:06.900 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:22:06.900+0000 7fb7e1f36880 1 bucket sync caught up with source: 2024-08-18T15:22:06.901 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000001.6.6, , 00000000001.4.6, , , , 00000000001.8.6, 00000000001.5.6, , ] 2024-08-18T15:22:06.901 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000001.6.6, , 00000000001.4.6, , , , 00000000001.8.6, 00000000001.5.6, , ] 2024-08-18T15:22:06.901 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:22:06.900+0000 7fb7e1f36880 0 bucket checkpoint complete 2024-08-18T15:22:06.916 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-10 zones={test-zone3, test-zone1} 2024-08-18T15:22:06.921 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:22:06.924 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-08-18T15:22:06.924 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-08-18T15:22:06.924 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-08-18T15:22:06.924 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-08-18T15:22:06.924 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:22:06.927 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-08-18T15:22:06.927 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-08-18T15:22:06.927 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-08-18T15:22:06.927 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-08-18T15:22:06.932 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-08-18T15:22:06.980 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-08-18T15:22:06.992 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-08-18T15:22:07.042 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-08-18T15:22:07.049 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-08-18T15:22:07.104 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-08-18T15:22:07.153 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-08-18T15:22:07.161 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-08-18T15:22:07.216 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-08-18T15:22:07.265 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-08-18T15:22:07.272 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-08-18T15:22:07.323 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-08-18T15:22:07.417 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-10 zones={test-zone3, test-zone1} 2024-08-18T15:22:07.418 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:22:07.418 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:22:07.418 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-11 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T15:22:07.476 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:22:07.476 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:22:07.514 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:22:07.514+0000 7ff6c7d26880 1 waiting to reach incremental sync.. 2024-08-18T15:22:12.515 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:22:12.514+0000 7ff6c7d26880 1 waiting to reach incremental sync.. 2024-08-18T15:22:17.516 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:22:17.515+0000 7ff6c7d26880 1 waiting to reach incremental sync.. 2024-08-18T15:22:22.518 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:22:22.516+0000 7ff6c7d26880 1 waiting to reach incremental sync.. 2024-08-18T15:22:27.517 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:22:27.516+0000 7ff6c7d26880 1 waiting to reach incremental sync.. 2024-08-18T15:22:32.518 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:22:32.517+0000 7ff6c7d26880 1 waiting to reach incremental sync.. 2024-08-18T15:22:37.518 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:22:37.517+0000 7ff6c7d26880 1 waiting to reach incremental sync.. 2024-08-18T15:22:42.519 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:22:42.518+0000 7ff6c7d26880 1 waiting to reach incremental sync.. 2024-08-18T15:22:47.519 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:22:47.518+0000 7ff6c7d26880 1 waiting to reach incremental sync.. 2024-08-18T15:22:52.519 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:22:52.519+0000 7ff6c7d26880 1 waiting to reach incremental sync.. 2024-08-18T15:22:57.520 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:22:57.519+0000 7ff6c7d26880 1 waiting to reach incremental sync.. 2024-08-18T15:23:02.520 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:23:02.520+0000 7ff6c7d26880 1 waiting to reach incremental sync.. 2024-08-18T15:23:07.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:23:07.521+0000 7ff6c7d26880 1 waiting to reach incremental sync.. 2024-08-18T15:23:12.522 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:23:12.522+0000 7ff6c7d26880 1 waiting to reach incremental sync.. 2024-08-18T15:23:17.524 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:23:17.523+0000 7ff6c7d26880 1 bucket sync caught up with source: 2024-08-18T15:23:17.524 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000001.17.6, , 00000000001.20.6, , , , 00000000001.12.6, 00000000001.20.6, , ] 2024-08-18T15:23:17.524 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000001.17.6, , 00000000001.20.6, , , , 00000000001.12.6, 00000000001.20.6, , ] 2024-08-18T15:23:17.525 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:23:17.523+0000 7ff6c7d26880 0 bucket checkpoint complete 2024-08-18T15:23:17.540 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-11 zones={test-zone1, test-zone2} 2024-08-18T15:23:17.554 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:23:17.557 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-08-18T15:23:17.557 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-08-18T15:23:17.557 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-08-18T15:23:17.557 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-08-18T15:23:17.557 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:23:17.560 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-08-18T15:23:17.560 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-08-18T15:23:17.560 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-08-18T15:23:17.560 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-08-18T15:23:17.566 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-08-18T15:23:17.574 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-08-18T15:23:17.586 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-08-18T15:23:17.636 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-08-18T15:23:17.643 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-08-18T15:23:17.701 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-08-18T15:23:17.761 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-08-18T15:23:17.769 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-08-18T15:23:17.825 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-08-18T15:23:17.873 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-08-18T15:23:17.880 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-08-18T15:23:17.933 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-08-18T15:23:17.990 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-11 zones={test-zone1, test-zone2} 2024-08-18T15:23:17.991 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:23:17.991 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:23:17.991 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-11 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T15:23:18.048 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:23:18.048 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:23:18.082 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:23:18.081+0000 7f64bd417880 1 bucket sync caught up with source: 2024-08-18T15:23:18.082 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000001.17.6, , 00000000001.20.6, , , , 00000000001.12.6, 00000000001.20.6, , ] 2024-08-18T15:23:18.082 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000001.17.6, , 00000000001.20.6, , , , 00000000001.12.6, 00000000001.20.6, , ] 2024-08-18T15:23:18.082 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:23:18.081+0000 7f64bd417880 0 bucket checkpoint complete 2024-08-18T15:23:18.094 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-11 zones={test-zone3, test-zone2} 2024-08-18T15:23:18.101 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:23:18.104 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-08-18T15:23:18.104 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-08-18T15:23:18.104 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-08-18T15:23:18.104 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-08-18T15:23:18.104 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:23:18.107 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-08-18T15:23:18.107 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-08-18T15:23:18.107 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-08-18T15:23:18.107 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-08-18T15:23:18.113 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-08-18T15:23:18.166 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-08-18T15:23:18.179 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-08-18T15:23:18.229 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-08-18T15:23:18.284 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-08-18T15:23:18.297 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-08-18T15:23:18.306 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-08-18T15:23:18.356 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-08-18T15:23:18.409 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-08-18T15:23:18.457 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-08-18T15:23:18.508 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-08-18T15:23:18.560 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-08-18T15:23:18.610 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-11 zones={test-zone3, test-zone2} 2024-08-18T15:23:18.610 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:23:18.610 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:23:18.610 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-12 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T15:23:18.665 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:23:18.666 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:23:18.700 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:23:18.698+0000 7fd72c738880 1 bucket sync caught up with source: 2024-08-18T15:23:18.700 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000001.17.6, , 00000000001.20.6, , , , 00000000001.35.6, 00000000001.6.6, , ] 2024-08-18T15:23:18.700 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000001.17.6, , 00000000001.20.6, , , , 00000000001.35.6, 00000000001.6.6, , ] 2024-08-18T15:23:18.700 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:23:18.698+0000 7fd72c738880 0 bucket checkpoint complete 2024-08-18T15:23:18.712 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-12 zones={test-zone1, test-zone3} 2024-08-18T15:23:18.716 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:23:18.718 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-08-18T15:23:18.718 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-08-18T15:23:18.718 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-08-18T15:23:18.719 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-08-18T15:23:18.719 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:23:18.721 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-08-18T15:23:18.721 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-08-18T15:23:18.721 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-08-18T15:23:18.721 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-08-18T15:23:18.725 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-08-18T15:23:18.731 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-08-18T15:23:18.788 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-08-18T15:23:18.836 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-08-18T15:23:18.842 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-08-18T15:23:18.853 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-08-18T15:23:18.860 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-08-18T15:23:18.866 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-08-18T15:23:18.877 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-08-18T15:23:18.929 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-08-18T15:23:18.935 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-08-18T15:23:19.035 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-08-18T15:23:19.085 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-12 zones={test-zone1, test-zone3} 2024-08-18T15:23:19.085 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:23:19.085 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:23:19.086 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-12 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T15:23:19.147 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:23:19.148 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:23:19.184 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:23:19.184+0000 7f5b81142880 1 bucket sync caught up with source: 2024-08-18T15:23:19.185 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, 00000000001.17.6, , 00000000001.20.6, , , , 00000000001.35.6, 00000000001.6.6, , ] 2024-08-18T15:23:19.185 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, 00000000001.17.6, , 00000000001.20.6, , , , 00000000001.35.6, 00000000001.6.6, , ] 2024-08-18T15:23:19.185 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:23:19.184+0000 7f5b81142880 0 bucket checkpoint complete 2024-08-18T15:23:19.197 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-12 zones={test-zone2, test-zone3} 2024-08-18T15:23:19.201 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:23:19.204 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-08-18T15:23:19.204 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-08-18T15:23:19.204 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-08-18T15:23:19.204 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-08-18T15:23:19.204 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:23:19.207 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-08-18T15:23:19.207 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-08-18T15:23:19.207 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-08-18T15:23:19.207 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-08-18T15:23:19.214 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-08-18T15:23:19.220 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-08-18T15:23:19.275 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-08-18T15:23:19.331 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-08-18T15:23:19.338 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-08-18T15:23:19.348 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-08-18T15:23:19.403 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-08-18T15:23:19.454 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-08-18T15:23:19.465 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-08-18T15:23:19.517 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-08-18T15:23:19.523 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-08-18T15:23:19.581 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-08-18T15:23:19.637 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-12 zones={test-zone2, test-zone3} 2024-08-18T15:23:19.638 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_object_sync ... ok 2024-08-18T15:23:19.638 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=kibgmb-13 2024-08-18T15:23:19.650 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=kibgmb-14 2024-08-18T15:23:19.674 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=kibgmb-15 2024-08-18T15:23:19.763 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:23:19.764 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:23:19.764 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:23:19.810 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:23:19.810 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:23:19.841 INFO:teuthology.orchestra.run.smithi134.stderr:No --period given, using current period=b0d74758-1871-4454-bc0f-e3e4b6c30105 2024-08-18T15:23:19.852 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T15:23:19.852 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:23:19.852 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:23:19.852 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:19:24.506194Z" 2024-08-18T15:23:19.852 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:23:19.852 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:23:19.852 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T15:23:19.852 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T15:23:19.853 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:23:19.853 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:23:19.853 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:23:19.853 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:21:34.952005Z" 2024-08-18T15:23:19.853 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:23:19.853 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:23:19.853 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994599.687927_92.1", 2024-08-18T15:23:19.853 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:23:19.687927Z" 2024-08-18T15:23:19.853 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:23:19.853 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T15:23:19.854 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-08-18T15:23:19.854 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:23:19.855 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:23:19.855 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:23:19.903 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:23:19.903 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:23:19.932 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:23:19.932+0000 7f1d09136880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:23:19.933 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:23:19.932+0000 7f1d09136880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:23:19.944 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:23:19.944 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:23:19.944 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:23:19.944 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:23:19.944 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:23:19.944 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "b0d74758-1871-4454-bc0f-e3e4b6c30105", 2024-08-18T15:23:19.945 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:23:19.945 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:23:19.945 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:23:19.945 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:23:19.945 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:23:19.945 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:23:19.945 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:23:19.945 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:23:19.945 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:23:19.945 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:23:19.945 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:23:19.945 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:23:19.945 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:23:19.946 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:23:19.946 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:23:19.946 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:23:19.946 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:23:19.946 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:23:19.946 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:23:19.946 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:23:19.946 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:23:19.946 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:23:19.946 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:23:19.946 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:23:19.946 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:23:19.946 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:23:19.947 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:23:19.947 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:23:19.947 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:23:19.947 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:23:19.947 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:23:19.947 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:23:19.947 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:23:19.947 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:23:19.947 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:23:19.947 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:23:19.947 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:23:19.947 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:23:19.948 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:23:19.948 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:23:19.948 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:23:19.948 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:23:19.948 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:23:19.948 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994599.685017_90.1", 2024-08-18T15:23:19.948 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:23:19.948 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:23:19.948 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:23:19.948 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:23:19.685017Z", 2024-08-18T15:23:19.948 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:23:19.948 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:23:19.948 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:23:19.948 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:23:19.949 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:23:19.949 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:23:19.949 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:23:19.949 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:23:19.949 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:23:19.949 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:23:19Z" 2024-08-18T15:23:19.949 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:23:19.950 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1723994599.687927_92.1 target=1_1723994599.685017_90.1 2024-08-18T15:23:24.951 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:23:24.952 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:23:24.952 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:23:25.007 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:23:25.007 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:23:25.039 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:23:25.038+0000 7f0e8ec15880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:23:25.040 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:23:25.039+0000 7f0e8ec15880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:23:25.052 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:23:25.052 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:23:25.052 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:23:25.052 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:23:25.052 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:23:25.053 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "b0d74758-1871-4454-bc0f-e3e4b6c30105", 2024-08-18T15:23:25.053 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:23:25.053 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:23:25.053 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:23:25.053 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:23:25.053 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:23:25.053 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:23:25.053 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:23:25.053 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:23:25.053 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:23:25.053 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:23:25.053 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:23:25.053 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:23:25.053 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:23:25.053 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:23:25.054 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:23:25.054 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:23:25.054 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:23:25.054 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:23:25.054 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:23:25.054 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:23:25.054 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:23:25.055 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:23:25.055 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:23:25.055 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:23:25.055 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:23:25.055 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:23:25.055 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:23:25.055 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:23:25.056 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:23:25.056 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:23:25.056 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:23:25.056 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:23:25.056 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:23:25.056 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:23:25.056 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:23:25.056 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:23:25.056 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:23:25.056 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:23:25.056 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:23:25.056 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:23:25.056 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:23:25.056 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:23:25.057 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:23:25.057 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994599.685017_90.1", 2024-08-18T15:23:25.057 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:23:25.057 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:23:25.057 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:23:25.057 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:23:19.685017Z", 2024-08-18T15:23:25.057 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:23:25.057 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:23:25.057 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:23:25.057 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:23:25.057 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:23:25.057 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:23:25.057 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:23:25.057 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:23:25.058 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:23:25.058 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:23:25Z" 2024-08-18T15:23:25.058 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:23:25.059 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1723994599.687927_92.1 target=1_1723994599.685017_90.1 2024-08-18T15:23:30.060 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:23:30.060 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:23:30.060 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:23:30.115 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:23:30.115 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:23:30.147 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:23:30.146+0000 7f01d8d2e880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:23:30.148 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:23:30.147+0000 7f01d8d2e880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:23:30.161 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:23:30.161 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:23:30.161 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:23:30.161 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:23:30.161 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:23:30.161 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "b0d74758-1871-4454-bc0f-e3e4b6c30105", 2024-08-18T15:23:30.161 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:23:30.161 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:23:30.162 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:23:30.162 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:23:30.162 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:23:30.162 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:23:30.162 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:23:30.162 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:23:30.162 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:23:30.162 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:23:30.162 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:23:30.162 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:23:30.162 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:23:30.162 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:23:30.162 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:23:30.163 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:23:30.163 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:23:30.163 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:23:30.163 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:23:30.163 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:23:30.163 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:23:30.163 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:23:30.163 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:23:30.163 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:23:30.163 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:23:30.163 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:23:30.163 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:23:30.163 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:23:30.163 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:23:30.164 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:23:30.164 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:23:30.164 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:23:30.164 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:23:30.164 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:23:30.164 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:23:30.164 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:23:30.164 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:23:30.164 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:23:30.164 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:23:30.164 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:23:30.164 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:23:30.164 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:23:30.165 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:23:30.165 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994599.687927_92.1", 2024-08-18T15:23:30.165 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:23:30.165 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:23:30.165 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:23:30.165 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:23:19.687927Z", 2024-08-18T15:23:30.165 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:23:30.165 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:23:30.165 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:23:30.165 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:23:30.165 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:23:30.165 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:23:30.166 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:23:30.166 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:23:30.166 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:23:30.166 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:23:30Z" 2024-08-18T15:23:30.166 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:23:30.167 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-08-18T15:23:30.167 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-08-18T15:23:30.167 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:23:30.167 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:23:30.167 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:23:30.220 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:23:30.220 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:23:30.252 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:23:30.252+0000 7f0130c62880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:23:30.253 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:23:30.253+0000 7f0130c62880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:23:30.265 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T15:23:30.265 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T15:23:30.265 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T15:23:30.265 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T15:23:30.265 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T15:23:30.266 INFO:teuthology.orchestra.run.smithi134.stdout: "period": "b0d74758-1871-4454-bc0f-e3e4b6c30105", 2024-08-18T15:23:30.266 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:23:30.266 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:23:30.266 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T15:23:30.266 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:23:30.266 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T15:23:30.266 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:23:30.266 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:23:30.266 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:23:30.266 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:23:30.266 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 1, 2024-08-18T15:23:30.266 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:23:30.266 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:23:30.267 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:23:30.267 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:23:30.267 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:23:30.267 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:23:30.267 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T15:23:30.267 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:23:30.267 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:23:30.267 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T15:23:30.267 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:23:30.267 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:23:30.267 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:23:30.267 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T15:23:30.267 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:23:30.267 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:23:30.268 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:23:30.268 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:23:30.268 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T15:23:30.268 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:23:30.268 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:23:30.268 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:23:30.268 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:23:30.268 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:23:30.268 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:23:30.268 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:23:30.268 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:23:30.268 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:23:30.268 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:23:30.269 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:23:30.269 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T15:23:30.269 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:23:30.269 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:23:30.269 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994599.687927_92.1", 2024-08-18T15:23:30.269 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:23:30.269 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:23:30.269 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:23:30.269 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:23:19.687927Z", 2024-08-18T15:23:30.269 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:23:30.269 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:23:30.269 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:23:30.269 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T15:23:30.269 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:23:30.270 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T15:23:30.270 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 1, 2024-08-18T15:23:30.270 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 1 2024-08-18T15:23:30.270 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:23:30.270 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T15:23:30Z" 2024-08-18T15:23:30.270 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T15:23:30.271 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-08-18T15:23:30.271 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:23:30.271 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:23:30.271 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T15:23:30.328 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:23:30.328 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:23:30.362 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:23:30.361+0000 7fb3c6c11880 1 waiting to reach incremental sync.. 2024-08-18T15:23:35.363 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:23:35.361+0000 7fb3c6c11880 1 waiting to reach incremental sync.. 2024-08-18T15:23:40.365 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:23:40.364+0000 7fb3c6c11880 1 bucket sync caught up with source: 2024-08-18T15:23:40.365 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, 00000000001.9.6, , , , , , , , , ] 2024-08-18T15:23:40.365 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, 00000000001.9.6, , , , , , , , , ] 2024-08-18T15:23:40.365 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:23:40.364+0000 7fb3c6c11880 0 bucket checkpoint complete 2024-08-18T15:23:40.381 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-13 zones={test-zone2, test-zone1} 2024-08-18T15:23:40.386 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:23:40.388 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-08-18T15:23:40.389 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:23:40.391 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-08-18T15:23:40.439 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-08-18T15:23:40.446 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-08-18T15:23:40.503 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-08-18T15:23:40.557 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-13 zones={test-zone2, test-zone1} 2024-08-18T15:23:40.558 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:23:40.558 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:23:40.558 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T15:23:40.618 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:23:40.619 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:23:40.654 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:23:40.654+0000 7f4b8e331880 1 bucket sync caught up with source: 2024-08-18T15:23:40.654 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000001.9.6, , , , , , , , , ] 2024-08-18T15:23:40.654 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000001.9.6, , , , , , , , , ] 2024-08-18T15:23:40.654 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:23:40.654+0000 7f4b8e331880 0 bucket checkpoint complete 2024-08-18T15:23:40.667 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-13 zones={test-zone3, test-zone1} 2024-08-18T15:23:40.671 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:23:40.674 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-08-18T15:23:40.674 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:23:40.719 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-08-18T15:23:40.723 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-08-18T15:23:40.729 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-08-18T15:23:40.782 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-08-18T15:23:40.789 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-13 zones={test-zone3, test-zone1} 2024-08-18T15:23:40.790 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:23:40.790 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:23:40.790 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T15:23:40.843 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:23:40.843 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:23:40.877 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:23:40.876+0000 7f9d95942880 1 bucket sync caught up with source: 2024-08-18T15:23:40.877 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000001.34.6, , , , , , , , , ] 2024-08-18T15:23:40.877 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000001.34.6, , , , , , , , , ] 2024-08-18T15:23:40.877 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:23:40.876+0000 7f9d95942880 0 bucket checkpoint complete 2024-08-18T15:23:40.891 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-14 zones={test-zone1, test-zone2} 2024-08-18T15:23:40.895 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:23:40.898 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-08-18T15:23:40.898 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:23:40.943 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-08-18T15:23:40.949 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-08-18T15:23:40.955 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-08-18T15:23:41.008 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-08-18T15:23:41.016 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-14 zones={test-zone1, test-zone2} 2024-08-18T15:23:41.017 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:23:41.017 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:23:41.017 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T15:23:41.073 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:23:41.073 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:23:41.109 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:23:41.109+0000 7f45f163f880 1 bucket sync caught up with source: 2024-08-18T15:23:41.109 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000001.34.6, , , , , , , , , ] 2024-08-18T15:23:41.110 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000001.34.6, , , , , , , , , ] 2024-08-18T15:23:41.110 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:23:41.109+0000 7f45f163f880 0 bucket checkpoint complete 2024-08-18T15:23:41.121 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-14 zones={test-zone3, test-zone2} 2024-08-18T15:23:41.127 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:23:41.129 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-08-18T15:23:41.130 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:23:41.175 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-08-18T15:23:41.181 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-08-18T15:23:41.189 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-08-18T15:23:41.249 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-08-18T15:23:41.306 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-14 zones={test-zone3, test-zone2} 2024-08-18T15:23:41.306 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:23:41.306 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:23:41.306 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T15:23:41.360 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:23:41.360 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:23:41.394 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:23:41.393+0000 7f2e09f20880 1 bucket sync caught up with source: 2024-08-18T15:23:41.394 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000001.12.6, , , , , , , , , ] 2024-08-18T15:23:41.394 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000001.12.6, , , , , , , , , ] 2024-08-18T15:23:41.394 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:23:41.393+0000 7f2e09f20880 0 bucket checkpoint complete 2024-08-18T15:23:41.406 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-15 zones={test-zone1, test-zone3} 2024-08-18T15:23:41.410 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:23:41.412 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-08-18T15:23:41.412 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:23:41.414 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-08-18T15:23:41.418 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-08-18T15:23:41.424 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-08-18T15:23:41.481 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-08-18T15:23:41.489 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-15 zones={test-zone1, test-zone3} 2024-08-18T15:23:41.489 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:23:41.489 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:23:41.489 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T15:23:41.551 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:23:41.551 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:23:41.589 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:23:41.589+0000 7f4e9c132880 1 bucket sync caught up with source: 2024-08-18T15:23:41.589 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, 00000000001.12.6, , , , , , , , , ] 2024-08-18T15:23:41.589 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, 00000000001.12.6, , , , , , , , , ] 2024-08-18T15:23:41.589 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:23:41.589+0000 7f4e9c132880 0 bucket checkpoint complete 2024-08-18T15:23:41.602 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-15 zones={test-zone2, test-zone3} 2024-08-18T15:23:41.606 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:23:41.608 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-08-18T15:23:41.609 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:23:41.611 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-08-18T15:23:41.659 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-08-18T15:23:41.666 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-08-18T15:23:41.677 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-08-18T15:23:41.729 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-15 zones={test-zone2, test-zone3} 2024-08-18T15:23:41.736 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:23:41.736 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:23:41.737 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T15:23:41.792 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:23:41.792 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:23:41.825 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:23:41.825+0000 7f1e3f724880 1 waiting for incremental sync to catch up: 2024-08-18T15:23:41.825 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, 00000000001.9.6, , , , , , , , , ] 2024-08-18T15:23:41.825 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, 00000000002.11.6, , , , , , , , , ] 2024-08-18T15:23:46.826 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:23:46.826+0000 7f1e3f724880 1 waiting for incremental sync to catch up: 2024-08-18T15:23:46.827 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, 00000000001.9.6, , , , , , , , , ] 2024-08-18T15:23:46.827 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, 00000000002.11.6, , , , , , , , , ] 2024-08-18T15:23:51.828 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:23:51.827+0000 7f1e3f724880 1 waiting for incremental sync to catch up: 2024-08-18T15:23:51.828 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, 00000000001.9.6, , , , , , , , , ] 2024-08-18T15:23:51.828 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, 00000000002.11.6, , , , , , , , , ] 2024-08-18T15:23:56.829 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:23:56.828+0000 7f1e3f724880 1 waiting for incremental sync to catch up: 2024-08-18T15:23:56.829 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, 00000000001.9.6, , , , , , , , , ] 2024-08-18T15:23:56.830 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, 00000000002.11.6, , , , , , , , , ] 2024-08-18T15:24:01.830 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:24:01.830+0000 7f1e3f724880 1 waiting for incremental sync to catch up: 2024-08-18T15:24:01.831 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, 00000000001.9.6, , , , , , , , , ] 2024-08-18T15:24:01.831 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, 00000000002.11.6, , , , , , , , , ] 2024-08-18T15:24:06.831 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:24:06.831+0000 7f1e3f724880 1 waiting for incremental sync to catch up: 2024-08-18T15:24:06.832 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, 00000000001.9.6, , , , , , , , , ] 2024-08-18T15:24:06.832 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, 00000000002.11.6, , , , , , , , , ] 2024-08-18T15:24:11.833 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:24:11.832+0000 7f1e3f724880 1 waiting for incremental sync to catch up: 2024-08-18T15:24:11.834 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, 00000000001.9.6, , , , , , , , , ] 2024-08-18T15:24:11.834 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, 00000000002.11.6, , , , , , , , , ] 2024-08-18T15:24:16.834 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:24:16.834+0000 7f1e3f724880 1 waiting for incremental sync to catch up: 2024-08-18T15:24:16.834 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, 00000000001.9.6, , , , , , , , , ] 2024-08-18T15:24:16.835 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, 00000000002.11.6, , , , , , , , , ] 2024-08-18T15:24:21.835 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:24:21.835+0000 7f1e3f724880 1 bucket sync caught up with source: 2024-08-18T15:24:21.836 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, 00000000002.11.6, , , , , , , , , ] 2024-08-18T15:24:21.836 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, 00000000002.11.6, , , , , , , , , ] 2024-08-18T15:24:21.836 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:24:21.835+0000 7f1e3f724880 0 bucket checkpoint complete 2024-08-18T15:24:21.852 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-13 zones={test-zone2, test-zone1} 2024-08-18T15:24:21.858 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:24:21.861 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:24:21.910 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-13 zones={test-zone2, test-zone1} 2024-08-18T15:24:21.911 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:24:21.911 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:24:21.911 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T15:24:21.973 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:24:21.973 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:24:22.010 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:24:22.010+0000 7f5d0a80f880 1 bucket sync caught up with source: 2024-08-18T15:24:22.011 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000002.11.6, , , , , , , , , ] 2024-08-18T15:24:22.011 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000002.11.6, , , , , , , , , ] 2024-08-18T15:24:22.011 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:24:22.010+0000 7f5d0a80f880 0 bucket checkpoint complete 2024-08-18T15:24:22.026 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-13 zones={test-zone3, test-zone1} 2024-08-18T15:24:22.030 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:24:22.032 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:24:22.038 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-13 zones={test-zone3, test-zone1} 2024-08-18T15:24:22.048 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:24:22.048 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:24:22.048 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T15:24:22.103 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:24:22.103 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:24:22.139 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:24:22.139+0000 7fb0f8f27880 1 waiting for incremental sync to catch up: 2024-08-18T15:24:22.140 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000001.34.6, , , , , , , , , ] 2024-08-18T15:24:22.140 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000002.37.6, , , , , , , , , ] 2024-08-18T15:24:27.141 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:24:27.140+0000 7fb0f8f27880 1 waiting for incremental sync to catch up: 2024-08-18T15:24:27.142 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000001.34.6, , , , , , , , , ] 2024-08-18T15:24:27.142 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000002.37.6, , , , , , , , , ] 2024-08-18T15:24:32.142 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:24:32.142+0000 7fb0f8f27880 1 waiting for incremental sync to catch up: 2024-08-18T15:24:32.143 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000001.34.6, , , , , , , , , ] 2024-08-18T15:24:32.143 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000002.37.6, , , , , , , , , ] 2024-08-18T15:24:37.143 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:24:37.143+0000 7fb0f8f27880 1 waiting for incremental sync to catch up: 2024-08-18T15:24:37.144 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000001.34.6, , , , , , , , , ] 2024-08-18T15:24:37.144 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000002.37.6, , , , , , , , , ] 2024-08-18T15:24:42.145 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:24:42.144+0000 7fb0f8f27880 1 bucket sync caught up with source: 2024-08-18T15:24:42.145 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000002.37.6, , , , , , , , , ] 2024-08-18T15:24:42.145 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000002.37.6, , , , , , , , , ] 2024-08-18T15:24:42.145 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:24:42.144+0000 7fb0f8f27880 0 bucket checkpoint complete 2024-08-18T15:24:42.162 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-14 zones={test-zone1, test-zone2} 2024-08-18T15:24:42.168 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:24:42.170 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:24:42.178 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-14 zones={test-zone1, test-zone2} 2024-08-18T15:24:42.178 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:24:42.178 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:24:42.179 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T15:24:42.238 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:24:42.238 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:24:42.276 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:24:42.276+0000 7fbb75e0f880 1 waiting for incremental sync to catch up: 2024-08-18T15:24:42.276 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000001.34.6, , , , , , , , , ] 2024-08-18T15:24:42.276 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000002.37.6, , , , , , , , , ] 2024-08-18T15:24:47.277 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:24:47.277+0000 7fbb75e0f880 1 waiting for incremental sync to catch up: 2024-08-18T15:24:47.278 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000001.34.6, , , , , , , , , ] 2024-08-18T15:24:47.278 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000002.37.6, , , , , , , , , ] 2024-08-18T15:24:52.279 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:24:52.278+0000 7fbb75e0f880 1 waiting for incremental sync to catch up: 2024-08-18T15:24:52.279 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000001.34.6, , , , , , , , , ] 2024-08-18T15:24:52.279 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000002.37.6, , , , , , , , , ] 2024-08-18T15:24:57.280 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:24:57.279+0000 7fbb75e0f880 1 waiting for incremental sync to catch up: 2024-08-18T15:24:57.280 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000001.34.6, , , , , , , , , ] 2024-08-18T15:24:57.280 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000002.37.6, , , , , , , , , ] 2024-08-18T15:25:02.281 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:25:02.281+0000 7fbb75e0f880 1 waiting for incremental sync to catch up: 2024-08-18T15:25:02.282 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000001.34.6, , , , , , , , , ] 2024-08-18T15:25:02.282 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000002.37.6, , , , , , , , , ] 2024-08-18T15:25:07.283 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:25:07.282+0000 7fbb75e0f880 1 bucket sync caught up with source: 2024-08-18T15:25:07.283 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000002.37.6, , , , , , , , , ] 2024-08-18T15:25:07.283 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000002.37.6, , , , , , , , , ] 2024-08-18T15:25:07.283 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:25:07.282+0000 7fbb75e0f880 0 bucket checkpoint complete 2024-08-18T15:25:07.298 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-14 zones={test-zone3, test-zone2} 2024-08-18T15:25:07.306 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:25:07.308 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:25:07.359 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-14 zones={test-zone3, test-zone2} 2024-08-18T15:25:07.369 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:25:07.369 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:25:07.369 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T15:25:07.428 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:25:07.429 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:25:07.464 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:25:07.464+0000 7f5690c0f880 1 waiting for incremental sync to catch up: 2024-08-18T15:25:07.465 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000001.12.6, , , , , , , , , ] 2024-08-18T15:25:07.465 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000002.14.6, , , , , , , , , ] 2024-08-18T15:25:12.466 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:25:12.465+0000 7f5690c0f880 1 waiting for incremental sync to catch up: 2024-08-18T15:25:12.467 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000001.12.6, , , , , , , , , ] 2024-08-18T15:25:12.467 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000002.14.6, , , , , , , , , ] 2024-08-18T15:25:17.468 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:25:17.467+0000 7f5690c0f880 1 waiting for incremental sync to catch up: 2024-08-18T15:25:17.469 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000001.12.6, , , , , , , , , ] 2024-08-18T15:25:17.469 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000002.14.6, , , , , , , , , ] 2024-08-18T15:25:22.469 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:25:22.468+0000 7f5690c0f880 1 bucket sync caught up with source: 2024-08-18T15:25:22.469 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000002.14.6, , , , , , , , , ] 2024-08-18T15:25:22.469 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000002.14.6, , , , , , , , , ] 2024-08-18T15:25:22.469 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:25:22.468+0000 7f5690c0f880 0 bucket checkpoint complete 2024-08-18T15:25:22.485 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-15 zones={test-zone1, test-zone3} 2024-08-18T15:25:22.490 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:25:22.493 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:25:22.543 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-15 zones={test-zone1, test-zone3} 2024-08-18T15:25:22.544 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:25:22.544 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:25:22.544 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T15:25:22.607 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:25:22.607 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:25:22.646 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:25:22.646+0000 7f6097738880 1 bucket sync caught up with source: 2024-08-18T15:25:22.646 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, 00000000002.14.6, , , , , , , , , ] 2024-08-18T15:25:22.646 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, 00000000002.14.6, , , , , , , , , ] 2024-08-18T15:25:22.646 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:25:22.646+0000 7f6097738880 0 bucket checkpoint complete 2024-08-18T15:25:22.659 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-15 zones={test-zone2, test-zone3} 2024-08-18T15:25:22.663 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:25:22.666 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:25:22.673 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-15 zones={test-zone2, test-zone3} 2024-08-18T15:25:22.673 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_object_delete ... ok 2024-08-18T15:25:22.674 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=kibgmb-16 2024-08-18T15:25:22.693 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=kibgmb-17 2024-08-18T15:25:22.726 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=kibgmb-18 2024-08-18T15:25:25.920 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:25:25.920 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:25:25.921 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:25:25.967 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:25:25.967 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:25:26.000 INFO:teuthology.orchestra.run.smithi134.stderr:No --period given, using current period=b0d74758-1871-4454-bc0f-e3e4b6c30105 2024-08-18T15:25:26.012 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T15:25:26.013 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:25:26.013 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:25:26.013 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:19:24.506194Z" 2024-08-18T15:25:26.013 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:25:26.013 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:25:26.013 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T15:25:26.013 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T15:25:26.013 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:25:26.013 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:25:26.013 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:25:26.014 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:21:34.952005Z" 2024-08-18T15:25:26.014 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:25:26.014 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:25:26.014 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994722.740810_107.1", 2024-08-18T15:25:26.014 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:25:22.740810Z" 2024-08-18T15:25:26.014 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:25:26.014 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T15:25:26.015 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-08-18T15:25:26.015 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:25:26.015 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:25:26.016 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:25:26.069 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:25:26.069 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:25:26.099 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:25:26.099+0000 7ffad483f880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:25:26.100 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:25:26.100+0000 7ffad483f880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:25:26.112 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:25:26.113 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:25:26.113 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:25:26.113 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:25:26.113 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:25:26.113 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "b0d74758-1871-4454-bc0f-e3e4b6c30105", 2024-08-18T15:25:26.113 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:25:26.113 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:25:26.113 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:25:26.113 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:25:26.113 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:25:26.113 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:25:26.114 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:25:26.114 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:25:26.114 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:25:26.114 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:25:26.114 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:25:26.114 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:25:26.114 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:25:26.114 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:25:26.114 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:25:26.114 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:25:26.114 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:25:26.114 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:25:26.114 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:25:26.115 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:25:26.115 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:25:26.115 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:25:26.115 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:25:26.115 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:25:26.115 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:25:26.115 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:25:26.115 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:25:26.115 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:25:26.115 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:25:26.115 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:25:26.115 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:25:26.115 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:25:26.116 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:25:26.116 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:25:26.116 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:25:26.116 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:25:26.116 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:25:26.116 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:25:26.116 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:25:26.116 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:25:26.116 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:25:26.116 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:25:26.116 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:25:26.116 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994722.736900_105.1", 2024-08-18T15:25:26.116 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:25:26.116 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:25:26.117 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:25:26.117 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:25:22.736900Z", 2024-08-18T15:25:26.117 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:25:26.117 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:25:26.117 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:25:26.117 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:25:26.117 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:25:26.117 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:25:26.117 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:25:26.117 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:25:26.117 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:25:26.117 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:25:26Z" 2024-08-18T15:25:26.117 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:25:26.118 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1723994722.740810_107.1 target=1_1723994722.736900_105.1 2024-08-18T15:25:31.119 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:25:31.120 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:25:31.120 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:25:31.176 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:25:31.176 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:25:31.208 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:25:31.207+0000 7f37620bb880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:25:31.209 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:25:31.209+0000 7f37620bb880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:25:31.221 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:25:31.221 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:25:31.222 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:25:31.222 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:25:31.222 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:25:31.222 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "b0d74758-1871-4454-bc0f-e3e4b6c30105", 2024-08-18T15:25:31.222 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:25:31.222 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:25:31.222 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:25:31.222 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:25:31.222 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:25:31.222 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:25:31.222 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:25:31.222 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:25:31.222 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:25:31.223 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:25:31.223 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:25:31.223 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:25:31.223 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:25:31.223 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:25:31.223 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:25:31.223 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:25:31.223 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:25:31.223 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:25:31.223 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:25:31.223 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:25:31.223 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:25:31.223 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:25:31.224 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:25:31.224 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:25:31.224 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:25:31.224 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:25:31.224 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:25:31.224 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:25:31.224 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:25:31.224 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:25:31.224 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:25:31.224 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:25:31.224 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:25:31.224 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:25:31.224 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:25:31.224 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:25:31.225 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:25:31.225 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:25:31.225 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:25:31.225 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:25:31.225 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:25:31.225 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:25:31.225 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:25:31.225 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994722.740810_107.1", 2024-08-18T15:25:31.225 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:25:31.225 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:25:31.225 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:25:31.225 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:25:22.740810Z", 2024-08-18T15:25:31.226 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:25:31.226 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:25:31.226 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:25:31.226 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:25:31.226 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:25:31.226 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:25:31.226 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:25:31.226 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:25:31.226 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:25:31.226 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:25:31Z" 2024-08-18T15:25:31.226 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:25:31.227 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-08-18T15:25:31.227 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-08-18T15:25:31.229 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:25:31.229 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:25:31.230 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:25:31.285 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:25:31.286 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:25:31.316 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:25:31.316+0000 7f7a0de35880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:25:31.317 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:25:31.317+0000 7f7a0de35880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:25:31.330 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T15:25:31.330 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T15:25:31.330 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T15:25:31.330 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T15:25:31.330 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T15:25:31.330 INFO:teuthology.orchestra.run.smithi134.stdout: "period": "b0d74758-1871-4454-bc0f-e3e4b6c30105", 2024-08-18T15:25:31.331 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:25:31.331 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:25:31.331 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T15:25:31.331 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:25:31.331 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T15:25:31.331 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:25:31.331 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:25:31.331 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:25:31.331 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:25:31.331 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 1, 2024-08-18T15:25:31.331 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:25:31.331 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:25:31.331 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:25:31.331 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:25:31.332 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:25:31.332 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:25:31.332 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T15:25:31.332 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:25:31.332 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:25:31.332 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T15:25:31.332 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:25:31.332 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:25:31.332 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:25:31.332 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T15:25:31.332 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:25:31.332 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:25:31.332 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:25:31.332 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:25:31.332 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T15:25:31.333 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:25:31.333 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:25:31.333 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:25:31.333 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:25:31.333 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:25:31.333 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:25:31.333 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:25:31.333 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:25:31.333 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:25:31.333 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:25:31.334 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:25:31.334 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T15:25:31.334 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:25:31.334 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:25:31.334 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994722.740810_107.1", 2024-08-18T15:25:31.334 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:25:31.334 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:25:31.334 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:25:31.334 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:25:22.740810Z", 2024-08-18T15:25:31.334 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:25:31.334 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:25:31.334 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:25:31.335 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T15:25:31.335 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:25:31.335 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T15:25:31.335 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 1, 2024-08-18T15:25:31.335 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 1 2024-08-18T15:25:31.335 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:25:31.335 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T15:25:31Z" 2024-08-18T15:25:31.335 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T15:25:31.336 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-08-18T15:25:31.336 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:25:31.336 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:25:31.336 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T15:25:31.392 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:25:31.392 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:25:31.426 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:25:31.426+0000 7f2845813880 1 waiting to reach incremental sync.. 2024-08-18T15:25:36.427 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:25:36.426+0000 7f2845813880 1 waiting to reach incremental sync.. 2024-08-18T15:25:41.428 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:25:41.426+0000 7f2845813880 1 waiting to reach incremental sync.. 2024-08-18T15:25:46.428 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:25:46.427+0000 7f2845813880 1 waiting to reach incremental sync.. 2024-08-18T15:25:51.429 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:25:51.428+0000 7f2845813880 1 waiting to reach incremental sync.. 2024-08-18T15:25:56.430 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:25:56.429+0000 7f2845813880 1 waiting to reach incremental sync.. 2024-08-18T15:26:01.430 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:26:01.430+0000 7f2845813880 1 waiting to reach incremental sync.. 2024-08-18T15:26:06.431 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:26:06.430+0000 7f2845813880 1 waiting to reach incremental sync.. 2024-08-18T15:26:11.431 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:26:11.431+0000 7f2845813880 1 waiting to reach incremental sync.. 2024-08-18T15:26:16.432 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:26:16.431+0000 7f2845813880 1 waiting to reach incremental sync.. 2024-08-18T15:26:21.434 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:26:21.433+0000 7f2845813880 1 bucket sync caught up with source: 2024-08-18T15:26:21.434 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000007.18.6, 00000000004.18.6, 00000000004.16.6, 00000000004.15.6, 00000000006.23.6, 00000000005.22.6, 00000000004.15.6, 00000000005.19.6, 00000000002.25.6, 00000000004.21.6, 00000000004.19.6] 2024-08-18T15:26:21.434 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000007.18.6, 00000000004.18.6, 00000000004.16.6, 00000000004.15.6, 00000000006.23.6, 00000000005.22.6, 00000000004.15.6, 00000000005.19.6, 00000000002.25.6, 00000000004.21.6, 00000000004.19.6] 2024-08-18T15:26:21.434 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:26:21.433+0000 7f2845813880 0 bucket checkpoint complete 2024-08-18T15:26:21.450 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-16 zones={test-zone2, test-zone1} 2024-08-18T15:26:21.456 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:26:21.461 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-08-18T15:26:21.461 DEBUG:tasks.rgw_multi.zone_rados:o=obj10 2024-08-18T15:26:21.461 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-08-18T15:26:21.461 DEBUG:tasks.rgw_multi.zone_rados:o=obj12 2024-08-18T15:26:21.461 DEBUG:tasks.rgw_multi.zone_rados:o=obj13 2024-08-18T15:26:21.461 DEBUG:tasks.rgw_multi.zone_rados:o=obj14 2024-08-18T15:26:21.461 DEBUG:tasks.rgw_multi.zone_rados:o=obj15 2024-08-18T15:26:21.462 DEBUG:tasks.rgw_multi.zone_rados:o=obj16 2024-08-18T15:26:21.462 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-08-18T15:26:21.462 DEBUG:tasks.rgw_multi.zone_rados:o=obj18 2024-08-18T15:26:21.462 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-08-18T15:26:21.462 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-08-18T15:26:21.462 DEBUG:tasks.rgw_multi.zone_rados:o=obj20 2024-08-18T15:26:21.462 DEBUG:tasks.rgw_multi.zone_rados:o=obj21 2024-08-18T15:26:21.462 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-08-18T15:26:21.462 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2024-08-18T15:26:21.462 DEBUG:tasks.rgw_multi.zone_rados:o=obj24 2024-08-18T15:26:21.462 DEBUG:tasks.rgw_multi.zone_rados:o=obj25 2024-08-18T15:26:21.462 DEBUG:tasks.rgw_multi.zone_rados:o=obj26 2024-08-18T15:26:21.462 DEBUG:tasks.rgw_multi.zone_rados:o=obj27 2024-08-18T15:26:21.462 DEBUG:tasks.rgw_multi.zone_rados:o=obj28 2024-08-18T15:26:21.463 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2024-08-18T15:26:21.463 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-08-18T15:26:21.463 DEBUG:tasks.rgw_multi.zone_rados:o=obj30 2024-08-18T15:26:21.463 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2024-08-18T15:26:21.463 DEBUG:tasks.rgw_multi.zone_rados:o=obj32 2024-08-18T15:26:21.463 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-08-18T15:26:21.463 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-08-18T15:26:21.463 DEBUG:tasks.rgw_multi.zone_rados:o=obj35 2024-08-18T15:26:21.463 DEBUG:tasks.rgw_multi.zone_rados:o=obj36 2024-08-18T15:26:21.463 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2024-08-18T15:26:21.463 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-08-18T15:26:21.463 DEBUG:tasks.rgw_multi.zone_rados:o=obj39 2024-08-18T15:26:21.463 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-08-18T15:26:21.463 DEBUG:tasks.rgw_multi.zone_rados:o=obj40 2024-08-18T15:26:21.463 DEBUG:tasks.rgw_multi.zone_rados:o=obj41 2024-08-18T15:26:21.464 DEBUG:tasks.rgw_multi.zone_rados:o=obj42 2024-08-18T15:26:21.464 DEBUG:tasks.rgw_multi.zone_rados:o=obj43 2024-08-18T15:26:21.464 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-08-18T15:26:21.464 DEBUG:tasks.rgw_multi.zone_rados:o=obj45 2024-08-18T15:26:21.464 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2024-08-18T15:26:21.464 DEBUG:tasks.rgw_multi.zone_rados:o=obj47 2024-08-18T15:26:21.464 DEBUG:tasks.rgw_multi.zone_rados:o=obj48 2024-08-18T15:26:21.464 DEBUG:tasks.rgw_multi.zone_rados:o=obj49 2024-08-18T15:26:21.464 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-08-18T15:26:21.464 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-08-18T15:26:21.464 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-08-18T15:26:21.464 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-08-18T15:26:21.464 DEBUG:tasks.rgw_multi.zone_rados:o=obj9 2024-08-18T15:26:21.464 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:26:21.469 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-08-18T15:26:21.469 DEBUG:tasks.rgw_multi.zone_rados:o=obj10 2024-08-18T15:26:21.469 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-08-18T15:26:21.469 DEBUG:tasks.rgw_multi.zone_rados:o=obj12 2024-08-18T15:26:21.469 DEBUG:tasks.rgw_multi.zone_rados:o=obj13 2024-08-18T15:26:21.469 DEBUG:tasks.rgw_multi.zone_rados:o=obj14 2024-08-18T15:26:21.469 DEBUG:tasks.rgw_multi.zone_rados:o=obj15 2024-08-18T15:26:21.469 DEBUG:tasks.rgw_multi.zone_rados:o=obj16 2024-08-18T15:26:21.470 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-08-18T15:26:21.470 DEBUG:tasks.rgw_multi.zone_rados:o=obj18 2024-08-18T15:26:21.470 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-08-18T15:26:21.470 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-08-18T15:26:21.470 DEBUG:tasks.rgw_multi.zone_rados:o=obj20 2024-08-18T15:26:21.470 DEBUG:tasks.rgw_multi.zone_rados:o=obj21 2024-08-18T15:26:21.470 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-08-18T15:26:21.470 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2024-08-18T15:26:21.470 DEBUG:tasks.rgw_multi.zone_rados:o=obj24 2024-08-18T15:26:21.470 DEBUG:tasks.rgw_multi.zone_rados:o=obj25 2024-08-18T15:26:21.470 DEBUG:tasks.rgw_multi.zone_rados:o=obj26 2024-08-18T15:26:21.470 DEBUG:tasks.rgw_multi.zone_rados:o=obj27 2024-08-18T15:26:21.470 DEBUG:tasks.rgw_multi.zone_rados:o=obj28 2024-08-18T15:26:21.470 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2024-08-18T15:26:21.470 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-08-18T15:26:21.471 DEBUG:tasks.rgw_multi.zone_rados:o=obj30 2024-08-18T15:26:21.471 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2024-08-18T15:26:21.471 DEBUG:tasks.rgw_multi.zone_rados:o=obj32 2024-08-18T15:26:21.471 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-08-18T15:26:21.471 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-08-18T15:26:21.471 DEBUG:tasks.rgw_multi.zone_rados:o=obj35 2024-08-18T15:26:21.471 DEBUG:tasks.rgw_multi.zone_rados:o=obj36 2024-08-18T15:26:21.471 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2024-08-18T15:26:21.471 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-08-18T15:26:21.471 DEBUG:tasks.rgw_multi.zone_rados:o=obj39 2024-08-18T15:26:21.471 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-08-18T15:26:21.471 DEBUG:tasks.rgw_multi.zone_rados:o=obj40 2024-08-18T15:26:21.471 DEBUG:tasks.rgw_multi.zone_rados:o=obj41 2024-08-18T15:26:21.471 DEBUG:tasks.rgw_multi.zone_rados:o=obj42 2024-08-18T15:26:21.471 DEBUG:tasks.rgw_multi.zone_rados:o=obj43 2024-08-18T15:26:21.471 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-08-18T15:26:21.472 DEBUG:tasks.rgw_multi.zone_rados:o=obj45 2024-08-18T15:26:21.472 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2024-08-18T15:26:21.472 DEBUG:tasks.rgw_multi.zone_rados:o=obj47 2024-08-18T15:26:21.472 DEBUG:tasks.rgw_multi.zone_rados:o=obj48 2024-08-18T15:26:21.472 DEBUG:tasks.rgw_multi.zone_rados:o=obj49 2024-08-18T15:26:21.472 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-08-18T15:26:21.472 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-08-18T15:26:21.472 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-08-18T15:26:21.472 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-08-18T15:26:21.472 DEBUG:tasks.rgw_multi.zone_rados:o=obj9 2024-08-18T15:26:21.481 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T15:26:21.530 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T15:26:21.542 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T15:26:21.594 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj10 2024-08-18T15:26:21.601 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj10 2024-08-18T15:26:21.656 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj10 2024-08-18T15:26:21.705 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T15:26:21.713 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T15:26:21.768 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T15:26:21.817 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj12 2024-08-18T15:26:21.826 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj12 2024-08-18T15:26:21.881 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj12 2024-08-18T15:26:21.933 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj13 2024-08-18T15:26:21.941 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj13 2024-08-18T15:26:21.997 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj13 2024-08-18T15:26:22.049 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj14 2024-08-18T15:26:22.056 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj14 2024-08-18T15:26:22.068 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj14 2024-08-18T15:26:22.076 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj15 2024-08-18T15:26:22.083 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj15 2024-08-18T15:26:22.138 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj15 2024-08-18T15:26:22.147 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj16 2024-08-18T15:26:22.194 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj16 2024-08-18T15:26:22.206 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj16 2024-08-18T15:26:22.257 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-08-18T15:26:22.264 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-08-18T15:26:22.316 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-08-18T15:26:22.369 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj18 2024-08-18T15:26:22.376 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj18 2024-08-18T15:26:22.430 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj18 2024-08-18T15:26:22.480 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-08-18T15:26:22.488 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-08-18T15:26:22.551 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-08-18T15:26:22.608 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T15:26:22.617 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T15:26:22.630 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T15:26:22.682 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj20 2024-08-18T15:26:22.690 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj20 2024-08-18T15:26:22.747 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj20 2024-08-18T15:26:22.800 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj21 2024-08-18T15:26:22.807 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj21 2024-08-18T15:26:22.863 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj21 2024-08-18T15:26:22.919 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T15:26:22.929 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T15:26:22.985 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T15:26:23.042 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-08-18T15:26:23.049 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-08-18T15:26:23.109 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-08-18T15:26:23.162 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj24 2024-08-18T15:26:23.169 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj24 2024-08-18T15:26:23.229 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj24 2024-08-18T15:26:23.285 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj25 2024-08-18T15:26:23.293 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj25 2024-08-18T15:26:23.353 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj25 2024-08-18T15:26:23.410 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj26 2024-08-18T15:26:23.418 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj26 2024-08-18T15:26:23.430 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj26 2024-08-18T15:26:23.480 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj27 2024-08-18T15:26:23.487 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj27 2024-08-18T15:26:23.543 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj27 2024-08-18T15:26:23.596 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj28 2024-08-18T15:26:23.604 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj28 2024-08-18T15:26:23.662 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj28 2024-08-18T15:26:23.672 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-08-18T15:26:23.720 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-08-18T15:26:23.732 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-08-18T15:26:23.782 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T15:26:23.789 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T15:26:23.802 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T15:26:23.855 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj30 2024-08-18T15:26:23.863 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj30 2024-08-18T15:26:23.915 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj30 2024-08-18T15:26:23.968 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2024-08-18T15:26:23.975 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2024-08-18T15:26:24.027 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2024-08-18T15:26:24.079 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj32 2024-08-18T15:26:24.087 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj32 2024-08-18T15:26:24.147 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj32 2024-08-18T15:26:24.200 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T15:26:24.207 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T15:26:24.263 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T15:26:24.316 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-08-18T15:26:24.323 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-08-18T15:26:24.380 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-08-18T15:26:24.434 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj35 2024-08-18T15:26:24.442 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj35 2024-08-18T15:26:24.498 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj35 2024-08-18T15:26:24.507 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj36 2024-08-18T15:26:24.560 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj36 2024-08-18T15:26:24.572 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj36 2024-08-18T15:26:24.580 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2024-08-18T15:26:24.588 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2024-08-18T15:26:24.600 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2024-08-18T15:26:24.654 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-08-18T15:26:24.702 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-08-18T15:26:24.714 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-08-18T15:26:24.723 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj39 2024-08-18T15:26:24.774 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj39 2024-08-18T15:26:24.786 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj39 2024-08-18T15:26:24.794 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T15:26:24.844 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T15:26:24.858 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T15:26:24.868 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj40 2024-08-18T15:26:24.923 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj40 2024-08-18T15:26:24.979 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj40 2024-08-18T15:26:25.035 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj41 2024-08-18T15:26:25.042 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj41 2024-08-18T15:26:25.053 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj41 2024-08-18T15:26:25.105 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj42 2024-08-18T15:26:25.112 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj42 2024-08-18T15:26:25.164 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj42 2024-08-18T15:26:25.214 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj43 2024-08-18T15:26:25.223 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj43 2024-08-18T15:26:25.279 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj43 2024-08-18T15:26:25.331 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T15:26:25.338 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T15:26:25.395 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T15:26:25.451 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj45 2024-08-18T15:26:25.458 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj45 2024-08-18T15:26:25.511 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj45 2024-08-18T15:26:25.559 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-08-18T15:26:25.566 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-08-18T15:26:25.618 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-08-18T15:26:25.667 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj47 2024-08-18T15:26:25.674 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj47 2024-08-18T15:26:25.730 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj47 2024-08-18T15:26:25.739 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj48 2024-08-18T15:26:25.788 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj48 2024-08-18T15:26:25.847 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj48 2024-08-18T15:26:25.899 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj49 2024-08-18T15:26:25.906 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj49 2024-08-18T15:26:25.918 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj49 2024-08-18T15:26:25.928 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-08-18T15:26:25.978 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-08-18T15:26:25.989 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-08-18T15:26:26.041 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-08-18T15:26:26.048 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-08-18T15:26:26.101 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-08-18T15:26:26.154 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-08-18T15:26:26.203 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-08-18T15:26:26.256 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-08-18T15:26:26.305 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-08-18T15:26:26.312 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-08-18T15:26:26.368 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-08-18T15:26:26.416 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj9 2024-08-18T15:26:26.423 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj9 2024-08-18T15:26:26.434 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj9 2024-08-18T15:26:26.483 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-16 zones={test-zone2, test-zone1} 2024-08-18T15:26:26.484 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:26:26.484 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:26:26.484 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T15:26:26.538 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:26:26.539 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:26:26.570 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:26:26.569+0000 7fd4d2522880 1 waiting to reach incremental sync.. 2024-08-18T15:26:31.571 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:26:31.570+0000 7fd4d2522880 1 waiting to reach incremental sync.. 2024-08-18T15:26:36.571 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:26:36.571+0000 7fd4d2522880 1 waiting to reach incremental sync.. 2024-08-18T15:26:41.572 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:26:41.571+0000 7fd4d2522880 1 waiting to reach incremental sync.. 2024-08-18T15:26:46.573 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:26:46.572+0000 7fd4d2522880 1 waiting to reach incremental sync.. 2024-08-18T15:26:51.574 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:26:51.573+0000 7fd4d2522880 1 waiting to reach incremental sync.. 2024-08-18T15:26:56.574 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:26:56.573+0000 7fd4d2522880 1 waiting to reach incremental sync.. 2024-08-18T15:27:01.575 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:27:01.574+0000 7fd4d2522880 1 waiting to reach incremental sync.. 2024-08-18T15:27:06.575 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:27:06.575+0000 7fd4d2522880 1 waiting to reach incremental sync.. 2024-08-18T15:27:11.576 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:27:11.575+0000 7fd4d2522880 1 waiting to reach incremental sync.. 2024-08-18T15:27:16.577 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:27:16.576+0000 7fd4d2522880 1 waiting to reach incremental sync.. 2024-08-18T15:27:21.577 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:27:21.577+0000 7fd4d2522880 1 waiting to reach incremental sync.. 2024-08-18T15:27:26.578 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:27:26.577+0000 7fd4d2522880 1 waiting to reach incremental sync.. 2024-08-18T15:27:31.579 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:27:31.578+0000 7fd4d2522880 1 waiting to reach incremental sync.. 2024-08-18T15:27:36.579 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:27:36.579+0000 7fd4d2522880 1 waiting to reach incremental sync.. 2024-08-18T15:27:41.580 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:27:41.579+0000 7fd4d2522880 1 waiting to reach incremental sync.. 2024-08-18T15:27:46.581 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:27:46.580+0000 7fd4d2522880 1 waiting to reach incremental sync.. 2024-08-18T15:27:51.581 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:27:51.580+0000 7fd4d2522880 1 waiting to reach incremental sync.. 2024-08-18T15:27:56.582 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:27:56.581+0000 7fd4d2522880 1 waiting to reach incremental sync.. 2024-08-18T15:28:01.583 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:28:01.582+0000 7fd4d2522880 1 waiting to reach incremental sync.. 2024-08-18T15:28:06.583 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:28:06.582+0000 7fd4d2522880 1 waiting to reach incremental sync.. 2024-08-18T15:28:11.584 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:28:11.583+0000 7fd4d2522880 1 waiting to reach incremental sync.. 2024-08-18T15:28:16.584 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:28:16.584+0000 7fd4d2522880 1 waiting to reach incremental sync.. 2024-08-18T15:28:21.585 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:28:21.584+0000 7fd4d2522880 1 waiting to reach incremental sync.. 2024-08-18T15:28:26.586 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:28:26.585+0000 7fd4d2522880 1 waiting to reach incremental sync.. 2024-08-18T15:28:31.587 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:28:31.586+0000 7fd4d2522880 1 waiting to reach incremental sync.. 2024-08-18T15:28:36.588 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:28:36.587+0000 7fd4d2522880 1 waiting to reach incremental sync.. 2024-08-18T15:28:41.588 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:28:41.587+0000 7fd4d2522880 1 waiting to reach incremental sync.. 2024-08-18T15:28:46.589 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:28:46.588+0000 7fd4d2522880 1 waiting to reach incremental sync.. 2024-08-18T15:28:51.589 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:28:51.588+0000 7fd4d2522880 1 waiting to reach incremental sync.. 2024-08-18T15:28:56.590 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:28:56.589+0000 7fd4d2522880 1 waiting to reach incremental sync.. 2024-08-18T15:29:01.590 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:29:01.590+0000 7fd4d2522880 1 waiting to reach incremental sync.. 2024-08-18T15:29:06.591 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:29:06.590+0000 7fd4d2522880 1 waiting to reach incremental sync.. 2024-08-18T15:29:11.592 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:29:11.591+0000 7fd4d2522880 1 waiting to reach incremental sync.. 2024-08-18T15:29:16.593 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:29:16.592+0000 7fd4d2522880 1 waiting to reach incremental sync.. 2024-08-18T15:29:21.593 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:29:21.593+0000 7fd4d2522880 1 waiting to reach incremental sync.. 2024-08-18T15:29:26.594 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:29:26.593+0000 7fd4d2522880 1 waiting to reach incremental sync.. 2024-08-18T15:29:31.595 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:29:31.594+0000 7fd4d2522880 1 waiting to reach incremental sync.. 2024-08-18T15:29:36.595 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:29:36.595+0000 7fd4d2522880 1 waiting to reach incremental sync.. 2024-08-18T15:29:41.596 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:29:41.595+0000 7fd4d2522880 1 waiting to reach incremental sync.. 2024-08-18T15:29:46.597 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:29:46.596+0000 7fd4d2522880 1 waiting to reach incremental sync.. 2024-08-18T15:29:51.597 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:29:51.597+0000 7fd4d2522880 1 waiting to reach incremental sync.. 2024-08-18T15:29:56.598 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:29:56.597+0000 7fd4d2522880 1 waiting to reach incremental sync.. 2024-08-18T15:30:01.602 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:30:01.598+0000 7fd4d2522880 1 waiting to reach incremental sync.. 2024-08-18T15:30:06.600 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:30:06.599+0000 7fd4d2522880 1 waiting to reach incremental sync.. 2024-08-18T15:30:11.600 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:30:11.599+0000 7fd4d2522880 1 waiting to reach incremental sync.. 2024-08-18T15:30:16.601 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:30:16.600+0000 7fd4d2522880 1 waiting to reach incremental sync.. 2024-08-18T15:30:21.602 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:30:21.601+0000 7fd4d2522880 1 waiting to reach incremental sync.. 2024-08-18T15:30:26.602 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:30:26.602+0000 7fd4d2522880 1 waiting to reach incremental sync.. 2024-08-18T15:30:31.603 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:30:31.602+0000 7fd4d2522880 1 waiting to reach incremental sync.. 2024-08-18T15:30:36.603 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:30:36.603+0000 7fd4d2522880 1 waiting to reach incremental sync.. 2024-08-18T15:30:41.604 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:30:41.603+0000 7fd4d2522880 1 waiting to reach incremental sync.. 2024-08-18T15:30:46.604 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:30:46.604+0000 7fd4d2522880 1 waiting to reach incremental sync.. 2024-08-18T15:30:51.605 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:30:51.604+0000 7fd4d2522880 1 waiting to reach incremental sync.. 2024-08-18T15:30:56.606 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:30:56.605+0000 7fd4d2522880 1 waiting to reach incremental sync.. 2024-08-18T15:31:01.607 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:31:01.606+0000 7fd4d2522880 1 waiting to reach incremental sync.. 2024-08-18T15:31:06.607 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:31:06.607+0000 7fd4d2522880 1 waiting to reach incremental sync.. 2024-08-18T15:31:11.608 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:31:11.607+0000 7fd4d2522880 1 waiting to reach incremental sync.. 2024-08-18T15:31:16.609 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:31:16.608+0000 7fd4d2522880 1 waiting to reach incremental sync.. 2024-08-18T15:31:21.609 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:31:21.609+0000 7fd4d2522880 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-08-18T15:31:21.610 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:31:21.609+0000 7fd4d2522880 0 bucket sync checkpoint failed: (110) Connection timed out 2024-08-18T15:31:21.610 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:31:21.609+0000 7fd4d2522880 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-08-18T15:31:21.624 DEBUG:teuthology.orchestra.run:got remote process result: 110 2024-08-18T15:31:21.635 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multi_object_delete ... FAIL 2024-08-18T15:31:21.635 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=kibgmb-19 2024-08-18T15:31:21.653 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=kibgmb-20 2024-08-18T15:31:21.680 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=kibgmb-21 2024-08-18T15:31:21.724 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:31:21.724 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:31:21.725 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:31:21.780 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:31:21.780 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:31:21.812 INFO:teuthology.orchestra.run.smithi134.stderr:No --period given, using current period=b0d74758-1871-4454-bc0f-e3e4b6c30105 2024-08-18T15:31:21.823 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T15:31:21.824 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:31:21.824 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:31:21.824 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:19:24.506194Z" 2024-08-18T15:31:21.824 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:31:21.824 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:31:21.824 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T15:31:21.824 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T15:31:21.824 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:31:21.824 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:31:21.824 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:31:21.825 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:21:34.952005Z" 2024-08-18T15:31:21.825 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:31:21.825 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:31:21.825 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723995081.722384_289.1", 2024-08-18T15:31:21.825 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:31:21.722384Z" 2024-08-18T15:31:21.825 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:31:21.825 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T15:31:21.826 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-08-18T15:31:21.826 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:31:21.827 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:31:21.827 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:31:21.880 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:31:21.880 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:31:21.910 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:31:21.910+0000 7fb24c319880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:31:21.911 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:31:21.910+0000 7fb24c319880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:31:21.923 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:31:21.923 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:31:21.923 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:31:21.923 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:31:21.923 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:31:21.923 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "b0d74758-1871-4454-bc0f-e3e4b6c30105", 2024-08-18T15:31:21.924 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:31:21.924 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:31:21.924 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:31:21.924 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:31:21.924 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:31:21.924 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:31:21.924 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:31:21.924 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:31:21.924 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:31:21.924 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:31:21.925 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:31:21.925 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:31:21.925 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:31:21.925 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:31:21.925 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:31:21.925 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:31:21.925 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:31:21.925 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:31:21.925 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:31:21.925 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:31:21.925 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:31:21.925 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:31:21.925 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:31:21.926 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:31:21.926 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:31:21.926 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:31:21.926 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:31:21.926 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:31:21.926 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:31:21.926 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:31:21.926 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:31:21.927 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:31:21.927 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:31:21.927 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:31:21.927 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:31:21.927 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:31:21.927 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:31:21.927 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:31:21.927 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:31:21.927 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:31:21.928 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:31:21.928 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:31:21.928 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:31:21.928 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723995081.645527_272.1", 2024-08-18T15:31:21.928 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:31:21.928 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:31:21.928 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:31:21.928 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:31:21.645527Z", 2024-08-18T15:31:21.928 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:31:21.928 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:31:21.929 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:31:21.929 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:31:21.929 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:31:21.929 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:31:21.929 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:31:21.929 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:31:21.929 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:31:21.929 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:31:21Z" 2024-08-18T15:31:21.929 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:31:21.930 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1723995081.722384_289.1 target=1_1723995081.645527_272.1 2024-08-18T15:31:26.931 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:31:26.932 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:31:26.932 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:31:26.987 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:31:26.987 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:31:27.021 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:31:27.020+0000 7f9a46415880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:31:27.022 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:31:27.021+0000 7f9a46415880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:31:27.034 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:31:27.034 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:31:27.034 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:31:27.034 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:31:27.034 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:31:27.034 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "b0d74758-1871-4454-bc0f-e3e4b6c30105", 2024-08-18T15:31:27.035 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:31:27.035 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:31:27.035 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:31:27.035 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:31:27.035 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:31:27.035 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:31:27.035 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:31:27.035 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:31:27.035 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:31:27.035 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:31:27.035 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:31:27.035 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:31:27.035 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:31:27.036 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:31:27.036 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:31:27.036 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:31:27.036 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:31:27.036 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:31:27.036 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:31:27.036 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:31:27.036 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:31:27.036 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:31:27.036 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:31:27.036 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:31:27.036 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:31:27.037 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:31:27.037 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:31:27.037 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:31:27.037 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:31:27.037 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:31:27.037 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:31:27.037 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:31:27.037 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:31:27.037 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:31:27.037 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:31:27.037 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:31:27.037 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:31:27.037 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:31:27.038 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:31:27.038 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:31:27.038 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:31:27.038 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:31:27.038 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:31:27.038 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723995081.722384_289.1", 2024-08-18T15:31:27.038 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:31:27.038 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:31:27.038 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:31:27.038 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:31:21.722384Z", 2024-08-18T15:31:27.038 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:31:27.038 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:31:27.038 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:31:27.039 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:31:27.039 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:31:27.039 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:31:27.039 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:31:27.039 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:31:27.039 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:31:27.039 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:31:27Z" 2024-08-18T15:31:27.039 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:31:27.040 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-08-18T15:31:27.040 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-08-18T15:31:27.040 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:31:27.041 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:31:27.041 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:31:27.096 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:31:27.096 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:31:27.126 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:31:27.125+0000 7fa5ddb42880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:31:27.126 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:31:27.126+0000 7fa5ddb42880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:31:27.138 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T15:31:27.138 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T15:31:27.138 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T15:31:27.138 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T15:31:27.138 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T15:31:27.139 INFO:teuthology.orchestra.run.smithi134.stdout: "period": "b0d74758-1871-4454-bc0f-e3e4b6c30105", 2024-08-18T15:31:27.139 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:31:27.139 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:31:27.139 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T15:31:27.139 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:31:27.139 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T15:31:27.139 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:31:27.139 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:31:27.139 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:31:27.139 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:31:27.139 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 1, 2024-08-18T15:31:27.139 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:31:27.139 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:31:27.140 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:31:27.140 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:31:27.140 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:31:27.140 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:31:27.140 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T15:31:27.140 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:31:27.140 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:31:27.140 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T15:31:27.140 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:31:27.140 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:31:27.140 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:31:27.140 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T15:31:27.140 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:31:27.140 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:31:27.141 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:31:27.141 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:31:27.141 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T15:31:27.141 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:31:27.141 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:31:27.141 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:31:27.141 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:31:27.141 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:31:27.141 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:31:27.141 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:31:27.141 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:31:27.141 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:31:27.141 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:31:27.141 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:31:27.142 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T15:31:27.142 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:31:27.142 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:31:27.142 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723995081.722384_289.1", 2024-08-18T15:31:27.142 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:31:27.142 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:31:27.142 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:31:27.142 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:31:21.722384Z", 2024-08-18T15:31:27.142 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:31:27.142 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:31:27.142 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:31:27.142 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T15:31:27.142 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:31:27.142 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T15:31:27.143 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 1, 2024-08-18T15:31:27.143 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 1 2024-08-18T15:31:27.143 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:31:27.143 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T15:31:27Z" 2024-08-18T15:31:27.143 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T15:31:27.144 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-08-18T15:31:27.156 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:31:27.156 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:31:27.157 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T15:31:27.214 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:31:27.214 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:31:27.244 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:31:27.244+0000 7faa01738880 1 waiting to reach incremental sync.. 2024-08-18T15:31:32.245 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:31:32.244+0000 7faa01738880 1 waiting to reach incremental sync.. 2024-08-18T15:31:37.245 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:31:37.245+0000 7faa01738880 1 waiting to reach incremental sync.. 2024-08-18T15:31:42.247 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:31:42.246+0000 7faa01738880 1 bucket sync caught up with source: 2024-08-18T15:31:42.247 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000002.34.11, , , , , ] 2024-08-18T15:31:42.247 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000002.34.11, , , , , ] 2024-08-18T15:31:42.248 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:31:42.246+0000 7faa01738880 0 bucket checkpoint complete 2024-08-18T15:31:42.262 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:31:42.262 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:31:42.262 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T15:31:42.323 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:31:42.323 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:31:42.358 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:31:42.357+0000 7f3d3ff42880 1 waiting to reach incremental sync.. 2024-08-18T15:31:47.360 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:31:47.359+0000 7f3d3ff42880 1 bucket sync caught up with source: 2024-08-18T15:31:47.360 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , 00000000002.34.11, , , , , ] 2024-08-18T15:31:47.360 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , 00000000002.34.11, , , , , ] 2024-08-18T15:31:47.360 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:31:47.359+0000 7f3d3ff42880 0 bucket checkpoint complete 2024-08-18T15:31:47.391 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:31:47.392 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:31:47.392 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T15:31:47.451 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:31:47.451 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:31:47.486 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:31:47.485+0000 7fecd4123880 1 waiting to reach incremental sync.. 2024-08-18T15:31:52.486 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:31:52.486+0000 7fecd4123880 1 waiting to reach incremental sync.. 2024-08-18T15:31:57.487 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:31:57.486+0000 7fecd4123880 1 waiting to reach incremental sync.. 2024-08-18T15:32:02.489 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:32:02.488+0000 7fecd4123880 1 bucket sync caught up with source: 2024-08-18T15:32:02.489 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , 00000000002.60.11, , , , , ] 2024-08-18T15:32:02.489 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , 00000000002.60.11, , , , , ] 2024-08-18T15:32:02.489 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:32:02.488+0000 7fecd4123880 0 bucket checkpoint complete 2024-08-18T15:32:02.504 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:32:02.504 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:32:02.505 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T15:32:02.563 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:32:02.563 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:32:02.598 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:32:02.597+0000 7f3a4bf38880 1 waiting to reach incremental sync.. 2024-08-18T15:32:07.600 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:32:07.599+0000 7f3a4bf38880 1 bucket sync caught up with source: 2024-08-18T15:32:07.600 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , 00000000002.60.11, , , , , ] 2024-08-18T15:32:07.600 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , 00000000002.60.11, , , , , ] 2024-08-18T15:32:07.600 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:32:07.599+0000 7f3a4bf38880 0 bucket checkpoint complete 2024-08-18T15:32:07.629 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:32:07.629 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:32:07.629 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-21 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T15:32:07.687 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:32:07.687 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:32:07.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:32:07.720+0000 7f90bb122880 1 waiting to reach incremental sync.. 2024-08-18T15:32:12.721 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:32:12.720+0000 7f90bb122880 1 waiting to reach incremental sync.. 2024-08-18T15:32:17.721 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:32:17.721+0000 7f90bb122880 1 waiting to reach incremental sync.. 2024-08-18T15:32:22.724 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:32:22.722+0000 7f90bb122880 1 bucket sync caught up with source: 2024-08-18T15:32:22.724 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , 00000000002.35.11, , , , , ] 2024-08-18T15:32:22.724 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , 00000000002.35.11, , , , , ] 2024-08-18T15:32:22.724 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:32:22.722+0000 7f90bb122880 0 bucket checkpoint complete 2024-08-18T15:32:22.739 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:32:22.739 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:32:22.739 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-21 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T15:32:22.805 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:32:22.805 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:32:22.843 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:32:22.842+0000 7fad6b136880 1 bucket sync caught up with source: 2024-08-18T15:32:22.843 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000002.35.11, , , , , ] 2024-08-18T15:32:22.843 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000002.35.11, , , , , ] 2024-08-18T15:32:22.843 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:32:22.842+0000 7fad6b136880 0 bucket checkpoint complete 2024-08-18T15:32:23.619 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:32:23.619 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:32:23.619 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T15:32:23.674 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:32:23.674 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:32:23.712 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:32:23.711+0000 7f21e16bb880 1 waiting for incremental sync to catch up: 2024-08-18T15:32:23.712 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000005.46.10, , , , , ] 2024-08-18T15:32:23.712 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, 00000000010.22.6, , , , 00000000005.46.10, , , , , ] 2024-08-18T15:32:28.713 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:32:28.712+0000 7f21e16bb880 1 waiting for incremental sync to catch up: 2024-08-18T15:32:28.713 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000005.46.10, , , , , ] 2024-08-18T15:32:28.713 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, 00000000010.22.6, , , , 00000000005.46.10, , , , , ] 2024-08-18T15:32:33.715 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:32:33.714+0000 7f21e16bb880 1 waiting for incremental sync to catch up: 2024-08-18T15:32:33.715 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000005.46.10, , , , , ] 2024-08-18T15:32:33.715 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, 00000000010.22.6, , , , 00000000005.46.10, , , , , ] 2024-08-18T15:32:38.716 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:32:38.715+0000 7f21e16bb880 1 waiting for incremental sync to catch up: 2024-08-18T15:32:38.716 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000005.46.10, , , , , ] 2024-08-18T15:32:38.716 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, 00000000010.22.6, , , , 00000000005.46.10, , , , , ] 2024-08-18T15:32:43.717 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:32:43.717+0000 7f21e16bb880 1 bucket sync caught up with source: 2024-08-18T15:32:43.718 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, 00000000010.22.6, , 00000000002.23.11, , 00000000007.52.11, , , , , ] 2024-08-18T15:32:43.718 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, 00000000010.22.6, , , , 00000000005.46.10, , , , , ] 2024-08-18T15:32:43.718 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:32:43.717+0000 7f21e16bb880 0 bucket checkpoint complete 2024-08-18T15:32:43.734 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:32:43.734 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:32:43.734 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T15:32:43.798 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:32:43.798 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:32:43.834 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:32:43.833+0000 7f1ff3530880 1 waiting for incremental sync to catch up: 2024-08-18T15:32:43.834 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000010.22.6, , , , 00000000005.46.10, , , , , ] 2024-08-18T15:32:43.834 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000010.22.6, , 00000000003.25.10, , 00000000007.52.11, , , , , ] 2024-08-18T15:32:48.836 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:32:48.835+0000 7f1ff3530880 1 bucket sync caught up with source: 2024-08-18T15:32:48.836 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000010.22.6, , 00000000003.25.10, , 00000000007.52.11, , , , , ] 2024-08-18T15:32:48.836 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000010.22.6, , 00000000003.25.10, , 00000000007.52.11, , , , , ] 2024-08-18T15:32:48.836 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:32:48.835+0000 7f1ff3530880 0 bucket checkpoint complete 2024-08-18T15:32:48.851 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:32:48.851 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:32:48.852 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T15:32:48.911 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:32:48.911 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:32:48.948 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:32:48.947+0000 7efe9d010880 1 waiting for incremental sync to catch up: 2024-08-18T15:32:48.948 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , 00000000010.59.6, , 00000000004.64.10, , , , , ] 2024-08-18T15:32:48.948 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000003.12.10, , 00000000010.59.6, , 00000000007.79.10, , , , , ] 2024-08-18T15:32:53.949 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:32:53.948+0000 7efe9d010880 1 waiting for incremental sync to catch up: 2024-08-18T15:32:53.949 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , 00000000010.59.6, , 00000000004.64.10, , , , , ] 2024-08-18T15:32:53.949 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000003.12.10, , 00000000010.59.6, , 00000000007.79.10, , , , , ] 2024-08-18T15:32:58.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:32:58.949+0000 7efe9d010880 1 waiting for incremental sync to catch up: 2024-08-18T15:32:58.950 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , 00000000010.59.6, , 00000000004.64.10, , , , , ] 2024-08-18T15:32:58.950 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000003.12.10, , 00000000010.59.6, , 00000000007.79.10, , , , , ] 2024-08-18T15:33:03.952 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:33:03.951+0000 7efe9d010880 1 bucket sync caught up with source: 2024-08-18T15:33:03.952 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000003.12.10, , 00000000010.59.6, , 00000000007.79.10, , , , , ] 2024-08-18T15:33:03.952 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000003.12.10, , 00000000010.59.6, , 00000000007.79.10, , , , , ] 2024-08-18T15:33:03.952 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:33:03.951+0000 7efe9d010880 0 bucket checkpoint complete 2024-08-18T15:33:03.968 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:33:03.968 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:33:03.968 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T15:33:04.027 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:33:04.027 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:33:04.063 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:33:04.062+0000 7ff8ba662880 1 bucket sync caught up with source: 2024-08-18T15:33:04.063 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000003.12.10, , 00000000010.59.6, , 00000000007.79.10, , , , , ] 2024-08-18T15:33:04.063 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000003.12.10, , 00000000010.59.6, , 00000000007.79.10, , , , , ] 2024-08-18T15:33:04.063 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:33:04.062+0000 7ff8ba662880 0 bucket checkpoint complete 2024-08-18T15:33:04.076 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:33:04.076 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:33:04.076 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T15:33:04.130 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:33:04.131 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:33:04.164 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:33:04.164+0000 7fc350c13880 1 bucket sync caught up with source: 2024-08-18T15:33:04.165 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000003.15.10, , 00000000003.45.10, , 00000000014.113.10, , , , , ] 2024-08-18T15:33:04.165 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000003.15.10, , 00000000003.45.10, , 00000000014.113.10, , , , , ] 2024-08-18T15:33:04.165 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:33:04.164+0000 7fc350c13880 0 bucket checkpoint complete 2024-08-18T15:33:04.177 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:33:04.177 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:33:04.177 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T15:33:04.238 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:33:04.238 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:33:04.274 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:33:04.274+0000 7f8da5724880 1 bucket sync caught up with source: 2024-08-18T15:33:04.275 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, 00000000003.15.10, , 00000000003.45.10, , 00000000014.113.10, , , , , ] 2024-08-18T15:33:04.275 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, 00000000003.15.10, , 00000000003.45.10, , 00000000014.113.10, , , , , ] 2024-08-18T15:33:04.275 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:33:04.274+0000 7f8da5724880 0 bucket checkpoint complete 2024-08-18T15:33:04.288 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-19 zones={test-zone2, test-zone1} 2024-08-18T15:33:04.293 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:33:04.296 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-08-18T15:33:04.296 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-08-18T15:33:04.296 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-08-18T15:33:04.297 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-08-18T15:33:04.297 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:33:04.299 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-08-18T15:33:04.299 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-08-18T15:33:04.299 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-08-18T15:33:04.299 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-08-18T15:33:04.304 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-08-18T15:33:04.311 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-08-18T15:33:04.365 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-08-18T15:33:04.413 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-08-18T15:33:04.465 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-08-18T15:33:04.565 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-08-18T15:33:04.662 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-08-18T15:33:04.670 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-08-18T15:33:04.771 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-08-18T15:33:04.778 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-08-18T15:33:04.785 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-08-18T15:33:04.841 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-08-18T15:33:04.897 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-19 zones={test-zone2, test-zone1} 2024-08-18T15:33:04.897 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-19 zones={test-zone3, test-zone1} 2024-08-18T15:33:04.901 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:33:04.903 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-08-18T15:33:04.903 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-08-18T15:33:04.903 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-08-18T15:33:04.903 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-08-18T15:33:04.903 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:33:04.906 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-08-18T15:33:04.906 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-08-18T15:33:04.906 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-08-18T15:33:04.906 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-08-18T15:33:04.910 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-08-18T15:33:04.916 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-08-18T15:33:04.969 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-08-18T15:33:04.976 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-08-18T15:33:04.982 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-08-18T15:33:05.080 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-08-18T15:33:05.169 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-08-18T15:33:05.175 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-08-18T15:33:05.276 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-08-18T15:33:05.414 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-08-18T15:33:05.421 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-08-18T15:33:05.475 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-08-18T15:33:05.524 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-19 zones={test-zone3, test-zone1} 2024-08-18T15:33:05.524 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-19 zones={test-zone3, test-zone2} 2024-08-18T15:33:05.528 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:33:05.531 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-08-18T15:33:05.532 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-08-18T15:33:05.532 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-08-18T15:33:05.532 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-08-18T15:33:05.532 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:33:05.535 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-08-18T15:33:05.535 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-08-18T15:33:05.535 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-08-18T15:33:05.535 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-08-18T15:33:05.540 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-08-18T15:33:05.548 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-08-18T15:33:05.603 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-08-18T15:33:05.611 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-08-18T15:33:05.658 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-08-18T15:33:05.805 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-08-18T15:33:05.983 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-08-18T15:33:05.991 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-08-18T15:33:06.050 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-08-18T15:33:06.102 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-08-18T15:33:06.109 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-08-18T15:33:06.259 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-08-18T15:33:06.351 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-19 zones={test-zone3, test-zone2} 2024-08-18T15:33:06.351 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:33:06.351 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:33:06.351 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T15:33:06.413 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:33:06.413 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:33:06.450 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:33:06.450+0000 7f3e1580e880 1 bucket sync caught up with source: 2024-08-18T15:33:06.451 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, 00000000010.38.6, , 00000000003.22.10, , 00000000007.59.10, , , , , ] 2024-08-18T15:33:06.451 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, 00000000010.38.6, , 00000000003.22.10, , 00000000007.59.10, , , , , ] 2024-08-18T15:33:06.451 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:33:06.450+0000 7f3e1580e880 0 bucket checkpoint complete 2024-08-18T15:33:06.464 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:33:06.464 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:33:06.464 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T15:33:06.519 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:33:06.520 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:33:06.552 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:33:06.552+0000 7f97b0230880 1 waiting for incremental sync to catch up: 2024-08-18T15:33:06.552 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000010.38.6, , 00000000003.22.10, , 00000000006.56.10, , , , , ] 2024-08-18T15:33:06.553 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000010.38.6, , 00000000003.22.10, , 00000000007.59.10, , , , , ] 2024-08-18T15:33:11.553 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:33:11.553+0000 7f97b0230880 1 bucket sync caught up with source: 2024-08-18T15:33:11.554 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000010.38.6, , 00000000003.22.10, , 00000000007.59.10, , , , , ] 2024-08-18T15:33:11.554 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000010.38.6, , 00000000003.22.10, , 00000000007.59.10, , , , , ] 2024-08-18T15:33:11.554 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:33:11.553+0000 7f97b0230880 0 bucket checkpoint complete 2024-08-18T15:33:11.569 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:33:11.569 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:33:11.569 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T15:33:11.626 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:33:11.626 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:33:11.662 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:33:11.661+0000 7fd4a5a15880 1 bucket sync caught up with source: 2024-08-18T15:33:11.662 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000003.34.10, , 00000000010.47.6, , 00000000007.78.10, , , , , ] 2024-08-18T15:33:11.662 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000003.34.10, , 00000000010.47.6, , 00000000007.78.10, , , , , ] 2024-08-18T15:33:11.662 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:33:11.661+0000 7fd4a5a15880 0 bucket checkpoint complete 2024-08-18T15:33:11.675 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:33:11.675 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:33:11.675 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T15:33:11.729 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:33:11.729 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:33:11.763 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:33:11.762+0000 7fb354612880 1 bucket sync caught up with source: 2024-08-18T15:33:11.763 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000003.34.10, , 00000000010.47.6, , 00000000007.78.10, , , , , ] 2024-08-18T15:33:11.763 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000003.34.10, , 00000000010.47.6, , 00000000007.78.10, , , , , ] 2024-08-18T15:33:11.763 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:33:11.762+0000 7fb354612880 0 bucket checkpoint complete 2024-08-18T15:33:11.775 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:33:11.775 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:33:11.775 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T15:33:11.828 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:33:11.828 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:33:11.861 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:33:11.861+0000 7f0a12534880 1 bucket sync caught up with source: 2024-08-18T15:33:11.861 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000003.38.10, , 00000000003.34.10, , 00000000014.111.10, , , , , ] 2024-08-18T15:33:11.862 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000003.38.10, , 00000000003.34.10, , 00000000014.111.10, , , , , ] 2024-08-18T15:33:11.862 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:33:11.861+0000 7f0a12534880 0 bucket checkpoint complete 2024-08-18T15:33:11.874 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:33:11.874 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:33:11.874 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T15:33:11.936 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:33:11.936 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:33:11.975 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:33:11.975+0000 7f8013a0f880 1 bucket sync caught up with source: 2024-08-18T15:33:11.975 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, 00000000003.38.10, , 00000000003.34.10, , 00000000014.111.10, , , , , ] 2024-08-18T15:33:11.976 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, 00000000003.38.10, , 00000000003.34.10, , 00000000014.111.10, , , , , ] 2024-08-18T15:33:11.976 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:33:11.975+0000 7f8013a0f880 0 bucket checkpoint complete 2024-08-18T15:33:11.989 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-20 zones={test-zone2, test-zone1} 2024-08-18T15:33:11.994 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:33:11.998 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-08-18T15:33:11.999 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-08-18T15:33:11.999 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-08-18T15:33:11.999 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-08-18T15:33:11.999 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:33:12.002 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-08-18T15:33:12.002 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-08-18T15:33:12.002 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-08-18T15:33:12.002 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-08-18T15:33:12.007 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-08-18T15:33:12.014 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-08-18T15:33:12.025 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-08-18T15:33:12.078 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-08-18T15:33:12.084 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-08-18T15:33:12.177 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-08-18T15:33:12.273 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-08-18T15:33:12.280 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-08-18T15:33:12.380 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-08-18T15:33:12.479 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-08-18T15:33:12.487 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-08-18T15:33:12.675 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-08-18T15:33:12.768 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-20 zones={test-zone2, test-zone1} 2024-08-18T15:33:12.768 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-20 zones={test-zone3, test-zone1} 2024-08-18T15:33:12.771 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:33:12.774 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-08-18T15:33:12.774 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-08-18T15:33:12.774 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-08-18T15:33:12.774 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-08-18T15:33:12.774 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:33:12.776 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-08-18T15:33:12.777 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-08-18T15:33:12.777 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-08-18T15:33:12.777 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-08-18T15:33:12.781 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-08-18T15:33:12.787 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-08-18T15:33:12.843 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-08-18T15:33:12.895 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-08-18T15:33:12.903 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-08-18T15:33:13.005 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-08-18T15:33:13.101 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-08-18T15:33:13.107 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-08-18T15:33:13.286 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-08-18T15:33:13.379 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-08-18T15:33:13.434 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-08-18T15:33:13.486 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-08-18T15:33:13.535 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-20 zones={test-zone3, test-zone1} 2024-08-18T15:33:13.535 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-20 zones={test-zone3, test-zone2} 2024-08-18T15:33:13.539 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:33:13.542 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-08-18T15:33:13.542 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-08-18T15:33:13.542 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-08-18T15:33:13.542 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-08-18T15:33:13.543 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:33:13.545 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-08-18T15:33:13.546 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-08-18T15:33:13.546 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-08-18T15:33:13.546 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-08-18T15:33:13.552 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-08-18T15:33:13.558 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-08-18T15:33:13.615 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-08-18T15:33:13.667 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-08-18T15:33:13.676 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-08-18T15:33:13.819 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-08-18T15:33:13.911 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-08-18T15:33:13.918 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-08-18T15:33:14.021 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-08-18T15:33:14.126 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-08-18T15:33:14.133 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-08-18T15:33:14.187 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-08-18T15:33:14.239 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-20 zones={test-zone3, test-zone2} 2024-08-18T15:33:14.239 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:33:14.239 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:33:14.240 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-21 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T15:33:14.300 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:33:14.300 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:33:14.337 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:33:14.337+0000 7f393d2b9880 1 bucket sync caught up with source: 2024-08-18T15:33:14.337 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, 00000000010.47.6, , 00000000003.23.10, , 00000000008.31.10, , , , , ] 2024-08-18T15:33:14.338 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, 00000000010.47.6, , 00000000003.23.10, , 00000000008.31.10, , , , , ] 2024-08-18T15:33:14.338 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:33:14.337+0000 7f393d2b9880 0 bucket checkpoint complete 2024-08-18T15:33:14.350 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:33:14.351 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:33:14.351 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-21 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T15:33:14.410 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:33:14.410 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:33:14.444 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:33:14.444+0000 7f2ebaf32880 1 bucket sync caught up with source: 2024-08-18T15:33:14.445 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000010.47.6, , 00000000003.23.10, , 00000000008.31.10, , , , , ] 2024-08-18T15:33:14.445 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000010.47.6, , 00000000003.23.10, , 00000000008.31.10, , , , , ] 2024-08-18T15:33:14.445 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:33:14.444+0000 7f2ebaf32880 0 bucket checkpoint complete 2024-08-18T15:33:14.457 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:33:14.458 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:33:14.458 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-21 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T15:33:14.515 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:33:14.515 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:33:14.549 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:33:14.548+0000 7f25e88bb880 1 bucket sync caught up with source: 2024-08-18T15:33:14.549 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000003.48.10, , 00000000010.48.6, , 00000000007.39.10, , , , , ] 2024-08-18T15:33:14.549 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000003.48.10, , 00000000010.48.6, , 00000000007.39.10, , , , , ] 2024-08-18T15:33:14.549 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:33:14.548+0000 7f25e88bb880 0 bucket checkpoint complete 2024-08-18T15:33:14.562 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:33:14.562 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:33:14.562 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-21 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T15:33:14.615 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:33:14.616 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:33:14.649 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:33:14.648+0000 7fde8fe17880 1 bucket sync caught up with source: 2024-08-18T15:33:14.649 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000003.48.10, , 00000000010.48.6, , 00000000007.39.10, , , , , ] 2024-08-18T15:33:14.649 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000003.48.10, , 00000000010.48.6, , 00000000007.39.10, , , , , ] 2024-08-18T15:33:14.649 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:33:14.648+0000 7fde8fe17880 0 bucket checkpoint complete 2024-08-18T15:33:14.661 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:33:14.661 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:33:14.661 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-21 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T15:33:14.714 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:33:14.715 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:33:14.747 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:33:14.746+0000 7f37a7e19880 1 bucket sync caught up with source: 2024-08-18T15:33:14.747 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000003.54.10, , 00000000003.40.10, , 00000000016.63.10, , , , , ] 2024-08-18T15:33:14.747 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000003.54.10, , 00000000003.40.10, , 00000000016.63.10, , , , , ] 2024-08-18T15:33:14.747 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:33:14.746+0000 7f37a7e19880 0 bucket checkpoint complete 2024-08-18T15:33:14.759 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:33:14.760 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:33:14.760 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-21 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T15:33:14.817 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:33:14.817 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:33:14.856 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:33:14.855+0000 7f2f75122880 1 bucket sync caught up with source: 2024-08-18T15:33:14.856 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, 00000000003.54.10, , 00000000003.40.10, , 00000000016.63.10, , , , , ] 2024-08-18T15:33:14.856 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, 00000000003.54.10, , 00000000003.40.10, , 00000000016.63.10, , , , , ] 2024-08-18T15:33:14.856 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:33:14.855+0000 7f2f75122880 0 bucket checkpoint complete 2024-08-18T15:33:14.868 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-21 zones={test-zone2, test-zone1} 2024-08-18T15:33:14.872 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:33:14.875 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-08-18T15:33:14.875 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-08-18T15:33:14.875 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-08-18T15:33:14.875 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-08-18T15:33:14.876 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:33:14.878 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-08-18T15:33:14.878 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-08-18T15:33:14.878 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-08-18T15:33:14.878 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-08-18T15:33:14.883 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-08-18T15:33:14.889 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-08-18T15:33:14.941 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-08-18T15:33:14.989 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-08-18T15:33:15.044 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-08-18T15:33:15.096 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-08-18T15:33:15.145 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-08-18T15:33:15.194 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-08-18T15:33:15.250 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-08-18T15:33:15.343 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-08-18T15:33:15.350 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-08-18T15:33:15.535 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-08-18T15:33:15.627 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-21 zones={test-zone2, test-zone1} 2024-08-18T15:33:15.627 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-21 zones={test-zone3, test-zone1} 2024-08-18T15:33:15.631 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:33:15.633 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-08-18T15:33:15.633 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-08-18T15:33:15.634 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-08-18T15:33:15.634 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-08-18T15:33:15.634 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:33:15.636 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-08-18T15:33:15.636 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-08-18T15:33:15.636 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-08-18T15:33:15.636 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-08-18T15:33:15.641 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-08-18T15:33:15.647 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-08-18T15:33:15.699 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-08-18T15:33:15.706 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-08-18T15:33:15.713 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-08-18T15:33:15.764 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-08-18T15:33:15.812 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-08-18T15:33:15.905 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-08-18T15:33:16.008 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-08-18T15:33:16.099 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-08-18T15:33:16.146 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-08-18T15:33:16.198 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-08-18T15:33:16.247 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-21 zones={test-zone3, test-zone1} 2024-08-18T15:33:16.247 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-21 zones={test-zone3, test-zone2} 2024-08-18T15:33:16.251 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:33:16.254 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-08-18T15:33:16.254 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-08-18T15:33:16.254 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-08-18T15:33:16.254 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-08-18T15:33:16.254 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:33:16.257 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-08-18T15:33:16.257 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-08-18T15:33:16.257 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-08-18T15:33:16.258 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-08-18T15:33:16.263 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-08-18T15:33:16.270 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-08-18T15:33:16.332 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-08-18T15:33:16.342 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-08-18T15:33:16.435 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-08-18T15:33:16.575 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-08-18T15:33:16.711 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-08-18T15:33:16.718 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-08-18T15:33:16.822 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-08-18T15:33:16.922 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-08-18T15:33:16.930 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-08-18T15:33:17.035 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-08-18T15:33:17.139 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-21 zones={test-zone3, test-zone2} 2024-08-18T15:33:17.359 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:33:17.360 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:33:17.360 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T15:33:17.423 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:33:17.424 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:33:17.461 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:33:17.460+0000 7fdb27536880 1 waiting for incremental sync to catch up: 2024-08-18T15:33:17.461 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, 00000000010.22.6, , 00000000003.25.10, , 00000000008.58.10, , , , , ] 2024-08-18T15:33:17.461 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, 00000000011.25.6, , 00000000003.25.10, , 00000000008.58.10, , , , , ] 2024-08-18T15:33:22.462 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:33:22.461+0000 7fdb27536880 1 bucket sync caught up with source: 2024-08-18T15:33:22.463 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, 00000000011.25.6, , 00000000003.25.10, , 00000000008.58.10, , , , , ] 2024-08-18T15:33:22.463 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, 00000000011.25.6, , 00000000003.25.10, , 00000000008.58.10, , , , , ] 2024-08-18T15:33:22.463 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:33:22.461+0000 7fdb27536880 0 bucket checkpoint complete 2024-08-18T15:33:22.477 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:33:22.477 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:33:22.478 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T15:33:22.542 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:33:22.542 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:33:22.579 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:33:22.578+0000 7ff1dbc86880 1 waiting for incremental sync to catch up: 2024-08-18T15:33:22.579 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000010.22.6, , 00000000003.25.10, , 00000000008.58.10, , , , , ] 2024-08-18T15:33:22.579 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000011.25.6, , 00000000003.25.10, , 00000000008.58.10, , , , , ] 2024-08-18T15:33:27.580 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:33:27.580+0000 7ff1dbc86880 1 waiting for incremental sync to catch up: 2024-08-18T15:33:27.581 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000010.22.6, , 00000000003.25.10, , 00000000008.58.10, , , , , ] 2024-08-18T15:33:27.581 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000011.25.6, , 00000000003.25.10, , 00000000008.58.10, , , , , ] 2024-08-18T15:33:32.582 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:33:32.581+0000 7ff1dbc86880 1 waiting for incremental sync to catch up: 2024-08-18T15:33:32.582 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000010.22.6, , 00000000003.25.10, , 00000000008.58.10, , , , , ] 2024-08-18T15:33:32.582 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000011.25.6, , 00000000003.25.10, , 00000000008.58.10, , , , , ] 2024-08-18T15:33:37.583 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:33:37.582+0000 7ff1dbc86880 1 waiting for incremental sync to catch up: 2024-08-18T15:33:37.583 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000010.22.6, , 00000000003.25.10, , 00000000008.58.10, , , , , ] 2024-08-18T15:33:37.584 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000011.25.6, , 00000000003.25.10, , 00000000008.58.10, , , , , ] 2024-08-18T15:33:42.585 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:33:42.584+0000 7ff1dbc86880 1 waiting for incremental sync to catch up: 2024-08-18T15:33:42.585 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000010.22.6, , 00000000003.25.10, , 00000000008.58.10, , , , , ] 2024-08-18T15:33:42.585 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000011.25.6, , 00000000003.25.10, , 00000000008.58.10, , , , , ] 2024-08-18T15:33:47.586 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:33:47.585+0000 7ff1dbc86880 1 waiting for incremental sync to catch up: 2024-08-18T15:33:47.586 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000010.22.6, , 00000000003.25.10, , 00000000008.58.10, , , , , ] 2024-08-18T15:33:47.586 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000011.25.6, , 00000000003.25.10, , 00000000008.58.10, , , , , ] 2024-08-18T15:33:52.587 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:33:52.586+0000 7ff1dbc86880 1 waiting for incremental sync to catch up: 2024-08-18T15:33:52.588 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000010.22.6, , 00000000003.25.10, , 00000000008.58.10, , , , , ] 2024-08-18T15:33:52.588 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000011.25.6, , 00000000003.25.10, , 00000000008.58.10, , , , , ] 2024-08-18T15:33:57.595 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:33:57.594+0000 7ff1dbc86880 1 waiting for incremental sync to catch up: 2024-08-18T15:33:57.595 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000010.22.6, , 00000000003.25.10, , 00000000008.58.10, , , , , ] 2024-08-18T15:33:57.596 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000011.25.6, , 00000000003.25.10, , 00000000008.58.10, , , , , ] 2024-08-18T15:34:02.596 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:34:02.596+0000 7ff1dbc86880 1 waiting for incremental sync to catch up: 2024-08-18T15:34:02.596 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000010.22.6, , 00000000003.25.10, , 00000000008.58.10, , , , , ] 2024-08-18T15:34:02.596 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000011.25.6, , 00000000003.25.10, , 00000000008.58.10, , , , , ] 2024-08-18T15:34:07.597 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:34:07.597+0000 7ff1dbc86880 1 bucket sync caught up with source: 2024-08-18T15:34:07.598 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000011.25.6, , 00000000003.25.10, , 00000000008.58.10, , , , , ] 2024-08-18T15:34:07.598 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000011.25.6, , 00000000003.25.10, , 00000000008.58.10, , , , , ] 2024-08-18T15:34:07.598 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:34:07.597+0000 7ff1dbc86880 0 bucket checkpoint complete 2024-08-18T15:34:07.611 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:34:07.611 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:34:07.612 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T15:34:07.666 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:34:07.667 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:34:07.700 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:34:07.699+0000 7ff49c415880 1 bucket sync caught up with source: 2024-08-18T15:34:07.700 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000003.12.10, , 00000000011.62.6, , 00000000007.79.10, , , , , ] 2024-08-18T15:34:07.700 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000003.12.10, , 00000000011.62.6, , 00000000007.79.10, , , , , ] 2024-08-18T15:34:07.700 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:34:07.699+0000 7ff49c415880 0 bucket checkpoint complete 2024-08-18T15:34:07.713 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:34:07.713 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:34:07.713 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T15:34:07.767 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:34:07.767 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:34:07.800 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:34:07.799+0000 7f87ece0f880 1 bucket sync caught up with source: 2024-08-18T15:34:07.800 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000003.12.10, , 00000000011.62.6, , 00000000007.79.10, , , , , ] 2024-08-18T15:34:07.800 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000003.12.10, , 00000000011.62.6, , 00000000007.79.10, , , , , ] 2024-08-18T15:34:07.800 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:34:07.799+0000 7f87ece0f880 0 bucket checkpoint complete 2024-08-18T15:34:07.812 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:34:07.812 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:34:07.813 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T15:34:07.866 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:34:07.866 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:34:07.898 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:34:07.898+0000 7f78f6e32880 1 bucket sync caught up with source: 2024-08-18T15:34:07.898 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000003.15.10, , 00000000003.45.10, , 00000000015.116.6, , , , , ] 2024-08-18T15:34:07.899 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000003.15.10, , 00000000003.45.10, , 00000000015.116.6, , , , , ] 2024-08-18T15:34:07.899 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:34:07.898+0000 7f78f6e32880 0 bucket checkpoint complete 2024-08-18T15:34:07.911 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:34:07.911 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:34:07.911 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T15:34:07.973 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:34:07.973 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:34:08.009 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:34:08.009+0000 7f0609726880 1 bucket sync caught up with source: 2024-08-18T15:34:08.010 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, 00000000003.15.10, , 00000000003.45.10, , 00000000015.116.6, , , , , ] 2024-08-18T15:34:08.010 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, 00000000003.15.10, , 00000000003.45.10, , 00000000015.116.6, , , , , ] 2024-08-18T15:34:08.010 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:34:08.009+0000 7f0609726880 0 bucket checkpoint complete 2024-08-18T15:34:08.023 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-19 zones={test-zone2, test-zone1} 2024-08-18T15:34:08.028 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:34:08.031 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-08-18T15:34:08.031 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-08-18T15:34:08.031 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-08-18T15:34:08.031 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-08-18T15:34:08.032 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:34:08.034 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-08-18T15:34:08.034 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-08-18T15:34:08.034 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-08-18T15:34:08.034 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-08-18T15:34:08.039 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-08-18T15:34:08.045 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-08-18T15:34:08.101 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-08-18T15:34:08.108 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-08-18T15:34:08.155 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-08-18T15:34:08.213 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-08-18T15:34:08.265 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-08-18T15:34:08.272 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-08-18T15:34:08.412 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-08-18T15:34:08.511 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-08-18T15:34:08.517 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-08-18T15:34:08.667 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-08-18T15:34:08.763 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-19 zones={test-zone2, test-zone1} 2024-08-18T15:34:08.764 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-19 zones={test-zone3, test-zone1} 2024-08-18T15:34:08.768 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:34:08.771 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-08-18T15:34:08.771 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-08-18T15:34:08.771 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-08-18T15:34:08.771 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-08-18T15:34:08.771 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:34:08.774 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-08-18T15:34:08.774 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-08-18T15:34:08.774 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-08-18T15:34:08.774 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-08-18T15:34:08.779 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-08-18T15:34:08.785 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-08-18T15:34:08.841 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-08-18T15:34:08.893 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-08-18T15:34:08.900 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-08-18T15:34:08.953 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-08-18T15:34:09.050 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-08-18T15:34:09.056 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-08-18T15:34:09.193 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-08-18T15:34:09.333 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-08-18T15:34:09.340 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-08-18T15:34:09.442 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-08-18T15:34:09.535 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-19 zones={test-zone3, test-zone1} 2024-08-18T15:34:09.535 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-19 zones={test-zone3, test-zone2} 2024-08-18T15:34:09.539 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:34:09.542 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-08-18T15:34:09.542 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-08-18T15:34:09.543 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-08-18T15:34:09.543 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-08-18T15:34:09.543 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:34:09.545 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-08-18T15:34:09.545 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-08-18T15:34:09.545 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-08-18T15:34:09.545 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-08-18T15:34:09.550 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-08-18T15:34:09.556 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-08-18T15:34:09.609 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-08-18T15:34:09.666 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-08-18T15:34:09.718 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-08-18T15:34:09.816 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-08-18T15:34:09.959 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-08-18T15:34:09.966 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-08-18T15:34:10.021 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-08-18T15:34:10.119 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-08-18T15:34:10.126 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-08-18T15:34:10.227 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-08-18T15:34:10.331 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-19 zones={test-zone3, test-zone2} 2024-08-18T15:34:10.331 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:34:10.331 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:34:10.331 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T15:34:10.390 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:34:10.391 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:34:10.427 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:34:10.427+0000 7f4fe9c12880 1 bucket sync caught up with source: 2024-08-18T15:34:10.427 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, 00000000011.41.6, , 00000000003.22.10, , 00000000007.59.10, , , , , ] 2024-08-18T15:34:10.427 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, 00000000011.41.6, , 00000000003.22.10, , 00000000007.59.10, , , , , ] 2024-08-18T15:34:10.428 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:34:10.427+0000 7f4fe9c12880 0 bucket checkpoint complete 2024-08-18T15:34:10.440 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:34:10.441 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:34:10.441 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T15:34:10.502 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:34:10.502 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:34:10.537 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:34:10.536+0000 7f3696015880 1 bucket sync caught up with source: 2024-08-18T15:34:10.537 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000011.41.6, , 00000000003.22.10, , 00000000007.59.10, , , , , ] 2024-08-18T15:34:10.537 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000011.41.6, , 00000000003.22.10, , 00000000007.59.10, , , , , ] 2024-08-18T15:34:10.538 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:34:10.536+0000 7f3696015880 0 bucket checkpoint complete 2024-08-18T15:34:10.550 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:34:10.550 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:34:10.550 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T15:34:10.604 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:34:10.604 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:34:10.638 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:34:10.638+0000 7f48a00b9880 1 bucket sync caught up with source: 2024-08-18T15:34:10.638 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000003.34.10, , 00000000011.50.6, , 00000000007.78.10, , , , , ] 2024-08-18T15:34:10.638 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000003.34.10, , 00000000011.50.6, , 00000000007.78.10, , , , , ] 2024-08-18T15:34:10.638 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:34:10.638+0000 7f48a00b9880 0 bucket checkpoint complete 2024-08-18T15:34:10.651 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:34:10.651 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:34:10.651 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T15:34:10.705 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:34:10.705 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:34:10.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:34:10.736+0000 7f6966342880 1 bucket sync caught up with source: 2024-08-18T15:34:10.737 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000003.34.10, , 00000000011.50.6, , 00000000007.78.10, , , , , ] 2024-08-18T15:34:10.737 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000003.34.10, , 00000000011.50.6, , 00000000007.78.10, , , , , ] 2024-08-18T15:34:10.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:34:10.736+0000 7f6966342880 0 bucket checkpoint complete 2024-08-18T15:34:10.749 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:34:10.749 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:34:10.749 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T15:34:10.803 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:34:10.803 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:34:10.835 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:34:10.834+0000 7f4ee45af880 1 bucket sync caught up with source: 2024-08-18T15:34:10.835 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000003.38.10, , 00000000003.34.10, , 00000000015.118.6, , , , , ] 2024-08-18T15:34:10.836 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000003.38.10, , 00000000003.34.10, , 00000000015.118.6, , , , , ] 2024-08-18T15:34:10.836 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:34:10.834+0000 7f4ee45af880 0 bucket checkpoint complete 2024-08-18T15:34:10.847 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:34:10.848 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:34:10.848 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T15:34:10.905 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:34:10.906 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:34:10.942 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:34:10.941+0000 7f1dae730880 1 bucket sync caught up with source: 2024-08-18T15:34:10.942 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, 00000000003.38.10, , 00000000003.34.10, , 00000000015.118.6, , , , , ] 2024-08-18T15:34:10.942 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, 00000000003.38.10, , 00000000003.34.10, , 00000000015.118.6, , , , , ] 2024-08-18T15:34:10.942 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:34:10.941+0000 7f1dae730880 0 bucket checkpoint complete 2024-08-18T15:34:10.954 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-20 zones={test-zone2, test-zone1} 2024-08-18T15:34:10.958 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:34:10.961 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-08-18T15:34:10.961 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-08-18T15:34:10.961 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-08-18T15:34:10.961 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-08-18T15:34:10.961 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:34:10.964 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-08-18T15:34:10.964 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-08-18T15:34:10.964 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-08-18T15:34:10.964 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-08-18T15:34:10.969 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-08-18T15:34:10.975 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-08-18T15:34:10.987 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-08-18T15:34:11.039 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-08-18T15:34:11.045 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-08-18T15:34:11.101 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-08-18T15:34:11.158 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-08-18T15:34:11.165 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-08-18T15:34:11.311 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-08-18T15:34:11.407 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-08-18T15:34:11.414 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-08-18T15:34:11.555 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-08-18T15:34:11.652 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-20 zones={test-zone2, test-zone1} 2024-08-18T15:34:11.652 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-20 zones={test-zone3, test-zone1} 2024-08-18T15:34:11.656 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:34:11.659 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-08-18T15:34:11.659 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-08-18T15:34:11.659 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-08-18T15:34:11.659 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-08-18T15:34:11.659 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:34:11.662 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-08-18T15:34:11.662 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-08-18T15:34:11.662 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-08-18T15:34:11.662 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-08-18T15:34:11.667 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-08-18T15:34:11.673 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-08-18T15:34:11.729 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-08-18T15:34:11.781 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-08-18T15:34:11.790 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-08-18T15:34:11.893 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-08-18T15:34:11.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-08-18T15:34:12.002 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-08-18T15:34:12.108 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-08-18T15:34:12.246 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-08-18T15:34:12.254 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-08-18T15:34:12.351 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-08-18T15:34:12.451 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-20 zones={test-zone3, test-zone1} 2024-08-18T15:34:12.451 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-20 zones={test-zone3, test-zone2} 2024-08-18T15:34:12.456 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:34:12.459 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-08-18T15:34:12.459 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-08-18T15:34:12.460 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-08-18T15:34:12.460 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-08-18T15:34:12.460 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:34:12.462 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-08-18T15:34:12.462 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-08-18T15:34:12.462 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-08-18T15:34:12.463 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-08-18T15:34:12.468 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-08-18T15:34:12.475 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-08-18T15:34:12.531 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-08-18T15:34:12.583 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-08-18T15:34:12.591 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-08-18T15:34:12.780 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-08-18T15:34:12.915 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-08-18T15:34:12.923 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-08-18T15:34:13.022 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-08-18T15:34:13.118 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-08-18T15:34:13.125 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-08-18T15:34:13.223 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-08-18T15:34:13.319 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-20 zones={test-zone3, test-zone2} 2024-08-18T15:34:13.319 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:34:13.319 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:34:13.319 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-21 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T15:34:13.381 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:34:13.382 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:34:13.422 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:34:13.421+0000 7f6c48c11880 1 bucket sync caught up with source: 2024-08-18T15:34:13.422 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, 00000000011.50.6, , 00000000003.23.10, , 00000000008.31.10, , , , , ] 2024-08-18T15:34:13.422 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, 00000000011.50.6, , 00000000003.23.10, , 00000000008.31.10, , , , , ] 2024-08-18T15:34:13.422 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:34:13.421+0000 7f6c48c11880 0 bucket checkpoint complete 2024-08-18T15:34:13.435 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:34:13.435 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:34:13.435 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-21 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T15:34:13.497 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:34:13.498 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:34:13.533 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:34:13.532+0000 7f5c60946880 1 waiting for incremental sync to catch up: 2024-08-18T15:34:13.533 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000010.47.6, , 00000000003.23.10, , 00000000008.31.10, , , , , ] 2024-08-18T15:34:13.533 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000011.50.6, , 00000000003.23.10, , 00000000008.31.10, , , , , ] 2024-08-18T15:34:18.534 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:34:18.533+0000 7f5c60946880 1 waiting for incremental sync to catch up: 2024-08-18T15:34:18.535 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000010.47.6, , 00000000003.23.10, , 00000000008.31.10, , , , , ] 2024-08-18T15:34:18.535 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000011.50.6, , 00000000003.23.10, , 00000000008.31.10, , , , , ] 2024-08-18T15:34:23.535 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:34:23.534+0000 7f5c60946880 1 waiting for incremental sync to catch up: 2024-08-18T15:34:23.536 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000010.47.6, , 00000000003.23.10, , 00000000008.31.10, , , , , ] 2024-08-18T15:34:23.536 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000011.50.6, , 00000000003.23.10, , 00000000008.31.10, , , , , ] 2024-08-18T15:34:28.535 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:34:28.535+0000 7f5c60946880 1 bucket sync caught up with source: 2024-08-18T15:34:28.536 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000011.50.6, , 00000000003.23.10, , 00000000008.31.10, , , , , ] 2024-08-18T15:34:28.536 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000011.50.6, , 00000000003.23.10, , 00000000008.31.10, , , , , ] 2024-08-18T15:34:28.536 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:34:28.535+0000 7f5c60946880 0 bucket checkpoint complete 2024-08-18T15:34:28.551 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:34:28.551 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:34:28.551 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-21 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T15:34:28.609 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:34:28.609 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:34:28.645 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:34:28.644+0000 7f25ffa3f880 1 bucket sync caught up with source: 2024-08-18T15:34:28.645 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000003.48.10, , 00000000011.51.6, , 00000000007.39.10, , , , , ] 2024-08-18T15:34:28.645 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000003.48.10, , 00000000011.51.6, , 00000000007.39.10, , , , , ] 2024-08-18T15:34:28.645 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:34:28.644+0000 7f25ffa3f880 0 bucket checkpoint complete 2024-08-18T15:34:28.658 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:34:28.658 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:34:28.658 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-21 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T15:34:28.712 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:34:28.713 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:34:28.745 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:34:28.744+0000 7fb137d26880 1 bucket sync caught up with source: 2024-08-18T15:34:28.745 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000003.48.10, , 00000000011.51.6, , 00000000007.39.10, , , , , ] 2024-08-18T15:34:28.746 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000003.48.10, , 00000000011.51.6, , 00000000007.39.10, , , , , ] 2024-08-18T15:34:28.746 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:34:28.744+0000 7fb137d26880 0 bucket checkpoint complete 2024-08-18T15:34:28.757 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:34:28.758 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:34:28.758 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-21 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T15:34:28.811 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:34:28.812 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:34:28.845 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:34:28.845+0000 7f2ef0928880 1 bucket sync caught up with source: 2024-08-18T15:34:28.845 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000003.54.10, , 00000000003.40.10, , 00000000017.66.6, , , , , ] 2024-08-18T15:34:28.845 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000003.54.10, , 00000000003.40.10, , 00000000017.66.6, , , , , ] 2024-08-18T15:34:28.845 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:34:28.845+0000 7f2ef0928880 0 bucket checkpoint complete 2024-08-18T15:34:28.859 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:34:28.859 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:34:28.859 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-21 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T15:34:28.920 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:34:28.921 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:34:28.958 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:34:28.958+0000 7f6b5c086880 1 bucket sync caught up with source: 2024-08-18T15:34:28.959 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, 00000000003.54.10, , 00000000003.40.10, , 00000000017.66.6, , , , , ] 2024-08-18T15:34:28.959 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, 00000000003.54.10, , 00000000003.40.10, , 00000000017.66.6, , , , , ] 2024-08-18T15:34:28.959 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:34:28.958+0000 7f6b5c086880 0 bucket checkpoint complete 2024-08-18T15:34:28.972 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-21 zones={test-zone2, test-zone1} 2024-08-18T15:34:28.977 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:34:28.979 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-08-18T15:34:28.980 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-08-18T15:34:28.980 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-08-18T15:34:28.980 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-08-18T15:34:28.980 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:34:28.983 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-08-18T15:34:28.983 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-08-18T15:34:28.983 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-08-18T15:34:28.983 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-08-18T15:34:28.987 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-08-18T15:34:28.993 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-08-18T15:34:29.005 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-08-18T15:34:29.053 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-08-18T15:34:29.104 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-08-18T15:34:29.201 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-08-18T15:34:29.297 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-08-18T15:34:29.346 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-08-18T15:34:29.399 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-08-18T15:34:29.447 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-08-18T15:34:29.454 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-08-18T15:34:29.551 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-08-18T15:34:29.731 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-21 zones={test-zone2, test-zone1} 2024-08-18T15:34:29.731 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-21 zones={test-zone3, test-zone1} 2024-08-18T15:34:29.735 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:34:29.737 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-08-18T15:34:29.737 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-08-18T15:34:29.738 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-08-18T15:34:29.738 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-08-18T15:34:29.738 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:34:29.740 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-08-18T15:34:29.740 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-08-18T15:34:29.740 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-08-18T15:34:29.740 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-08-18T15:34:29.745 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-08-18T15:34:29.751 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-08-18T15:34:29.805 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-08-18T15:34:29.812 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-08-18T15:34:29.860 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-08-18T15:34:29.912 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-08-18T15:34:29.960 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-08-18T15:34:30.010 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-08-18T15:34:30.154 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-08-18T15:34:30.339 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-08-18T15:34:30.345 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-08-18T15:34:30.450 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-08-18T15:34:30.547 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-21 zones={test-zone3, test-zone1} 2024-08-18T15:34:30.547 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-21 zones={test-zone3, test-zone2} 2024-08-18T15:34:30.552 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:34:30.554 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-08-18T15:34:30.555 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-08-18T15:34:30.555 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-08-18T15:34:30.555 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-08-18T15:34:30.555 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:34:30.558 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-08-18T15:34:30.558 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-08-18T15:34:30.558 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-08-18T15:34:30.558 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-08-18T15:34:30.563 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-08-18T15:34:30.570 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-08-18T15:34:30.627 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-08-18T15:34:30.679 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-08-18T15:34:30.734 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-08-18T15:34:30.875 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-08-18T15:34:31.051 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-08-18T15:34:31.059 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-08-18T15:34:31.114 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-08-18T15:34:31.166 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-08-18T15:34:31.174 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-08-18T15:34:31.230 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-08-18T15:34:31.328 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-21 zones={test-zone3, test-zone2} 2024-08-18T15:34:31.329 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_versioned_object_incremental_sync ... ok 2024-08-18T15:34:31.350 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:34:31.351 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:34:31.351 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:34:31.403 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:34:31.403 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:34:31.435 INFO:teuthology.orchestra.run.smithi134.stderr:No --period given, using current period=b0d74758-1871-4454-bc0f-e3e4b6c30105 2024-08-18T15:34:31.447 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T15:34:31.447 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:34:31.447 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:34:31.447 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:19:24.506194Z" 2024-08-18T15:34:31.447 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:34:31.447 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:34:31.447 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T15:34:31.447 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T15:34:31.447 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:34:31.447 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:34:31.447 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:34:31.448 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:21:34.952005Z" 2024-08-18T15:34:31.448 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:34:31.448 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:34:31.448 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723995271.350113_383.1", 2024-08-18T15:34:31.448 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:34:31.350113Z" 2024-08-18T15:34:31.448 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:34:31.448 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T15:34:31.449 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-08-18T15:34:31.449 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:34:31.449 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:34:31.450 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:34:31.507 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:34:31.507 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:34:31.542 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:34:31.541+0000 7f2351013880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:34:31.543 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:34:31.542+0000 7f2351013880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:34:31.556 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:34:31.556 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:34:31.556 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:34:31.556 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:34:31.556 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:34:31.556 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "b0d74758-1871-4454-bc0f-e3e4b6c30105", 2024-08-18T15:34:31.556 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:34:31.557 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:34:31.557 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:34:31.557 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:34:31.557 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:34:31.557 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:34:31.557 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:34:31.557 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:34:31.557 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:34:31.557 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:34:31.557 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:34:31.557 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:34:31.557 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:34:31.557 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:34:31.558 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:34:31.558 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:34:31.558 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:34:31.558 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:34:31.558 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:34:31.558 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:34:31.558 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:34:31.558 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:34:31.558 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:34:31.558 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:34:31.558 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:34:31.558 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:34:31.559 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:34:31.559 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:34:31.559 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:34:31.559 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:34:31.559 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:34:31.559 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:34:31.559 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:34:31.559 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:34:31.559 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:34:31.559 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:34:31.559 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:34:31.559 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:34:31.560 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:34:31.560 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:34:31.560 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:34:31.560 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:34:31.560 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:34:31.560 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723995271.336261_378.1", 2024-08-18T15:34:31.560 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:34:31.560 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:34:31.560 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:34:31.560 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:34:31.336261Z", 2024-08-18T15:34:31.560 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:34:31.560 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:34:31.560 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:34:31.561 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:34:31.561 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:34:31.561 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:34:31.561 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:34:31.561 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:34:31.561 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:34:31.561 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:34:31Z" 2024-08-18T15:34:31.561 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:34:31.562 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1723995271.350113_383.1 target=1_1723995271.336261_378.1 2024-08-18T15:34:36.563 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:34:36.563 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:34:36.564 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:34:36.621 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:34:36.621 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:34:36.654 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:34:36.654+0000 7fcb48c26880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:34:36.655 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:34:36.655+0000 7fcb48c26880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:34:36.668 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:34:36.668 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:34:36.668 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:34:36.668 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:34:36.668 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:34:36.668 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "b0d74758-1871-4454-bc0f-e3e4b6c30105", 2024-08-18T15:34:36.669 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:34:36.669 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:34:36.669 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:34:36.669 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:34:36.669 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:34:36.669 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:34:36.669 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:34:36.669 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:34:36.669 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:34:36.669 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:34:36.670 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:34:36.670 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:34:36.670 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:34:36.670 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:34:36.670 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:34:36.670 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:34:36.670 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:34:36.670 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:34:36.670 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:34:36.670 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:34:36.670 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:34:36.671 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:34:36.671 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:34:36.671 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:34:36.671 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:34:36.671 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:34:36.671 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:34:36.671 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:34:36.671 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:34:36.671 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:34:36.672 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:34:36.672 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:34:36.672 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:34:36.672 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:34:36.672 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:34:36.672 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:34:36.672 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:34:36.672 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:34:36.672 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:34:36.672 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:34:36.672 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:34:36.673 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:34:36.673 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:34:36.673 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723995271.350113_383.1", 2024-08-18T15:34:36.673 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:34:36.673 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:34:36.673 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:34:36.673 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:34:31.350113Z", 2024-08-18T15:34:36.673 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:34:36.673 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:34:36.673 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:34:36.673 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:34:36.674 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:34:36.674 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:34:36.674 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:34:36.674 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:34:36.674 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:34:36.674 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:34:36Z" 2024-08-18T15:34:36.674 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:34:36.675 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-08-18T15:34:36.675 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-08-18T15:34:36.676 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:34:36.676 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:34:36.676 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:34:36.731 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:34:36.731 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:34:36.762 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:34:36.761+0000 7f72a8d24880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:34:36.763 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:34:36.762+0000 7f72a8d24880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:34:36.775 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T15:34:36.775 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T15:34:36.775 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T15:34:36.775 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T15:34:36.775 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T15:34:36.776 INFO:teuthology.orchestra.run.smithi134.stdout: "period": "b0d74758-1871-4454-bc0f-e3e4b6c30105", 2024-08-18T15:34:36.776 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:34:36.776 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:34:36.776 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T15:34:36.776 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:34:36.776 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T15:34:36.776 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:34:36.776 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:34:36.776 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:34:36.776 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:34:36.776 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 1, 2024-08-18T15:34:36.776 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:34:36.776 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:34:36.777 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:34:36.777 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:34:36.777 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:34:36.777 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:34:36.777 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T15:34:36.777 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:34:36.777 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:34:36.777 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T15:34:36.777 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:34:36.777 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:34:36.777 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:34:36.777 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T15:34:36.777 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:34:36.777 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:34:36.777 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:34:36.777 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:34:36.778 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T15:34:36.778 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:34:36.778 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:34:36.778 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:34:36.778 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:34:36.778 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:34:36.778 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:34:36.778 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:34:36.778 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:34:36.778 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:34:36.778 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:34:36.778 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:34:36.779 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T15:34:36.779 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:34:36.779 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:34:36.779 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723995271.350113_383.1", 2024-08-18T15:34:36.779 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:34:36.779 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:34:36.779 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:34:36.779 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:34:31.350113Z", 2024-08-18T15:34:36.779 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:34:36.779 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:34:36.779 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:34:36.779 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T15:34:36.779 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:34:36.779 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T15:34:36.779 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 1, 2024-08-18T15:34:36.779 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 1 2024-08-18T15:34:36.780 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:34:36.780 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T15:34:36Z" 2024-08-18T15:34:36.780 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T15:34:36.781 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-08-18T15:34:36.793 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:34:36.794 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:34:36.794 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-22 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T15:34:36.849 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:34:36.849 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:34:36.883 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:34:36.882+0000 7fd012886880 1 waiting to reach incremental sync.. 2024-08-18T15:34:41.884 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:34:41.883+0000 7fd012886880 1 waiting to reach incremental sync.. 2024-08-18T15:34:46.885 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:34:46.884+0000 7fd012886880 1 bucket sync caught up with source: 2024-08-18T15:34:46.885 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000002.48.11, , , , , ] 2024-08-18T15:34:46.885 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000002.48.11, , , , , ] 2024-08-18T15:34:46.885 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:34:46.884+0000 7fd012886880 0 bucket checkpoint complete 2024-08-18T15:34:46.902 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:34:46.902 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:34:46.902 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-22 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T15:34:46.963 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:34:46.963 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:34:46.999 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:34:46.999+0000 7fc954e15880 1 waiting to reach incremental sync.. 2024-08-18T15:34:52.001 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:34:52.000+0000 7fc954e15880 1 bucket sync caught up with source: 2024-08-18T15:34:52.001 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , 00000000002.48.11, , , , , ] 2024-08-18T15:34:52.002 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , 00000000002.48.11, , , , , ] 2024-08-18T15:34:52.002 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:34:52.000+0000 7fc954e15880 0 bucket checkpoint complete 2024-08-18T15:34:52.016 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:34:52.016 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:34:52.017 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-22 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T15:34:52.074 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:34:52.075 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:34:52.109 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:34:52.109+0000 7f8cc781f880 1 waiting to reach incremental sync.. 2024-08-18T15:34:57.110 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:34:57.109+0000 7f8cc781f880 1 waiting to reach incremental sync.. 2024-08-18T15:35:02.114 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:35:02.113+0000 7f8cc781f880 1 bucket sync caught up with source: 2024-08-18T15:35:02.114 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , 00000000002.60.11, , , , , ] 2024-08-18T15:35:02.115 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , 00000000002.60.11, , , , , ] 2024-08-18T15:35:02.115 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:35:02.113+0000 7f8cc781f880 0 bucket checkpoint complete 2024-08-18T15:35:02.154 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:35:02.154 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:35:02.155 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-22 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T15:35:02.213 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:35:02.214 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:35:02.248 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:35:02.247+0000 7fb11e8bb880 1 waiting for incremental sync to catch up: 2024-08-18T15:35:02.248 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , 00000000002.60.11, , , , , ] 2024-08-18T15:35:02.248 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , 00000000003.62.10, , , , , ] 2024-08-18T15:35:07.249 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:35:07.248+0000 7fb11e8bb880 1 bucket sync caught up with source: 2024-08-18T15:35:07.249 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , 00000000003.62.10, , , , , ] 2024-08-18T15:35:07.249 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , 00000000003.62.10, , , , , ] 2024-08-18T15:35:07.249 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:35:07.248+0000 7fb11e8bb880 0 bucket checkpoint complete 2024-08-18T15:35:07.262 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:35:07.263 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:35:07.263 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-22 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T15:35:07.318 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:35:07.318 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:35:07.352 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:35:07.351+0000 7fb420690880 1 bucket sync caught up with source: 2024-08-18T15:35:07.352 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , 00000000003.63.10, , , , , ] 2024-08-18T15:35:07.352 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , 00000000003.63.10, , , , , ] 2024-08-18T15:35:07.352 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:35:07.351+0000 7fb420690880 0 bucket checkpoint complete 2024-08-18T15:35:07.364 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:35:07.364 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:35:07.365 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-22 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T15:35:07.427 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:35:07.427 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:35:07.466 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:35:07.464+0000 7f7db4323880 1 bucket sync caught up with source: 2024-08-18T15:35:07.466 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000003.63.10, , , , , ] 2024-08-18T15:35:07.466 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000003.63.10, , , , , ] 2024-08-18T15:35:07.466 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:35:07.464+0000 7f7db4323880 0 bucket checkpoint complete 2024-08-18T15:35:07.479 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-22 zones={test-zone2, test-zone1} 2024-08-18T15:35:07.487 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:35:07.490 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-08-18T15:35:07.490 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:35:07.539 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-08-18T15:35:07.543 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-08-18T15:35:07.550 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-08-18T15:35:07.606 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-08-18T15:35:07.614 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-22 zones={test-zone2, test-zone1} 2024-08-18T15:35:07.615 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-22 zones={test-zone3, test-zone1} 2024-08-18T15:35:07.620 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:35:07.623 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-08-18T15:35:07.623 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:35:07.671 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-08-18T15:35:07.676 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-08-18T15:35:07.682 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-08-18T15:35:07.734 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-08-18T15:35:07.742 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-22 zones={test-zone3, test-zone1} 2024-08-18T15:35:07.742 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-22 zones={test-zone3, test-zone2} 2024-08-18T15:35:07.746 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:35:07.749 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-08-18T15:35:07.749 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:35:07.752 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-08-18T15:35:07.799 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-08-18T15:35:07.807 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-08-18T15:35:07.863 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-08-18T15:35:07.915 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-22 zones={test-zone3, test-zone2} 2024-08-18T15:35:08.484 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:35:08.484 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:35:08.484 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-22 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T15:35:08.538 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:35:08.538 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:35:08.576 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:35:08.575+0000 7f46dfb42880 1 waiting for incremental sync to catch up: 2024-08-18T15:35:08.577 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000002.48.11, , , , , ] 2024-08-18T15:35:08.577 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000004.52.10, , , 00000000020.47.12, , ] 2024-08-18T15:35:13.578 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:35:13.577+0000 7f46dfb42880 1 waiting for incremental sync to catch up: 2024-08-18T15:35:13.578 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000002.48.11, , , , , ] 2024-08-18T15:35:13.578 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000004.52.10, , , 00000000020.47.12, , ] 2024-08-18T15:35:18.579 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:35:18.578+0000 7f46dfb42880 1 waiting for incremental sync to catch up: 2024-08-18T15:35:18.579 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000002.48.11, , , , , ] 2024-08-18T15:35:18.579 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000004.52.10, , , 00000000020.47.12, , ] 2024-08-18T15:35:23.580 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:35:23.580+0000 7f46dfb42880 1 waiting for incremental sync to catch up: 2024-08-18T15:35:23.581 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000002.48.11, , , 00000000023.56.6, , ] 2024-08-18T15:35:23.581 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000004.52.10, , , 00000000020.47.12, , ] 2024-08-18T15:35:28.582 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:35:28.581+0000 7f46dfb42880 1 waiting for incremental sync to catch up: 2024-08-18T15:35:28.582 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000002.48.11, , , 00000000023.56.6, , ] 2024-08-18T15:35:28.582 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000004.52.10, , , 00000000020.47.12, , ] 2024-08-18T15:35:33.583 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:35:33.582+0000 7f46dfb42880 1 waiting for incremental sync to catch up: 2024-08-18T15:35:33.583 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000002.48.11, , , 00000000023.56.6, , ] 2024-08-18T15:35:33.583 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000004.52.10, , , 00000000020.47.12, , ] 2024-08-18T15:35:38.584 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:35:38.583+0000 7f46dfb42880 1 waiting for incremental sync to catch up: 2024-08-18T15:35:38.584 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000002.48.11, , , 00000000023.56.6, , ] 2024-08-18T15:35:38.584 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000004.52.10, , , 00000000020.47.12, , ] 2024-08-18T15:35:43.585 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:35:43.584+0000 7f46dfb42880 1 bucket sync caught up with source: 2024-08-18T15:35:43.585 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000005.54.10, , , 00000000028.71.6, , ] 2024-08-18T15:35:43.585 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000004.52.10, , , 00000000020.47.12, , ] 2024-08-18T15:35:43.586 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:35:43.584+0000 7f46dfb42880 0 bucket checkpoint complete 2024-08-18T15:35:43.603 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:35:43.604 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:35:43.604 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-22 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T15:35:43.663 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:35:43.663 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:35:43.698 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:35:43.698+0000 7f5be9412880 1 waiting for incremental sync to catch up: 2024-08-18T15:35:43.699 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , 00000000005.54.10, , , 00000000005.18.6, , ] 2024-08-18T15:35:43.699 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , 00000000005.54.10, , , 00000000042.109.9, , ] 2024-08-18T15:35:48.700 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:35:48.699+0000 7f5be9412880 1 waiting for incremental sync to catch up: 2024-08-18T15:35:48.700 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , 00000000005.54.10, , , 00000000005.18.6, , ] 2024-08-18T15:35:48.700 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , 00000000005.54.10, , , 00000000042.109.9, , ] 2024-08-18T15:35:53.701 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:35:53.701+0000 7f5be9412880 1 waiting for incremental sync to catch up: 2024-08-18T15:35:53.702 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , 00000000005.54.10, , , 00000000005.18.6, , ] 2024-08-18T15:35:53.702 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , 00000000005.54.10, , , 00000000042.109.9, , ] 2024-08-18T15:35:58.702 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:35:58.702+0000 7f5be9412880 1 waiting for incremental sync to catch up: 2024-08-18T15:35:58.703 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , 00000000005.54.10, , , 00000000005.18.6, , ] 2024-08-18T15:35:58.703 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , 00000000005.54.10, , , 00000000042.109.9, , ] 2024-08-18T15:36:03.704 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:36:03.703+0000 7f5be9412880 1 waiting for incremental sync to catch up: 2024-08-18T15:36:03.704 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , 00000000005.54.10, , , 00000000005.18.6, , ] 2024-08-18T15:36:03.704 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , 00000000005.54.10, , , 00000000042.109.9, , ] 2024-08-18T15:36:08.706 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:36:08.705+0000 7f5be9412880 1 waiting for incremental sync to catch up: 2024-08-18T15:36:08.706 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , 00000000005.54.10, , , 00000000017.42.6, , ] 2024-08-18T15:36:08.706 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , 00000000005.54.10, , , 00000000042.109.9, , ] 2024-08-18T15:36:13.707 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:36:13.706+0000 7f5be9412880 1 waiting for incremental sync to catch up: 2024-08-18T15:36:13.707 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , 00000000005.54.10, , , 00000000017.42.6, , ] 2024-08-18T15:36:13.707 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , 00000000005.54.10, , , 00000000042.109.9, , ] 2024-08-18T15:36:18.709 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:36:18.708+0000 7f5be9412880 1 waiting for incremental sync to catch up: 2024-08-18T15:36:18.709 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , 00000000005.54.10, , , 00000000017.42.6, , ] 2024-08-18T15:36:18.709 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , 00000000005.54.10, , , 00000000042.109.9, , ] 2024-08-18T15:36:23.713 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:36:23.710+0000 7f5be9412880 1 waiting for incremental sync to catch up: 2024-08-18T15:36:23.714 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , 00000000005.54.10, , , 00000000017.42.6, , ] 2024-08-18T15:36:23.714 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , 00000000005.54.10, , , 00000000042.109.9, , ] 2024-08-18T15:36:28.711 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:36:28.711+0000 7f5be9412880 1 bucket sync caught up with source: 2024-08-18T15:36:28.712 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , 00000000005.54.10, , , 00000000044.113.10, , ] 2024-08-18T15:36:28.712 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , 00000000005.54.10, , , 00000000042.109.9, , ] 2024-08-18T15:36:28.712 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:36:28.711+0000 7f5be9412880 0 bucket checkpoint complete 2024-08-18T15:36:28.727 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:36:28.728 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:36:28.728 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-22 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T15:36:28.787 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:36:28.787 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:36:28.824 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:36:28.823+0000 7fa85762f880 1 waiting for incremental sync to catch up: 2024-08-18T15:36:28.824 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , 00000000004.64.10, , , 00000000031.86.6, , ] 2024-08-18T15:36:28.824 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , 00000000004.64.10, , , 00000000043.117.10, , ] 2024-08-18T15:36:33.825 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:36:33.824+0000 7fa85762f880 1 waiting for incremental sync to catch up: 2024-08-18T15:36:33.825 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , 00000000004.64.10, , , 00000000031.86.6, , ] 2024-08-18T15:36:33.825 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , 00000000004.64.10, , , 00000000043.117.10, , ] 2024-08-18T15:36:38.826 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:36:38.825+0000 7fa85762f880 1 waiting for incremental sync to catch up: 2024-08-18T15:36:38.826 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , 00000000004.64.10, , , 00000000031.86.6, , ] 2024-08-18T15:36:38.826 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , 00000000004.64.10, , , 00000000043.117.10, , ] 2024-08-18T15:36:43.827 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:36:43.827+0000 7fa85762f880 1 bucket sync caught up with source: 2024-08-18T15:36:43.828 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , 00000000004.64.10, , , 00000000043.117.10, , ] 2024-08-18T15:36:43.828 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , 00000000004.64.10, , , 00000000043.117.10, , ] 2024-08-18T15:36:43.828 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:36:43.827+0000 7fa85762f880 0 bucket checkpoint complete 2024-08-18T15:36:43.844 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:36:43.844 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:36:43.844 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-22 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T15:36:43.905 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:36:43.905 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:36:43.941 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:36:43.940+0000 7f6bf5d42880 1 bucket sync caught up with source: 2024-08-18T15:36:43.941 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , 00000000004.64.10, , , 00000000043.117.10, , ] 2024-08-18T15:36:43.941 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , 00000000004.64.10, , , 00000000043.117.10, , ] 2024-08-18T15:36:43.941 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:36:43.940+0000 7f6bf5d42880 0 bucket checkpoint complete 2024-08-18T15:36:43.954 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:36:43.954 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:36:43.954 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-22 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T15:36:44.010 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:36:44.010 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:36:44.043 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:36:44.042+0000 7fd75e634880 1 bucket sync caught up with source: 2024-08-18T15:36:44.043 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , 00000000004.65.10, , , 00000000044.122.10, , ] 2024-08-18T15:36:44.043 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , 00000000004.65.10, , , 00000000044.122.10, , ] 2024-08-18T15:36:44.044 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:36:44.042+0000 7fd75e634880 0 bucket checkpoint complete 2024-08-18T15:36:44.055 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:36:44.055 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:36:44.056 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-22 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T15:36:44.118 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:36:44.118 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:36:44.158 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:36:44.158+0000 7fb068742880 1 waiting for incremental sync to catch up: 2024-08-18T15:36:44.159 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000004.65.10, , , 00000000042.111.12, , ] 2024-08-18T15:36:44.159 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000004.65.10, , , 00000000044.122.10, , ] 2024-08-18T15:36:49.160 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:36:49.159+0000 7fb068742880 1 waiting for incremental sync to catch up: 2024-08-18T15:36:49.160 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000004.65.10, , , 00000000042.111.12, , ] 2024-08-18T15:36:49.160 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000004.65.10, , , 00000000044.122.10, , ] 2024-08-18T15:36:54.161 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:36:54.160+0000 7fb068742880 1 waiting for incremental sync to catch up: 2024-08-18T15:36:54.161 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000004.65.10, , , 00000000042.111.12, , ] 2024-08-18T15:36:54.161 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000004.65.10, , , 00000000044.122.10, , ] 2024-08-18T15:36:59.162 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:36:59.162+0000 7fb068742880 1 waiting for incremental sync to catch up: 2024-08-18T15:36:59.163 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000004.65.10, , , 00000000042.111.12, , ] 2024-08-18T15:36:59.163 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000004.65.10, , , 00000000044.122.10, , ] 2024-08-18T15:37:04.164 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:37:04.163+0000 7fb068742880 1 waiting for incremental sync to catch up: 2024-08-18T15:37:04.164 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000004.65.10, , , 00000000042.111.12, , ] 2024-08-18T15:37:04.164 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000004.65.10, , , 00000000044.122.10, , ] 2024-08-18T15:37:09.165 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:37:09.163+0000 7fb068742880 1 waiting for incremental sync to catch up: 2024-08-18T15:37:09.165 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000004.65.10, , , 00000000042.111.12, , ] 2024-08-18T15:37:09.165 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000004.65.10, , , 00000000044.122.10, , ] 2024-08-18T15:37:14.166 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:37:14.165+0000 7fb068742880 1 waiting for incremental sync to catch up: 2024-08-18T15:37:14.167 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000004.65.10, , , 00000000042.111.12, , ] 2024-08-18T15:37:14.167 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000004.65.10, , , 00000000044.122.10, , ] 2024-08-18T15:37:19.168 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:37:19.167+0000 7fb068742880 1 waiting for incremental sync to catch up: 2024-08-18T15:37:19.168 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000004.65.10, , , 00000000042.111.12, , ] 2024-08-18T15:37:19.168 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000004.65.10, , , 00000000044.122.10, , ] 2024-08-18T15:37:24.169 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:37:24.168+0000 7fb068742880 1 bucket sync caught up with source: 2024-08-18T15:37:24.169 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000004.65.10, , , 00000000044.122.10, , ] 2024-08-18T15:37:24.169 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000004.65.10, , , 00000000044.122.10, , ] 2024-08-18T15:37:24.169 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:37:24.168+0000 7fb068742880 0 bucket checkpoint complete 2024-08-18T15:37:24.185 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-22 zones={test-zone2, test-zone1} 2024-08-18T15:37:24.196 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:37:24.201 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-08-18T15:37:24.201 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:24.202 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:24.202 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:24.202 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:24.202 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:24.202 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:24.202 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:24.202 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:24.202 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:24.202 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:24.202 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:24.202 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:24.202 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:24.202 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:24.202 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:24.203 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:24.203 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:24.203 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:24.203 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:24.203 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:24.203 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:24.203 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:24.203 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:24.203 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:24.203 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:24.203 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:24.203 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:24.203 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:24.203 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:24.204 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:24.204 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:37:24.208 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-08-18T15:37:24.208 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:24.208 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:24.208 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:24.208 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:24.208 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:24.208 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:24.208 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:24.208 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:24.208 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:24.208 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:24.209 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:24.209 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:24.209 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:24.209 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:24.209 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:24.209 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:24.209 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:24.209 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:24.209 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:24.209 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:24.209 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:24.209 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:24.209 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:24.209 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:24.209 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:24.209 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:24.210 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:24.210 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:24.210 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:24.210 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:24.218 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-08-18T15:37:24.226 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-08-18T15:37:24.284 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-08-18T15:37:24.335 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:24.342 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:24.397 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:24.450 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:24.458 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:24.559 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:24.610 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:24.618 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:24.718 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:24.726 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:24.776 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:24.840 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:24.894 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:24.902 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:24.961 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:24.974 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:24.984 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:25.042 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:25.052 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:25.106 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:25.121 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:25.178 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:25.186 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:25.201 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:25.253 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:25.260 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:25.321 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:25.376 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:25.384 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:25.441 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:25.494 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:25.501 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:25.563 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:25.615 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:25.622 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:25.637 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:25.688 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:25.695 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:25.754 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:25.806 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:25.813 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:25.873 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:25.882 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:25.932 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:25.951 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:26.003 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:26.054 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:26.070 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:26.121 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:26.129 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:26.188 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:26.289 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:26.297 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:26.358 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:26.410 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:26.416 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:26.473 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:26.573 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:26.581 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:26.640 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:26.733 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:26.741 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:26.799 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:26.850 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:26.857 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:26.916 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:27.014 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:27.066 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:27.124 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:27.178 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:27.186 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:27.288 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:27.337 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:27.388 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:27.448 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:27.503 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:27.552 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:27.612 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:27.661 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:27.669 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:27.732 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:27.786 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:27.794 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:27.896 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:27.998 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:28.006 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:28.064 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:28.121 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:28.130 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:28.146 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:28.198 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-22 zones={test-zone2, test-zone1} 2024-08-18T15:37:28.198 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-22 zones={test-zone3, test-zone1} 2024-08-18T15:37:28.204 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:37:28.208 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-08-18T15:37:28.208 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:28.208 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:28.208 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:28.208 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:28.208 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:28.209 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:28.209 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:28.209 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:28.209 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:28.209 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:28.209 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:28.209 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:28.209 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:28.209 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:28.209 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:28.209 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:28.209 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:28.209 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:28.209 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:28.209 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:28.210 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:28.210 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:28.210 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:28.210 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:28.210 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:28.210 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:28.210 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:28.210 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:28.210 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:28.210 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:28.210 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:37:28.214 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-08-18T15:37:28.214 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:28.214 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:28.215 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:28.215 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:28.215 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:28.215 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:28.215 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:28.215 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:28.215 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:28.215 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:28.215 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:28.215 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:28.215 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:28.215 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:28.215 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:28.215 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:28.215 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:28.215 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:28.216 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:28.216 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:28.216 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:28.216 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:28.216 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:28.216 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:28.216 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:28.216 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:28.216 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:28.216 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:28.216 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:28.216 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:28.223 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-08-18T15:37:28.230 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-08-18T15:37:28.284 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-08-18T15:37:28.291 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:28.297 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:28.350 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:28.357 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:28.407 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:28.463 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:28.515 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:28.521 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:28.533 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:28.585 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:28.595 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:28.654 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:28.705 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:28.711 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:28.766 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:28.816 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:28.822 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:28.834 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:28.885 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:28.891 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:29.040 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:29.048 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:29.055 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:29.113 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:29.123 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:29.178 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:29.192 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:29.201 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:29.207 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:29.357 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:29.365 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:29.372 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:29.426 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:29.477 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:29.483 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:29.580 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:29.631 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:29.637 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:29.696 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:29.745 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:29.795 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:29.849 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:29.941 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:29.947 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:30.000 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:30.098 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:30.117 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:30.264 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:30.353 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:30.360 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:30.416 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:30.509 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:30.559 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:30.654 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:30.745 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:30.751 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:30.808 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:30.897 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:30.947 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:31.046 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:31.137 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:31.143 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:31.282 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:31.374 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:31.380 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:31.565 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:31.618 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:31.625 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:31.685 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:31.782 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:31.832 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:31.936 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:31.986 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:32.040 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:32.189 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:32.286 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:32.336 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:32.392 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:32.490 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:32.497 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:32.553 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:32.605 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:32.656 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:32.713 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:32.813 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:32.820 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:32.919 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:33.018 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:33.024 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:33.205 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:33.257 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-22 zones={test-zone3, test-zone1} 2024-08-18T15:37:33.258 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-22 zones={test-zone3, test-zone2} 2024-08-18T15:37:33.262 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:37:33.267 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-08-18T15:37:33.267 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:33.267 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:33.267 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:33.267 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:33.267 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:33.267 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:33.267 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:33.268 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:33.268 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:33.268 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:33.268 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:33.268 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:33.268 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:33.268 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:33.268 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:33.268 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:33.268 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:33.268 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:33.268 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:33.268 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:33.268 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:33.269 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:33.269 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:33.269 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:33.269 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:33.269 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:33.269 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:33.269 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:33.269 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:33.269 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:33.269 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:37:33.274 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-08-18T15:37:33.274 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:33.274 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:33.274 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:33.274 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:33.274 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:33.274 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:33.274 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:33.274 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:33.275 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:33.275 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:33.275 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:33.275 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:33.275 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:33.275 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:33.275 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:33.275 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:33.275 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:33.275 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:33.275 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:33.275 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:33.275 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:33.275 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:33.275 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:33.276 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:33.276 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:33.276 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:33.276 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:33.276 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:33.276 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:33.276 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:37:33.284 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-08-18T15:37:33.292 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-08-18T15:37:33.346 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-08-18T15:37:33.354 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:33.404 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:33.514 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:33.651 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:33.658 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:33.760 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:33.947 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:34.004 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:34.105 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:34.200 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:34.251 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:34.353 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:34.449 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:34.457 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:34.516 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:34.691 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:34.740 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:34.888 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:35.021 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:35.070 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:35.169 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:35.264 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:35.314 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:35.415 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:35.550 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:35.601 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:35.748 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:35.844 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:35.898 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:35.960 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:36.052 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:36.142 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:36.243 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:36.379 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:36.386 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:36.443 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:36.536 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:36.587 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:36.688 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:36.826 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:36.876 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:37.026 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:37.124 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:37.177 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:37.234 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:37.328 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:37.336 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:37.392 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:37.579 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:37.590 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:37.693 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:37.791 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:37.882 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:38.024 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:38.167 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:38.175 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:38.238 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:38.375 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:38.425 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:38.576 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:38.668 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:38.675 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:38.780 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:38.923 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:38.931 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:39.035 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:39.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:39.224 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:39.287 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:39.389 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:39.444 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:39.597 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:39.699 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:39.750 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:39.847 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:39.985 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:40.041 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:40.184 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:40.280 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:40.330 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:40.388 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:40.480 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:40.488 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:40.545 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:40.731 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:40.738 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:40.884 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:40.975 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:41.022 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:41.127 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:37:41.261 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-22 zones={test-zone3, test-zone2} 2024-08-18T15:37:41.262 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:37:41.262 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:37:41.262 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:37:41.316 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:37:41.316 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:37:41.362 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T15:37:41.362 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:37:41.362 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000003197", 2024-08-18T15:37:41.362 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:26:15.364847Z" 2024-08-18T15:37:41.362 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:37:41.362 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:37:41.362 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000004037", 2024-08-18T15:37:41.362 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:32:51.378244Z" 2024-08-18T15:37:41.362 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:37:41.363 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:37:41.363 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000004319", 2024-08-18T15:37:41.363 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:36:53.395554Z" 2024-08-18T15:37:41.363 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:37:41.363 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:37:41.363 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000008126", 2024-08-18T15:37:41.363 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:35:25.388487Z" 2024-08-18T15:37:41.363 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:37:41.363 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T15:37:41.364 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-08-18T15:37:41.364 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:37:41.364 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:37:41.365 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:37:41.422 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:37:41.422 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:37:41.469 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:37:41.469 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:37:41.469 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:37:41.469 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:37:41.470 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:37:41.470 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 8935070562734594544 2024-08-18T15:37:41.470 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:37:41.470 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:37:41.470 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:37:41.470 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:37:41.470 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:37:41.470 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T15:37:41.470 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000003197", 2024-08-18T15:37:41.470 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:37:41.470 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:37:41.470 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:37:41.470 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:26:15.364847Z" 2024-08-18T15:37:41.471 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:37:41.471 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:37:41.471 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:37:41.471 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:37:41.471 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:37:41.471 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T15:37:41.471 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000004037", 2024-08-18T15:37:41.471 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:37:41.471 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:37:41.471 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:37:41.471 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:32:51.378244Z" 2024-08-18T15:37:41.471 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:37:41.471 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:37:41.472 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:37:41.472 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:37:41.472 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:37:41.472 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T15:37:41.472 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000004319", 2024-08-18T15:37:41.472 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:37:41.472 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:37:41.472 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:37:41.472 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:36:53.395554Z" 2024-08-18T15:37:41.472 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:37:41.472 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:37:41.472 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:37:41.472 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:37:41.473 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:37:41.473 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T15:37:41.473 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000008126", 2024-08-18T15:37:41.473 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:37:41.473 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:37:41.473 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:37:41.473 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:35:25.388487Z" 2024-08-18T15:37:41.473 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:37:41.473 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:37:41.473 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:37:41.473 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:37:41.473 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:37:41.473 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 0, 2024-08-18T15:37:41.474 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 0 2024-08-18T15:37:41.474 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:37:41.474 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:37:41Z" 2024-08-18T15:37:41.474 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:37:41.475 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-08-18T15:37:41.475 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:37:41.475 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:37:41.475 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:37:41.521 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:37:41.522 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:37:41.566 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T15:37:41.566 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:37:41.567 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000003197", 2024-08-18T15:37:41.567 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:26:15.364847Z" 2024-08-18T15:37:41.567 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:37:41.567 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:37:41.567 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000004037", 2024-08-18T15:37:41.567 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:32:51.378244Z" 2024-08-18T15:37:41.567 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:37:41.567 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:37:41.567 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000004319", 2024-08-18T15:37:41.567 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:36:53.395554Z" 2024-08-18T15:37:41.567 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:37:41.567 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:37:41.567 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000008126", 2024-08-18T15:37:41.568 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:35:25.388487Z" 2024-08-18T15:37:41.568 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:37:41.568 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T15:37:41.569 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2024-08-18T15:37:41.569 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:37:41.569 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:37:41.569 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:37:41.617 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:37:41.618 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:37:41.657 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T15:37:41.658 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T15:37:41.658 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T15:37:41.658 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T15:37:41.658 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T15:37:41.658 INFO:teuthology.orchestra.run.smithi134.stdout: "instance_id": 6908254841848037526 2024-08-18T15:37:41.658 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:37:41.658 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T15:37:41.658 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:37:41.658 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T15:37:41.658 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:37:41.658 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T15:37:41.659 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000003197", 2024-08-18T15:37:41.659 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:37:41.659 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:37:41.659 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:37:41.659 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:26:15.364847Z" 2024-08-18T15:37:41.659 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:37:41.659 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:37:41.659 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:37:41.659 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T15:37:41.659 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:37:41.659 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T15:37:41.659 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000004037", 2024-08-18T15:37:41.660 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:37:41.660 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:37:41.660 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:37:41.660 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:32:51.378244Z" 2024-08-18T15:37:41.660 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:37:41.660 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:37:41.660 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:37:41.660 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T15:37:41.660 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:37:41.660 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T15:37:41.660 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000004319", 2024-08-18T15:37:41.660 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:37:41.661 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:37:41.661 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:37:41.661 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:36:53.395554Z" 2024-08-18T15:37:41.661 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:37:41.661 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:37:41.661 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:37:41.662 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T15:37:41.662 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:37:41.662 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T15:37:41.662 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000008126", 2024-08-18T15:37:41.662 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:37:41.662 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:37:41.663 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:37:41.663 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:35:25.388487Z" 2024-08-18T15:37:41.663 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:37:41.663 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:37:41.663 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T15:37:41.663 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:37:41.664 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T15:37:41.664 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 0, 2024-08-18T15:37:41.664 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 0 2024-08-18T15:37:41.664 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:37:41.664 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T15:37:41Z" 2024-08-18T15:37:41.664 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T15:37:41.666 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2024-08-18T15:37:41.666 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:37:41.666 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:37:41.667 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:37:41.716 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:37:41.716 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:37:41.762 INFO:teuthology.orchestra.run.smithi193.stdout:[ 2024-08-18T15:37:41.763 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:37:41.763 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000002914", 2024-08-18T15:37:41.763 INFO:teuthology.orchestra.run.smithi193.stdout: "last_update": "2024-08-18T15:25:53.340491Z" 2024-08-18T15:37:41.763 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:37:41.763 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:37:41.764 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000004178", 2024-08-18T15:37:41.764 INFO:teuthology.orchestra.run.smithi193.stdout: "last_update": "2024-08-18T15:33:17.304450Z" 2024-08-18T15:37:41.764 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:37:41.764 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:37:41.764 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000004178", 2024-08-18T15:37:41.764 INFO:teuthology.orchestra.run.smithi193.stdout: "last_update": "2024-08-18T15:37:19.759410Z" 2024-08-18T15:37:41.765 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:37:41.765 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:37:41.765 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000007139", 2024-08-18T15:37:41.765 INFO:teuthology.orchestra.run.smithi193.stdout: "last_update": "2024-08-18T15:35:43.061910Z" 2024-08-18T15:37:41.765 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:37:41.765 INFO:teuthology.orchestra.run.smithi193.stdout:] 2024-08-18T15:37:41.767 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2024-08-18T15:37:41.767 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:37:41.767 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:37:41.768 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:37:41.818 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:37:41.818 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:37:41.860 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T15:37:41.860 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T15:37:41.860 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T15:37:41.860 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T15:37:41.861 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T15:37:41.861 INFO:teuthology.orchestra.run.smithi134.stdout: "instance_id": 15510818030922206205 2024-08-18T15:37:41.861 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:37:41.861 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T15:37:41.861 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:37:41.862 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T15:37:41.862 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:37:41.862 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T15:37:41.862 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000002914", 2024-08-18T15:37:41.862 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:37:41.863 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:37:41.863 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:37:41.863 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:25:53.340491Z" 2024-08-18T15:37:41.863 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:37:41.863 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:37:41.863 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:37:41.864 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T15:37:41.864 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:37:41.864 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T15:37:41.864 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000004178", 2024-08-18T15:37:41.864 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:37:41.865 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:37:41.865 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:37:41.865 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:33:17.304450Z" 2024-08-18T15:37:41.865 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:37:41.865 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:37:41.866 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:37:41.866 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T15:37:41.866 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:37:41.866 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T15:37:41.866 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000004178", 2024-08-18T15:37:41.866 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:37:41.867 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:37:41.867 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:37:41.867 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:37:19.759410Z" 2024-08-18T15:37:41.868 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:37:41.868 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:37:41.868 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:37:41.868 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T15:37:41.868 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:37:41.868 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T15:37:41.869 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000007139", 2024-08-18T15:37:41.869 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:37:41.869 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:37:41.869 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:37:41.869 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:35:43.061910Z" 2024-08-18T15:37:41.870 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:37:41.870 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:37:41.870 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T15:37:41.870 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:37:41.870 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T15:37:41.870 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 0, 2024-08-18T15:37:41.871 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 0 2024-08-18T15:37:41.871 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:37:41.871 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T15:37:41Z" 2024-08-18T15:37:41.871 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T15:37:41.873 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2024-08-18T15:37:41.873 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:37:41.873 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:37:41.874 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:37:41.921 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:37:41.922 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:37:41.971 INFO:teuthology.orchestra.run.smithi193.stdout:[ 2024-08-18T15:37:41.971 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:37:41.971 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000002914", 2024-08-18T15:37:41.971 INFO:teuthology.orchestra.run.smithi193.stdout: "last_update": "2024-08-18T15:25:53.340491Z" 2024-08-18T15:37:41.971 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:37:41.972 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:37:41.972 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000004178", 2024-08-18T15:37:41.972 INFO:teuthology.orchestra.run.smithi193.stdout: "last_update": "2024-08-18T15:33:17.304450Z" 2024-08-18T15:37:41.972 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:37:41.972 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:37:41.972 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000004178", 2024-08-18T15:37:41.972 INFO:teuthology.orchestra.run.smithi193.stdout: "last_update": "2024-08-18T15:37:19.759410Z" 2024-08-18T15:37:41.972 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:37:41.972 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:37:41.972 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000007139", 2024-08-18T15:37:41.972 INFO:teuthology.orchestra.run.smithi193.stdout: "last_update": "2024-08-18T15:35:43.061910Z" 2024-08-18T15:37:41.972 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:37:41.972 INFO:teuthology.orchestra.run.smithi193.stdout:] 2024-08-18T15:37:41.973 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2024-08-18T15:37:41.974 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:37:41.974 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:37:41.974 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:37:42.025 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:37:42.025 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:37:42.087 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T15:37:42.088 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T15:37:42.088 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T15:37:42.088 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T15:37:42.088 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T15:37:42.088 INFO:teuthology.orchestra.run.smithi134.stdout: "instance_id": 9282142518545740118 2024-08-18T15:37:42.088 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:37:42.088 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T15:37:42.088 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:37:42.088 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T15:37:42.088 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:37:42.088 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T15:37:42.088 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000002914", 2024-08-18T15:37:42.088 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:37:42.089 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:37:42.089 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:37:42.089 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:25:53.340491Z" 2024-08-18T15:37:42.089 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:37:42.089 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:37:42.089 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:37:42.089 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T15:37:42.089 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:37:42.089 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T15:37:42.089 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000004178", 2024-08-18T15:37:42.089 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:37:42.089 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:37:42.089 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:37:42.090 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:33:17.304450Z" 2024-08-18T15:37:42.090 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:37:42.090 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:37:42.090 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:37:42.090 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T15:37:42.090 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:37:42.090 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T15:37:42.090 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000004178", 2024-08-18T15:37:42.090 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:37:42.090 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:37:42.090 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:37:42.090 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:37:19.759410Z" 2024-08-18T15:37:42.091 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:37:42.091 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:37:42.091 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:37:42.091 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T15:37:42.091 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:37:42.091 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T15:37:42.091 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000007139", 2024-08-18T15:37:42.091 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:37:42.091 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:37:42.091 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:37:42.091 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:35:43.061910Z" 2024-08-18T15:37:42.091 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:37:42.091 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:37:42.091 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T15:37:42.092 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:37:42.092 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T15:37:42.092 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 0, 2024-08-18T15:37:42.092 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 0 2024-08-18T15:37:42.092 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:37:42.092 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T15:37:42Z" 2024-08-18T15:37:42.092 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T15:37:42.093 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2024-08-18T15:37:42.093 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:37:42.093 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:37:42.093 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:37:42.171 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:37:42.171 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:37:42.217 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T15:37:42.217 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:37:42.218 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000002914", 2024-08-18T15:37:42.218 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:25:59.441712Z" 2024-08-18T15:37:42.218 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:37:42.218 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:37:42.218 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000004460", 2024-08-18T15:37:42.218 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:32:57.456234Z" 2024-08-18T15:37:42.218 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:37:42.218 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:37:42.218 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000004319", 2024-08-18T15:37:42.218 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:36:37.469826Z" 2024-08-18T15:37:42.219 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:37:42.219 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:37:42.219 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000008408", 2024-08-18T15:37:42.219 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:35:09.464574Z" 2024-08-18T15:37:42.219 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:37:42.219 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T15:37:42.220 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2024-08-18T15:37:42.220 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:37:42.220 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:37:42.221 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:37:42.269 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:37:42.269 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:37:42.360 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T15:37:42.360 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T15:37:42.360 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T15:37:42.360 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T15:37:42.360 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T15:37:42.360 INFO:teuthology.orchestra.run.smithi134.stdout: "instance_id": 16929744281786545893 2024-08-18T15:37:42.360 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:37:42.360 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T15:37:42.361 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:37:42.361 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T15:37:42.361 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:37:42.361 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T15:37:42.361 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000002914", 2024-08-18T15:37:42.361 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:37:42.361 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:37:42.361 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:37:42.361 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:25:59.441712Z" 2024-08-18T15:37:42.361 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:37:42.361 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:37:42.362 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:37:42.362 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T15:37:42.362 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:37:42.362 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T15:37:42.362 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000004460", 2024-08-18T15:37:42.362 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:37:42.362 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:37:42.362 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:37:42.362 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:32:57.456234Z" 2024-08-18T15:37:42.362 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:37:42.362 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:37:42.362 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:37:42.363 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T15:37:42.363 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:37:42.363 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T15:37:42.363 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000004319", 2024-08-18T15:37:42.363 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:37:42.363 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:37:42.363 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:37:42.363 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:36:37.469826Z" 2024-08-18T15:37:42.363 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:37:42.363 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:37:42.363 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:37:42.363 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T15:37:42.363 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:37:42.364 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T15:37:42.364 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000008408", 2024-08-18T15:37:42.364 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:37:42.364 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:37:42.364 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:37:42.364 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:35:09.464574Z" 2024-08-18T15:37:42.364 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:37:42.364 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:37:42.364 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T15:37:42.364 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:37:42.364 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T15:37:42.364 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 0, 2024-08-18T15:37:42.364 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 0 2024-08-18T15:37:42.365 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:37:42.365 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T15:37:42Z" 2024-08-18T15:37:42.365 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T15:37:42.366 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2024-08-18T15:37:42.366 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:37:42.366 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:37:42.366 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:37:42.412 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:37:42.412 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:37:42.457 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T15:37:42.457 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:37:42.457 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000002914", 2024-08-18T15:37:42.457 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:25:59.441712Z" 2024-08-18T15:37:42.458 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:37:42.458 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:37:42.458 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000004460", 2024-08-18T15:37:42.458 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:32:57.456234Z" 2024-08-18T15:37:42.458 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:37:42.458 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:37:42.458 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000004319", 2024-08-18T15:37:42.458 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:36:37.469826Z" 2024-08-18T15:37:42.459 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:37:42.459 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:37:42.459 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000008408", 2024-08-18T15:37:42.459 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:35:09.464574Z" 2024-08-18T15:37:42.459 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:37:42.459 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T15:37:42.460 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2024-08-18T15:37:42.460 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:37:42.460 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:37:42.460 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:37:42.512 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:37:42.513 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:37:42.557 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:37:42.557 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:37:42.557 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:37:42.557 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:37:42.557 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:37:42.557 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17055221296585213178 2024-08-18T15:37:42.557 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:37:42.557 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:37:42.557 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:37:42.558 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:37:42.558 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:37:42.558 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T15:37:42.558 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000002914", 2024-08-18T15:37:42.558 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:37:42.558 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:37:42.558 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:37:42.558 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:25:59.441712Z" 2024-08-18T15:37:42.558 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:37:42.558 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:37:42.558 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:37:42.559 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:37:42.559 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:37:42.559 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T15:37:42.559 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000004460", 2024-08-18T15:37:42.559 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:37:42.559 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:37:42.559 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:37:42.559 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:32:57.456234Z" 2024-08-18T15:37:42.559 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:37:42.559 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:37:42.559 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:37:42.559 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:37:42.559 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:37:42.560 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T15:37:42.560 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000004319", 2024-08-18T15:37:42.560 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:37:42.560 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:37:42.560 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:37:42.560 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:36:37.469826Z" 2024-08-18T15:37:42.560 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:37:42.560 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:37:42.560 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:37:42.560 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:37:42.560 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:37:42.560 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T15:37:42.560 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000008408", 2024-08-18T15:37:42.561 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:37:42.561 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:37:42.561 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:37:42.561 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:35:09.464574Z" 2024-08-18T15:37:42.561 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:37:42.561 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:37:42.561 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:37:42.561 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:37:42.561 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:37:42.561 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 0, 2024-08-18T15:37:42.561 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 0 2024-08-18T15:37:42.561 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:37:42.561 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:37:42Z" 2024-08-18T15:37:42.562 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:37:42.563 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2024-08-18T15:37:42.563 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_concurrent_versioned_object_incremental_sync ... ok 2024-08-18T15:37:42.576 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:37:42.576 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:37:42.576 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:37:42.622 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:37:42.622 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:37:42.652 INFO:teuthology.orchestra.run.smithi134.stderr:No --period given, using current period=b0d74758-1871-4454-bc0f-e3e4b6c30105 2024-08-18T15:37:42.664 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T15:37:42.664 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:37:42.664 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:37:42.664 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:19:24.506194Z" 2024-08-18T15:37:42.664 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:37:42.664 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:37:42.664 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T15:37:42.664 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T15:37:42.664 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:37:42.664 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:37:42.665 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:37:42.665 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:21:34.952005Z" 2024-08-18T15:37:42.665 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:37:42.665 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:37:42.665 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723995462.575132_476.1", 2024-08-18T15:37:42.665 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:37:42.575132Z" 2024-08-18T15:37:42.665 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:37:42.665 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T15:37:42.666 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-08-18T15:37:42.666 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:37:42.666 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:37:42.666 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:37:42.715 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:37:42.715 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:37:42.744 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:37:42.744+0000 7f58d7e0f880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:37:42.745 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:37:42.744+0000 7f58d7e0f880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:37:42.757 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:37:42.757 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:37:42.757 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:37:42.757 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:37:42.757 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:37:42.757 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "b0d74758-1871-4454-bc0f-e3e4b6c30105", 2024-08-18T15:37:42.757 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:37:42.757 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:37:42.757 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:37:42.757 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:37:42.758 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:37:42.758 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:37:42.758 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:37:42.758 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:37:42.758 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:37:42.758 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:37:42.758 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:37:42.758 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:37:42.758 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:37:42.758 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:37:42.758 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:37:42.758 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:37:42.759 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:37:42.759 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:37:42.759 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:37:42.759 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:37:42.759 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:37:42.759 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:37:42.759 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:37:42.759 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:37:42.759 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:37:42.759 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:37:42.759 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:37:42.759 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:37:42.760 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:37:42.760 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:37:42.760 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:37:42.760 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:37:42.760 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:37:42.760 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:37:42.760 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:37:42.760 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:37:42.760 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:37:42.760 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:37:42.760 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:37:42.760 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:37:42.761 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:37:42.761 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:37:42.761 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:37:42.761 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723995271.350113_383.1", 2024-08-18T15:37:42.761 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:37:42.761 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:37:42.761 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:37:42.761 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:34:31.350113Z", 2024-08-18T15:37:42.761 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:37:42.761 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:37:42.761 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:37:42.761 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:37:42.762 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:37:42.762 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:37:42.762 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:37:42.762 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:37:42.762 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:37:42.762 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:37:42Z" 2024-08-18T15:37:42.762 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:37:42.763 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1723995462.575132_476.1 target=1_1723995271.350113_383.1 2024-08-18T15:37:47.764 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:37:47.764 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:37:47.764 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:37:47.819 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:37:47.819 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:37:47.850 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:37:47.850+0000 7f3a5640f880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:37:47.851 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:37:47.851+0000 7f3a5640f880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:37:47.866 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:37:47.866 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:37:47.867 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:37:47.867 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:37:47.867 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:37:47.867 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "b0d74758-1871-4454-bc0f-e3e4b6c30105", 2024-08-18T15:37:47.867 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:37:47.867 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:37:47.867 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:37:47.867 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:37:47.867 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:37:47.867 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:37:47.867 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:37:47.867 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:37:47.868 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:37:47.868 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:37:47.868 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:37:47.868 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:37:47.868 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:37:47.868 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:37:47.868 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:37:47.868 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:37:47.868 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:37:47.868 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:37:47.868 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:37:47.868 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:37:47.868 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:37:47.869 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:37:47.869 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:37:47.869 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:37:47.869 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:37:47.869 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:37:47.869 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:37:47.869 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:37:47.869 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:37:47.869 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:37:47.869 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:37:47.869 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:37:47.869 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:37:47.869 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:37:47.870 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:37:47.870 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:37:47.870 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:37:47.870 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:37:47.870 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:37:47.870 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:37:47.870 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:37:47.870 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:37:47.870 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:37:47.870 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723995462.575132_476.1", 2024-08-18T15:37:47.870 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:37:47.870 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:37:47.870 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:37:47.870 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:37:42.575132Z", 2024-08-18T15:37:47.871 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:37:47.871 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:37:47.871 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:37:47.871 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:37:47.871 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:37:47.871 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:37:47.871 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:37:47.871 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:37:47.871 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:37:47.871 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:37:47Z" 2024-08-18T15:37:47.871 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:37:47.872 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-08-18T15:37:47.872 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-08-18T15:37:47.872 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:37:47.873 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:37:47.873 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:37:47.927 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:37:47.927 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:37:47.959 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:37:47.958+0000 7fab9d6bb880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:37:47.960 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:37:47.959+0000 7fab9d6bb880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:37:47.972 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T15:37:47.972 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T15:37:47.973 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T15:37:47.973 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T15:37:47.973 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T15:37:47.973 INFO:teuthology.orchestra.run.smithi134.stdout: "period": "b0d74758-1871-4454-bc0f-e3e4b6c30105", 2024-08-18T15:37:47.973 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:37:47.973 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:37:47.973 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T15:37:47.973 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:37:47.973 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T15:37:47.973 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:37:47.973 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:37:47.973 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:37:47.973 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:37:47.974 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 1, 2024-08-18T15:37:47.974 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:37:47.974 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:37:47.974 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:37:47.974 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:37:47.974 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:37:47.974 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:37:47.974 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T15:37:47.974 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:37:47.974 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:37:47.974 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T15:37:47.974 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:37:47.974 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:37:47.975 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:37:47.975 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T15:37:47.975 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:37:47.975 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:37:47.975 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:37:47.975 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:37:47.975 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T15:37:47.975 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:37:47.975 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:37:47.975 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:37:47.975 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:37:47.975 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:37:47.975 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:37:47.975 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:37:47.976 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:37:47.976 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:37:47.976 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:37:47.976 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:37:47.976 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T15:37:47.976 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:37:47.976 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:37:47.976 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723995462.575132_476.1", 2024-08-18T15:37:47.976 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:37:47.976 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:37:47.976 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:37:47.976 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:37:42.575132Z", 2024-08-18T15:37:47.976 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:37:47.976 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:37:47.977 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:37:47.977 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T15:37:47.977 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:37:47.977 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T15:37:47.977 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 1, 2024-08-18T15:37:47.977 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 1 2024-08-18T15:37:47.977 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:37:47.977 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T15:37:47Z" 2024-08-18T15:37:47.977 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T15:37:47.978 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-08-18T15:37:47.987 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:37:47.987 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:37:47.987 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T15:37:48.043 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:37:48.043 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:37:48.078 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:37:48.077+0000 7fa064f29880 1 waiting to reach incremental sync.. 2024-08-18T15:37:53.079 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:37:53.078+0000 7fa064f29880 1 waiting to reach incremental sync.. 2024-08-18T15:37:58.079 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:37:58.078+0000 7fa064f29880 1 waiting to reach incremental sync.. 2024-08-18T15:38:03.081 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:38:03.080+0000 7fa064f29880 1 bucket sync caught up with source: 2024-08-18T15:38:03.081 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , 00000000001.36.6, , ] 2024-08-18T15:38:03.081 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , , , 00000000001.36.6, , ] 2024-08-18T15:38:03.081 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:38:03.080+0000 7fa064f29880 0 bucket checkpoint complete 2024-08-18T15:38:03.098 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:38:03.098 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:38:03.098 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T15:38:03.161 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:38:03.161 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:38:03.196 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:38:03.195+0000 7ffa92530880 1 waiting to reach incremental sync.. 2024-08-18T15:38:08.198 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:38:08.197+0000 7ffa92530880 1 bucket sync caught up with source: 2024-08-18T15:38:08.198 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , , , 00000000001.36.6, , ] 2024-08-18T15:38:08.198 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , , , , 00000000001.36.6, , ] 2024-08-18T15:38:08.198 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:38:08.197+0000 7ffa92530880 0 bucket checkpoint complete 2024-08-18T15:38:08.212 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:38:08.212 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:38:08.212 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-23 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T15:38:08.297 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:38:08.298 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:38:08.335 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:38:08.335+0000 7fdc68f42880 1 waiting to reach incremental sync.. 2024-08-18T15:38:13.336 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:38:13.336+0000 7fdc68f42880 1 waiting to reach incremental sync.. 2024-08-18T15:38:18.337 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:38:18.336+0000 7fdc68f42880 1 waiting to reach incremental sync.. 2024-08-18T15:38:23.339 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:38:23.338+0000 7fdc68f42880 1 bucket sync caught up with source: 2024-08-18T15:38:23.339 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , , , 00000000001.50.6, , ] 2024-08-18T15:38:23.339 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , , , , 00000000001.50.6, , ] 2024-08-18T15:38:23.339 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:38:23.338+0000 7fdc68f42880 0 bucket checkpoint complete 2024-08-18T15:38:23.355 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:38:23.355 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:38:23.355 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-23 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T15:38:23.414 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:38:23.414 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:38:23.451 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:38:23.449+0000 7f3bccebb880 1 bucket sync caught up with source: 2024-08-18T15:38:23.451 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , , , 00000000001.50.6, , ] 2024-08-18T15:38:23.451 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , , , , 00000000001.50.6, , ] 2024-08-18T15:38:23.451 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:38:23.449+0000 7f3bccebb880 0 bucket checkpoint complete 2024-08-18T15:38:23.464 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:38:23.464 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:38:23.464 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-23 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T15:38:23.517 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:38:23.518 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:38:23.551 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:38:23.550+0000 7f7937035880 1 bucket sync caught up with source: 2024-08-18T15:38:23.552 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , , , 00000000001.78.6, , ] 2024-08-18T15:38:23.552 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , , , , 00000000001.78.6, , ] 2024-08-18T15:38:23.552 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:38:23.550+0000 7f7937035880 0 bucket checkpoint complete 2024-08-18T15:38:23.564 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:38:23.564 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:38:23.565 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-23 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T15:38:23.627 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:38:23.627 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:38:23.665 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:38:23.664+0000 7f51be812880 1 bucket sync caught up with source: 2024-08-18T15:38:23.665 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , 00000000001.78.6, , ] 2024-08-18T15:38:23.665 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , , , 00000000001.78.6, , ] 2024-08-18T15:38:23.665 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:38:23.664+0000 7f51be812880 0 bucket checkpoint complete 2024-08-18T15:38:23.678 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-23 zones={test-zone2, test-zone1} 2024-08-18T15:38:23.684 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:38:23.686 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:38:23.687 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:38:23.689 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:38:23.693 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:38:23.699 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:38:23.752 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:38:23.760 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-23 zones={test-zone2, test-zone1} 2024-08-18T15:38:23.760 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-23 zones={test-zone3, test-zone1} 2024-08-18T15:38:23.763 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:38:23.766 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:38:23.767 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:38:23.769 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:38:23.774 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:38:23.780 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:38:23.834 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:38:23.841 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-23 zones={test-zone3, test-zone1} 2024-08-18T15:38:23.841 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-23 zones={test-zone3, test-zone2} 2024-08-18T15:38:23.848 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:38:23.891 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:38:23.891 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:38:23.893 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:38:23.897 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:38:23.904 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:38:23.957 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:38:23.965 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-23 zones={test-zone3, test-zone2} 2024-08-18T15:38:23.970 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:38:23.970 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:38:23.970 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:38:24.017 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:38:24.017 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:38:24.050 INFO:teuthology.orchestra.run.smithi134.stderr:No --period given, using current period=b0d74758-1871-4454-bc0f-e3e4b6c30105 2024-08-18T15:38:24.061 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T15:38:24.061 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:38:24.061 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:38:24.061 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:19:24.506194Z" 2024-08-18T15:38:24.061 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:38:24.061 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:38:24.062 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T15:38:24.062 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T15:38:24.062 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:38:24.062 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:38:24.062 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:38:24.062 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:21:34.952005Z" 2024-08-18T15:38:24.062 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:38:24.062 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:38:24.062 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723995503.969777_487.1", 2024-08-18T15:38:24.062 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:38:23.969777Z" 2024-08-18T15:38:24.062 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:38:24.062 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T15:38:24.063 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-08-18T15:38:24.064 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:38:24.064 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:38:24.064 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:38:24.111 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:38:24.111 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:38:24.138 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:38:24.138+0000 7f9d2cf3c880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:38:24.139 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:38:24.139+0000 7f9d2cf3c880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:38:24.151 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:38:24.151 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:38:24.151 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:38:24.151 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:38:24.151 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:38:24.151 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "b0d74758-1871-4454-bc0f-e3e4b6c30105", 2024-08-18T15:38:24.151 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:38:24.151 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:38:24.151 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:38:24.152 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:38:24.152 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:38:24.152 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:38:24.152 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:38:24.152 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:38:24.152 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:38:24.152 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:38:24.152 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:38:24.152 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:38:24.152 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:38:24.152 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:38:24.152 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:38:24.152 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:38:24.152 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:38:24.153 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:38:24.153 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:38:24.153 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:38:24.153 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:38:24.153 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:38:24.153 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:38:24.153 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:38:24.153 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:38:24.153 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:38:24.153 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:38:24.153 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:38:24.153 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:38:24.153 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:38:24.154 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:38:24.154 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:38:24.154 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:38:24.154 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:38:24.154 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:38:24.154 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:38:24.154 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:38:24.154 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:38:24.154 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:38:24.154 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:38:24.154 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:38:24.154 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:38:24.155 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:38:24.155 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723995503.969777_487.1", 2024-08-18T15:38:24.155 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:38:24.155 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:38:24.155 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:38:24.155 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:38:23.969777Z", 2024-08-18T15:38:24.155 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:38:24.155 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:38:24.155 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:38:24.155 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:38:24.155 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:38:24.155 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:38:24.155 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:38:24.155 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:38:24.156 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:38:24.156 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:38:24Z" 2024-08-18T15:38:24.156 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:38:24.157 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-08-18T15:38:24.157 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-08-18T15:38:24.157 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:38:24.157 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:38:24.157 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:38:24.205 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:38:24.205 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:38:24.232 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:38:24.231+0000 7fae34c10880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:38:24.233 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:38:24.232+0000 7fae34c10880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:38:24.245 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T15:38:24.245 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T15:38:24.245 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T15:38:24.245 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T15:38:24.245 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T15:38:24.245 INFO:teuthology.orchestra.run.smithi134.stdout: "period": "b0d74758-1871-4454-bc0f-e3e4b6c30105", 2024-08-18T15:38:24.245 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:38:24.245 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:38:24.245 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T15:38:24.245 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:38:24.245 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T15:38:24.246 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:38:24.246 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:38:24.246 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:38:24.246 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:38:24.246 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 1, 2024-08-18T15:38:24.246 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:38:24.246 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:38:24.246 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:38:24.246 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:38:24.246 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:38:24.246 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:38:24.246 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T15:38:24.246 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:38:24.247 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:38:24.247 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T15:38:24.247 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:38:24.247 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:38:24.247 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:38:24.247 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T15:38:24.247 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:38:24.247 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:38:24.247 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:38:24.247 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:38:24.247 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T15:38:24.247 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:38:24.247 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:38:24.248 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:38:24.248 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:38:24.248 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:38:24.248 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:38:24.248 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:38:24.248 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:38:24.248 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:38:24.248 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:38:24.248 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:38:24.248 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T15:38:24.248 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:38:24.248 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:38:24.248 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723995503.969777_487.1", 2024-08-18T15:38:24.248 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:38:24.249 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:38:24.249 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:38:24.249 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:38:23.969777Z", 2024-08-18T15:38:24.249 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:38:24.249 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:38:24.249 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:38:24.249 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T15:38:24.249 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:38:24.249 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T15:38:24.249 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 1, 2024-08-18T15:38:24.249 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 1 2024-08-18T15:38:24.249 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:38:24.249 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T15:38:24Z" 2024-08-18T15:38:24.249 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T15:38:24.250 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-08-18T15:38:24.261 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:38:24.262 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:38:24.262 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T15:38:24.315 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:38:24.315 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:38:24.353 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:38:24.352+0000 7efc40329880 1 waiting for incremental sync to catch up: 2024-08-18T15:38:24.353 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , 00000000001.36.6, , ] 2024-08-18T15:38:24.353 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , , , 00000000003.39.13, , ] 2024-08-18T15:38:29.354 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:38:29.353+0000 7efc40329880 1 waiting for incremental sync to catch up: 2024-08-18T15:38:29.355 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , 00000000001.36.6, , ] 2024-08-18T15:38:29.355 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , , , 00000000003.39.13, , ] 2024-08-18T15:38:34.356 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:38:34.355+0000 7efc40329880 1 waiting for incremental sync to catch up: 2024-08-18T15:38:34.356 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , 00000000001.36.6, , ] 2024-08-18T15:38:34.356 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , , , 00000000003.39.13, , ] 2024-08-18T15:38:39.357 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:38:39.356+0000 7efc40329880 1 waiting for incremental sync to catch up: 2024-08-18T15:38:39.357 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , 00000000001.36.6, , ] 2024-08-18T15:38:39.357 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , , , 00000000003.39.13, , ] 2024-08-18T15:38:44.358 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:38:44.357+0000 7efc40329880 1 bucket sync caught up with source: 2024-08-18T15:38:44.358 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , 00000000003.39.13, , ] 2024-08-18T15:38:44.358 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , , , 00000000003.39.13, , ] 2024-08-18T15:38:44.358 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:38:44.357+0000 7efc40329880 0 bucket checkpoint complete 2024-08-18T15:38:44.376 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:38:44.376 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:38:44.377 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T15:38:44.437 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:38:44.438 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:38:44.474 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:38:44.473+0000 7f4db5530880 1 bucket sync caught up with source: 2024-08-18T15:38:44.474 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , , , 00000000003.39.13, , ] 2024-08-18T15:38:44.474 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , , , , 00000000003.39.13, , ] 2024-08-18T15:38:44.474 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:38:44.473+0000 7f4db5530880 0 bucket checkpoint complete 2024-08-18T15:38:44.487 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:38:44.487 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:38:44.487 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-23 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T15:38:44.543 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:38:44.543 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:38:44.578 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:38:44.577+0000 7fe8c7133880 1 waiting for incremental sync to catch up: 2024-08-18T15:38:44.578 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , , , 00000000001.50.6, , ] 2024-08-18T15:38:44.578 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , , , , 00000000003.54.13, , ] 2024-08-18T15:38:49.579 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:38:49.578+0000 7fe8c7133880 1 waiting for incremental sync to catch up: 2024-08-18T15:38:49.579 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , , , 00000000001.50.6, , ] 2024-08-18T15:38:49.579 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , , , , 00000000003.54.13, , ] 2024-08-18T15:38:54.580 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:38:54.579+0000 7fe8c7133880 1 waiting for incremental sync to catch up: 2024-08-18T15:38:54.580 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , , , 00000000001.50.6, , ] 2024-08-18T15:38:54.581 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , , , , 00000000003.54.13, , ] 2024-08-18T15:38:59.581 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:38:59.581+0000 7fe8c7133880 1 waiting for incremental sync to catch up: 2024-08-18T15:38:59.582 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , , , 00000000001.50.6, , ] 2024-08-18T15:38:59.582 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , , , , 00000000003.54.13, , ] 2024-08-18T15:39:04.583 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:39:04.582+0000 7fe8c7133880 1 bucket sync caught up with source: 2024-08-18T15:39:04.583 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , , , 00000000003.54.13, , ] 2024-08-18T15:39:04.583 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , , , , 00000000003.54.13, , ] 2024-08-18T15:39:04.583 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:39:04.582+0000 7fe8c7133880 0 bucket checkpoint complete 2024-08-18T15:39:04.599 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:39:04.599 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:39:04.599 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-23 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T15:39:04.659 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:39:04.660 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:39:04.697 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:39:04.696+0000 7fc6a4662880 1 bucket sync caught up with source: 2024-08-18T15:39:04.697 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , , , 00000000003.54.13, , ] 2024-08-18T15:39:04.697 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , , , , 00000000003.54.13, , ] 2024-08-18T15:39:04.697 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:39:04.696+0000 7fc6a4662880 0 bucket checkpoint complete 2024-08-18T15:39:04.710 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:39:04.710 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:39:04.710 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-23 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T15:39:04.763 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:39:04.763 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:39:04.796 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:39:04.796+0000 7fc494696880 1 bucket sync caught up with source: 2024-08-18T15:39:04.796 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , , , 00000000004.84.10, , ] 2024-08-18T15:39:04.797 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , , , , 00000000004.84.10, , ] 2024-08-18T15:39:04.797 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:39:04.796+0000 7fc494696880 0 bucket checkpoint complete 2024-08-18T15:39:04.810 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:39:04.810 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:39:04.810 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-23 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T15:39:04.872 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:39:04.872 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:39:04.909 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:39:04.909+0000 7fa2ef413880 1 waiting for incremental sync to catch up: 2024-08-18T15:39:04.909 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , 00000000002.81.6, , ] 2024-08-18T15:39:04.910 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , , , 00000000004.84.10, , ] 2024-08-18T15:39:09.911 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:39:09.910+0000 7fa2ef413880 1 waiting for incremental sync to catch up: 2024-08-18T15:39:09.911 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , 00000000002.81.6, , ] 2024-08-18T15:39:09.911 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , , , 00000000004.84.10, , ] 2024-08-18T15:39:14.912 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:39:14.911+0000 7fa2ef413880 1 waiting for incremental sync to catch up: 2024-08-18T15:39:14.913 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , 00000000002.81.6, , ] 2024-08-18T15:39:14.913 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , , , 00000000004.84.10, , ] 2024-08-18T15:39:19.914 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:39:19.912+0000 7fa2ef413880 1 waiting for incremental sync to catch up: 2024-08-18T15:39:19.914 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , 00000000002.81.6, , ] 2024-08-18T15:39:19.914 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , , , 00000000004.84.10, , ] 2024-08-18T15:39:24.915 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:39:24.914+0000 7fa2ef413880 1 bucket sync caught up with source: 2024-08-18T15:39:24.915 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , 00000000004.84.10, , ] 2024-08-18T15:39:24.915 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , , , 00000000004.84.10, , ] 2024-08-18T15:39:24.915 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:39:24.914+0000 7fa2ef413880 0 bucket checkpoint complete 2024-08-18T15:39:24.931 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-23 zones={test-zone2, test-zone1} 2024-08-18T15:39:24.940 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:39:24.944 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:39:24.944 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:39:24.944 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:39:24.946 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:39:24.946 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:39:24.951 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:39:24.958 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:39:24.970 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:39:25.019 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:39:25.026 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:39:25.085 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:39:25.138 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-23 zones={test-zone2, test-zone1} 2024-08-18T15:39:25.138 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-23 zones={test-zone3, test-zone1} 2024-08-18T15:39:25.143 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:39:25.146 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:39:25.146 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:39:25.146 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:39:25.148 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:39:25.148 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:39:25.153 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:39:25.159 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:39:25.214 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:39:25.222 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:39:25.228 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:39:25.281 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:39:25.329 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-23 zones={test-zone3, test-zone1} 2024-08-18T15:39:25.329 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-23 zones={test-zone3, test-zone2} 2024-08-18T15:39:25.333 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:39:25.335 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:39:25.335 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:39:25.336 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:39:25.338 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:39:25.338 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:39:25.385 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:39:25.392 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:39:25.445 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:39:25.497 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:39:25.505 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:39:25.558 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:39:25.567 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-23 zones={test-zone3, test-zone2} 2024-08-18T15:39:25.572 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:39:25.572 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:39:25.572 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:39:25.619 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:39:25.619 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:39:25.649 INFO:teuthology.orchestra.run.smithi134.stderr:No --period given, using current period=b0d74758-1871-4454-bc0f-e3e4b6c30105 2024-08-18T15:39:25.660 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T15:39:25.661 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:39:25.661 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:39:25.661 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:19:24.506194Z" 2024-08-18T15:39:25.661 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:39:25.661 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:39:25.661 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T15:39:25.661 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T15:39:25.661 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:39:25.661 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:39:25.661 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:39:25.661 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:21:34.952005Z" 2024-08-18T15:39:25.662 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:39:25.662 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:39:25.662 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723995565.571903_531.1", 2024-08-18T15:39:25.662 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:39:25.571903Z" 2024-08-18T15:39:25.662 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:39:25.662 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T15:39:25.663 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-08-18T15:39:25.663 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:39:25.663 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:39:25.664 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:39:25.720 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:39:25.720 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:39:25.754 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:39:25.753+0000 7ff070811880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:39:25.754 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:39:25.754+0000 7ff070811880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:39:25.767 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:39:25.768 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:39:25.768 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:39:25.768 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:39:25.768 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:39:25.768 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "b0d74758-1871-4454-bc0f-e3e4b6c30105", 2024-08-18T15:39:25.768 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:39:25.768 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:39:25.768 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:39:25.768 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:39:25.768 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:39:25.768 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:39:25.768 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:39:25.769 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:39:25.769 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:39:25.769 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:39:25.769 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:39:25.769 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:39:25.769 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:39:25.769 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:39:25.769 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:39:25.769 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:39:25.769 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:39:25.769 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:39:25.769 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:39:25.769 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:39:25.770 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:39:25.770 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:39:25.770 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:39:25.770 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:39:25.770 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:39:25.771 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:39:25.771 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:39:25.771 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:39:25.771 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:39:25.771 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:39:25.771 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:39:25.771 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:39:25.771 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:39:25.771 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:39:25.771 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:39:25.771 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:39:25.771 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:39:25.772 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:39:25.772 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:39:25.772 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:39:25.772 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:39:25.772 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:39:25.772 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:39:25.772 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723995565.571903_531.1", 2024-08-18T15:39:25.772 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:39:25.772 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:39:25.772 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:39:25.773 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:39:25.571903Z", 2024-08-18T15:39:25.773 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:39:25.773 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:39:25.773 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:39:25.773 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:39:25.773 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:39:25.773 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:39:25.773 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:39:25.773 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:39:25.773 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:39:25.773 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:39:25Z" 2024-08-18T15:39:25.774 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:39:25.775 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-08-18T15:39:25.775 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-08-18T15:39:25.775 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:39:25.775 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:39:25.775 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:39:25.822 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:39:25.823 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:39:25.850 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:39:25.849+0000 7ff904a35880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:39:25.851 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:39:25.850+0000 7ff904a35880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:39:25.862 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T15:39:25.862 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T15:39:25.862 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T15:39:25.862 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T15:39:25.863 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T15:39:25.863 INFO:teuthology.orchestra.run.smithi134.stdout: "period": "b0d74758-1871-4454-bc0f-e3e4b6c30105", 2024-08-18T15:39:25.863 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:39:25.863 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:39:25.863 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T15:39:25.863 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:39:25.863 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T15:39:25.863 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:39:25.863 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:39:25.863 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:39:25.863 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:39:25.863 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 1, 2024-08-18T15:39:25.863 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:39:25.864 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:39:25.864 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:39:25.864 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:39:25.864 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:39:25.864 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:39:25.864 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T15:39:25.864 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:39:25.864 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:39:25.864 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T15:39:25.864 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:39:25.864 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:39:25.864 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:39:25.864 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T15:39:25.865 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:39:25.865 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:39:25.865 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:39:25.865 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:39:25.865 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T15:39:25.865 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:39:25.865 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:39:25.865 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:39:25.865 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:39:25.865 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:39:25.865 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:39:25.865 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:39:25.865 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:39:25.866 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:39:25.866 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:39:25.866 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:39:25.866 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T15:39:25.866 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:39:25.866 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:39:25.866 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723995565.571903_531.1", 2024-08-18T15:39:25.866 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:39:25.866 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:39:25.866 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:39:25.866 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:39:25.571903Z", 2024-08-18T15:39:25.866 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:39:25.867 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:39:25.867 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:39:25.867 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T15:39:25.867 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:39:25.867 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T15:39:25.867 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 1, 2024-08-18T15:39:25.867 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 1 2024-08-18T15:39:25.867 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:39:25.867 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T15:39:25Z" 2024-08-18T15:39:25.867 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T15:39:25.868 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-08-18T15:39:25.881 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:39:25.881 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:39:25.881 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T15:39:25.937 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:39:25.937 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:39:25.975 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:39:25.974+0000 7f33e7213880 1 waiting for incremental sync to catch up: 2024-08-18T15:39:25.975 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , 00000000003.39.13, , ] 2024-08-18T15:39:25.975 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , , , 00000000005.43.18, , ] 2024-08-18T15:39:30.976 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:39:30.976+0000 7f33e7213880 1 waiting for incremental sync to catch up: 2024-08-18T15:39:30.977 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , 00000000003.39.13, , ] 2024-08-18T15:39:30.977 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , , , 00000000005.43.18, , ] 2024-08-18T15:39:35.978 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:39:35.977+0000 7f33e7213880 1 waiting for incremental sync to catch up: 2024-08-18T15:39:35.978 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , 00000000003.39.13, , ] 2024-08-18T15:39:35.978 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , , , 00000000005.43.18, , ] 2024-08-18T15:39:40.979 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:39:40.978+0000 7f33e7213880 1 waiting for incremental sync to catch up: 2024-08-18T15:39:40.979 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , 00000000003.39.13, , ] 2024-08-18T15:39:40.979 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , , , 00000000005.43.18, , ] 2024-08-18T15:39:45.980 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:39:45.979+0000 7f33e7213880 1 bucket sync caught up with source: 2024-08-18T15:39:45.980 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , 00000000005.43.18, , ] 2024-08-18T15:39:45.980 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , , , 00000000005.43.18, , ] 2024-08-18T15:39:45.980 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:39:45.979+0000 7f33e7213880 0 bucket checkpoint complete 2024-08-18T15:39:45.995 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:39:45.996 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:39:45.996 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T15:39:46.058 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:39:46.058 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:39:46.094 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:39:46.093+0000 7f7518a3f880 1 bucket sync caught up with source: 2024-08-18T15:39:46.094 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , , , 00000000005.43.18, , ] 2024-08-18T15:39:46.094 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , , , , 00000000005.43.18, , ] 2024-08-18T15:39:46.094 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:39:46.093+0000 7f7518a3f880 0 bucket checkpoint complete 2024-08-18T15:39:46.107 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:39:46.107 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:39:46.108 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-23 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T15:39:46.163 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:39:46.164 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:39:46.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:39:46.198+0000 7f9007c11880 1 waiting for incremental sync to catch up: 2024-08-18T15:39:46.199 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , , , 00000000004.56.10, , ] 2024-08-18T15:39:46.199 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , , , , 00000000007.63.15, , ] 2024-08-18T15:39:51.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:39:51.200+0000 7f9007c11880 1 waiting for incremental sync to catch up: 2024-08-18T15:39:51.201 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , , , 00000000004.56.10, , ] 2024-08-18T15:39:51.201 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , , , , 00000000007.63.15, , ] 2024-08-18T15:39:56.202 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:39:56.201+0000 7f9007c11880 1 waiting for incremental sync to catch up: 2024-08-18T15:39:56.202 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , , , 00000000004.56.10, , ] 2024-08-18T15:39:56.202 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , , , , 00000000007.63.15, , ] 2024-08-18T15:40:01.203 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:40:01.202+0000 7f9007c11880 1 waiting for incremental sync to catch up: 2024-08-18T15:40:01.203 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , , , 00000000004.56.10, , ] 2024-08-18T15:40:01.204 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , , , , 00000000007.63.15, , ] 2024-08-18T15:40:06.204 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:40:06.204+0000 7f9007c11880 1 bucket sync caught up with source: 2024-08-18T15:40:06.205 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , , , 00000000007.63.15, , ] 2024-08-18T15:40:06.205 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , , , , 00000000007.63.15, , ] 2024-08-18T15:40:06.205 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:40:06.204+0000 7f9007c11880 0 bucket checkpoint complete 2024-08-18T15:40:06.223 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:40:06.223 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:40:06.224 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-23 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T15:40:06.276 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:40:06.277 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:40:06.311 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:40:06.310+0000 7fa55c20e880 1 waiting for incremental sync to catch up: 2024-08-18T15:40:06.311 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , , , 00000000004.56.10, , ] 2024-08-18T15:40:06.311 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , , , , 00000000007.63.15, , ] 2024-08-18T15:40:11.312 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:40:11.311+0000 7fa55c20e880 1 bucket sync caught up with source: 2024-08-18T15:40:11.313 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , , , 00000000007.63.15, , ] 2024-08-18T15:40:11.313 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , , , , 00000000007.63.15, , ] 2024-08-18T15:40:11.313 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:40:11.311+0000 7fa55c20e880 0 bucket checkpoint complete 2024-08-18T15:40:11.328 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:40:11.329 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:40:11.329 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-23 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T15:40:11.387 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:40:11.388 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:40:11.423 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:40:11.422+0000 7f0d4680f880 1 waiting for incremental sync to catch up: 2024-08-18T15:40:11.423 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , , , 00000000006.89.18, , ] 2024-08-18T15:40:11.423 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , , , , 00000000007.91.15, , ] 2024-08-18T15:40:16.424 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:40:16.423+0000 7f0d4680f880 1 waiting for incremental sync to catch up: 2024-08-18T15:40:16.424 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , , , 00000000006.89.18, , ] 2024-08-18T15:40:16.425 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , , , , 00000000007.91.15, , ] 2024-08-18T15:40:21.426 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:40:21.425+0000 7f0d4680f880 1 waiting for incremental sync to catch up: 2024-08-18T15:40:21.426 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , , , 00000000006.89.18, , ] 2024-08-18T15:40:21.426 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , , , , 00000000007.91.15, , ] 2024-08-18T15:40:26.427 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:40:26.426+0000 7f0d4680f880 1 bucket sync caught up with source: 2024-08-18T15:40:26.427 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , , , 00000000007.91.15, , ] 2024-08-18T15:40:26.427 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , , , , 00000000007.91.15, , ] 2024-08-18T15:40:26.427 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:40:26.426+0000 7f0d4680f880 0 bucket checkpoint complete 2024-08-18T15:40:26.444 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:40:26.444 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:40:26.444 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-23 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T15:40:26.506 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:40:26.506 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:40:26.544 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:40:26.543+0000 7fc2a3936880 1 bucket sync caught up with source: 2024-08-18T15:40:26.544 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , 00000000007.91.15, , ] 2024-08-18T15:40:26.544 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , , , 00000000007.91.15, , ] 2024-08-18T15:40:26.544 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:40:26.543+0000 7fc2a3936880 0 bucket checkpoint complete 2024-08-18T15:40:26.557 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-23 zones={test-zone2, test-zone1} 2024-08-18T15:40:26.570 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:40:26.573 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:40:26.573 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:40:26.573 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:40:26.576 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:40:26.576 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:40:26.581 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:40:26.588 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:40:26.599 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:40:26.647 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:40:26.654 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:40:26.665 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:40:26.718 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-23 zones={test-zone2, test-zone1} 2024-08-18T15:40:26.718 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-23 zones={test-zone3, test-zone1} 2024-08-18T15:40:26.724 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:40:26.727 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:40:26.727 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:40:26.727 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:40:26.730 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:40:26.730 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:40:26.735 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:40:26.741 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:40:26.794 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:40:26.802 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:40:26.808 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:40:26.860 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:40:26.868 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-23 zones={test-zone3, test-zone1} 2024-08-18T15:40:26.868 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-23 zones={test-zone3, test-zone2} 2024-08-18T15:40:26.873 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:40:26.923 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:40:26.923 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:40:26.923 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:40:26.926 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:40:26.926 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:40:26.931 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:40:26.938 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:40:26.998 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:40:27.055 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:40:27.063 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:40:27.118 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:40:27.127 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-23 zones={test-zone3, test-zone2} 2024-08-18T15:40:27.127 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_version_suspended_incremental_sync ... ok 2024-08-18T15:40:27.128 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=kibgmb-24 2024-08-18T15:40:27.142 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=kibgmb-25 2024-08-18T15:40:27.166 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=kibgmb-26 2024-08-18T15:40:27.207 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:40:27.208 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:40:27.208 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:40:27.258 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:40:27.258 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:40:27.293 INFO:teuthology.orchestra.run.smithi134.stderr:No --period given, using current period=b0d74758-1871-4454-bc0f-e3e4b6c30105 2024-08-18T15:40:27.305 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T15:40:27.305 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:40:27.305 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:40:27.305 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:19:24.506194Z" 2024-08-18T15:40:27.305 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:40:27.305 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:40:27.306 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T15:40:27.306 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T15:40:27.306 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:40:27.306 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:40:27.306 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:40:27.306 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:21:34.952005Z" 2024-08-18T15:40:27.306 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:40:27.306 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:40:27.306 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723995627.205732_576.1", 2024-08-18T15:40:27.306 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:40:27.205732Z" 2024-08-18T15:40:27.306 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:40:27.306 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T15:40:27.308 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-08-18T15:40:27.308 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:40:27.308 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:40:27.308 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:40:27.358 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:40:27.358 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:40:27.387 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:40:27.386+0000 7f265332a880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:40:27.388 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:40:27.387+0000 7f265332a880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:40:27.399 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:40:27.400 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:40:27.400 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:40:27.400 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:40:27.400 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:40:27.400 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "b0d74758-1871-4454-bc0f-e3e4b6c30105", 2024-08-18T15:40:27.400 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:40:27.400 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:40:27.400 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:40:27.400 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:40:27.400 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:40:27.400 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:40:27.400 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:40:27.401 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:40:27.401 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:40:27.401 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:40:27.401 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:40:27.401 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:40:27.401 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:40:27.401 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:40:27.401 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:40:27.402 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:40:27.402 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:40:27.402 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:40:27.402 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:40:27.402 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:40:27.402 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:40:27.402 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:40:27.402 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:40:27.402 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:40:27.402 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:40:27.403 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:40:27.403 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:40:27.403 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:40:27.403 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:40:27.403 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:40:27.403 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:40:27.403 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:40:27.403 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:40:27.403 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:40:27.403 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:40:27.403 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:40:27.403 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:40:27.403 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:40:27.404 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:40:27.404 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:40:27.404 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:40:27.404 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:40:27.404 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:40:27.404 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723995627.134415_559.1", 2024-08-18T15:40:27.404 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:40:27.404 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:40:27.405 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:40:27.405 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:40:27.134415Z", 2024-08-18T15:40:27.405 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:40:27.405 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:40:27.405 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:40:27.405 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:40:27.405 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:40:27.405 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:40:27.405 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:40:27.405 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:40:27.405 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:40:27.406 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:40:27Z" 2024-08-18T15:40:27.406 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:40:27.407 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1723995627.205732_576.1 target=1_1723995627.134415_559.1 2024-08-18T15:40:32.408 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:40:32.408 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:40:32.408 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:40:32.464 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:40:32.464 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:40:32.495 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:40:32.495+0000 7f41f0d42880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:40:32.496 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:40:32.496+0000 7f41f0d42880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:40:32.509 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:40:32.509 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:40:32.509 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:40:32.509 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:40:32.509 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:40:32.509 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "b0d74758-1871-4454-bc0f-e3e4b6c30105", 2024-08-18T15:40:32.509 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:40:32.509 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:40:32.509 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:40:32.510 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:40:32.510 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:40:32.510 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:40:32.510 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:40:32.510 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:40:32.510 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:40:32.510 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:40:32.510 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:40:32.510 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:40:32.510 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:40:32.510 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:40:32.511 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:40:32.511 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:40:32.511 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:40:32.511 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:40:32.511 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:40:32.511 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:40:32.511 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:40:32.511 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:40:32.511 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:40:32.511 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:40:32.511 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:40:32.511 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:40:32.511 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:40:32.512 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:40:32.512 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:40:32.512 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:40:32.512 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:40:32.512 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:40:32.512 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:40:32.512 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:40:32.512 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:40:32.512 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:40:32.512 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:40:32.512 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:40:32.512 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:40:32.512 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:40:32.513 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:40:32.513 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:40:32.513 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:40:32.513 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723995627.205732_576.1", 2024-08-18T15:40:32.513 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:40:32.513 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:40:32.513 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:40:32.513 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:40:27.205732Z", 2024-08-18T15:40:32.513 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:40:32.513 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:40:32.513 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:40:32.513 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:40:32.513 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:40:32.513 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:40:32.514 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:40:32.514 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:40:32.514 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:40:32.514 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:40:32Z" 2024-08-18T15:40:32.514 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:40:32.515 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-08-18T15:40:32.515 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-08-18T15:40:32.516 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:40:32.516 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:40:32.516 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:40:32.575 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:40:32.575 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:40:32.602 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:40:32.602+0000 7f301bd20880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:40:32.603 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:40:32.603+0000 7f301bd20880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:40:32.616 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T15:40:32.616 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T15:40:32.616 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T15:40:32.616 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T15:40:32.616 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T15:40:32.616 INFO:teuthology.orchestra.run.smithi134.stdout: "period": "b0d74758-1871-4454-bc0f-e3e4b6c30105", 2024-08-18T15:40:32.616 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:40:32.616 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:40:32.616 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T15:40:32.616 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:40:32.617 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T15:40:32.617 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:40:32.617 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:40:32.617 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:40:32.617 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:40:32.617 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 1, 2024-08-18T15:40:32.618 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:40:32.618 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:40:32.618 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:40:32.618 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:40:32.618 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:40:32.618 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:40:32.618 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T15:40:32.618 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:40:32.618 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:40:32.618 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T15:40:32.618 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:40:32.619 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:40:32.619 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:40:32.619 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T15:40:32.619 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:40:32.619 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:40:32.619 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:40:32.619 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:40:32.619 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T15:40:32.619 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:40:32.619 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:40:32.619 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:40:32.619 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:40:32.619 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:40:32.620 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:40:32.620 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:40:32.620 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:40:32.620 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:40:32.620 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:40:32.620 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:40:32.620 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T15:40:32.620 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:40:32.620 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:40:32.620 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723995627.134415_559.1", 2024-08-18T15:40:32.620 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:40:32.620 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:40:32.620 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:40:32.621 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:40:27.134415Z", 2024-08-18T15:40:32.621 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:40:32.621 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:40:32.621 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:40:32.621 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T15:40:32.621 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:40:32.621 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T15:40:32.621 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 1, 2024-08-18T15:40:32.621 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 1 2024-08-18T15:40:32.621 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:40:32.621 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T15:40:32Z" 2024-08-18T15:40:32.621 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T15:40:32.622 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1723995627.205732_576.1 target=1_1723995627.134415_559.1 2024-08-18T15:40:37.624 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:40:37.624 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:40:37.624 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:40:37.676 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:40:37.676 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:40:37.706 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:40:37.705+0000 7f994cc35880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:40:37.707 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:40:37.706+0000 7f994cc35880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:40:37.718 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T15:40:37.719 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T15:40:37.719 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T15:40:37.719 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T15:40:37.719 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T15:40:37.719 INFO:teuthology.orchestra.run.smithi134.stdout: "period": "b0d74758-1871-4454-bc0f-e3e4b6c30105", 2024-08-18T15:40:37.719 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:40:37.719 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:40:37.719 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T15:40:37.720 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:40:37.720 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T15:40:37.720 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:40:37.720 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:40:37.720 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:40:37.720 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:40:37.720 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 1, 2024-08-18T15:40:37.720 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:40:37.720 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:40:37.720 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:40:37.720 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:40:37.720 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:40:37.720 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:40:37.721 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T15:40:37.721 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:40:37.721 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:40:37.721 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T15:40:37.721 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:40:37.721 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:40:37.721 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:40:37.721 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T15:40:37.721 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:40:37.721 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:40:37.721 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:40:37.721 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:40:37.721 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T15:40:37.721 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:40:37.722 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:40:37.722 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:40:37.722 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:40:37.722 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:40:37.722 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:40:37.722 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:40:37.722 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:40:37.722 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:40:37.722 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:40:37.722 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:40:37.722 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T15:40:37.722 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:40:37.723 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:40:37.723 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723995627.205732_576.1", 2024-08-18T15:40:37.723 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:40:37.723 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:40:37.723 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:40:37.723 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:40:27.205732Z", 2024-08-18T15:40:37.724 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:40:37.724 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:40:37.724 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:40:37.724 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T15:40:37.724 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:40:37.724 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T15:40:37.725 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 1, 2024-08-18T15:40:37.725 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 1 2024-08-18T15:40:37.725 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:40:37.725 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T15:40:37Z" 2024-08-18T15:40:37.725 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T15:40:37.726 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-08-18T15:40:37.789 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:40:37.790 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:40:37.790 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-24 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T15:40:37.850 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:40:37.850 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:40:37.885 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:40:37.884+0000 7f3014526880 1 waiting to reach incremental sync.. 2024-08-18T15:40:42.887 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:40:42.886+0000 7f3014526880 1 bucket sync caught up with source: 2024-08-18T15:40:42.887 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , 00000000003.9.12, , ] 2024-08-18T15:40:42.887 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , , , 00000000003.9.12, , ] 2024-08-18T15:40:42.887 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:40:42.886+0000 7f3014526880 0 bucket checkpoint complete 2024-08-18T15:40:42.903 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:40:42.903 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:40:42.904 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-24 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T15:40:42.964 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:40:42.964 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:40:42.999 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:40:42.999+0000 7ff1a1d42880 1 waiting to reach incremental sync.. 2024-08-18T15:40:48.000 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:40:47.999+0000 7ff1a1d42880 1 waiting to reach incremental sync.. 2024-08-18T15:40:53.002 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:40:53.001+0000 7ff1a1d42880 1 bucket sync caught up with source: 2024-08-18T15:40:53.002 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , , , 00000000003.9.12, , ] 2024-08-18T15:40:53.002 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , , , , 00000000003.9.12, , ] 2024-08-18T15:40:53.002 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:40:53.001+0000 7ff1a1d42880 0 bucket checkpoint complete 2024-08-18T15:40:53.018 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:40:53.019 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:40:53.019 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-24 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T15:40:53.078 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:40:53.078 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:40:53.117 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:40:53.116+0000 7f5c79813880 1 waiting to reach incremental sync.. 2024-08-18T15:40:58.118 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:40:58.117+0000 7f5c79813880 1 waiting to reach incremental sync.. 2024-08-18T15:41:03.120 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:41:03.119+0000 7f5c79813880 1 bucket sync caught up with source: 2024-08-18T15:41:03.120 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , , , 00000000002.22.6, , ] 2024-08-18T15:41:03.120 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , , , , 00000000002.22.6, , ] 2024-08-18T15:41:03.120 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:41:03.119+0000 7f5c79813880 0 bucket checkpoint complete 2024-08-18T15:41:03.136 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:41:03.136 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:41:03.136 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-24 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T15:41:03.195 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:41:03.196 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:41:03.233 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:41:03.232+0000 7f3cad719880 1 bucket sync caught up with source: 2024-08-18T15:41:03.233 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , , , 00000000002.22.6, , ] 2024-08-18T15:41:03.233 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , , , , 00000000002.22.6, , ] 2024-08-18T15:41:03.233 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:41:03.232+0000 7f3cad719880 0 bucket checkpoint complete 2024-08-18T15:41:03.246 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:41:03.246 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:41:03.246 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-24 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T15:41:03.300 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:41:03.300 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:41:03.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:41:03.332+0000 7fec8c835880 1 waiting to reach incremental sync.. 2024-08-18T15:41:08.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:41:08.332+0000 7fec8c835880 1 waiting to reach incremental sync.. 2024-08-18T15:41:13.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:41:13.333+0000 7fec8c835880 1 waiting to reach incremental sync.. 2024-08-18T15:41:18.335 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:41:18.334+0000 7fec8c835880 1 waiting to reach incremental sync.. 2024-08-18T15:41:23.335 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:41:23.334+0000 7fec8c835880 1 waiting to reach incremental sync.. 2024-08-18T15:41:28.337 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:41:28.336+0000 7fec8c835880 1 bucket sync caught up with source: 2024-08-18T15:41:28.337 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , , , 00000000002.27.6, , ] 2024-08-18T15:41:28.337 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , , , , 00000000002.27.6, , ] 2024-08-18T15:41:28.337 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:41:28.336+0000 7fec8c835880 0 bucket checkpoint complete 2024-08-18T15:41:28.354 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:41:28.354 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:41:28.354 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-24 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T15:41:28.416 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:41:28.416 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:41:28.454 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:41:28.454+0000 7fd57cc96880 1 bucket sync caught up with source: 2024-08-18T15:41:28.455 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , 00000000002.27.6, , ] 2024-08-18T15:41:28.455 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , , , 00000000002.27.6, , ] 2024-08-18T15:41:28.455 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:41:28.454+0000 7fd57cc96880 0 bucket checkpoint complete 2024-08-18T15:41:28.469 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-24 zones={test-zone2, test-zone1} 2024-08-18T15:41:28.493 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:41:28.539 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:41:28.539 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:41:28.539 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:41:28.542 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:41:28.542 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:41:28.547 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:41:28.550 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:41:28.557 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:41:28.610 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-24 zones={test-zone2, test-zone1} 2024-08-18T15:41:28.610 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-24 zones={test-zone3, test-zone1} 2024-08-18T15:41:28.614 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:41:28.616 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:41:28.617 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:41:28.617 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:41:28.619 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:41:28.619 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:41:28.624 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:41:28.627 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:41:28.634 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:41:28.686 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-24 zones={test-zone3, test-zone1} 2024-08-18T15:41:28.687 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-24 zones={test-zone3, test-zone2} 2024-08-18T15:41:28.691 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:41:28.694 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:41:28.694 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:41:28.694 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:41:28.696 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:41:28.697 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:41:28.701 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:41:28.705 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:41:28.711 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:41:28.762 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-24 zones={test-zone3, test-zone2} 2024-08-18T15:41:28.762 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:41:28.762 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:41:28.763 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-25 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T15:41:28.817 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:41:28.818 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:41:28.852 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:41:28.851+0000 7f66b2a12880 1 bucket sync caught up with source: 2024-08-18T15:41:28.852 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , 00000000002.23.6, , ] 2024-08-18T15:41:28.852 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , , , 00000000002.23.6, , ] 2024-08-18T15:41:28.852 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:41:28.851+0000 7f66b2a12880 0 bucket checkpoint complete 2024-08-18T15:41:28.865 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:41:28.865 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:41:28.865 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-25 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T15:41:28.919 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:41:28.920 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:41:28.953 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:41:28.952+0000 7f1a4dd32880 1 bucket sync caught up with source: 2024-08-18T15:41:28.953 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , , , 00000000002.23.6, , ] 2024-08-18T15:41:28.953 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , , , , 00000000002.23.6, , ] 2024-08-18T15:41:28.953 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:41:28.952+0000 7f1a4dd32880 0 bucket checkpoint complete 2024-08-18T15:41:28.965 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:41:28.965 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:41:28.965 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-25 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T15:41:29.019 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:41:29.019 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:41:29.054 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:41:29.053+0000 7f129af32880 1 bucket sync caught up with source: 2024-08-18T15:41:29.054 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , , , 00000000003.69.12, , ] 2024-08-18T15:41:29.054 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , , , , 00000000003.69.12, , ] 2024-08-18T15:41:29.054 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:41:29.053+0000 7f129af32880 0 bucket checkpoint complete 2024-08-18T15:41:29.067 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:41:29.067 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:41:29.067 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-25 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T15:41:29.122 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:41:29.122 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:41:29.155 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:41:29.154+0000 7f9b5e286880 1 bucket sync caught up with source: 2024-08-18T15:41:29.155 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , , , 00000000003.69.12, , ] 2024-08-18T15:41:29.155 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , , , , 00000000003.69.12, , ] 2024-08-18T15:41:29.155 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:41:29.154+0000 7f9b5e286880 0 bucket checkpoint complete 2024-08-18T15:41:29.168 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:41:29.168 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:41:29.168 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-25 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T15:41:29.221 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:41:29.221 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:41:29.254 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:41:29.254+0000 7f8cd5012880 1 bucket sync caught up with source: 2024-08-18T15:41:29.254 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , , , 00000000002.74.6, , ] 2024-08-18T15:41:29.255 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , , , , 00000000002.74.6, , ] 2024-08-18T15:41:29.255 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:41:29.254+0000 7f8cd5012880 0 bucket checkpoint complete 2024-08-18T15:41:29.267 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:41:29.267 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:41:29.267 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-25 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T15:41:29.322 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:41:29.322 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:41:29.356 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:41:29.355+0000 7fe04260f880 1 bucket sync caught up with source: 2024-08-18T15:41:29.356 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , 00000000002.74.6, , ] 2024-08-18T15:41:29.356 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , , , 00000000002.74.6, , ] 2024-08-18T15:41:29.356 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:41:29.355+0000 7fe04260f880 0 bucket checkpoint complete 2024-08-18T15:41:29.369 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-25 zones={test-zone2, test-zone1} 2024-08-18T15:41:29.373 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:41:29.376 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:41:29.376 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:41:29.376 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:41:29.378 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:41:29.378 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:41:29.430 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:41:29.434 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:41:29.440 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:41:29.450 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-25 zones={test-zone2, test-zone1} 2024-08-18T15:41:29.451 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-25 zones={test-zone3, test-zone1} 2024-08-18T15:41:29.455 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:41:29.457 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:41:29.457 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:41:29.457 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:41:29.503 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:41:29.503 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:41:29.507 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:41:29.511 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:41:29.517 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:41:29.572 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-25 zones={test-zone3, test-zone1} 2024-08-18T15:41:29.573 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-25 zones={test-zone3, test-zone2} 2024-08-18T15:41:29.576 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:41:29.623 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:41:29.623 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:41:29.623 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:41:29.625 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:41:29.625 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:41:29.630 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:41:29.634 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:41:29.640 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:41:29.693 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-25 zones={test-zone3, test-zone2} 2024-08-18T15:41:29.693 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:41:29.693 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:41:29.693 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-26 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T15:41:29.750 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:41:29.750 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:41:29.785 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:41:29.784+0000 7f4f11626880 1 bucket sync caught up with source: 2024-08-18T15:41:29.785 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , 00000000002.56.6, , ] 2024-08-18T15:41:29.785 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , , , 00000000002.56.6, , ] 2024-08-18T15:41:29.785 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:41:29.784+0000 7f4f11626880 0 bucket checkpoint complete 2024-08-18T15:41:29.798 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:41:29.798 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:41:29.798 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-26 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T15:41:29.852 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:41:29.852 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:41:29.885 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:41:29.884+0000 7f0c03129880 1 bucket sync caught up with source: 2024-08-18T15:41:29.885 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , , , 00000000002.56.6, , ] 2024-08-18T15:41:29.885 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , , , , 00000000002.56.6, , ] 2024-08-18T15:41:29.885 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:41:29.884+0000 7f0c03129880 0 bucket checkpoint complete 2024-08-18T15:41:29.897 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:41:29.898 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:41:29.898 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-26 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T15:41:29.951 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:41:29.951 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:41:29.985 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:41:29.984+0000 7effef432880 1 bucket sync caught up with source: 2024-08-18T15:41:29.985 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , , , 00000000002.71.6, , ] 2024-08-18T15:41:29.985 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , , , , 00000000002.71.6, , ] 2024-08-18T15:41:29.985 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:41:29.984+0000 7effef432880 0 bucket checkpoint complete 2024-08-18T15:41:29.998 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:41:29.998 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:41:29.998 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-26 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T15:41:30.051 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:41:30.052 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:41:30.085 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:41:30.085+0000 7f847dd42880 1 bucket sync caught up with source: 2024-08-18T15:41:30.086 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , , , 00000000002.71.6, , ] 2024-08-18T15:41:30.086 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , , , , 00000000002.71.6, , ] 2024-08-18T15:41:30.086 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:41:30.085+0000 7f847dd42880 0 bucket checkpoint complete 2024-08-18T15:41:30.098 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:41:30.098 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:41:30.098 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-26 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T15:41:30.151 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:41:30.151 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:41:30.185 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:41:30.184+0000 7f1234826880 1 bucket sync caught up with source: 2024-08-18T15:41:30.185 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , , , 00000000003.96.12, , ] 2024-08-18T15:41:30.186 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , , , , 00000000003.96.12, , ] 2024-08-18T15:41:30.186 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:41:30.184+0000 7f1234826880 0 bucket checkpoint complete 2024-08-18T15:41:30.199 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:41:30.199 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:41:30.199 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-26 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T15:41:30.254 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:41:30.254 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:41:30.289 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:41:30.288+0000 7f304d3af880 1 bucket sync caught up with source: 2024-08-18T15:41:30.289 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , 00000000003.96.12, , ] 2024-08-18T15:41:30.289 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , , , 00000000003.96.12, , ] 2024-08-18T15:41:30.289 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:41:30.288+0000 7f304d3af880 0 bucket checkpoint complete 2024-08-18T15:41:30.301 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-26 zones={test-zone2, test-zone1} 2024-08-18T15:41:30.305 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:41:30.355 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:41:30.355 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:41:30.355 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:41:30.359 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:41:30.359 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:41:30.364 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:41:30.368 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:41:30.374 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:41:30.386 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-26 zones={test-zone2, test-zone1} 2024-08-18T15:41:30.386 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-26 zones={test-zone3, test-zone1} 2024-08-18T15:41:30.390 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:41:30.393 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:41:30.393 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:41:30.393 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:41:30.395 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:41:30.395 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:41:30.443 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:41:30.446 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:41:30.452 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:41:30.463 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-26 zones={test-zone3, test-zone1} 2024-08-18T15:41:30.463 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-26 zones={test-zone3, test-zone2} 2024-08-18T15:41:30.467 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:41:30.470 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:41:30.470 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:41:30.470 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:41:30.515 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:41:30.515 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:41:30.520 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:41:30.523 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:41:30.530 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:41:30.540 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-26 zones={test-zone3, test-zone2} 2024-08-18T15:41:30.541 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_delete_marker_full_sync ... ok 2024-08-18T15:41:30.541 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=kibgmb-27 2024-08-18T15:41:30.554 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=kibgmb-28 2024-08-18T15:41:30.576 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=kibgmb-29 2024-08-18T15:41:30.631 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:41:30.632 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:41:30.632 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:41:30.678 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:41:30.678 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:41:30.714 INFO:teuthology.orchestra.run.smithi134.stderr:No --period given, using current period=b0d74758-1871-4454-bc0f-e3e4b6c30105 2024-08-18T15:41:30.727 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T15:41:30.727 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:41:30.727 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:41:30.727 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:19:24.506194Z" 2024-08-18T15:41:30.727 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:41:30.727 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:41:30.727 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T15:41:30.727 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T15:41:30.728 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:41:30.728 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:41:30.728 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:41:30.728 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:21:34.952005Z" 2024-08-18T15:41:30.728 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:41:30.728 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:41:30.728 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723995690.630163_633.1", 2024-08-18T15:41:30.728 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:41:30.630163Z" 2024-08-18T15:41:30.728 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:41:30.728 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T15:41:30.729 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-08-18T15:41:30.729 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:41:30.730 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:41:30.730 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:41:30.779 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:41:30.779 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:41:30.807 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:41:30.806+0000 7f3b0e7af880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:41:30.808 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:41:30.807+0000 7f3b0e7af880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:41:30.819 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:41:30.820 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:41:30.820 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:41:30.820 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:41:30.820 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:41:30.820 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "b0d74758-1871-4454-bc0f-e3e4b6c30105", 2024-08-18T15:41:30.820 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:41:30.820 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:41:30.820 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:41:30.820 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:41:30.820 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:41:30.820 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:41:30.820 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:41:30.821 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:41:30.821 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:41:30.821 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:41:30.821 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:41:30.821 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:41:30.821 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:41:30.821 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:41:30.821 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:41:30.821 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:41:30.821 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:41:30.821 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:41:30.821 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:41:30.821 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:41:30.822 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:41:30.822 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:41:30.822 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:41:30.822 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:41:30.822 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:41:30.822 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:41:30.822 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:41:30.822 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:41:30.823 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:41:30.823 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:41:30.823 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:41:30.823 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:41:30.823 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:41:30.823 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:41:30.823 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:41:30.823 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:41:30.823 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:41:30.823 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:41:30.824 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:41:30.824 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:41:30.824 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:41:30.824 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:41:30.824 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:41:30.824 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723995690.585913_619.1", 2024-08-18T15:41:30.824 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:41:30.824 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:41:30.824 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:41:30.824 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:41:30.585913Z", 2024-08-18T15:41:30.824 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:41:30.824 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:41:30.825 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:41:30.825 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:41:30.825 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:41:30.825 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:41:30.825 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:41:30.825 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:41:30.825 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:41:30.825 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:41:30Z" 2024-08-18T15:41:30.825 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:41:30.826 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1723995690.630163_633.1 target=1_1723995690.585913_619.1 2024-08-18T15:41:35.827 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:41:35.828 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:41:35.828 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:41:35.881 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:41:35.881 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:41:35.912 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:41:35.911+0000 7fb238211880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:41:35.913 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:41:35.912+0000 7fb238211880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:41:35.925 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:41:35.925 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:41:35.925 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:41:35.925 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:41:35.926 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:41:35.926 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "b0d74758-1871-4454-bc0f-e3e4b6c30105", 2024-08-18T15:41:35.926 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:41:35.926 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:41:35.926 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:41:35.926 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:41:35.926 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:41:35.926 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:41:35.926 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:41:35.926 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:41:35.926 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:41:35.926 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:41:35.926 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:41:35.927 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:41:35.927 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:41:35.927 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:41:35.927 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:41:35.927 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:41:35.927 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:41:35.927 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:41:35.927 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:41:35.927 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:41:35.927 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:41:35.927 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:41:35.927 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:41:35.927 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:41:35.927 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:41:35.928 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:41:35.928 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:41:35.928 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:41:35.928 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:41:35.928 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:41:35.928 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:41:35.928 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:41:35.928 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:41:35.928 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:41:35.928 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:41:35.929 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:41:35.929 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:41:35.929 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:41:35.929 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:41:35.929 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:41:35.929 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:41:35.929 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:41:35.929 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:41:35.929 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723995690.586806_620.1", 2024-08-18T15:41:35.929 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:41:35.929 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:41:35.930 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:41:35.930 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:41:30.586806Z", 2024-08-18T15:41:35.930 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:41:35.930 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:41:35.930 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:41:35.930 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:41:35.930 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:41:35.930 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:41:35.930 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:41:35.930 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:41:35.930 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:41:35.930 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:41:35Z" 2024-08-18T15:41:35.930 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:41:35.931 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1723995690.630163_633.1 target=1_1723995690.586806_620.1 2024-08-18T15:41:40.934 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:41:40.934 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:41:40.935 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:41:40.990 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:41:40.990 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:41:41.021 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:41:41.021+0000 7f405ff22880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:41:41.022 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:41:41.022+0000 7f405ff22880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:41:41.035 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:41:41.035 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:41:41.035 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:41:41.035 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:41:41.035 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:41:41.035 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "b0d74758-1871-4454-bc0f-e3e4b6c30105", 2024-08-18T15:41:41.035 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:41:41.035 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:41:41.036 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:41:41.036 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:41:41.036 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:41:41.036 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:41:41.036 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:41:41.036 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:41:41.036 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:41:41.036 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:41:41.036 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:41:41.036 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:41:41.036 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:41:41.036 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:41:41.036 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:41:41.037 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:41:41.037 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:41:41.037 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:41:41.037 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:41:41.037 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:41:41.037 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:41:41.037 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:41:41.037 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:41:41.037 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:41:41.037 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:41:41.037 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:41:41.037 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:41:41.037 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:41:41.037 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:41:41.038 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:41:41.038 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:41:41.038 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:41:41.038 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:41:41.038 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:41:41.038 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:41:41.038 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:41:41.038 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:41:41.038 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:41:41.038 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:41:41.038 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:41:41.038 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:41:41.039 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:41:41.039 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:41:41.039 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723995690.630163_633.1", 2024-08-18T15:41:41.039 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:41:41.039 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:41:41.039 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:41:41.039 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:41:30.630163Z", 2024-08-18T15:41:41.039 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:41:41.039 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:41:41.039 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:41:41.039 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:41:41.039 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:41:41.039 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:41:41.040 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:41:41.040 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:41:41.040 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:41:41.040 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:41:41Z" 2024-08-18T15:41:41.040 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:41:41.041 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-08-18T15:41:41.041 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-08-18T15:41:41.041 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:41:41.041 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:41:41.041 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:41:41.100 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:41:41.100 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:41:41.131 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:41:41.130+0000 7f3435731880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:41:41.132 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:41:41.131+0000 7f3435731880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:41:41.144 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T15:41:41.145 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T15:41:41.145 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T15:41:41.145 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T15:41:41.145 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T15:41:41.145 INFO:teuthology.orchestra.run.smithi134.stdout: "period": "b0d74758-1871-4454-bc0f-e3e4b6c30105", 2024-08-18T15:41:41.145 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:41:41.145 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:41:41.145 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T15:41:41.145 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:41:41.145 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T15:41:41.146 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:41:41.146 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:41:41.146 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:41:41.146 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:41:41.146 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 1, 2024-08-18T15:41:41.146 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:41:41.146 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:41:41.146 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:41:41.146 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:41:41.146 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:41:41.146 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:41:41.146 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T15:41:41.146 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:41:41.147 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:41:41.147 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T15:41:41.147 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:41:41.147 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:41:41.147 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:41:41.147 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T15:41:41.147 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:41:41.147 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:41:41.147 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:41:41.147 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:41:41.147 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T15:41:41.147 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:41:41.147 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:41:41.147 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:41:41.148 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:41:41.148 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:41:41.148 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:41:41.148 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:41:41.148 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:41:41.148 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:41:41.148 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:41:41.148 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:41:41.148 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T15:41:41.148 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:41:41.148 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:41:41.148 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723995690.630163_633.1", 2024-08-18T15:41:41.148 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:41:41.149 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:41:41.149 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:41:41.149 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:41:30.630163Z", 2024-08-18T15:41:41.149 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:41:41.149 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:41:41.149 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:41:41.149 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T15:41:41.149 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:41:41.149 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T15:41:41.149 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 1, 2024-08-18T15:41:41.149 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 1 2024-08-18T15:41:41.149 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:41:41.149 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T15:41:41Z" 2024-08-18T15:41:41.149 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T15:41:41.150 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-08-18T15:41:41.205 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-27', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:41:41.206 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-27', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:41:41.206 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-27 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T15:41:41.261 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:41:41.320 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:41:41.321 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:41:41.294+0000 7fb25f522880 1 waiting to reach incremental sync.. 2024-08-18T15:41:46.297 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:41:46.296+0000 7fb25f522880 1 bucket sync caught up with source: 2024-08-18T15:41:46.297 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , 00000000003.17.7, , ] 2024-08-18T15:41:46.297 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , , , 00000000003.17.7, , ] 2024-08-18T15:41:46.297 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:41:46.296+0000 7fb25f522880 0 bucket checkpoint complete 2024-08-18T15:41:46.313 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-27', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:41:46.314 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-27', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:41:46.314 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-27 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T15:41:46.375 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:41:46.376 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:41:46.411 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:41:46.410+0000 7ff9b8033880 1 waiting to reach incremental sync.. 2024-08-18T15:41:51.413 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:41:51.412+0000 7ff9b8033880 1 bucket sync caught up with source: 2024-08-18T15:41:51.413 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , , , 00000000003.17.7, , ] 2024-08-18T15:41:51.413 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , , , , 00000000003.17.7, , ] 2024-08-18T15:41:51.413 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:41:51.412+0000 7ff9b8033880 0 bucket checkpoint complete 2024-08-18T15:41:51.429 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-27', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:41:51.430 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-27', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:41:51.430 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-27 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T15:41:51.489 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:41:51.489 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:41:51.526 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:41:51.525+0000 7f2c5dabb880 1 waiting to reach incremental sync.. 2024-08-18T15:41:56.527 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:41:56.526+0000 7f2c5dabb880 1 waiting to reach incremental sync.. 2024-08-18T15:42:01.527 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:42:01.526+0000 7f2c5dabb880 1 waiting to reach incremental sync.. 2024-08-18T15:42:06.529 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:42:06.528+0000 7f2c5dabb880 1 bucket sync caught up with source: 2024-08-18T15:42:06.529 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , , , 00000000001.47.14, , ] 2024-08-18T15:42:06.529 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , , , , 00000000001.47.14, , ] 2024-08-18T15:42:06.529 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:42:06.528+0000 7f2c5dabb880 0 bucket checkpoint complete 2024-08-18T15:42:06.544 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-27', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:42:06.544 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-27', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:42:06.545 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-27 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T15:42:06.601 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:42:06.602 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:42:06.637 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:42:06.636+0000 7f79d1c0f880 1 bucket sync caught up with source: 2024-08-18T15:42:06.637 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , , , 00000000001.47.14, , ] 2024-08-18T15:42:06.637 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , , , , 00000000001.47.14, , ] 2024-08-18T15:42:06.637 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:42:06.636+0000 7f79d1c0f880 0 bucket checkpoint complete 2024-08-18T15:42:06.649 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-27', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:42:06.650 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-27', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:42:06.650 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-27 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T15:42:06.702 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:42:06.703 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:42:06.735 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:42:06.734+0000 7ff0556bb880 1 bucket sync caught up with source: 2024-08-18T15:42:06.735 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , , , 00000000001.61.14, , ] 2024-08-18T15:42:06.735 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , , , , 00000000001.61.14, , ] 2024-08-18T15:42:06.735 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:42:06.734+0000 7ff0556bb880 0 bucket checkpoint complete 2024-08-18T15:42:06.748 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-27', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:42:06.748 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-27', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:42:06.748 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-27 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T15:42:06.810 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:42:06.810 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:42:06.847 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:42:06.846+0000 7f0f4243f880 1 waiting to reach incremental sync.. 2024-08-18T15:42:11.848 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:42:11.847+0000 7f0f4243f880 1 waiting to reach incremental sync.. 2024-08-18T15:42:16.848 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:42:16.848+0000 7f0f4243f880 1 waiting to reach incremental sync.. 2024-08-18T15:42:21.849 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:42:21.848+0000 7f0f4243f880 1 waiting to reach incremental sync.. 2024-08-18T15:42:26.850 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:42:26.849+0000 7f0f4243f880 1 bucket sync caught up with source: 2024-08-18T15:42:26.851 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , 00000000001.61.14, , ] 2024-08-18T15:42:26.851 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , , , 00000000001.61.14, , ] 2024-08-18T15:42:26.851 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:42:26.849+0000 7f0f4243f880 0 bucket checkpoint complete 2024-08-18T15:42:26.867 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-27 zones={test-zone2, test-zone1} 2024-08-18T15:42:26.874 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:42:26.919 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:42:26.919 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:42:26.922 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:42:26.928 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:42:26.931 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-27 zones={test-zone2, test-zone1} 2024-08-18T15:42:26.931 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-27 zones={test-zone3, test-zone1} 2024-08-18T15:42:26.935 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:42:26.938 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:42:26.938 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:42:26.983 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:42:26.988 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:42:26.991 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-27 zones={test-zone3, test-zone1} 2024-08-18T15:42:26.992 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-27 zones={test-zone3, test-zone2} 2024-08-18T15:42:26.996 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:42:26.999 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:42:26.999 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:42:27.001 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:42:27.007 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:42:27.010 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-27 zones={test-zone3, test-zone2} 2024-08-18T15:42:27.010 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-28', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:42:27.010 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-28', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:42:27.011 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-28 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T15:42:27.071 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:42:27.071 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:42:27.110 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:42:27.108+0000 7f532bb26880 1 bucket sync caught up with source: 2024-08-18T15:42:27.110 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , 00000000001.18.14, , ] 2024-08-18T15:42:27.110 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , , , 00000000001.18.14, , ] 2024-08-18T15:42:27.110 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:42:27.108+0000 7f532bb26880 0 bucket checkpoint complete 2024-08-18T15:42:27.123 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-28', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:42:27.124 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-28', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:42:27.124 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-28 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T15:42:27.184 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:42:27.184 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:42:27.220 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:42:27.220+0000 7f34d412e880 1 bucket sync caught up with source: 2024-08-18T15:42:27.221 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , , , 00000000001.18.14, , ] 2024-08-18T15:42:27.221 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , , , , 00000000001.18.14, , ] 2024-08-18T15:42:27.221 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:42:27.220+0000 7f34d412e880 0 bucket checkpoint complete 2024-08-18T15:42:27.233 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-28', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:42:27.234 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-28', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:42:27.234 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-28 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T15:42:27.288 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:42:27.288 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:42:27.322 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:42:27.321+0000 7f9c67926880 1 bucket sync caught up with source: 2024-08-18T15:42:27.322 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , , , 00000000003.58.7, , ] 2024-08-18T15:42:27.322 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , , , , 00000000003.58.7, , ] 2024-08-18T15:42:27.322 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:42:27.321+0000 7f9c67926880 0 bucket checkpoint complete 2024-08-18T15:42:27.335 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-28', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:42:27.335 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-28', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:42:27.335 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-28 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T15:42:27.388 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:42:27.388 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:42:27.422 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:42:27.421+0000 7ff7f6917880 1 bucket sync caught up with source: 2024-08-18T15:42:27.422 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , , , 00000000003.58.7, , ] 2024-08-18T15:42:27.422 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , , , , 00000000003.58.7, , ] 2024-08-18T15:42:27.422 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:42:27.421+0000 7ff7f6917880 0 bucket checkpoint complete 2024-08-18T15:42:27.435 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-28', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:42:27.435 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-28', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:42:27.435 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-28 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T15:42:27.488 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:42:27.488 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:42:27.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:42:27.520+0000 7ff763936880 1 bucket sync caught up with source: 2024-08-18T15:42:27.521 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , , , 00000000001.37.14, , ] 2024-08-18T15:42:27.521 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , , , , 00000000001.37.14, , ] 2024-08-18T15:42:27.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:42:27.520+0000 7ff763936880 0 bucket checkpoint complete 2024-08-18T15:42:27.534 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-28', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:42:27.534 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-28', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:42:27.534 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-28 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T15:42:27.592 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:42:27.592 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:42:27.627 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:42:27.625+0000 7f7dd7919880 1 bucket sync caught up with source: 2024-08-18T15:42:27.627 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , 00000000001.37.14, , ] 2024-08-18T15:42:27.627 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , , , 00000000001.37.14, , ] 2024-08-18T15:42:27.627 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:42:27.625+0000 7f7dd7919880 0 bucket checkpoint complete 2024-08-18T15:42:27.640 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-28 zones={test-zone2, test-zone1} 2024-08-18T15:42:27.644 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:42:27.647 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:42:27.647 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:42:27.649 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:42:27.653 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:42:27.656 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-28 zones={test-zone2, test-zone1} 2024-08-18T15:42:27.656 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-28 zones={test-zone3, test-zone1} 2024-08-18T15:42:27.660 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:42:27.662 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:42:27.662 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:42:27.665 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:42:27.669 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:42:27.672 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-28 zones={test-zone3, test-zone1} 2024-08-18T15:42:27.672 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-28 zones={test-zone3, test-zone2} 2024-08-18T15:42:27.676 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:42:27.679 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:42:27.679 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:42:27.723 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:42:27.728 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:42:27.731 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-28 zones={test-zone3, test-zone2} 2024-08-18T15:42:27.731 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-29', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:42:27.731 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-29', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:42:27.731 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-29 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T15:42:27.785 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:42:27.785 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:42:27.819 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:42:27.818+0000 7f6b7a733880 1 bucket sync caught up with source: 2024-08-18T15:42:27.820 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , 00000000001.58.14, , ] 2024-08-18T15:42:27.820 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , , , 00000000001.58.14, , ] 2024-08-18T15:42:27.820 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:42:27.818+0000 7f6b7a733880 0 bucket checkpoint complete 2024-08-18T15:42:27.833 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-29', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:42:27.833 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-29', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:42:27.833 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-29 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T15:42:27.887 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:42:27.887 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:42:27.920 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:42:27.919+0000 7efc94013880 1 bucket sync caught up with source: 2024-08-18T15:42:27.920 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , , , 00000000001.58.14, , ] 2024-08-18T15:42:27.920 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , , , , 00000000001.58.14, , ] 2024-08-18T15:42:27.920 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:42:27.919+0000 7efc94013880 0 bucket checkpoint complete 2024-08-18T15:42:27.932 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-29', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:42:27.932 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-29', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:42:27.933 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-29 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T15:42:27.985 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:42:27.985 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:42:28.022 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:42:28.021+0000 7fbc85932880 1 bucket sync caught up with source: 2024-08-18T15:42:28.022 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , , , 00000000001.68.14, , ] 2024-08-18T15:42:28.022 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , , , , 00000000001.68.14, , ] 2024-08-18T15:42:28.023 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:42:28.021+0000 7fbc85932880 0 bucket checkpoint complete 2024-08-18T15:42:28.035 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-29', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:42:28.035 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-29', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:42:28.035 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-29 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T15:42:28.088 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:42:28.089 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:42:28.124 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:42:28.123+0000 7f2002138880 1 bucket sync caught up with source: 2024-08-18T15:42:28.124 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , , , 00000000001.68.14, , ] 2024-08-18T15:42:28.124 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , , , , 00000000001.68.14, , ] 2024-08-18T15:42:28.124 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:42:28.123+0000 7f2002138880 0 bucket checkpoint complete 2024-08-18T15:42:28.136 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-29', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:42:28.137 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-29', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:42:28.137 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-29 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T15:42:28.190 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:42:28.191 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:42:28.224 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:42:28.223+0000 7f2c75a2f880 1 bucket sync caught up with source: 2024-08-18T15:42:28.224 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , , , 00000000003.121.7, , ] 2024-08-18T15:42:28.224 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , , , , 00000000003.121.7, , ] 2024-08-18T15:42:28.224 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:42:28.223+0000 7f2c75a2f880 0 bucket checkpoint complete 2024-08-18T15:42:28.237 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-29', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:42:28.237 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-29', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:42:28.237 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-29 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T15:42:28.295 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:42:28.296 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:42:28.329 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:42:28.327+0000 7f3471932880 1 bucket sync caught up with source: 2024-08-18T15:42:28.329 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , 00000000003.121.7, , ] 2024-08-18T15:42:28.329 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , , , 00000000003.121.7, , ] 2024-08-18T15:42:28.329 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:42:28.327+0000 7f3471932880 0 bucket checkpoint complete 2024-08-18T15:42:28.341 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-29 zones={test-zone2, test-zone1} 2024-08-18T15:42:28.346 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:42:28.391 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:42:28.391 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:42:28.394 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:42:28.401 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:42:28.405 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-29 zones={test-zone2, test-zone1} 2024-08-18T15:42:28.405 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-29 zones={test-zone3, test-zone1} 2024-08-18T15:42:28.413 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:42:28.416 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:42:28.416 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:42:28.419 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:42:28.474 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:42:28.484 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-29 zones={test-zone3, test-zone1} 2024-08-18T15:42:28.484 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-29 zones={test-zone3, test-zone2} 2024-08-18T15:42:28.490 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:42:28.494 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:42:28.495 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:42:28.497 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-08-18T15:42:28.506 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-08-18T15:42:28.510 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-29 zones={test-zone3, test-zone2} 2024-08-18T15:42:28.511 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_suspended_delete_marker_full_sync ... ok 2024-08-18T15:42:28.511 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=kibgmb-30 2024-08-18T15:42:28.529 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=kibgmb-31 2024-08-18T15:42:28.552 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=kibgmb-32 2024-08-18T15:42:28.679 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_versioning ... ok 2024-08-18T15:42:28.679 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=kibgmb-33 2024-08-18T15:42:28.692 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=kibgmb-34 2024-08-18T15:42:28.714 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=kibgmb-35 2024-08-18T15:42:28.855 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_acl ... ok 2024-08-18T15:42:28.855 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=kibgmb-36 2024-08-18T15:42:28.869 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=kibgmb-37 2024-08-18T15:42:28.891 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=kibgmb-38 2024-08-18T15:42:31.035 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_cors ... ok 2024-08-18T15:42:31.036 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=kibgmb-39 2024-08-18T15:42:31.051 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=kibgmb-40 2024-08-18T15:42:31.081 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=kibgmb-41 2024-08-18T15:42:31.107 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:42:31.107 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:42:31.107 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:42:31.163 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:42:31.163 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:42:31.198 INFO:teuthology.orchestra.run.smithi134.stderr:No --period given, using current period=b0d74758-1871-4454-bc0f-e3e4b6c30105 2024-08-18T15:42:31.210 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T15:42:31.210 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:42:31.210 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:42:31.210 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:19:24.506194Z" 2024-08-18T15:42:31.210 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:42:31.210 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:42:31.210 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T15:42:31.210 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T15:42:31.210 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:42:31.210 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:42:31.210 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:42:31.211 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:21:34.952005Z" 2024-08-18T15:42:31.211 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:42:31.211 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:42:31.211 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723995751.097661_712.1", 2024-08-18T15:42:31.211 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:42:31.097661Z" 2024-08-18T15:42:31.211 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:42:31.211 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T15:42:31.212 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-08-18T15:42:31.212 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:42:31.212 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:42:31.213 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:42:31.264 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:42:31.264 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:42:31.294 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:42:31.293+0000 7f72a4f2a880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:42:31.295 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:42:31.294+0000 7f72a4f2a880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:42:31.307 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:42:31.307 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:42:31.307 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:42:31.307 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:42:31.307 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:42:31.308 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "b0d74758-1871-4454-bc0f-e3e4b6c30105", 2024-08-18T15:42:31.308 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:42:31.308 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:42:31.308 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:42:31.308 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:42:31.308 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:42:31.308 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:42:31.308 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:42:31.309 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:42:31.309 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:42:31.309 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:42:31.309 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:42:31.309 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:42:31.309 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:42:31.309 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:42:31.309 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:42:31.309 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:42:31.309 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:42:31.309 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:42:31.309 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:42:31.309 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:42:31.310 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:42:31.310 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:42:31.310 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:42:31.310 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:42:31.310 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:42:31.310 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:42:31.310 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:42:31.310 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:42:31.310 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:42:31.310 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:42:31.310 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:42:31.310 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:42:31.311 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:42:31.311 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:42:31.311 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:42:31.311 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:42:31.311 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:42:31.311 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:42:31.311 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:42:31.311 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:42:31.311 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:42:31.311 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:42:31.311 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:42:31.311 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723995748.877645_687.1", 2024-08-18T15:42:31.312 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:42:31.312 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:42:31.312 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:42:31.312 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:28.877645Z", 2024-08-18T15:42:31.312 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:42:31.312 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:42:31.312 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:42:31.312 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:42:31.312 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:42:31.312 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:42:31.312 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:42:31.312 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:42:31.313 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:42:31.313 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:42:31Z" 2024-08-18T15:42:31.313 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:42:31.315 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1723995751.097661_712.1 target=1_1723995748.877645_687.1 2024-08-18T15:42:36.316 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:42:36.316 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:42:36.316 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:42:36.371 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:42:36.371 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:42:36.403 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:42:36.402+0000 7f339ce94880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:42:36.403 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:42:36.403+0000 7f339ce94880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:42:36.417 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:42:36.417 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:42:36.417 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:42:36.417 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:42:36.417 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:42:36.417 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "b0d74758-1871-4454-bc0f-e3e4b6c30105", 2024-08-18T15:42:36.417 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:42:36.418 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:42:36.418 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:42:36.418 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:42:36.418 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:42:36.418 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:42:36.418 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:42:36.418 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:42:36.418 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:42:36.418 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:42:36.418 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:42:36.418 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:42:36.419 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:42:36.419 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:42:36.419 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:42:36.419 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:42:36.419 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:42:36.419 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:42:36.419 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:42:36.419 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:42:36.419 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:42:36.419 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:42:36.419 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:42:36.419 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:42:36.419 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:42:36.419 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:42:36.420 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:42:36.420 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:42:36.420 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:42:36.420 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:42:36.420 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:42:36.420 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:42:36.420 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:42:36.420 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:42:36.420 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:42:36.420 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:42:36.420 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:42:36.420 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:42:36.420 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:42:36.421 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:42:36.421 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:42:36.421 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:42:36.421 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:42:36.421 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723995748.899500_691.1", 2024-08-18T15:42:36.421 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:42:36.421 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:42:36.421 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:42:36.421 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:28.899500Z", 2024-08-18T15:42:36.421 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:42:36.421 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:42:36.421 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:42:36.421 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:42:36.421 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:42:36.422 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:42:36.422 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:42:36.422 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:42:36.422 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:42:36.422 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:42:36Z" 2024-08-18T15:42:36.422 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:42:36.426 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1723995751.097661_712.1 target=1_1723995748.899500_691.1 2024-08-18T15:42:41.427 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:42:41.428 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:42:41.428 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:42:41.492 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:42:41.493 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:42:41.537 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:42:41.536+0000 7fc591936880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:42:41.538 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:42:41.537+0000 7fc591936880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:42:41.550 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:42:41.551 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:42:41.551 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:42:41.551 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:42:41.551 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:42:41.551 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "b0d74758-1871-4454-bc0f-e3e4b6c30105", 2024-08-18T15:42:41.551 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:42:41.551 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:42:41.551 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:42:41.551 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:42:41.551 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:42:41.551 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:42:41.551 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:42:41.551 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:42:41.552 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:42:41.552 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:42:41.552 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:42:41.552 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:42:41.552 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:42:41.552 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:42:41.552 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:42:41.552 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:42:41.552 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:42:41.552 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:42:41.552 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:42:41.552 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:42:41.552 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:42:41.553 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:42:41.553 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:42:41.553 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:42:41.553 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:42:41.553 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:42:41.553 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:42:41.553 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:42:41.553 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:42:41.553 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:42:41.553 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:42:41.553 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:42:41.553 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:42:41.553 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:42:41.554 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:42:41.554 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:42:41.554 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:42:41.554 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:42:41.554 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:42:41.554 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:42:41.554 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:42:41.554 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:42:41.554 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:42:41.554 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723995748.899500_691.1", 2024-08-18T15:42:41.554 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:42:41.554 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:42:41.554 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:42:41.555 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:28.899500Z", 2024-08-18T15:42:41.555 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:42:41.555 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:42:41.555 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:42:41.555 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:42:41.555 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:42:41.555 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:42:41.555 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:42:41.555 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:42:41.555 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:42:41.555 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:42:41Z" 2024-08-18T15:42:41.555 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:42:41.556 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1723995751.097661_712.1 target=1_1723995748.899500_691.1 2024-08-18T15:42:46.557 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:42:46.557 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:42:46.558 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:42:46.613 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:42:46.613 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:42:46.645 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:42:46.645+0000 7f00feb22880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:42:46.646 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:42:46.646+0000 7f00feb22880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:42:46.662 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:42:46.662 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:42:46.662 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:42:46.662 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:42:46.663 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:42:46.663 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "b0d74758-1871-4454-bc0f-e3e4b6c30105", 2024-08-18T15:42:46.663 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:42:46.663 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:42:46.663 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:42:46.663 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:42:46.663 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:42:46.663 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:42:46.663 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:42:46.663 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:42:46.663 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:42:46.663 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:42:46.663 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:42:46.664 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:42:46.664 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:42:46.664 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:42:46.664 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:42:46.664 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:42:46.664 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:42:46.664 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:42:46.664 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:42:46.664 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:42:46.664 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:42:46.664 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:42:46.664 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:42:46.664 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:42:46.665 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:42:46.665 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:42:46.665 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:42:46.665 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:42:46.665 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:42:46.665 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:42:46.665 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:42:46.665 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:42:46.665 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:42:46.665 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:42:46.665 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:42:46.666 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:42:46.666 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:42:46.666 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:42:46.666 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:42:46.666 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:42:46.666 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:42:46.666 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:42:46.666 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:42:46.666 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723995748.877645_687.1", 2024-08-18T15:42:46.666 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:42:46.666 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:42:46.666 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:42:46.666 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:28.877645Z", 2024-08-18T15:42:46.667 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:42:46.667 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:42:46.667 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:42:46.667 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:42:46.667 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:42:46.667 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:42:46.667 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:42:46.667 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:42:46.667 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:42:46.668 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:42:46Z" 2024-08-18T15:42:46.668 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:42:46.669 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1723995751.097661_712.1 target=1_1723995748.877645_687.1 2024-08-18T15:42:51.671 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:42:51.671 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:42:51.671 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:42:51.725 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:42:51.726 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:42:51.757 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:42:51.756+0000 7f6b54332880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:42:51.758 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:42:51.758+0000 7f6b54332880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:42:51.771 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:42:51.771 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:42:51.771 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:42:51.772 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:42:51.772 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:42:51.772 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "b0d74758-1871-4454-bc0f-e3e4b6c30105", 2024-08-18T15:42:51.772 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:42:51.772 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:42:51.772 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:42:51.772 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:42:51.772 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:42:51.772 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:42:51.772 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:42:51.773 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:42:51.773 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:42:51.773 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:42:51.773 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:42:51.773 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:42:51.773 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:42:51.773 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:42:51.773 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:42:51.773 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:42:51.773 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:42:51.773 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:42:51.773 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:42:51.773 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:42:51.774 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:42:51.774 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:42:51.774 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:42:51.774 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:42:51.774 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:42:51.774 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:42:51.774 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:42:51.774 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:42:51.774 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:42:51.774 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:42:51.774 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:42:51.774 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:42:51.774 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:42:51.775 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:42:51.775 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:42:51.775 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:42:51.775 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:42:51.775 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:42:51.775 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:42:51.775 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:42:51.775 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:42:51.775 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:42:51.775 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:42:51.775 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723995751.097661_712.1", 2024-08-18T15:42:51.775 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:42:51.775 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:42:51.776 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:42:51.776 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:31.097661Z", 2024-08-18T15:42:51.776 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:42:51.776 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:42:51.776 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:42:51.776 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:42:51.776 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:42:51.776 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:42:51.776 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:42:51.776 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:42:51.776 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:42:51.776 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:42:51Z" 2024-08-18T15:42:51.776 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:42:51.777 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-08-18T15:42:51.778 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-08-18T15:42:51.778 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:42:51.778 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:42:51.778 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:42:51.833 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:42:51.833 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:42:51.864 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:42:51.863+0000 7f4aaee35880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:42:51.864 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:42:51.864+0000 7f4aaee35880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:42:51.877 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T15:42:51.878 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T15:42:51.878 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T15:42:51.878 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T15:42:51.878 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T15:42:51.878 INFO:teuthology.orchestra.run.smithi134.stdout: "period": "b0d74758-1871-4454-bc0f-e3e4b6c30105", 2024-08-18T15:42:51.878 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:42:51.878 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:42:51.878 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T15:42:51.878 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:42:51.878 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T15:42:51.879 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:42:51.879 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:42:51.879 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:42:51.879 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:42:51.879 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 1, 2024-08-18T15:42:51.879 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:42:51.879 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:42:51.879 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:42:51.879 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:42:51.879 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:42:51.879 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:42:51.880 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T15:42:51.880 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:42:51.880 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:42:51.880 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T15:42:51.880 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:42:51.880 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:42:51.880 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:42:51.881 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T15:42:51.881 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:42:51.881 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:42:51.881 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:42:51.881 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:42:51.881 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T15:42:51.882 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:42:51.882 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:42:51.882 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:42:51.882 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:42:51.882 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:42:51.883 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:42:51.883 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:42:51.883 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:42:51.883 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:42:51.883 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:42:51.883 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:42:51.883 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T15:42:51.884 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:42:51.884 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:42:51.884 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723995751.097661_712.1", 2024-08-18T15:42:51.884 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:42:51.884 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:42:51.885 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:42:51.885 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:31.097661Z", 2024-08-18T15:42:51.885 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:42:51.885 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:42:51.885 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:42:51.885 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T15:42:51.886 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:42:51.886 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T15:42:51.886 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 1, 2024-08-18T15:42:51.886 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 1 2024-08-18T15:42:51.886 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:42:51.886 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T15:42:51Z" 2024-08-18T15:42:51.887 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T15:42:51.888 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-08-18T15:42:51.981 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_delete_notempty ... ok 2024-08-18T15:42:51.981 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=kibgmb-42 2024-08-18T15:42:51.995 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=kibgmb-43 2024-08-18T15:42:52.022 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=kibgmb-44 2024-08-18T15:42:52.046 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:42:52.046 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:42:52.047 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:42:52.093 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:42:52.093 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:42:52.122 INFO:teuthology.orchestra.run.smithi134.stderr:No --period given, using current period=b0d74758-1871-4454-bc0f-e3e4b6c30105 2024-08-18T15:42:52.135 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T15:42:52.135 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:42:52.135 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:42:52.135 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:19:24.506194Z" 2024-08-18T15:42:52.135 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:42:52.135 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:42:52.135 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T15:42:52.136 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T15:42:52.136 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:42:52.136 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:42:52.136 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:42:52.136 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:21:34.952005Z" 2024-08-18T15:42:52.136 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:42:52.136 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:42:52.136 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723995772.038282_740.1", 2024-08-18T15:42:52.136 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:42:52.038282Z" 2024-08-18T15:42:52.136 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:42:52.136 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T15:42:52.137 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-08-18T15:42:52.137 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:42:52.137 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:42:52.138 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:42:52.197 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:42:52.197 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:42:52.229 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:42:52.228+0000 7f08b3815880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:42:52.229 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:42:52.229+0000 7f08b3815880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:42:52.242 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:42:52.243 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:42:52.243 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:42:52.243 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:42:52.243 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:42:52.243 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "b0d74758-1871-4454-bc0f-e3e4b6c30105", 2024-08-18T15:42:52.243 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:42:52.243 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:42:52.243 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:42:52.243 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:42:52.243 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:42:52.244 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:42:52.244 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:42:52.244 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:42:52.244 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:42:52.244 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:42:52.244 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:42:52.244 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:42:52.244 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:42:52.244 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:42:52.244 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:42:52.244 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:42:52.244 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:42:52.244 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:42:52.244 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:42:52.245 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:42:52.245 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:42:52.245 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:42:52.245 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:42:52.245 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:42:52.245 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:42:52.245 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:42:52.245 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:42:52.245 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:42:52.245 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:42:52.245 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:42:52.245 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:42:52.245 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:42:52.245 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:42:52.246 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:42:52.246 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:42:52.246 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:42:52.246 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:42:52.246 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:42:52.246 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:42:52.246 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:42:52.246 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:42:52.246 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:42:52.247 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:42:52.247 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723995771.988164_729.1", 2024-08-18T15:42:52.247 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:42:52.247 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:42:52.247 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:42:52.247 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:51.988164Z", 2024-08-18T15:42:52.247 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:42:52.247 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:42:52.247 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:42:52.247 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:42:52.247 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:42:52.247 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:42:52.247 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:42:52.248 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:42:52.248 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:42:52.248 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:42:52Z" 2024-08-18T15:42:52.248 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:42:52.249 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1723995772.038282_740.1 target=1_1723995771.988164_729.1 2024-08-18T15:42:57.250 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:42:57.250 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:42:57.251 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:42:57.305 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:42:57.306 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:42:57.338 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:42:57.337+0000 7fc0a6f23880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:42:57.338 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:42:57.338+0000 7fc0a6f23880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:42:57.351 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:42:57.351 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:42:57.352 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:42:57.352 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:42:57.352 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:42:57.352 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "b0d74758-1871-4454-bc0f-e3e4b6c30105", 2024-08-18T15:42:57.352 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:42:57.352 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:42:57.352 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:42:57.352 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:42:57.352 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:42:57.352 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:42:57.352 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:42:57.352 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:42:57.353 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:42:57.353 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:42:57.353 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:42:57.353 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:42:57.353 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:42:57.353 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:42:57.353 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:42:57.353 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:42:57.353 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:42:57.353 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:42:57.353 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:42:57.353 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:42:57.353 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:42:57.354 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:42:57.354 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:42:57.354 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:42:57.354 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:42:57.354 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:42:57.354 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:42:57.354 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:42:57.354 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:42:57.354 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:42:57.354 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:42:57.354 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:42:57.354 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:42:57.354 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:42:57.355 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:42:57.355 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:42:57.355 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:42:57.355 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:42:57.355 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:42:57.355 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:42:57.355 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:42:57.355 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:42:57.355 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:42:57.355 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723995772.038282_740.1", 2024-08-18T15:42:57.355 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:42:57.355 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:42:57.355 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:42:57.356 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:52.038282Z", 2024-08-18T15:42:57.356 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:42:57.356 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:42:57.356 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:42:57.356 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:42:57.356 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:42:57.356 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:42:57.356 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:42:57.356 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:42:57.356 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:42:57.356 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:42:57Z" 2024-08-18T15:42:57.356 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:42:57.357 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-08-18T15:42:57.358 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-08-18T15:42:57.358 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:42:57.358 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:42:57.358 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:42:57.410 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:42:57.410 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:42:57.445 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:42:57.444+0000 7efec6536880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:42:57.446 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:42:57.445+0000 7efec6536880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:42:57.461 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T15:42:57.461 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T15:42:57.461 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T15:42:57.461 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T15:42:57.461 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T15:42:57.461 INFO:teuthology.orchestra.run.smithi134.stdout: "period": "b0d74758-1871-4454-bc0f-e3e4b6c30105", 2024-08-18T15:42:57.461 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:42:57.461 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:42:57.462 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T15:42:57.462 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:42:57.462 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T15:42:57.462 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:42:57.462 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:42:57.462 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:42:57.462 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:42:57.462 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 1, 2024-08-18T15:42:57.462 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:42:57.462 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:42:57.463 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:42:57.463 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:42:57.463 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:42:57.463 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:42:57.463 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T15:42:57.463 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:42:57.463 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:42:57.463 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T15:42:57.463 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:42:57.463 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:42:57.463 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:42:57.463 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T15:42:57.463 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:42:57.464 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:42:57.464 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:42:57.464 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:42:57.464 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T15:42:57.464 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:42:57.464 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:42:57.464 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:42:57.464 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:42:57.464 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:42:57.464 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:42:57.464 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:42:57.464 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:42:57.465 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:42:57.465 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:42:57.465 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:42:57.465 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T15:42:57.465 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:42:57.465 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:42:57.465 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723995771.988164_729.1", 2024-08-18T15:42:57.465 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:42:57.465 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:42:57.465 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:42:57.465 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:51.988164Z", 2024-08-18T15:42:57.465 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:42:57.465 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:42:57.466 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:42:57.466 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T15:42:57.466 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:42:57.466 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T15:42:57.466 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 1, 2024-08-18T15:42:57.466 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 1 2024-08-18T15:42:57.466 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:42:57.466 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T15:42:57Z" 2024-08-18T15:42:57.466 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T15:42:57.467 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1723995772.038282_740.1 target=1_1723995771.988164_729.1 2024-08-18T15:43:02.468 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:43:02.469 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:43:02.469 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:43:02.525 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:43:02.525 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:43:02.557 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:43:02.556+0000 7f7a6961f880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:43:02.557 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:43:02.557+0000 7f7a6961f880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:43:02.571 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T15:43:02.571 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T15:43:02.571 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T15:43:02.571 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T15:43:02.571 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T15:43:02.571 INFO:teuthology.orchestra.run.smithi134.stdout: "period": "b0d74758-1871-4454-bc0f-e3e4b6c30105", 2024-08-18T15:43:02.571 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:43:02.571 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:43:02.571 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T15:43:02.572 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:43:02.572 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T15:43:02.572 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:43:02.572 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:43:02.572 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:43:02.572 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:43:02.572 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 1, 2024-08-18T15:43:02.572 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:43:02.572 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:43:02.572 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:43:02.572 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:43:02.573 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:43:02.573 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:43:02.573 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T15:43:02.573 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:43:02.573 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:43:02.573 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T15:43:02.573 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:43:02.573 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:43:02.573 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:43:02.573 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T15:43:02.573 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:43:02.574 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:43:02.574 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:43:02.574 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:43:02.574 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T15:43:02.574 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:43:02.574 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:43:02.574 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:43:02.574 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:43:02.574 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:43:02.574 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:43:02.574 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:43:02.575 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:43:02.575 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:43:02.575 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:43:02.575 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:43:02.575 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T15:43:02.575 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:43:02.575 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:43:02.575 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723995772.038282_740.1", 2024-08-18T15:43:02.575 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:43:02.575 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:43:02.575 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:43:02.575 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:52.038282Z", 2024-08-18T15:43:02.575 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:43:02.575 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:43:02.576 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:43:02.576 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T15:43:02.576 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:43:02.576 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T15:43:02.576 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 1, 2024-08-18T15:43:02.576 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 1 2024-08-18T15:43:02.576 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:43:02.576 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T15:43:02Z" 2024-08-18T15:43:02.576 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T15:43:02.577 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-08-18T15:43:02.577 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2024-08-18T15:43:02.578 INFO:teuthology.orchestra.run:waiting for 300 2024-08-18T15:43:02.578 INFO:tasks.rgw.c1.client.1.smithi134.stdout:2024-08-18T15:43:02.578+0000 7fd5b8247640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 40925) UID: 0 2024-08-18T15:43:02.578 INFO:tasks.rgw.c1.client.1.smithi134.stdout:2024-08-18T15:43:02.578+0000 7fd5bbaa6880 -1 shutting down 2024-08-18T15:43:07.570 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2024-08-18T15:43:08.679 INFO:tasks.rgw.c1.client.1:Stopped 2024-08-18T15:43:08.679 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:43:08.679 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:43:08.680 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --master --cluster c2 --debug-rgw 0 --debug-ms 0 2024-08-18T15:43:08.734 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:43:08.734 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:43:08.753 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:43:08.752+0000 7f39f5d32880 0 NOTICE: overriding master zone: 96bf8502-2f39-4d53-8196-3378c28da3b4 2024-08-18T15:43:08.757 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:43:08.757 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "10090ccf-aad6-416d-9775-df8b7b27345a", 2024-08-18T15:43:08.757 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "test-zone2", 2024-08-18T15:43:08.757 INFO:teuthology.orchestra.run.smithi193.stdout: "domain_root": "test-zone2.rgw.meta:root", 2024-08-18T15:43:08.757 INFO:teuthology.orchestra.run.smithi193.stdout: "control_pool": "test-zone2.rgw.control", 2024-08-18T15:43:08.757 INFO:teuthology.orchestra.run.smithi193.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2024-08-18T15:43:08.757 INFO:teuthology.orchestra.run.smithi193.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2024-08-18T15:43:08.757 INFO:teuthology.orchestra.run.smithi193.stdout: "log_pool": "test-zone2.rgw.log", 2024-08-18T15:43:08.757 INFO:teuthology.orchestra.run.smithi193.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2024-08-18T15:43:08.757 INFO:teuthology.orchestra.run.smithi193.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2024-08-18T15:43:08.757 INFO:teuthology.orchestra.run.smithi193.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2024-08-18T15:43:08.758 INFO:teuthology.orchestra.run.smithi193.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2024-08-18T15:43:08.758 INFO:teuthology.orchestra.run.smithi193.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2024-08-18T15:43:08.758 INFO:teuthology.orchestra.run.smithi193.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2024-08-18T15:43:08.758 INFO:teuthology.orchestra.run.smithi193.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2024-08-18T15:43:08.758 INFO:teuthology.orchestra.run.smithi193.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2024-08-18T15:43:08.758 INFO:teuthology.orchestra.run.smithi193.stdout: "otp_pool": "test-zone2.rgw.otp", 2024-08-18T15:43:08.758 INFO:teuthology.orchestra.run.smithi193.stdout: "system_key": { 2024-08-18T15:43:08.758 INFO:teuthology.orchestra.run.smithi193.stdout: "access_key": "MRX9HX7RYBDXO1UM", 2024-08-18T15:43:08.758 INFO:teuthology.orchestra.run.smithi193.stdout: "secret_key": "hGUsvhUdpGZAYWjQwu3StBaYXJPhuCEj" 2024-08-18T15:43:08.758 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:43:08.758 INFO:teuthology.orchestra.run.smithi193.stdout: "placement_pools": [ 2024-08-18T15:43:08.758 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:43:08.759 INFO:teuthology.orchestra.run.smithi193.stdout: "key": "default-placement", 2024-08-18T15:43:08.759 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:43:08.759 INFO:teuthology.orchestra.run.smithi193.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2024-08-18T15:43:08.759 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_classes": { 2024-08-18T15:43:08.759 INFO:teuthology.orchestra.run.smithi193.stdout: "STANDARD": { 2024-08-18T15:43:08.759 INFO:teuthology.orchestra.run.smithi193.stdout: "data_pool": "test-zone2.rgw.buckets.data", 2024-08-18T15:43:08.759 INFO:teuthology.orchestra.run.smithi193.stdout: "compression_type": "random" 2024-08-18T15:43:08.759 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:43:08.759 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:43:08.759 INFO:teuthology.orchestra.run.smithi193.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2024-08-18T15:43:08.759 INFO:teuthology.orchestra.run.smithi193.stdout: "index_type": 0, 2024-08-18T15:43:08.760 INFO:teuthology.orchestra.run.smithi193.stdout: "inline_data": true 2024-08-18T15:43:08.760 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:43:08.760 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:43:08.760 INFO:teuthology.orchestra.run.smithi193.stdout: ], 2024-08-18T15:43:08.760 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_id": "2827c6f0-bf0d-4836-944c-9e7fcf22457b", 2024-08-18T15:43:08.760 INFO:teuthology.orchestra.run.smithi193.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2024-08-18T15:43:08.760 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:43:08.761 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:43:08.761 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:43:08.762 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2024-08-18T15:43:08.810 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:43:08.810 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:43:08.851 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:43:08.851+0000 7febe0f29880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:43:08.852 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:43:08.851+0000 7febe0f29880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:43:08.859 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:43:08.859 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "64309209-7567-46e5-8881-65783d84cf3c", 2024-08-18T15:43:08.859 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 1, 2024-08-18T15:43:08.859 INFO:teuthology.orchestra.run.smithi193.stdout: "predecessor_uuid": "b0d74758-1871-4454-bc0f-e3e4b6c30105", 2024-08-18T15:43:08.859 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": [ 2024-08-18T15:43:08.860 INFO:teuthology.orchestra.run.smithi193.stdout: "1_1723994364.506194_23.1", 2024-08-18T15:43:08.860 INFO:teuthology.orchestra.run.smithi193.stdout: "", 2024-08-18T15:43:08.860 INFO:teuthology.orchestra.run.smithi193.stdout: "1_1723994494.952005_96.1", 2024-08-18T15:43:08.860 INFO:teuthology.orchestra.run.smithi193.stdout: "1_1723995772.038282_740.1" 2024-08-18T15:43:08.860 INFO:teuthology.orchestra.run.smithi193.stdout: ], 2024-08-18T15:43:08.860 INFO:teuthology.orchestra.run.smithi193.stdout: "period_map": { 2024-08-18T15:43:08.860 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "64309209-7567-46e5-8881-65783d84cf3c", 2024-08-18T15:43:08.860 INFO:teuthology.orchestra.run.smithi193.stdout: "zonegroups": [ 2024-08-18T15:43:08.860 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:43:08.860 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "test-zonegroup", 2024-08-18T15:43:08.860 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "test-zonegroup", 2024-08-18T15:43:08.860 INFO:teuthology.orchestra.run.smithi193.stdout: "api_name": "", 2024-08-18T15:43:08.860 INFO:teuthology.orchestra.run.smithi193.stdout: "is_master": true, 2024-08-18T15:43:08.860 INFO:teuthology.orchestra.run.smithi193.stdout: "endpoints": [ 2024-08-18T15:43:08.861 INFO:teuthology.orchestra.run.smithi193.stdout: "http://smithi134.front.sepia.ceph.com:8000" 2024-08-18T15:43:08.861 INFO:teuthology.orchestra.run.smithi193.stdout: ], 2024-08-18T15:43:08.861 INFO:teuthology.orchestra.run.smithi193.stdout: "hostnames": [], 2024-08-18T15:43:08.861 INFO:teuthology.orchestra.run.smithi193.stdout: "hostnames_s3website": [], 2024-08-18T15:43:08.861 INFO:teuthology.orchestra.run.smithi193.stdout: "master_zone": "10090ccf-aad6-416d-9775-df8b7b27345a", 2024-08-18T15:43:08.861 INFO:teuthology.orchestra.run.smithi193.stdout: "zones": [ 2024-08-18T15:43:08.861 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:43:08.861 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T15:43:08.861 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "test-zone3", 2024-08-18T15:43:08.861 INFO:teuthology.orchestra.run.smithi193.stdout: "endpoints": [ 2024-08-18T15:43:08.861 INFO:teuthology.orchestra.run.smithi193.stdout: "http://smithi134.front.sepia.ceph.com:8001" 2024-08-18T15:43:08.861 INFO:teuthology.orchestra.run.smithi193.stdout: ], 2024-08-18T15:43:08.861 INFO:teuthology.orchestra.run.smithi193.stdout: "log_meta": false, 2024-08-18T15:43:08.861 INFO:teuthology.orchestra.run.smithi193.stdout: "log_data": true, 2024-08-18T15:43:08.862 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket_index_max_shards": 11, 2024-08-18T15:43:08.862 INFO:teuthology.orchestra.run.smithi193.stdout: "read_only": false, 2024-08-18T15:43:08.862 INFO:teuthology.orchestra.run.smithi193.stdout: "tier_type": "", 2024-08-18T15:43:08.862 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_from_all": true, 2024-08-18T15:43:08.862 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_from": [], 2024-08-18T15:43:08.862 INFO:teuthology.orchestra.run.smithi193.stdout: "redirect_zone": "", 2024-08-18T15:43:08.862 INFO:teuthology.orchestra.run.smithi193.stdout: "supported_features": [ 2024-08-18T15:43:08.862 INFO:teuthology.orchestra.run.smithi193.stdout: "compress-encrypted", 2024-08-18T15:43:08.862 INFO:teuthology.orchestra.run.smithi193.stdout: "resharding" 2024-08-18T15:43:08.862 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:43:08.862 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:43:08.862 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:43:08.862 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "10090ccf-aad6-416d-9775-df8b7b27345a", 2024-08-18T15:43:08.863 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "test-zone2", 2024-08-18T15:43:08.863 INFO:teuthology.orchestra.run.smithi193.stdout: "endpoints": [ 2024-08-18T15:43:08.863 INFO:teuthology.orchestra.run.smithi193.stdout: "http://smithi193.front.sepia.ceph.com:8000" 2024-08-18T15:43:08.863 INFO:teuthology.orchestra.run.smithi193.stdout: ], 2024-08-18T15:43:08.863 INFO:teuthology.orchestra.run.smithi193.stdout: "log_meta": false, 2024-08-18T15:43:08.863 INFO:teuthology.orchestra.run.smithi193.stdout: "log_data": true, 2024-08-18T15:43:08.863 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket_index_max_shards": 11, 2024-08-18T15:43:08.863 INFO:teuthology.orchestra.run.smithi193.stdout: "read_only": false, 2024-08-18T15:43:08.863 INFO:teuthology.orchestra.run.smithi193.stdout: "tier_type": "", 2024-08-18T15:43:08.863 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_from_all": true, 2024-08-18T15:43:08.863 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_from": [], 2024-08-18T15:43:08.863 INFO:teuthology.orchestra.run.smithi193.stdout: "redirect_zone": "", 2024-08-18T15:43:08.863 INFO:teuthology.orchestra.run.smithi193.stdout: "supported_features": [ 2024-08-18T15:43:08.863 INFO:teuthology.orchestra.run.smithi193.stdout: "compress-encrypted", 2024-08-18T15:43:08.864 INFO:teuthology.orchestra.run.smithi193.stdout: "resharding" 2024-08-18T15:43:08.864 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:43:08.864 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:43:08.864 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:43:08.864 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:43:08.864 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "test-zone1", 2024-08-18T15:43:08.864 INFO:teuthology.orchestra.run.smithi193.stdout: "endpoints": [ 2024-08-18T15:43:08.864 INFO:teuthology.orchestra.run.smithi193.stdout: "http://smithi134.front.sepia.ceph.com:8000" 2024-08-18T15:43:08.864 INFO:teuthology.orchestra.run.smithi193.stdout: ], 2024-08-18T15:43:08.864 INFO:teuthology.orchestra.run.smithi193.stdout: "log_meta": false, 2024-08-18T15:43:08.864 INFO:teuthology.orchestra.run.smithi193.stdout: "log_data": true, 2024-08-18T15:43:08.864 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket_index_max_shards": 11, 2024-08-18T15:43:08.864 INFO:teuthology.orchestra.run.smithi193.stdout: "read_only": false, 2024-08-18T15:43:08.865 INFO:teuthology.orchestra.run.smithi193.stdout: "tier_type": "", 2024-08-18T15:43:08.865 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_from_all": true, 2024-08-18T15:43:08.865 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_from": [], 2024-08-18T15:43:08.865 INFO:teuthology.orchestra.run.smithi193.stdout: "redirect_zone": "", 2024-08-18T15:43:08.865 INFO:teuthology.orchestra.run.smithi193.stdout: "supported_features": [ 2024-08-18T15:43:08.865 INFO:teuthology.orchestra.run.smithi193.stdout: "compress-encrypted", 2024-08-18T15:43:08.865 INFO:teuthology.orchestra.run.smithi193.stdout: "resharding" 2024-08-18T15:43:08.865 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:43:08.865 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:43:08.865 INFO:teuthology.orchestra.run.smithi193.stdout: ], 2024-08-18T15:43:08.865 INFO:teuthology.orchestra.run.smithi193.stdout: "placement_targets": [ 2024-08-18T15:43:08.865 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:43:08.865 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "default-placement", 2024-08-18T15:43:08.866 INFO:teuthology.orchestra.run.smithi193.stdout: "tags": [], 2024-08-18T15:43:08.866 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_classes": [ 2024-08-18T15:43:08.866 INFO:teuthology.orchestra.run.smithi193.stdout: "STANDARD" 2024-08-18T15:43:08.866 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:43:08.866 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:43:08.866 INFO:teuthology.orchestra.run.smithi193.stdout: ], 2024-08-18T15:43:08.866 INFO:teuthology.orchestra.run.smithi193.stdout: "default_placement": "default-placement", 2024-08-18T15:43:08.866 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_id": "2827c6f0-bf0d-4836-944c-9e7fcf22457b", 2024-08-18T15:43:08.866 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_policy": { 2024-08-18T15:43:08.866 INFO:teuthology.orchestra.run.smithi193.stdout: "groups": [] 2024-08-18T15:43:08.866 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:43:08.866 INFO:teuthology.orchestra.run.smithi193.stdout: "enabled_features": [ 2024-08-18T15:43:08.866 INFO:teuthology.orchestra.run.smithi193.stdout: "resharding" 2024-08-18T15:43:08.867 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:43:08.867 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:43:08.867 INFO:teuthology.orchestra.run.smithi193.stdout: ], 2024-08-18T15:43:08.867 INFO:teuthology.orchestra.run.smithi193.stdout: "short_zone_ids": [ 2024-08-18T15:43:08.867 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:43:08.867 INFO:teuthology.orchestra.run.smithi193.stdout: "key": "018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T15:43:08.867 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 462036791 2024-08-18T15:43:08.867 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:43:08.871 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:43:08.871 INFO:teuthology.orchestra.run.smithi193.stdout: "key": "10090ccf-aad6-416d-9775-df8b7b27345a", 2024-08-18T15:43:08.872 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 874289624 2024-08-18T15:43:08.872 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:43:08.872 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:43:08.872 INFO:teuthology.orchestra.run.smithi193.stdout: "key": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:43:08.872 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 1721399886 2024-08-18T15:43:08.872 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:43:08.872 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:43:08.872 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:43:08.872 INFO:teuthology.orchestra.run.smithi193.stdout: "master_zonegroup": "test-zonegroup", 2024-08-18T15:43:08.872 INFO:teuthology.orchestra.run.smithi193.stdout: "master_zone": "10090ccf-aad6-416d-9775-df8b7b27345a", 2024-08-18T15:43:08.872 INFO:teuthology.orchestra.run.smithi193.stdout: "period_config": { 2024-08-18T15:43:08.872 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket_quota": { 2024-08-18T15:43:08.872 INFO:teuthology.orchestra.run.smithi193.stdout: "enabled": false, 2024-08-18T15:43:08.873 INFO:teuthology.orchestra.run.smithi193.stdout: "check_on_raw": false, 2024-08-18T15:43:08.873 INFO:teuthology.orchestra.run.smithi193.stdout: "max_size": -1, 2024-08-18T15:43:08.873 INFO:teuthology.orchestra.run.smithi193.stdout: "max_size_kb": 0, 2024-08-18T15:43:08.873 INFO:teuthology.orchestra.run.smithi193.stdout: "max_objects": -1 2024-08-18T15:43:08.873 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:43:08.873 INFO:teuthology.orchestra.run.smithi193.stdout: "user_quota": { 2024-08-18T15:43:08.873 INFO:teuthology.orchestra.run.smithi193.stdout: "enabled": false, 2024-08-18T15:43:08.873 INFO:teuthology.orchestra.run.smithi193.stdout: "check_on_raw": false, 2024-08-18T15:43:08.873 INFO:teuthology.orchestra.run.smithi193.stdout: "max_size": -1, 2024-08-18T15:43:08.873 INFO:teuthology.orchestra.run.smithi193.stdout: "max_size_kb": 0, 2024-08-18T15:43:08.873 INFO:teuthology.orchestra.run.smithi193.stdout: "max_objects": -1 2024-08-18T15:43:08.873 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:43:08.874 INFO:teuthology.orchestra.run.smithi193.stdout: "user_ratelimit": { 2024-08-18T15:43:08.874 INFO:teuthology.orchestra.run.smithi193.stdout: "max_read_ops": 0, 2024-08-18T15:43:08.874 INFO:teuthology.orchestra.run.smithi193.stdout: "max_write_ops": 0, 2024-08-18T15:43:08.874 INFO:teuthology.orchestra.run.smithi193.stdout: "max_read_bytes": 0, 2024-08-18T15:43:08.874 INFO:teuthology.orchestra.run.smithi193.stdout: "max_write_bytes": 0, 2024-08-18T15:43:08.874 INFO:teuthology.orchestra.run.smithi193.stdout: "enabled": false 2024-08-18T15:43:08.874 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:43:08.874 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket_ratelimit": { 2024-08-18T15:43:08.874 INFO:teuthology.orchestra.run.smithi193.stdout: "max_read_ops": 0, 2024-08-18T15:43:08.874 INFO:teuthology.orchestra.run.smithi193.stdout: "max_write_ops": 0, 2024-08-18T15:43:08.874 INFO:teuthology.orchestra.run.smithi193.stdout: "max_read_bytes": 0, 2024-08-18T15:43:08.874 INFO:teuthology.orchestra.run.smithi193.stdout: "max_write_bytes": 0, 2024-08-18T15:43:08.874 INFO:teuthology.orchestra.run.smithi193.stdout: "enabled": false 2024-08-18T15:43:08.875 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:43:08.875 INFO:teuthology.orchestra.run.smithi193.stdout: "anonymous_ratelimit": { 2024-08-18T15:43:08.875 INFO:teuthology.orchestra.run.smithi193.stdout: "max_read_ops": 0, 2024-08-18T15:43:08.875 INFO:teuthology.orchestra.run.smithi193.stdout: "max_write_ops": 0, 2024-08-18T15:43:08.875 INFO:teuthology.orchestra.run.smithi193.stdout: "max_read_bytes": 0, 2024-08-18T15:43:08.875 INFO:teuthology.orchestra.run.smithi193.stdout: "max_write_bytes": 0, 2024-08-18T15:43:08.875 INFO:teuthology.orchestra.run.smithi193.stdout: "enabled": false 2024-08-18T15:43:08.875 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:43:08.875 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:43:08.875 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_id": "2827c6f0-bf0d-4836-944c-9e7fcf22457b", 2024-08-18T15:43:08.875 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_name": "test-realm", 2024-08-18T15:43:08.875 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 3 2024-08-18T15:43:08.875 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:43:08.876 INFO:rgw_multi.tests:Set master zone=test-zone2, waiting 60s for reconfiguration.. 2024-08-18T15:43:13.175 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2024-08-18T15:44:08.878 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=kibgmb-45 2024-08-18T15:44:08.913 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=kibgmb-46 2024-08-18T15:44:08.932 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:44:08.932 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:44:08.933 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:44:08.985 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:44:08.986 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:44:09.018 INFO:teuthology.orchestra.run.smithi193.stderr:No --period given, using current period=64309209-7567-46e5-8881-65783d84cf3c 2024-08-18T15:44:09.032 INFO:teuthology.orchestra.run.smithi193.stdout:[ 2024-08-18T15:44:09.032 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:44:09.032 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:44:09.032 INFO:teuthology.orchestra.run.smithi193.stdout: "last_update": "0.000000" 2024-08-18T15:44:09.032 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:44:09.033 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:44:09.033 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:44:09.033 INFO:teuthology.orchestra.run.smithi193.stdout: "last_update": "0.000000" 2024-08-18T15:44:09.033 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:44:09.033 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:44:09.034 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:44:09.034 INFO:teuthology.orchestra.run.smithi193.stdout: "last_update": "0.000000" 2024-08-18T15:44:09.034 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:44:09.034 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:44:09.034 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723995848.930494_187.1", 2024-08-18T15:44:09.035 INFO:teuthology.orchestra.run.smithi193.stdout: "last_update": "2024-08-18T15:44:08.930494Z" 2024-08-18T15:44:09.035 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:44:09.035 INFO:teuthology.orchestra.run.smithi193.stdout:] 2024-08-18T15:44:09.036 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone1 2024-08-18T15:44:09.037 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:44:09.037 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:44:09.037 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:44:09.093 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:44:09.094 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:44:09.125 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:44:09.123+0000 7f6cde6bb880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:44:09.126 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:44:09.125+0000 7f6cde6bb880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:44:09.139 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T15:44:09.139 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T15:44:09.139 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T15:44:09.139 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T15:44:09.139 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T15:44:09.139 INFO:teuthology.orchestra.run.smithi134.stdout: "period": "64309209-7567-46e5-8881-65783d84cf3c", 2024-08-18T15:44:09.139 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 3 2024-08-18T15:44:09.139 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:44:09.139 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T15:44:09.140 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:44:09.140 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T15:44:09.140 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:44:09.140 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 0, 2024-08-18T15:44:09.140 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "bucket.instance:kibgmb-4:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.4", 2024-08-18T15:44:09.140 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:44:09.140 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 87, 2024-08-18T15:44:09.140 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 39, 2024-08-18T15:44:09.140 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T15:44:09.140 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 0 2024-08-18T15:44:09.140 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:44:09.140 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:44:09.140 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:44:09.141 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T15:44:09.141 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:44:09.141 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:44:09.141 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T15:44:09.141 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:44:09.141 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:44:09.142 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:44:09.142 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T15:44:09.142 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 3 2024-08-18T15:44:09.142 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:44:09.142 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:44:09.142 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:44:09.142 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T15:44:09.142 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:44:09.142 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:44:09.142 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T15:44:09.142 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:44:09.142 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:44:09.143 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:44:09.143 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T15:44:09.143 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 3 2024-08-18T15:44:09.143 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:44:09.143 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:44:09.143 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:44:09.143 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T15:44:09.143 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:44:09.143 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:44:09.143 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T15:44:09.143 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:44:09.143 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:44:09.143 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:44:09.144 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T15:44:09.144 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 3 2024-08-18T15:44:09.144 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:44:09.144 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:44:09.144 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T15:44:09.144 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:44:09.144 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T15:44:09.144 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 87, 2024-08-18T15:44:09.144 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 39 2024-08-18T15:44:09.145 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:44:09.145 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T15:44:09Z" 2024-08-18T15:44:09.145 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T15:44:09.146 WARNING:rgw_multi.tests:zone test-zone1 behind master: shard=3 master=1_1723995848.930494_187.1 target= 2024-08-18T15:44:14.147 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:44:14.147 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:44:14.148 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:44:14.205 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:44:14.205 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:44:14.236 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:44:14.236+0000 7ff1c8d42880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:44:14.237 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:44:14.237+0000 7ff1c8d42880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:44:14.250 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T15:44:14.250 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T15:44:14.250 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T15:44:14.250 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T15:44:14.251 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T15:44:14.251 INFO:teuthology.orchestra.run.smithi134.stdout: "period": "64309209-7567-46e5-8881-65783d84cf3c", 2024-08-18T15:44:14.251 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 3 2024-08-18T15:44:14.251 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:44:14.251 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T15:44:14.251 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:44:14.251 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T15:44:14.251 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:44:14.251 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 0, 2024-08-18T15:44:14.251 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "bucket.instance:kibgmb-4:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.4", 2024-08-18T15:44:14.251 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:44:14.251 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 87, 2024-08-18T15:44:14.251 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 39, 2024-08-18T15:44:14.252 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T15:44:14.252 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 0 2024-08-18T15:44:14.252 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:44:14.252 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:44:14.252 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:44:14.252 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T15:44:14.252 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:44:14.252 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:44:14.252 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T15:44:14.252 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:44:14.252 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:44:14.252 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:44:14.252 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T15:44:14.252 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 3 2024-08-18T15:44:14.253 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:44:14.253 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:44:14.253 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:44:14.253 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T15:44:14.253 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:44:14.253 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:44:14.253 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T15:44:14.253 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:44:14.253 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:44:14.253 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:44:14.253 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T15:44:14.253 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 3 2024-08-18T15:44:14.253 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:44:14.254 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:44:14.254 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:44:14.254 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T15:44:14.254 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:44:14.254 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:44:14.254 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723995848.930494_187.1", 2024-08-18T15:44:14.254 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:44:14.254 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:44:14.254 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:44:14.254 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:44:08.930494Z", 2024-08-18T15:44:14.254 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 3 2024-08-18T15:44:14.254 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:44:14.254 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:44:14.254 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T15:44:14.255 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:44:14.255 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T15:44:14.255 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 87, 2024-08-18T15:44:14.255 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 39 2024-08-18T15:44:14.255 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:44:14.255 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T15:44:14Z" 2024-08-18T15:44:14.255 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T15:44:14.256 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone1 2024-08-18T15:44:14.256 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:44:14.256 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'modify', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:44:14.256 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone modify --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --master --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T15:44:14.303 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:44:14.303 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:44:14.319 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:44:14.319+0000 7fc08172d880 0 NOTICE: overriding master zone: 10090ccf-aad6-416d-9775-df8b7b27345a 2024-08-18T15:44:14.323 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T15:44:14.323 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:44:14.323 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "test-zone1", 2024-08-18T15:44:14.323 INFO:teuthology.orchestra.run.smithi134.stdout: "domain_root": "test-zone1.rgw.meta:root", 2024-08-18T15:44:14.323 INFO:teuthology.orchestra.run.smithi134.stdout: "control_pool": "test-zone1.rgw.control", 2024-08-18T15:44:14.323 INFO:teuthology.orchestra.run.smithi134.stdout: "gc_pool": "test-zone1.rgw.log:gc", 2024-08-18T15:44:14.323 INFO:teuthology.orchestra.run.smithi134.stdout: "lc_pool": "test-zone1.rgw.log:lc", 2024-08-18T15:44:14.324 INFO:teuthology.orchestra.run.smithi134.stdout: "log_pool": "test-zone1.rgw.log", 2024-08-18T15:44:14.324 INFO:teuthology.orchestra.run.smithi134.stdout: "intent_log_pool": "test-zone1.rgw.log:intent", 2024-08-18T15:44:14.324 INFO:teuthology.orchestra.run.smithi134.stdout: "usage_log_pool": "test-zone1.rgw.log:usage", 2024-08-18T15:44:14.324 INFO:teuthology.orchestra.run.smithi134.stdout: "roles_pool": "test-zone1.rgw.meta:roles", 2024-08-18T15:44:14.324 INFO:teuthology.orchestra.run.smithi134.stdout: "reshard_pool": "test-zone1.rgw.log:reshard", 2024-08-18T15:44:14.324 INFO:teuthology.orchestra.run.smithi134.stdout: "user_keys_pool": "test-zone1.rgw.meta:users.keys", 2024-08-18T15:44:14.324 INFO:teuthology.orchestra.run.smithi134.stdout: "user_email_pool": "test-zone1.rgw.meta:users.email", 2024-08-18T15:44:14.324 INFO:teuthology.orchestra.run.smithi134.stdout: "user_swift_pool": "test-zone1.rgw.meta:users.swift", 2024-08-18T15:44:14.324 INFO:teuthology.orchestra.run.smithi134.stdout: "user_uid_pool": "test-zone1.rgw.meta:users.uid", 2024-08-18T15:44:14.324 INFO:teuthology.orchestra.run.smithi134.stdout: "otp_pool": "test-zone1.rgw.otp", 2024-08-18T15:44:14.324 INFO:teuthology.orchestra.run.smithi134.stdout: "system_key": { 2024-08-18T15:44:14.324 INFO:teuthology.orchestra.run.smithi134.stdout: "access_key": "MRX9HX7RYBDXO1UM", 2024-08-18T15:44:14.324 INFO:teuthology.orchestra.run.smithi134.stdout: "secret_key": "hGUsvhUdpGZAYWjQwu3StBaYXJPhuCEj" 2024-08-18T15:44:14.324 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:44:14.325 INFO:teuthology.orchestra.run.smithi134.stdout: "placement_pools": [ 2024-08-18T15:44:14.325 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:44:14.325 INFO:teuthology.orchestra.run.smithi134.stdout: "key": "default-placement", 2024-08-18T15:44:14.325 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:44:14.325 INFO:teuthology.orchestra.run.smithi134.stdout: "index_pool": "test-zone1.rgw.buckets.index", 2024-08-18T15:44:14.325 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_classes": { 2024-08-18T15:44:14.325 INFO:teuthology.orchestra.run.smithi134.stdout: "STANDARD": { 2024-08-18T15:44:14.325 INFO:teuthology.orchestra.run.smithi134.stdout: "data_pool": "test-zone1.rgw.buckets.data", 2024-08-18T15:44:14.325 INFO:teuthology.orchestra.run.smithi134.stdout: "compression_type": "random" 2024-08-18T15:44:14.325 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:44:14.325 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:44:14.325 INFO:teuthology.orchestra.run.smithi134.stdout: "data_extra_pool": "test-zone1.rgw.buckets.non-ec", 2024-08-18T15:44:14.325 INFO:teuthology.orchestra.run.smithi134.stdout: "index_type": 0, 2024-08-18T15:44:14.325 INFO:teuthology.orchestra.run.smithi134.stdout: "inline_data": true 2024-08-18T15:44:14.326 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:44:14.326 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:44:14.326 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T15:44:14.326 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_id": "2827c6f0-bf0d-4836-944c-9e7fcf22457b", 2024-08-18T15:44:14.326 INFO:teuthology.orchestra.run.smithi134.stdout: "notif_pool": "test-zone1.rgw.log:notif" 2024-08-18T15:44:14.326 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T15:44:14.327 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:44:14.327 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:44:14.327 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T15:44:14.373 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:44:14.373 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:44:14.412 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:44:14.411+0000 7f6223f3e880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:44:14.412 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:44:14.412+0000 7f6223f3e880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:44:14.419 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T15:44:14.419 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T15:44:14.419 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 1, 2024-08-18T15:44:14.419 INFO:teuthology.orchestra.run.smithi134.stdout: "predecessor_uuid": "64309209-7567-46e5-8881-65783d84cf3c", 2024-08-18T15:44:14.420 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": [ 2024-08-18T15:44:14.420 INFO:teuthology.orchestra.run.smithi134.stdout: "", 2024-08-18T15:44:14.420 INFO:teuthology.orchestra.run.smithi134.stdout: "", 2024-08-18T15:44:14.420 INFO:teuthology.orchestra.run.smithi134.stdout: "", 2024-08-18T15:44:14.420 INFO:teuthology.orchestra.run.smithi134.stdout: "1_1723995848.930494_187.1" 2024-08-18T15:44:14.420 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T15:44:14.420 INFO:teuthology.orchestra.run.smithi134.stdout: "period_map": { 2024-08-18T15:44:14.420 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T15:44:14.420 INFO:teuthology.orchestra.run.smithi134.stdout: "zonegroups": [ 2024-08-18T15:44:14.420 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:44:14.420 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "test-zonegroup", 2024-08-18T15:44:14.420 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "test-zonegroup", 2024-08-18T15:44:14.420 INFO:teuthology.orchestra.run.smithi134.stdout: "api_name": "", 2024-08-18T15:44:14.421 INFO:teuthology.orchestra.run.smithi134.stdout: "is_master": true, 2024-08-18T15:44:14.421 INFO:teuthology.orchestra.run.smithi134.stdout: "endpoints": [ 2024-08-18T15:44:14.421 INFO:teuthology.orchestra.run.smithi134.stdout: "http://smithi134.front.sepia.ceph.com:8000" 2024-08-18T15:44:14.421 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T15:44:14.421 INFO:teuthology.orchestra.run.smithi134.stdout: "hostnames": [], 2024-08-18T15:44:14.421 INFO:teuthology.orchestra.run.smithi134.stdout: "hostnames_s3website": [], 2024-08-18T15:44:14.421 INFO:teuthology.orchestra.run.smithi134.stdout: "master_zone": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:44:14.421 INFO:teuthology.orchestra.run.smithi134.stdout: "zones": [ 2024-08-18T15:44:14.421 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:44:14.421 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T15:44:14.421 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "test-zone3", 2024-08-18T15:44:14.421 INFO:teuthology.orchestra.run.smithi134.stdout: "endpoints": [ 2024-08-18T15:44:14.421 INFO:teuthology.orchestra.run.smithi134.stdout: "http://smithi134.front.sepia.ceph.com:8001" 2024-08-18T15:44:14.422 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T15:44:14.422 INFO:teuthology.orchestra.run.smithi134.stdout: "log_meta": false, 2024-08-18T15:44:14.422 INFO:teuthology.orchestra.run.smithi134.stdout: "log_data": true, 2024-08-18T15:44:14.422 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_index_max_shards": 11, 2024-08-18T15:44:14.422 INFO:teuthology.orchestra.run.smithi134.stdout: "read_only": false, 2024-08-18T15:44:14.422 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_type": "", 2024-08-18T15:44:14.422 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from_all": true, 2024-08-18T15:44:14.422 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from": [], 2024-08-18T15:44:14.422 INFO:teuthology.orchestra.run.smithi134.stdout: "redirect_zone": "", 2024-08-18T15:44:14.422 INFO:teuthology.orchestra.run.smithi134.stdout: "supported_features": [ 2024-08-18T15:44:14.422 INFO:teuthology.orchestra.run.smithi134.stdout: "compress-encrypted", 2024-08-18T15:44:14.422 INFO:teuthology.orchestra.run.smithi134.stdout: "resharding" 2024-08-18T15:44:14.423 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T15:44:14.423 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:44:14.423 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:44:14.423 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "10090ccf-aad6-416d-9775-df8b7b27345a", 2024-08-18T15:44:14.423 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "test-zone2", 2024-08-18T15:44:14.423 INFO:teuthology.orchestra.run.smithi134.stdout: "endpoints": [ 2024-08-18T15:44:14.423 INFO:teuthology.orchestra.run.smithi134.stdout: "http://smithi193.front.sepia.ceph.com:8000" 2024-08-18T15:44:14.423 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T15:44:14.423 INFO:teuthology.orchestra.run.smithi134.stdout: "log_meta": false, 2024-08-18T15:44:14.423 INFO:teuthology.orchestra.run.smithi134.stdout: "log_data": true, 2024-08-18T15:44:14.423 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_index_max_shards": 11, 2024-08-18T15:44:14.423 INFO:teuthology.orchestra.run.smithi134.stdout: "read_only": false, 2024-08-18T15:44:14.423 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_type": "", 2024-08-18T15:44:14.424 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from_all": true, 2024-08-18T15:44:14.424 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from": [], 2024-08-18T15:44:14.424 INFO:teuthology.orchestra.run.smithi134.stdout: "redirect_zone": "", 2024-08-18T15:44:14.424 INFO:teuthology.orchestra.run.smithi134.stdout: "supported_features": [ 2024-08-18T15:44:14.424 INFO:teuthology.orchestra.run.smithi134.stdout: "compress-encrypted", 2024-08-18T15:44:14.424 INFO:teuthology.orchestra.run.smithi134.stdout: "resharding" 2024-08-18T15:44:14.424 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T15:44:14.424 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:44:14.424 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:44:14.424 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:44:14.424 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "test-zone1", 2024-08-18T15:44:14.424 INFO:teuthology.orchestra.run.smithi134.stdout: "endpoints": [ 2024-08-18T15:44:14.424 INFO:teuthology.orchestra.run.smithi134.stdout: "http://smithi134.front.sepia.ceph.com:8000" 2024-08-18T15:44:14.425 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T15:44:14.425 INFO:teuthology.orchestra.run.smithi134.stdout: "log_meta": false, 2024-08-18T15:44:14.425 INFO:teuthology.orchestra.run.smithi134.stdout: "log_data": true, 2024-08-18T15:44:14.425 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_index_max_shards": 11, 2024-08-18T15:44:14.425 INFO:teuthology.orchestra.run.smithi134.stdout: "read_only": false, 2024-08-18T15:44:14.425 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_type": "", 2024-08-18T15:44:14.425 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from_all": true, 2024-08-18T15:44:14.425 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from": [], 2024-08-18T15:44:14.425 INFO:teuthology.orchestra.run.smithi134.stdout: "redirect_zone": "", 2024-08-18T15:44:14.425 INFO:teuthology.orchestra.run.smithi134.stdout: "supported_features": [ 2024-08-18T15:44:14.425 INFO:teuthology.orchestra.run.smithi134.stdout: "compress-encrypted", 2024-08-18T15:44:14.425 INFO:teuthology.orchestra.run.smithi134.stdout: "resharding" 2024-08-18T15:44:14.425 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T15:44:14.425 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:44:14.426 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T15:44:14.426 INFO:teuthology.orchestra.run.smithi134.stdout: "placement_targets": [ 2024-08-18T15:44:14.426 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:44:14.426 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "default-placement", 2024-08-18T15:44:14.426 INFO:teuthology.orchestra.run.smithi134.stdout: "tags": [], 2024-08-18T15:44:14.426 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_classes": [ 2024-08-18T15:44:14.426 INFO:teuthology.orchestra.run.smithi134.stdout: "STANDARD" 2024-08-18T15:44:14.426 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T15:44:14.426 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:44:14.426 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T15:44:14.426 INFO:teuthology.orchestra.run.smithi134.stdout: "default_placement": "default-placement", 2024-08-18T15:44:14.426 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_id": "2827c6f0-bf0d-4836-944c-9e7fcf22457b", 2024-08-18T15:44:14.426 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_policy": { 2024-08-18T15:44:14.427 INFO:teuthology.orchestra.run.smithi134.stdout: "groups": [] 2024-08-18T15:44:14.427 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:44:14.427 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled_features": [ 2024-08-18T15:44:14.427 INFO:teuthology.orchestra.run.smithi134.stdout: "resharding" 2024-08-18T15:44:14.427 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T15:44:14.427 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:44:14.427 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T15:44:14.427 INFO:teuthology.orchestra.run.smithi134.stdout: "short_zone_ids": [ 2024-08-18T15:44:14.427 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:44:14.427 INFO:teuthology.orchestra.run.smithi134.stdout: "key": "018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T15:44:14.427 INFO:teuthology.orchestra.run.smithi134.stdout: "val": 462036791 2024-08-18T15:44:14.427 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:44:14.427 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:44:14.431 INFO:teuthology.orchestra.run.smithi134.stdout: "key": "10090ccf-aad6-416d-9775-df8b7b27345a", 2024-08-18T15:44:14.431 INFO:teuthology.orchestra.run.smithi134.stdout: "val": 874289624 2024-08-18T15:44:14.431 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:44:14.431 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:44:14.431 INFO:teuthology.orchestra.run.smithi134.stdout: "key": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:44:14.431 INFO:teuthology.orchestra.run.smithi134.stdout: "val": 1721399886 2024-08-18T15:44:14.431 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:44:14.432 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T15:44:14.432 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:44:14.432 INFO:teuthology.orchestra.run.smithi134.stdout: "master_zonegroup": "test-zonegroup", 2024-08-18T15:44:14.432 INFO:teuthology.orchestra.run.smithi134.stdout: "master_zone": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:44:14.432 INFO:teuthology.orchestra.run.smithi134.stdout: "period_config": { 2024-08-18T15:44:14.432 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_quota": { 2024-08-18T15:44:14.432 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false, 2024-08-18T15:44:14.432 INFO:teuthology.orchestra.run.smithi134.stdout: "check_on_raw": false, 2024-08-18T15:44:14.432 INFO:teuthology.orchestra.run.smithi134.stdout: "max_size": -1, 2024-08-18T15:44:14.432 INFO:teuthology.orchestra.run.smithi134.stdout: "max_size_kb": 0, 2024-08-18T15:44:14.432 INFO:teuthology.orchestra.run.smithi134.stdout: "max_objects": -1 2024-08-18T15:44:14.432 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:44:14.432 INFO:teuthology.orchestra.run.smithi134.stdout: "user_quota": { 2024-08-18T15:44:14.433 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false, 2024-08-18T15:44:14.433 INFO:teuthology.orchestra.run.smithi134.stdout: "check_on_raw": false, 2024-08-18T15:44:14.433 INFO:teuthology.orchestra.run.smithi134.stdout: "max_size": -1, 2024-08-18T15:44:14.433 INFO:teuthology.orchestra.run.smithi134.stdout: "max_size_kb": 0, 2024-08-18T15:44:14.433 INFO:teuthology.orchestra.run.smithi134.stdout: "max_objects": -1 2024-08-18T15:44:14.433 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:44:14.433 INFO:teuthology.orchestra.run.smithi134.stdout: "user_ratelimit": { 2024-08-18T15:44:14.433 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_ops": 0, 2024-08-18T15:44:14.433 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_ops": 0, 2024-08-18T15:44:14.433 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_bytes": 0, 2024-08-18T15:44:14.433 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_bytes": 0, 2024-08-18T15:44:14.433 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false 2024-08-18T15:44:14.433 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:44:14.434 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_ratelimit": { 2024-08-18T15:44:14.434 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_ops": 0, 2024-08-18T15:44:14.434 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_ops": 0, 2024-08-18T15:44:14.434 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_bytes": 0, 2024-08-18T15:44:14.434 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_bytes": 0, 2024-08-18T15:44:14.434 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false 2024-08-18T15:44:14.434 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:44:14.434 INFO:teuthology.orchestra.run.smithi134.stdout: "anonymous_ratelimit": { 2024-08-18T15:44:14.434 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_ops": 0, 2024-08-18T15:44:14.434 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_ops": 0, 2024-08-18T15:44:14.434 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_bytes": 0, 2024-08-18T15:44:14.434 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_bytes": 0, 2024-08-18T15:44:14.435 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false 2024-08-18T15:44:14.435 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:44:14.435 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:44:14.435 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_id": "2827c6f0-bf0d-4836-944c-9e7fcf22457b", 2024-08-18T15:44:14.435 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_name": "test-realm", 2024-08-18T15:44:14.435 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T15:44:14.435 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T15:44:14.436 INFO:rgw_multi.tests:Set master zone=test-zone1, waiting 60s for reconfiguration.. 2024-08-18T15:45:14.437 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=kibgmb-47 2024-08-18T15:45:14.461 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=kibgmb-48 2024-08-18T15:45:14.495 INFO:tasks.rgw.c1.client.1:Restarting daemon 2024-08-18T15:45:14.495 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2024-08-18T15:45:14.498 INFO:tasks.rgw.c1.client.1:Started 2024-08-18T15:45:14.499 DEBUG:teuthology.orchestra.run.smithi134:> curl http://smithi134.front.sepia.ceph.com:8001 2024-08-18T15:45:14.520 INFO:teuthology.orchestra.run.smithi134.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-08-18T15:45:14.520 INFO:teuthology.orchestra.run.smithi134.stderr: Dload Upload Total Spent Left Speed 2024-08-18T15:45:14.520 INFO:teuthology.orchestra.run.smithi134.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2024-08-18T15:45:14.521 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-08-18T15:45:14.522 INFO:teuthology.orchestra.run.smithi134.stderr:curl: (7) Failed to connect to smithi134.front.sepia.ceph.com port 8001: Connection refused 2024-08-18T15:45:15.523 DEBUG:teuthology.orchestra.run.smithi134:> curl http://smithi134.front.sepia.ceph.com:8001 2024-08-18T15:45:15.544 INFO:teuthology.orchestra.run.smithi134.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-08-18T15:45:15.544 INFO:teuthology.orchestra.run.smithi134.stderr: Dload Upload Total Spent Left Speed 2024-08-18T15:45:15.546 INFO:teuthology.orchestra.run.smithi134.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-08-18T15:45:15.546 INFO:teuthology.orchestra.run.smithi134.stdout:anonymous 2024-08-18T15:45:15.546 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:45:15.547 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:45:15.547 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:45:15.634 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:45:15.634 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:45:15.664 INFO:teuthology.orchestra.run.smithi134.stderr:No --period given, using current period=01050e30-179b-43be-bdbb-3bb23cf37ae8 2024-08-18T15:45:15.677 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T15:45:15.677 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:15.677 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T15:45:15.677 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T15:45:15.677 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:15.677 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:15.677 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T15:45:15.677 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T15:45:15.677 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:15.678 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:15.678 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T15:45:15.678 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T15:45:15.678 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:15.678 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:15.678 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723995914.484839_255.1", 2024-08-18T15:45:15.678 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:45:14.484839Z" 2024-08-18T15:45:15.678 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:15.678 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T15:45:15.682 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-08-18T15:45:15.682 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:45:15.682 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:45:15.682 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:45:15.742 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:45:15.742 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:45:15.799 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:45:15.798+0000 7f3d86c15880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:45:15.801 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:45:15.800+0000 7f3d86c15880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:45:15.819 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:45:15.819 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:45:15.819 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:45:15.819 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:45:15.820 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:45:15.820 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T15:45:15.820 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:45:15.820 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:15.820 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:45:15.820 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:15.820 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:45:15.820 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:45:15.820 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:45:15.820 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:45:15.820 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:45:15.820 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:45:15.821 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:45:15.821 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:45:15.821 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:45:15.821 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:15.821 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:15.821 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:15.821 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:45:15.821 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:45:15.821 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:45:15.821 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:45:15.821 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:45:15.821 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:45:15.822 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:45:15.822 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:45:15.822 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:45:15.822 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:15.822 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:15.822 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:15.822 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:45:15.822 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:45:15.822 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:45:15.822 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:45:15.822 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:45:15.822 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:45:15.823 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:45:15.823 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:45:15.823 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:45:15.823 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:15.823 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:15.823 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:15.823 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:45:15.823 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:45:15.823 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:45:15.823 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723995848.930494_187.1", 2024-08-18T15:45:15.823 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:45:15.823 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:45:15.824 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:45:15.824 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:44:08.930494Z", 2024-08-18T15:45:15.824 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 3 2024-08-18T15:45:15.824 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:15.824 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:15.824 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:45:15.824 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:15.824 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:45:15.824 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:45:15.824 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:45:15.824 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:15.824 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:45:15Z" 2024-08-18T15:45:15.824 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:45:15.825 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1723995914.484839_255.1 target= 2024-08-18T15:45:20.826 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:45:20.826 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:45:20.827 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:45:20.882 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:45:20.883 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:45:20.914 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:45:20.914+0000 7f95ba328880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:45:20.915 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:45:20.915+0000 7f95ba328880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:45:20.928 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:45:20.928 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:45:20.929 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:45:20.929 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:45:20.929 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:45:20.929 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T15:45:20.929 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:45:20.929 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:20.929 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:45:20.929 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:20.929 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:45:20.929 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:45:20.929 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:45:20.929 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:45:20.929 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:45:20.930 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:45:20.930 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:45:20.930 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:45:20.930 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:45:20.930 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:20.930 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:20.930 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:20.930 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:45:20.930 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:45:20.930 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:45:20.930 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:45:20.930 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:45:20.930 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:45:20.931 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:45:20.931 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:45:20.931 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:45:20.931 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:20.931 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:20.931 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:20.931 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:45:20.931 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:45:20.931 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:45:20.931 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:45:20.931 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:45:20.931 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:45:20.931 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:45:20.932 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:45:20.932 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:45:20.932 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:20.932 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:20.932 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:20.932 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:45:20.932 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:45:20.932 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:45:20.932 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723995914.484839_255.1", 2024-08-18T15:45:20.932 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:45:20.932 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:45:20.932 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:45:20.932 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:45:14.484839Z", 2024-08-18T15:45:20.933 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:45:20.933 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:20.933 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:20.933 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:45:20.933 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:20.933 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:45:20.933 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:45:20.933 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:45:20.934 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:20.934 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:45:20Z" 2024-08-18T15:45:20.934 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:45:20.935 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-08-18T15:45:20.935 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-08-18T15:45:20.935 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:45:20.935 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:45:20.936 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:45:20.990 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:45:20.990 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:45:21.022 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:45:21.021+0000 7f6bab01f880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:45:21.022 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:45:21.022+0000 7f6bab01f880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:45:21.035 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T15:45:21.035 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T15:45:21.035 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T15:45:21.035 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T15:45:21.035 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T15:45:21.035 INFO:teuthology.orchestra.run.smithi134.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T15:45:21.035 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T15:45:21.036 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:21.036 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T15:45:21.036 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:21.036 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T15:45:21.036 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:45:21.036 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:45:21.036 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:45:21.036 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:45:21.036 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 1, 2024-08-18T15:45:21.036 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:45:21.036 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:45:21.036 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:45:21.037 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:21.037 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:21.037 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:21.037 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T15:45:21.037 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:45:21.037 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:45:21.037 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T15:45:21.037 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:45:21.037 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:45:21.037 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:45:21.037 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T15:45:21.037 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:45:21.037 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:21.038 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:21.038 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:21.038 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T15:45:21.038 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:45:21.038 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:45:21.038 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:45:21.038 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:45:21.038 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:45:21.038 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:45:21.038 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:45:21.038 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:45:21.038 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:21.038 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:21.039 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:21.039 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T15:45:21.039 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:45:21.039 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:45:21.039 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723995914.484839_255.1", 2024-08-18T15:45:21.039 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:45:21.039 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:45:21.039 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:45:21.039 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:45:14.484839Z", 2024-08-18T15:45:21.039 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T15:45:21.039 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:21.039 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:21.039 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T15:45:21.039 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:21.040 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T15:45:21.040 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 1, 2024-08-18T15:45:21.040 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 1 2024-08-18T15:45:21.040 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:21.040 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T15:45:21Z" 2024-08-18T15:45:21.040 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T15:45:21.041 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-08-18T15:45:21.041 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-42 zones={test-zone2, test-zone1} 2024-08-18T15:45:21.047 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:45:21.050 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:45:21.056 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-42 zones={test-zone2, test-zone1} 2024-08-18T15:45:21.057 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-42 zones={test-zone3, test-zone1} 2024-08-18T15:45:21.063 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:45:21.066 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:45:21.072 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-42 zones={test-zone3, test-zone1} 2024-08-18T15:45:21.072 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-42 zones={test-zone1, test-zone2} 2024-08-18T15:45:21.075 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:45:21.077 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:45:21.127 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-42 zones={test-zone1, test-zone2} 2024-08-18T15:45:21.127 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-42 zones={test-zone3, test-zone2} 2024-08-18T15:45:21.132 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:45:21.134 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:45:21.140 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-42 zones={test-zone3, test-zone2} 2024-08-18T15:45:21.140 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-42 zones={test-zone1, test-zone3} 2024-08-18T15:45:21.144 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:45:21.146 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:45:21.152 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-42 zones={test-zone1, test-zone3} 2024-08-18T15:45:21.153 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-42 zones={test-zone2, test-zone3} 2024-08-18T15:45:21.156 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:45:21.158 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:45:21.205 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-42 zones={test-zone2, test-zone3} 2024-08-18T15:45:21.205 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-43 zones={test-zone2, test-zone1} 2024-08-18T15:45:21.211 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:45:21.213 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:45:21.263 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-43 zones={test-zone2, test-zone1} 2024-08-18T15:45:21.263 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-43 zones={test-zone3, test-zone1} 2024-08-18T15:45:21.268 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:45:21.270 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:45:21.276 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-43 zones={test-zone3, test-zone1} 2024-08-18T15:45:21.276 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-43 zones={test-zone1, test-zone2} 2024-08-18T15:45:21.281 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:45:21.327 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:45:21.333 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-43 zones={test-zone1, test-zone2} 2024-08-18T15:45:21.333 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-43 zones={test-zone3, test-zone2} 2024-08-18T15:45:21.337 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:45:21.340 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:45:21.389 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-43 zones={test-zone3, test-zone2} 2024-08-18T15:45:21.389 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-43 zones={test-zone1, test-zone3} 2024-08-18T15:45:21.396 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:45:21.398 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:45:21.404 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-43 zones={test-zone1, test-zone3} 2024-08-18T15:45:21.404 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-43 zones={test-zone2, test-zone3} 2024-08-18T15:45:21.408 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:45:21.455 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:45:21.461 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-43 zones={test-zone2, test-zone3} 2024-08-18T15:45:21.461 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-44 zones={test-zone2, test-zone1} 2024-08-18T15:45:21.466 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:45:21.511 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:45:21.517 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-44 zones={test-zone2, test-zone1} 2024-08-18T15:45:21.517 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-44 zones={test-zone3, test-zone1} 2024-08-18T15:45:21.521 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:45:21.523 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:45:21.573 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-44 zones={test-zone3, test-zone1} 2024-08-18T15:45:21.573 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-44 zones={test-zone1, test-zone2} 2024-08-18T15:45:21.577 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:45:21.627 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:45:21.633 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-44 zones={test-zone1, test-zone2} 2024-08-18T15:45:21.633 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-44 zones={test-zone3, test-zone2} 2024-08-18T15:45:21.637 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:45:21.639 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:45:21.646 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-44 zones={test-zone3, test-zone2} 2024-08-18T15:45:21.646 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-44 zones={test-zone1, test-zone3} 2024-08-18T15:45:21.649 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:45:21.651 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:45:21.701 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-44 zones={test-zone1, test-zone3} 2024-08-18T15:45:21.701 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-44 zones={test-zone2, test-zone3} 2024-08-18T15:45:21.705 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:45:21.707 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:45:21.757 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-44 zones={test-zone2, test-zone3} 2024-08-18T15:45:21.757 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-45 zones={test-zone2, test-zone1} 2024-08-18T15:45:21.762 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:45:21.764 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:45:21.813 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-45 zones={test-zone2, test-zone1} 2024-08-18T15:45:21.813 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-45 zones={test-zone3, test-zone1} 2024-08-18T15:45:21.817 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:45:21.863 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:45:21.870 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-45 zones={test-zone3, test-zone1} 2024-08-18T15:45:21.870 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-45 zones={test-zone1, test-zone2} 2024-08-18T15:45:21.874 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:45:21.877 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:45:21.883 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-45 zones={test-zone1, test-zone2} 2024-08-18T15:45:21.883 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-45 zones={test-zone3, test-zone2} 2024-08-18T15:45:21.887 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:45:21.889 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:45:21.940 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-45 zones={test-zone3, test-zone2} 2024-08-18T15:45:21.940 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-45 zones={test-zone1, test-zone3} 2024-08-18T15:45:21.943 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:45:21.946 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:45:21.951 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-45 zones={test-zone1, test-zone3} 2024-08-18T15:45:21.951 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-45 zones={test-zone2, test-zone3} 2024-08-18T15:45:21.955 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:45:21.957 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:45:22.008 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-45 zones={test-zone2, test-zone3} 2024-08-18T15:45:22.008 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-46 zones={test-zone2, test-zone1} 2024-08-18T15:45:22.012 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:45:22.014 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:45:22.020 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-46 zones={test-zone2, test-zone1} 2024-08-18T15:45:22.021 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-46 zones={test-zone3, test-zone1} 2024-08-18T15:45:22.025 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:45:22.028 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:45:22.077 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-46 zones={test-zone3, test-zone1} 2024-08-18T15:45:22.077 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-46 zones={test-zone1, test-zone2} 2024-08-18T15:45:22.081 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:45:22.083 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:45:22.090 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-46 zones={test-zone1, test-zone2} 2024-08-18T15:45:22.090 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-46 zones={test-zone3, test-zone2} 2024-08-18T15:45:22.094 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:45:22.096 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:45:22.102 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-46 zones={test-zone3, test-zone2} 2024-08-18T15:45:22.102 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-46 zones={test-zone1, test-zone3} 2024-08-18T15:45:22.106 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:45:22.151 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:45:22.157 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-46 zones={test-zone1, test-zone3} 2024-08-18T15:45:22.157 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-46 zones={test-zone2, test-zone3} 2024-08-18T15:45:22.161 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:45:22.207 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:45:22.216 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-46 zones={test-zone2, test-zone3} 2024-08-18T15:45:22.216 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-47 zones={test-zone2, test-zone1} 2024-08-18T15:45:22.221 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:45:22.223 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:45:22.273 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-47 zones={test-zone2, test-zone1} 2024-08-18T15:45:22.273 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-47 zones={test-zone3, test-zone1} 2024-08-18T15:45:22.277 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:45:22.323 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:45:22.329 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-47 zones={test-zone3, test-zone1} 2024-08-18T15:45:22.329 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-47 zones={test-zone1, test-zone2} 2024-08-18T15:45:22.333 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:45:22.335 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:45:22.385 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-47 zones={test-zone1, test-zone2} 2024-08-18T15:45:22.386 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-47 zones={test-zone3, test-zone2} 2024-08-18T15:45:22.390 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:45:22.392 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:45:22.399 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-47 zones={test-zone3, test-zone2} 2024-08-18T15:45:22.399 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-47 zones={test-zone1, test-zone3} 2024-08-18T15:45:22.402 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:45:22.405 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:45:22.455 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-47 zones={test-zone1, test-zone3} 2024-08-18T15:45:22.455 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-47 zones={test-zone2, test-zone3} 2024-08-18T15:45:22.459 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:45:22.461 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:45:22.511 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-47 zones={test-zone2, test-zone3} 2024-08-18T15:45:22.511 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-48 zones={test-zone2, test-zone1} 2024-08-18T15:45:22.516 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:45:22.518 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:45:22.526 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-48 zones={test-zone2, test-zone1} 2024-08-18T15:45:22.526 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-48 zones={test-zone3, test-zone1} 2024-08-18T15:45:22.530 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:45:22.532 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:45:22.578 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-48 zones={test-zone3, test-zone1} 2024-08-18T15:45:22.579 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-48 zones={test-zone1, test-zone2} 2024-08-18T15:45:22.583 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:45:22.585 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:45:22.634 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-48 zones={test-zone1, test-zone2} 2024-08-18T15:45:22.635 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-48 zones={test-zone3, test-zone2} 2024-08-18T15:45:22.639 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:45:22.642 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:45:22.648 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-48 zones={test-zone3, test-zone2} 2024-08-18T15:45:22.649 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-48 zones={test-zone1, test-zone3} 2024-08-18T15:45:22.653 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:45:22.655 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:45:22.705 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-48 zones={test-zone1, test-zone3} 2024-08-18T15:45:22.705 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-48 zones={test-zone2, test-zone3} 2024-08-18T15:45:22.709 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:45:22.755 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:45:22.762 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-48 zones={test-zone2, test-zone3} 2024-08-18T15:45:22.762 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'b0d74758-1871-4454-bc0f-e3e4b6c30105', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:45:22.762 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'b0d74758-1871-4454-bc0f-e3e4b6c30105', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:45:22.762 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period b0d74758-1871-4454-bc0f-e3e4b6c30105 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:45:22.807 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:45:22.808 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:45:22.838 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T15:45:22.838 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.838 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994364.503712_22.1", 2024-08-18T15:45:22.839 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "user", 2024-08-18T15:45:22.839 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "rgw-multisite-test-user", 2024-08-18T15:45:22.839 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:19:24.503712511Z", 2024-08-18T15:45:22.839 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.839 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.839 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:22.839 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:22.839 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.839 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.839 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_MXHnfxZIHwQEXOVDZVD1osf", 2024-08-18T15:45:22.839 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.839 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.840 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.840 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:22.840 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.840 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.840 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.840 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.840 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994364.506194_23.1", 2024-08-18T15:45:22.840 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "user", 2024-08-18T15:45:22.840 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "rgw-multisite-test-user", 2024-08-18T15:45:22.840 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:19:24.506194765Z", 2024-08-18T15:45:22.840 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.840 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.840 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:22.841 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:22.841 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.841 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.841 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_MXHnfxZIHwQEXOVDZVD1osf", 2024-08-18T15:45:22.841 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.841 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.841 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.841 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:22.841 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.841 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.841 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.841 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.842 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994485.499738_49.1", 2024-08-18T15:45:22.842 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:22.842 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-1:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.1", 2024-08-18T15:45:22.842 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:25.499738805Z", 2024-08-18T15:45:22.842 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.842 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.842 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:22.842 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:22.842 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.842 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.842 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_RWKCfRB1w1J8Q2l4ptlOal-", 2024-08-18T15:45:22.842 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.842 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.843 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.843 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:22.843 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.843 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.843 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.843 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.843 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994485.513055_50.1", 2024-08-18T15:45:22.843 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:22.843 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-1:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.1", 2024-08-18T15:45:22.843 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:25.513055821Z", 2024-08-18T15:45:22.843 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.843 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.843 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:22.844 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:22.844 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.844 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.844 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_RWKCfRB1w1J8Q2l4ptlOal-", 2024-08-18T15:45:22.844 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.844 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.844 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.844 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:22.844 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.844 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.844 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.844 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.845 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994485.514799_51.1", 2024-08-18T15:45:22.845 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:22.845 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-1", 2024-08-18T15:45:22.845 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:25.514799747Z", 2024-08-18T15:45:22.845 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.845 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.845 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:22.845 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:22.845 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.845 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.845 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_zN4WlRRiwX-IH3XcKWBWZ7B", 2024-08-18T15:45:22.845 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.845 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.846 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.846 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:22.846 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.846 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.846 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.846 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.846 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994485.517793_52.1", 2024-08-18T15:45:22.846 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:22.846 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-1", 2024-08-18T15:45:22.847 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:25.517793650Z", 2024-08-18T15:45:22.847 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.847 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.847 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:22.847 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:22.847 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.847 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.847 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_zN4WlRRiwX-IH3XcKWBWZ7B", 2024-08-18T15:45:22.847 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.847 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.847 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.847 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:22.847 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.848 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.848 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.848 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.848 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994485.530711_53.1", 2024-08-18T15:45:22.848 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:22.848 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-2:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.2", 2024-08-18T15:45:22.848 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:25.530711866Z", 2024-08-18T15:45:22.848 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.848 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.848 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:22.848 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:22.848 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.848 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.849 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_kZfzVgY3qrQmzEpOy-gNh0z", 2024-08-18T15:45:22.849 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.849 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.849 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.849 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:22.849 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.849 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.849 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.849 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.849 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994485.533125_54.1", 2024-08-18T15:45:22.849 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:22.849 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-2:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.2", 2024-08-18T15:45:22.849 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:25.533125204Z", 2024-08-18T15:45:22.850 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.850 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.850 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:22.850 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:22.850 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.850 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.850 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_kZfzVgY3qrQmzEpOy-gNh0z", 2024-08-18T15:45:22.850 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.850 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.850 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.850 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:22.850 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.851 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.851 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.851 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.851 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994485.534087_55.1", 2024-08-18T15:45:22.851 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:22.851 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-2", 2024-08-18T15:45:22.851 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:25.534087256Z", 2024-08-18T15:45:22.851 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.851 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.851 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:22.851 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:22.851 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.851 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.852 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_4VibGFZMXb9COC_5_LGnUYq", 2024-08-18T15:45:22.852 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.852 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.852 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.852 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:22.852 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.852 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.852 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.852 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.852 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994485.536327_56.1", 2024-08-18T15:45:22.852 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:22.852 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-2", 2024-08-18T15:45:22.852 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:25.536327238Z", 2024-08-18T15:45:22.852 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.853 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.853 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:22.853 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:22.853 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.853 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.853 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_4VibGFZMXb9COC_5_LGnUYq", 2024-08-18T15:45:22.853 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.853 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.853 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.853 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:22.853 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.853 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.853 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.854 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.854 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994488.093196_57.1", 2024-08-18T15:45:22.854 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:22.854 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-3:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.3", 2024-08-18T15:45:22.854 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:28.093196831Z", 2024-08-18T15:45:22.854 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.854 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.854 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:22.854 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:22.854 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.854 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.854 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_7WKtlNtV_hilpzaA5UW3kBl", 2024-08-18T15:45:22.855 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.855 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.855 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.855 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:22.855 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.855 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.855 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.855 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.855 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994488.096301_58.1", 2024-08-18T15:45:22.855 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:22.855 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-3:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.3", 2024-08-18T15:45:22.855 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:28.096301217Z", 2024-08-18T15:45:22.855 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.856 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.856 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:22.856 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:22.856 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.856 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.856 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_7WKtlNtV_hilpzaA5UW3kBl", 2024-08-18T15:45:22.856 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.856 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.856 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.856 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:22.856 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.856 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.856 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.857 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.857 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994488.097328_59.1", 2024-08-18T15:45:22.857 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:22.857 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-3", 2024-08-18T15:45:22.857 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:28.097328364Z", 2024-08-18T15:45:22.857 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.857 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.857 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:22.857 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:22.857 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.857 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.857 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_T0NcIXjN0Rm7yrm1EC_xxAs", 2024-08-18T15:45:22.857 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.857 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.858 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.858 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:22.858 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.858 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.858 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.858 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.858 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994488.100145_60.1", 2024-08-18T15:45:22.858 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:22.858 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-3", 2024-08-18T15:45:22.858 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:28.100145189Z", 2024-08-18T15:45:22.858 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.858 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.858 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:22.859 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:22.859 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.859 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.859 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_T0NcIXjN0Rm7yrm1EC_xxAs", 2024-08-18T15:45:22.859 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.859 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.859 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.859 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:22.859 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.859 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.859 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.859 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.859 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994493.565103_61.1", 2024-08-18T15:45:22.860 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:22.860 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-4:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.4", 2024-08-18T15:45:22.860 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:33.565103683Z", 2024-08-18T15:45:22.860 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.860 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.860 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:22.860 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:22.860 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.860 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.860 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_NVsrONxJz03-nbFIVh9-7HH", 2024-08-18T15:45:22.860 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.860 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.860 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.860 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:22.861 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.861 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.861 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.861 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.861 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994493.567515_62.1", 2024-08-18T15:45:22.861 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:22.861 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-4:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.4", 2024-08-18T15:45:22.861 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:33.567515090Z", 2024-08-18T15:45:22.861 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.861 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.861 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:22.861 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:22.861 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.862 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.862 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_NVsrONxJz03-nbFIVh9-7HH", 2024-08-18T15:45:22.862 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.862 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.862 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.862 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:22.862 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.862 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.862 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.862 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.862 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994493.568350_63.1", 2024-08-18T15:45:22.862 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:22.863 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-4", 2024-08-18T15:45:22.863 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:33.568350483Z", 2024-08-18T15:45:22.863 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.863 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.863 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:22.863 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:22.863 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.863 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.863 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_5jHEePfx-6SZjjYqolXPBLX", 2024-08-18T15:45:22.863 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.863 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.863 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.863 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:22.864 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.864 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.864 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.864 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.864 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994493.570568_64.1", 2024-08-18T15:45:22.864 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:22.864 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-4", 2024-08-18T15:45:22.864 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:33.570568110Z", 2024-08-18T15:45:22.864 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.864 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.864 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:22.864 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:22.864 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.864 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.865 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_5jHEePfx-6SZjjYqolXPBLX", 2024-08-18T15:45:22.865 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.865 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.865 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.865 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:22.865 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.865 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.865 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.865 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.865 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994493.583318_65.1", 2024-08-18T15:45:22.865 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:22.865 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-5:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.5", 2024-08-18T15:45:22.865 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:33.583318078Z", 2024-08-18T15:45:22.866 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.866 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.866 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:22.866 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:22.866 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.866 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.866 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_gGmjq-oCwB5WheV6EKKwPBK", 2024-08-18T15:45:22.866 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.866 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.866 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.866 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:22.866 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.866 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.867 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.867 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.867 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994493.586220_66.1", 2024-08-18T15:45:22.867 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:22.867 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-5:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.5", 2024-08-18T15:45:22.867 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:33.586220697Z", 2024-08-18T15:45:22.867 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.867 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.867 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:22.867 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:22.867 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.867 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.867 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_gGmjq-oCwB5WheV6EKKwPBK", 2024-08-18T15:45:22.868 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.868 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.868 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.868 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:22.868 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.868 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.868 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.868 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.868 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994493.587090_67.1", 2024-08-18T15:45:22.868 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:22.868 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-5", 2024-08-18T15:45:22.868 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:33.587090828Z", 2024-08-18T15:45:22.868 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.868 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.869 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:22.869 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:22.869 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.869 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.869 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_d2Y8p1KjRuhShSVV0OrvphE", 2024-08-18T15:45:22.869 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.869 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.869 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.869 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:22.869 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.869 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.869 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.869 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.870 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994493.589499_68.1", 2024-08-18T15:45:22.870 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:22.870 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-5", 2024-08-18T15:45:22.870 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:33.589499147Z", 2024-08-18T15:45:22.870 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.870 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.870 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:22.870 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:22.870 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.870 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.870 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_d2Y8p1KjRuhShSVV0OrvphE", 2024-08-18T15:45:22.870 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.870 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.871 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.871 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:22.871 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.871 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.871 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.871 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.871 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994493.608922_69.1", 2024-08-18T15:45:22.871 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:22.871 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-6:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.6", 2024-08-18T15:45:22.871 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:33.608922657Z", 2024-08-18T15:45:22.871 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.871 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.871 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:22.871 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:22.872 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.872 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.872 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_ZjXDBXiyWYi6JTPxve1UMrv", 2024-08-18T15:45:22.872 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.872 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.872 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.872 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:22.872 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.872 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.872 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.872 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.872 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994493.613023_70.1", 2024-08-18T15:45:22.872 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:22.873 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-6:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.6", 2024-08-18T15:45:22.873 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:33.613023410Z", 2024-08-18T15:45:22.873 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.873 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.873 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:22.873 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:22.873 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.873 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.873 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_ZjXDBXiyWYi6JTPxve1UMrv", 2024-08-18T15:45:22.873 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.873 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.873 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.873 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:22.874 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.874 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.874 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.874 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.874 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994493.614132_71.1", 2024-08-18T15:45:22.874 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:22.874 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-6", 2024-08-18T15:45:22.874 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:33.614132419Z", 2024-08-18T15:45:22.874 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.874 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.874 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:22.874 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:22.875 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.875 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.875 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_ZBRWYIGyUuGJdOA0PdgsHNt", 2024-08-18T15:45:22.875 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.875 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.875 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.875 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:22.875 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.875 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.875 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.875 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.875 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994493.616159_72.1", 2024-08-18T15:45:22.875 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:22.876 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-6", 2024-08-18T15:45:22.876 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:33.616159354Z", 2024-08-18T15:45:22.876 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.876 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.876 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:22.876 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:22.876 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.876 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.876 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_ZBRWYIGyUuGJdOA0PdgsHNt", 2024-08-18T15:45:22.876 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.876 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.876 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.876 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:22.877 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.877 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.877 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.877 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.877 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994494.468832_73.1", 2024-08-18T15:45:22.877 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:22.877 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-7:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.7", 2024-08-18T15:45:22.877 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:34.468832363Z", 2024-08-18T15:45:22.877 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.877 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.877 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:22.877 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:22.877 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.877 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.878 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_qRFDABc-dwq6RziTweT5YtT", 2024-08-18T15:45:22.878 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.878 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.878 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.878 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:22.878 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.878 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.878 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.878 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.878 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994494.471921_74.1", 2024-08-18T15:45:22.878 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:22.878 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-7:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.7", 2024-08-18T15:45:22.879 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:34.471921559Z", 2024-08-18T15:45:22.879 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.879 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.879 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:22.879 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:22.879 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.879 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.879 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_qRFDABc-dwq6RziTweT5YtT", 2024-08-18T15:45:22.879 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.879 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.879 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.879 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:22.879 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.880 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.880 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.880 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.880 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994494.472699_75.1", 2024-08-18T15:45:22.880 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:22.880 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-7", 2024-08-18T15:45:22.880 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:34.472699971Z", 2024-08-18T15:45:22.880 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.880 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.880 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:22.880 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:22.880 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.880 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.881 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_tLmChDVmX3sImKf0993KFB1", 2024-08-18T15:45:22.881 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.881 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.881 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.881 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:22.881 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.881 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.881 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.881 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.881 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994494.475169_76.1", 2024-08-18T15:45:22.881 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:22.881 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-7", 2024-08-18T15:45:22.881 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:34.475169889Z", 2024-08-18T15:45:22.882 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.882 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.882 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:22.882 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:22.882 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.882 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.882 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_tLmChDVmX3sImKf0993KFB1", 2024-08-18T15:45:22.882 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.882 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.882 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.882 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:22.882 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.882 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.883 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.883 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.883 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994494.483917_77.1", 2024-08-18T15:45:22.883 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:22.883 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-8:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.8", 2024-08-18T15:45:22.883 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:34.483917175Z", 2024-08-18T15:45:22.883 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.883 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.883 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:22.883 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:22.883 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.883 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.883 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_Zi2-lA3qHEM3t_BsL4AeXRU", 2024-08-18T15:45:22.883 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.884 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.884 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.884 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:22.884 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.884 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.884 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.884 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.884 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994494.487103_78.1", 2024-08-18T15:45:22.884 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:22.884 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-8:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.8", 2024-08-18T15:45:22.884 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:34.487103946Z", 2024-08-18T15:45:22.884 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.884 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.885 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:22.885 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:22.885 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.885 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.885 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_Zi2-lA3qHEM3t_BsL4AeXRU", 2024-08-18T15:45:22.885 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.885 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.885 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.885 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:22.885 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.885 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.885 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.885 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.886 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994494.487887_79.1", 2024-08-18T15:45:22.886 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:22.886 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-8", 2024-08-18T15:45:22.886 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:34.487887903Z", 2024-08-18T15:45:22.886 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.886 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.886 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:22.886 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:22.886 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.886 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.886 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_7oG9nLitKlO5mrAHdrnWcCe", 2024-08-18T15:45:22.886 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.886 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.887 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.887 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:22.887 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.887 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.887 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.887 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.887 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994494.489837_80.1", 2024-08-18T15:45:22.887 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:22.887 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-8", 2024-08-18T15:45:22.887 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:34.489837888Z", 2024-08-18T15:45:22.887 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.887 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.887 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:22.888 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:22.888 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.888 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.888 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_7oG9nLitKlO5mrAHdrnWcCe", 2024-08-18T15:45:22.888 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.888 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.888 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.888 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:22.888 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.888 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.888 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.888 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.888 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994494.511942_81.1", 2024-08-18T15:45:22.888 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:22.889 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-9:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.9", 2024-08-18T15:45:22.889 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:34.511942269Z", 2024-08-18T15:45:22.889 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.889 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.889 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:22.889 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:22.889 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.889 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.889 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_M-lw4sdJUfr_hU7bn6pQCkY", 2024-08-18T15:45:22.889 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.889 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.889 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.889 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:22.890 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.890 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.890 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.890 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.890 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994494.520062_82.1", 2024-08-18T15:45:22.890 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:22.890 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-9:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.9", 2024-08-18T15:45:22.890 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:34.520062161Z", 2024-08-18T15:45:22.890 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.890 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.890 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:22.890 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:22.890 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.891 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.891 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_M-lw4sdJUfr_hU7bn6pQCkY", 2024-08-18T15:45:22.891 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.891 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.891 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.891 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:22.891 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.891 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.891 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.891 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.891 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994494.520843_83.1", 2024-08-18T15:45:22.891 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:22.891 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-9", 2024-08-18T15:45:22.892 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:34.520843453Z", 2024-08-18T15:45:22.892 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.892 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.892 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:22.892 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:22.892 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.892 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.892 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_WwzPQ6LnfxKTsseathJHhf2", 2024-08-18T15:45:22.892 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.893 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.893 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.893 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:22.893 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.893 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.893 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.893 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.893 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994494.523006_84.1", 2024-08-18T15:45:22.893 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:22.893 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-9", 2024-08-18T15:45:22.893 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:34.523006825Z", 2024-08-18T15:45:22.893 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.894 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.894 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:22.894 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:22.894 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.894 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.894 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_WwzPQ6LnfxKTsseathJHhf2", 2024-08-18T15:45:22.894 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.894 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.894 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.894 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:22.894 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.894 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.895 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.895 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.895 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994494.861305_85.1", 2024-08-18T15:45:22.895 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:22.895 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-7:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.7", 2024-08-18T15:45:22.895 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:34.861305566Z", 2024-08-18T15:45:22.895 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.895 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.895 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_qRFDABc-dwq6RziTweT5YtT", 2024-08-18T15:45:22.895 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.895 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.895 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.895 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_qRFDABc-dwq6RziTweT5YtT", 2024-08-18T15:45:22.895 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:22.896 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.896 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.896 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:22.896 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.896 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.896 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.896 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.896 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994494.863517_86.1", 2024-08-18T15:45:22.896 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:22.896 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-7:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.7", 2024-08-18T15:45:22.896 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:34.863517491Z", 2024-08-18T15:45:22.896 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.896 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.897 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_qRFDABc-dwq6RziTweT5YtT", 2024-08-18T15:45:22.897 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.897 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.897 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.897 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_qRFDABc-dwq6RziTweT5YtT", 2024-08-18T15:45:22.897 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:22.897 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.897 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.897 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:22.897 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.897 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.897 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.897 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.897 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994494.870867_87.1", 2024-08-18T15:45:22.898 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:22.898 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-7", 2024-08-18T15:45:22.898 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:34.870867850Z", 2024-08-18T15:45:22.898 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.898 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.898 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_tLmChDVmX3sImKf0993KFB1", 2024-08-18T15:45:22.898 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.898 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.898 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.898 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_tLmChDVmX3sImKf0993KFB1", 2024-08-18T15:45:22.898 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:22.898 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.898 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.899 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "remove" 2024-08-18T15:45:22.899 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.899 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.899 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.899 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.899 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994494.872883_88.1", 2024-08-18T15:45:22.899 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:22.899 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-7", 2024-08-18T15:45:22.899 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:34.872883708Z", 2024-08-18T15:45:22.899 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.899 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.899 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_tLmChDVmX3sImKf0993KFB1", 2024-08-18T15:45:22.899 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.900 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.900 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.900 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_tLmChDVmX3sImKf0993KFB1", 2024-08-18T15:45:22.900 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:22.900 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.900 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.900 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:22.900 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.900 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.900 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.900 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.900 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994494.890261_89.1", 2024-08-18T15:45:22.901 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:22.901 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-8:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.8", 2024-08-18T15:45:22.901 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:34.890261024Z", 2024-08-18T15:45:22.901 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.901 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.901 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_Zi2-lA3qHEM3t_BsL4AeXRU", 2024-08-18T15:45:22.901 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.901 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.901 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.901 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_Zi2-lA3qHEM3t_BsL4AeXRU", 2024-08-18T15:45:22.901 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:22.901 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.901 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.902 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:22.902 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.902 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.902 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.902 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.902 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994494.892455_90.1", 2024-08-18T15:45:22.902 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:22.902 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-8:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.8", 2024-08-18T15:45:22.902 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:34.892455177Z", 2024-08-18T15:45:22.902 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.902 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.902 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_Zi2-lA3qHEM3t_BsL4AeXRU", 2024-08-18T15:45:22.902 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.903 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.903 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.903 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_Zi2-lA3qHEM3t_BsL4AeXRU", 2024-08-18T15:45:22.903 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:22.903 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.903 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.903 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:22.903 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.903 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.903 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.903 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.903 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994494.899743_91.1", 2024-08-18T15:45:22.903 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:22.903 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-8", 2024-08-18T15:45:22.904 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:34.899743512Z", 2024-08-18T15:45:22.904 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.904 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.904 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_7oG9nLitKlO5mrAHdrnWcCe", 2024-08-18T15:45:22.904 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.904 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.904 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.904 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_7oG9nLitKlO5mrAHdrnWcCe", 2024-08-18T15:45:22.904 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:22.904 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.904 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.904 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "remove" 2024-08-18T15:45:22.904 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.905 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.905 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.905 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.905 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994494.901752_92.1", 2024-08-18T15:45:22.905 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:22.905 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-8", 2024-08-18T15:45:22.905 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:34.901752069Z", 2024-08-18T15:45:22.905 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.905 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.905 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_7oG9nLitKlO5mrAHdrnWcCe", 2024-08-18T15:45:22.905 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.905 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.905 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.905 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_7oG9nLitKlO5mrAHdrnWcCe", 2024-08-18T15:45:22.906 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:22.906 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.906 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.906 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:22.906 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.906 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.906 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.906 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.906 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994494.940403_93.1", 2024-08-18T15:45:22.906 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:22.906 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-9:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.9", 2024-08-18T15:45:22.906 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:34.940403695Z", 2024-08-18T15:45:22.906 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.907 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.907 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_M-lw4sdJUfr_hU7bn6pQCkY", 2024-08-18T15:45:22.907 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.907 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.907 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.907 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_M-lw4sdJUfr_hU7bn6pQCkY", 2024-08-18T15:45:22.907 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:22.907 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.907 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.907 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:22.907 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.907 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.907 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.908 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.908 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994494.942560_94.1", 2024-08-18T15:45:22.908 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:22.908 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-9:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.9", 2024-08-18T15:45:22.908 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:34.942560102Z", 2024-08-18T15:45:22.908 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.908 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.908 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_M-lw4sdJUfr_hU7bn6pQCkY", 2024-08-18T15:45:22.908 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.909 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.909 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.909 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_M-lw4sdJUfr_hU7bn6pQCkY", 2024-08-18T15:45:22.909 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:22.909 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.909 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.909 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:22.909 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.909 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.909 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.909 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.910 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994494.949797_95.1", 2024-08-18T15:45:22.910 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:22.910 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-9", 2024-08-18T15:45:22.910 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:34.949797571Z", 2024-08-18T15:45:22.910 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.910 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.910 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_WwzPQ6LnfxKTsseathJHhf2", 2024-08-18T15:45:22.910 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.910 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.910 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.910 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_WwzPQ6LnfxKTsseathJHhf2", 2024-08-18T15:45:22.910 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:22.911 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.911 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.911 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "remove" 2024-08-18T15:45:22.911 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.911 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.911 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.911 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.911 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994494.952005_96.1", 2024-08-18T15:45:22.911 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:22.911 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-9", 2024-08-18T15:45:22.911 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:34.952005923Z", 2024-08-18T15:45:22.911 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.911 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.912 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_WwzPQ6LnfxKTsseathJHhf2", 2024-08-18T15:45:22.912 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.912 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.912 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.912 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_WwzPQ6LnfxKTsseathJHhf2", 2024-08-18T15:45:22.912 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:22.912 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.912 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.912 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:22.912 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.912 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.912 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.912 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.913 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994505.517635_51.1", 2024-08-18T15:45:22.913 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:22.913 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:45:22.913 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:45.517635733Z", 2024-08-18T15:45:22.913 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.913 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.913 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:22.913 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:22.913 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.913 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.913 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "__WpsUsCter7eN_yzr7oojfP", 2024-08-18T15:45:22.913 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.913 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.914 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.914 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:22.914 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.914 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.914 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.914 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.914 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994505.521539_52.1", 2024-08-18T15:45:22.914 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:22.914 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:45:22.914 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:45.521539522Z", 2024-08-18T15:45:22.914 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.914 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.915 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:22.915 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:22.915 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.915 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.915 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "__WpsUsCter7eN_yzr7oojfP", 2024-08-18T15:45:22.915 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.915 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.915 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.915 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:22.915 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.915 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.915 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.915 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.916 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994505.522484_53.1", 2024-08-18T15:45:22.916 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:22.916 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-10", 2024-08-18T15:45:22.916 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:45.522484813Z", 2024-08-18T15:45:22.916 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.916 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.916 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:22.916 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:22.916 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.916 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.916 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_u-jdXqkvtd3BndfjmYUBTit", 2024-08-18T15:45:22.916 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.916 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.916 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.917 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:22.917 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.917 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.917 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.917 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.917 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994505.524516_54.1", 2024-08-18T15:45:22.917 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:22.917 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-10", 2024-08-18T15:45:22.917 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:45.524516821Z", 2024-08-18T15:45:22.917 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.917 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.917 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:22.917 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:22.918 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.918 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.918 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_u-jdXqkvtd3BndfjmYUBTit", 2024-08-18T15:45:22.918 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.918 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.918 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.918 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:22.918 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.918 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.918 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.918 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.918 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994505.535896_55.1", 2024-08-18T15:45:22.918 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:22.919 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:45:22.919 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:45.535896117Z", 2024-08-18T15:45:22.919 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.919 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.919 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:22.919 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:22.919 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.919 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.919 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_XktQF5h6nGNC5v3w-DrHnvQ", 2024-08-18T15:45:22.919 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.919 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.919 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.919 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:22.920 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.920 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.920 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.920 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.920 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994505.539494_56.1", 2024-08-18T15:45:22.920 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:22.920 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:45:22.920 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:45.539494049Z", 2024-08-18T15:45:22.920 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.920 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.920 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:22.920 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:22.920 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.921 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.921 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_XktQF5h6nGNC5v3w-DrHnvQ", 2024-08-18T15:45:22.921 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.921 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.921 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.921 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:22.921 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.921 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.921 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.921 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.921 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994505.540711_57.1", 2024-08-18T15:45:22.921 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:22.921 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-11", 2024-08-18T15:45:22.922 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:45.540711150Z", 2024-08-18T15:45:22.922 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.922 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.922 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:22.922 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:22.922 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.922 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.922 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_y9YtDg7ol2vKL0ge54w1-u0", 2024-08-18T15:45:22.922 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.922 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.922 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.922 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:22.922 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.923 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.923 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.923 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.923 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994505.542807_58.1", 2024-08-18T15:45:22.923 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:22.923 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-11", 2024-08-18T15:45:22.923 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:45.542807482Z", 2024-08-18T15:45:22.923 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.923 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.923 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:22.923 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:22.923 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.923 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.924 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_y9YtDg7ol2vKL0ge54w1-u0", 2024-08-18T15:45:22.924 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.924 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.924 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.924 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:22.924 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.924 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.924 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.924 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.924 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994505.560557_59.1", 2024-08-18T15:45:22.924 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:22.924 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:45:22.924 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:45.560557573Z", 2024-08-18T15:45:22.925 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.925 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.925 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:22.925 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:22.925 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.925 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.925 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_EyCJst2_YMPiGsXCrb7chb3", 2024-08-18T15:45:22.925 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.925 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.925 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.925 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:22.925 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.925 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.926 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.926 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.926 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994505.563007_60.1", 2024-08-18T15:45:22.926 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:22.926 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:45:22.926 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:45.563007502Z", 2024-08-18T15:45:22.926 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.926 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.926 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:22.926 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:22.926 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.926 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.927 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_EyCJst2_YMPiGsXCrb7chb3", 2024-08-18T15:45:22.927 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.927 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.927 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.927 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:22.927 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.927 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.927 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.927 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.927 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994505.563839_61.1", 2024-08-18T15:45:22.927 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:22.927 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-12", 2024-08-18T15:45:22.927 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:45.563839125Z", 2024-08-18T15:45:22.928 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.928 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.928 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:22.928 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:22.928 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.928 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.928 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_O6k9owNlHrC6zA06qCpRJJZ", 2024-08-18T15:45:22.928 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.928 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.928 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.928 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:22.928 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.928 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.928 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.929 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.929 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994505.566000_62.1", 2024-08-18T15:45:22.929 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:22.929 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-12", 2024-08-18T15:45:22.929 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:45.566000292Z", 2024-08-18T15:45:22.929 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.929 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.929 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:22.929 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:22.929 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.929 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.929 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_O6k9owNlHrC6zA06qCpRJJZ", 2024-08-18T15:45:22.929 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.930 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.930 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.930 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:22.930 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.930 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.930 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.930 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.930 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994599.644114_81.1", 2024-08-18T15:45:22.930 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:22.930 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:45:22.930 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:23:19.644114701Z", 2024-08-18T15:45:22.930 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.930 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.931 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:22.931 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:22.931 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.931 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.931 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_JA4U4fOsjO9zivYOJOBkWF7", 2024-08-18T15:45:22.931 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.931 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.931 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.931 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:22.931 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.931 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.931 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.931 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.932 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994599.646276_82.1", 2024-08-18T15:45:22.935 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:22.935 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:45:22.935 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:23:19.646276235Z", 2024-08-18T15:45:22.935 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.935 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.935 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:22.936 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:22.936 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.936 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.936 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_JA4U4fOsjO9zivYOJOBkWF7", 2024-08-18T15:45:22.936 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.936 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.936 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.936 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:22.936 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.936 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.936 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.936 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.936 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994599.647062_83.1", 2024-08-18T15:45:22.937 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:22.937 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-13", 2024-08-18T15:45:22.937 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:23:19.647062932Z", 2024-08-18T15:45:22.937 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.937 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.937 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:22.937 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:22.937 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.937 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.937 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_qqt9_2CdbPwt_U6wcKZCsvw", 2024-08-18T15:45:22.937 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.937 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.937 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.938 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:22.938 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.938 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.938 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.938 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.938 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994599.649062_84.1", 2024-08-18T15:45:22.938 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:22.938 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-13", 2024-08-18T15:45:22.938 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:23:19.649062828Z", 2024-08-18T15:45:22.938 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.938 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.938 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:22.938 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:22.939 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.939 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.939 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_qqt9_2CdbPwt_U6wcKZCsvw", 2024-08-18T15:45:22.939 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.939 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.939 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.939 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:22.939 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.939 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.939 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.939 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.939 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994599.657472_85.1", 2024-08-18T15:45:22.939 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:22.940 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:45:22.940 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:23:19.657472231Z", 2024-08-18T15:45:22.940 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.940 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.940 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:22.940 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:22.940 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.940 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.940 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_hVfHZOUQ7KCzUbpwOhp8fwo", 2024-08-18T15:45:22.940 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.940 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.940 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.940 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:22.940 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.941 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.941 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.941 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.941 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994599.659687_86.1", 2024-08-18T15:45:22.941 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:22.941 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:45:22.941 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:23:19.659687294Z", 2024-08-18T15:45:22.941 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.941 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.941 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:22.941 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:22.941 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.941 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.942 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_hVfHZOUQ7KCzUbpwOhp8fwo", 2024-08-18T15:45:22.942 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.942 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.942 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.942 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:22.942 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.942 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.942 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.942 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.942 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994599.660446_87.1", 2024-08-18T15:45:22.943 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:22.943 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-14", 2024-08-18T15:45:22.943 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:23:19.660446107Z", 2024-08-18T15:45:22.943 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.943 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.943 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:22.943 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:22.943 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.943 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.943 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_Xr1Z4jV7ZXypVMLCsCytQag", 2024-08-18T15:45:22.943 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.943 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.943 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.944 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:22.944 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.944 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.944 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.944 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.944 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994599.662685_88.1", 2024-08-18T15:45:22.944 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:22.944 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-14", 2024-08-18T15:45:22.944 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:23:19.662685027Z", 2024-08-18T15:45:22.944 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.944 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.944 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:22.944 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:22.945 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.945 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.945 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_Xr1Z4jV7ZXypVMLCsCytQag", 2024-08-18T15:45:22.945 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.945 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.945 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.945 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:22.945 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.945 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.945 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.945 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.945 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994599.682718_89.1", 2024-08-18T15:45:22.945 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:22.945 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:45:22.946 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:23:19.682718683Z", 2024-08-18T15:45:22.946 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.946 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.946 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:22.946 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:22.946 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.946 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.946 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_tZckUgZHpjdeXtjxrO3FDKV", 2024-08-18T15:45:22.946 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.946 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.946 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.946 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:22.946 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.947 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.947 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.947 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.947 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994599.685017_90.1", 2024-08-18T15:45:22.947 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:22.947 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:45:22.947 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:23:19.685017287Z", 2024-08-18T15:45:22.947 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.947 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.947 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:22.947 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:22.947 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.947 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.947 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_tZckUgZHpjdeXtjxrO3FDKV", 2024-08-18T15:45:22.948 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.948 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.948 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.948 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:22.948 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.948 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.948 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.948 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.948 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994599.685790_91.1", 2024-08-18T15:45:22.948 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:22.948 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-15", 2024-08-18T15:45:22.948 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:23:19.685790914Z", 2024-08-18T15:45:22.948 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.948 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.949 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:22.949 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:22.949 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.949 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.949 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_z4wReHTq2_y91Bl7PqzxJPS", 2024-08-18T15:45:22.949 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.949 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.949 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.949 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:22.949 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.949 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.949 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.949 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.949 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994599.687927_92.1", 2024-08-18T15:45:22.950 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:22.950 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-15", 2024-08-18T15:45:22.950 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:23:19.687927319Z", 2024-08-18T15:45:22.950 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.950 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.950 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:22.950 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:22.950 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.950 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.950 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_z4wReHTq2_y91Bl7PqzxJPS", 2024-08-18T15:45:22.950 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.950 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.950 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.951 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:22.951 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.951 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.951 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.951 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.951 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994722.684594_96.1", 2024-08-18T15:45:22.951 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:22.951 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:45:22.951 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:25:22.684594373Z", 2024-08-18T15:45:22.951 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.951 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.952 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:22.952 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:22.952 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.952 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.952 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_Y8QIfzmW8naLImEFza7--rg", 2024-08-18T15:45:22.952 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.952 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.952 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.952 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:22.952 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.952 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.952 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.952 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.952 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994722.687709_97.1", 2024-08-18T15:45:22.953 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:22.953 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:45:22.953 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:25:22.687709684Z", 2024-08-18T15:45:22.953 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.953 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.953 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:22.953 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:22.953 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.953 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.953 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_Y8QIfzmW8naLImEFza7--rg", 2024-08-18T15:45:22.953 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.953 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.953 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.953 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:22.954 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.954 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.954 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.954 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.954 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994722.688892_98.1", 2024-08-18T15:45:22.954 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:22.954 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-16", 2024-08-18T15:45:22.954 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:25:22.688892164Z", 2024-08-18T15:45:22.954 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.954 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.954 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:22.954 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:22.954 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.955 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.955 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_pogOAh2gLfyKg4IGe5K-1It", 2024-08-18T15:45:22.955 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.955 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.955 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.955 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:22.955 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.955 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.955 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.955 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.955 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994722.692127_99.1", 2024-08-18T15:45:22.955 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:22.955 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-16", 2024-08-18T15:45:22.955 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:25:22.692127591Z", 2024-08-18T15:45:22.956 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.956 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.956 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:22.956 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:22.956 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.956 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.956 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_pogOAh2gLfyKg4IGe5K-1It", 2024-08-18T15:45:22.956 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.956 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.956 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.956 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:22.956 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.956 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.956 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.957 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.957 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994722.705107_100.1", 2024-08-18T15:45:22.957 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:22.957 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:45:22.957 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:25:22.705107630Z", 2024-08-18T15:45:22.957 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.957 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.957 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:22.957 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:22.958 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.958 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.958 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_TizKQmOZjxDm3jCjcVr_PHs", 2024-08-18T15:45:22.958 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.958 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.958 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.958 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:22.958 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.958 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.958 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.959 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.959 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994722.707889_101.1", 2024-08-18T15:45:22.959 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:22.959 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:45:22.959 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:25:22.707889768Z", 2024-08-18T15:45:22.959 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.960 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.960 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:22.960 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:22.960 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.960 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.960 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_TizKQmOZjxDm3jCjcVr_PHs", 2024-08-18T15:45:22.960 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.960 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.960 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.960 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:22.960 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.960 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.960 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.960 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.961 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994722.708890_102.1", 2024-08-18T15:45:22.961 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:22.961 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-17", 2024-08-18T15:45:22.961 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:25:22.708890200Z", 2024-08-18T15:45:22.961 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.961 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.961 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:22.961 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:22.961 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.962 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.962 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_bG8oC8bXHaoSssmVUfs6qfz", 2024-08-18T15:45:22.962 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.962 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.962 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.962 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:22.962 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.962 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.962 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.962 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.962 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994722.711644_103.1", 2024-08-18T15:45:22.962 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:22.962 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-17", 2024-08-18T15:45:22.963 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:25:22.711644822Z", 2024-08-18T15:45:22.963 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.963 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.963 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:22.963 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:22.963 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.963 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.963 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_bG8oC8bXHaoSssmVUfs6qfz", 2024-08-18T15:45:22.963 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.964 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.964 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.964 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:22.964 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.964 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.964 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.964 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.964 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994722.734205_104.1", 2024-08-18T15:45:22.964 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:22.964 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:45:22.964 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:25:22.734205817Z", 2024-08-18T15:45:22.964 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.964 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.965 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:22.965 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:22.965 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.965 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.965 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_dLDL_K-k27vccG78KCPIIzE", 2024-08-18T15:45:22.965 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.965 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.965 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.965 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:22.965 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.965 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.965 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.965 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.965 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994722.736900_105.1", 2024-08-18T15:45:22.966 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:22.966 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:45:22.966 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:25:22.736900697Z", 2024-08-18T15:45:22.966 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.966 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.966 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:22.966 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:22.966 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.966 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.966 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_dLDL_K-k27vccG78KCPIIzE", 2024-08-18T15:45:22.966 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.966 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.966 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.967 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:22.967 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.967 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.967 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.967 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.967 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994722.737991_106.1", 2024-08-18T15:45:22.967 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:22.967 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-18", 2024-08-18T15:45:22.967 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:25:22.737991682Z", 2024-08-18T15:45:22.967 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.967 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.967 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:22.967 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:22.967 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.968 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.968 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_HPNnX1tYqBsOgWp8mgW7Qb0", 2024-08-18T15:45:22.968 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.968 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.968 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.968 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:22.968 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.968 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.968 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.968 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.968 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994722.740810_107.1", 2024-08-18T15:45:22.968 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:22.968 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-18", 2024-08-18T15:45:22.968 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:25:22.740810965Z", 2024-08-18T15:45:22.969 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.969 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.969 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:22.969 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:22.969 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.969 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.969 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_HPNnX1tYqBsOgWp8mgW7Qb0", 2024-08-18T15:45:22.969 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.969 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.969 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.969 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:22.969 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.969 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.970 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.970 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.970 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995081.645527_272.1", 2024-08-18T15:45:22.970 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:22.970 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:45:22.970 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:31:21.645527834Z", 2024-08-18T15:45:22.970 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.970 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.970 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:22.970 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:22.970 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.970 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.970 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_tgDMSIYDTOiTpHaxXYDAzVY", 2024-08-18T15:45:22.971 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.971 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.971 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.971 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:22.971 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.971 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.971 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.971 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.971 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995081.648393_273.1", 2024-08-18T15:45:22.971 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:22.971 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:45:22.971 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:31:21.648393707Z", 2024-08-18T15:45:22.971 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.972 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.972 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:22.972 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:22.972 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.972 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.972 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_tgDMSIYDTOiTpHaxXYDAzVY", 2024-08-18T15:45:22.972 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.972 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.972 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.972 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:22.972 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.972 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.972 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.972 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.973 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995081.649470_274.1", 2024-08-18T15:45:22.973 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:22.973 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-19", 2024-08-18T15:45:22.973 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:31:21.649470258Z", 2024-08-18T15:45:22.973 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.973 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.973 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:22.973 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:22.973 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.973 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.973 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_K6QtNcOVkdATFYGx2kwzf4z", 2024-08-18T15:45:22.973 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.973 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.974 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.974 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:22.974 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.974 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.974 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.974 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.974 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995081.652067_275.1", 2024-08-18T15:45:22.974 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:22.974 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-19", 2024-08-18T15:45:22.974 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:31:21.652067379Z", 2024-08-18T15:45:22.974 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.974 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.974 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:22.974 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:22.975 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.975 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.975 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_K6QtNcOVkdATFYGx2kwzf4z", 2024-08-18T15:45:22.975 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.975 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.975 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.975 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:22.975 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.975 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.975 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.975 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.975 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995081.663969_276.1", 2024-08-18T15:45:22.975 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:22.976 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:45:22.976 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:31:21.663969084Z", 2024-08-18T15:45:22.976 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.976 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.976 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:22.976 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:22.976 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.976 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.976 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_Tq_y8CH-R5JTA5co_8cOHJr", 2024-08-18T15:45:22.976 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.976 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.976 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.976 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:22.976 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.977 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.977 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.977 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.977 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995081.666599_277.1", 2024-08-18T15:45:22.977 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:22.977 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:45:22.977 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:31:21.666599931Z", 2024-08-18T15:45:22.977 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.977 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.977 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:22.977 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:22.977 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.977 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.977 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_Tq_y8CH-R5JTA5co_8cOHJr", 2024-08-18T15:45:22.978 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.978 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.978 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.978 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:22.978 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.978 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.978 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.978 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.978 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995081.667569_278.1", 2024-08-18T15:45:22.978 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:22.978 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-20", 2024-08-18T15:45:22.978 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:31:21.667569296Z", 2024-08-18T15:45:22.978 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.979 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.979 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:22.979 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:22.979 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.979 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.979 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_wlqM8_vGXOeMXMyiNpDyzb0", 2024-08-18T15:45:22.979 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.979 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.979 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.979 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:22.979 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.979 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.979 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.979 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.980 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995081.670137_279.1", 2024-08-18T15:45:22.980 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:22.980 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-20", 2024-08-18T15:45:22.980 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:31:21.670137868Z", 2024-08-18T15:45:22.980 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.980 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.980 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:22.980 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:22.980 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.980 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.980 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_wlqM8_vGXOeMXMyiNpDyzb0", 2024-08-18T15:45:22.980 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.980 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.980 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.981 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:22.981 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.981 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.981 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.981 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.981 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995081.689780_280.1", 2024-08-18T15:45:22.981 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:22.981 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:45:22.981 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:31:21.689780751Z", 2024-08-18T15:45:22.981 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.981 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.981 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:22.981 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:22.982 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.982 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.982 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_tuo9kCpofi-fHLKLK3syMDP", 2024-08-18T15:45:22.982 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.982 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.982 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.982 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:22.982 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.982 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.982 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.982 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.982 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995081.692441_281.1", 2024-08-18T15:45:22.982 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:22.983 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:45:22.983 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:31:21.692441804Z", 2024-08-18T15:45:22.983 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.983 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.983 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:22.983 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:22.983 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.983 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.983 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_tuo9kCpofi-fHLKLK3syMDP", 2024-08-18T15:45:22.983 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.983 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.983 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.983 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:22.984 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.984 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.984 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.984 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.984 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995081.693378_282.1", 2024-08-18T15:45:22.984 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:22.984 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-21", 2024-08-18T15:45:22.984 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:31:21.693378470Z", 2024-08-18T15:45:22.984 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.984 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.984 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:22.984 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:22.984 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.985 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.985 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_ntCDeW1uH7FFuKrSyPKYHJm", 2024-08-18T15:45:22.985 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.985 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.985 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.985 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:22.985 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.985 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.985 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.985 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.985 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995081.695800_283.1", 2024-08-18T15:45:22.985 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:22.985 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-21", 2024-08-18T15:45:22.985 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:31:21.695800030Z", 2024-08-18T15:45:22.986 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.986 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.986 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:22.986 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:22.986 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.986 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.986 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_ntCDeW1uH7FFuKrSyPKYHJm", 2024-08-18T15:45:22.986 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.986 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.986 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.986 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:22.986 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.987 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.987 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.987 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.987 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995081.706620_284.1", 2024-08-18T15:45:22.987 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:22.987 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:45:22.987 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:31:21.706620679Z", 2024-08-18T15:45:22.987 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.987 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.987 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_tgDMSIYDTOiTpHaxXYDAzVY", 2024-08-18T15:45:22.987 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.987 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.987 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.987 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_tgDMSIYDTOiTpHaxXYDAzVY", 2024-08-18T15:45:22.988 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:22.988 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.988 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.988 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:22.988 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.988 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.988 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.988 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.988 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995081.709128_285.1", 2024-08-18T15:45:22.988 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:22.988 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:45:22.988 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:31:21.709128917Z", 2024-08-18T15:45:22.988 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.988 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.989 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_tgDMSIYDTOiTpHaxXYDAzVY", 2024-08-18T15:45:22.989 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.989 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.989 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.989 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_tgDMSIYDTOiTpHaxXYDAzVY", 2024-08-18T15:45:22.989 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:22.989 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.989 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.989 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:22.989 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.989 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.989 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.989 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.990 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995081.712458_286.1", 2024-08-18T15:45:22.990 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:22.990 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:45:22.990 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:31:21.712458353Z", 2024-08-18T15:45:22.990 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.990 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.990 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_Tq_y8CH-R5JTA5co_8cOHJr", 2024-08-18T15:45:22.990 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.990 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.990 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.990 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_Tq_y8CH-R5JTA5co_8cOHJr", 2024-08-18T15:45:22.990 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:22.990 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.990 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.991 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:22.991 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.991 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.991 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.991 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.991 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995081.714923_287.1", 2024-08-18T15:45:22.991 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:22.991 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:45:22.991 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:31:21.714923303Z", 2024-08-18T15:45:22.991 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.991 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.991 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_Tq_y8CH-R5JTA5co_8cOHJr", 2024-08-18T15:45:22.991 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.992 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.992 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.992 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_Tq_y8CH-R5JTA5co_8cOHJr", 2024-08-18T15:45:22.992 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:22.992 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.992 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.992 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:22.992 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.992 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.992 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.992 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.992 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995081.719853_288.1", 2024-08-18T15:45:22.992 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:22.992 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:45:22.993 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:31:21.719853955Z", 2024-08-18T15:45:22.993 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.993 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.993 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_tuo9kCpofi-fHLKLK3syMDP", 2024-08-18T15:45:22.993 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.993 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.993 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.993 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_tuo9kCpofi-fHLKLK3syMDP", 2024-08-18T15:45:22.993 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:22.993 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.993 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.993 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:22.993 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.994 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.994 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.994 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.994 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995081.722384_289.1", 2024-08-18T15:45:22.994 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:22.994 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:45:22.994 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:31:21.722384196Z", 2024-08-18T15:45:22.994 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.994 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.994 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_tuo9kCpofi-fHLKLK3syMDP", 2024-08-18T15:45:22.994 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.994 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.994 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.995 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_tuo9kCpofi-fHLKLK3syMDP", 2024-08-18T15:45:22.995 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:22.995 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.995 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.995 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:22.995 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.995 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.995 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.995 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.995 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995271.336261_378.1", 2024-08-18T15:45:22.995 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:22.995 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:45:22.995 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:34:31.336261807Z", 2024-08-18T15:45:22.996 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.996 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.996 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:22.996 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:22.996 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.996 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.996 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_pdWHtmdUtmS11GVZZnyUudX", 2024-08-18T15:45:22.996 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.996 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.996 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.996 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:22.996 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.996 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.996 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.997 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.997 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995271.339446_379.1", 2024-08-18T15:45:22.997 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:22.997 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:45:22.997 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:34:31.339446352Z", 2024-08-18T15:45:22.997 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.997 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.997 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:22.997 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:22.997 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.997 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.997 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_pdWHtmdUtmS11GVZZnyUudX", 2024-08-18T15:45:22.997 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.997 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.998 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.998 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:22.998 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.998 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.998 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.998 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.998 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995271.340841_380.1", 2024-08-18T15:45:22.998 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:22.998 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-22", 2024-08-18T15:45:22.998 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:34:31.340841665Z", 2024-08-18T15:45:22.998 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:22.998 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:22.998 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:22.999 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:22.999 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.999 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:22.999 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_O9sHsrAa3NkoKI1YNBKaQpY", 2024-08-18T15:45:22.999 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:22.999 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.999 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:22.999 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:22.999 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.999 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:22.999 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:22.999 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:22.999 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995271.343670_381.1", 2024-08-18T15:45:23.000 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.000 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-22", 2024-08-18T15:45:23.000 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:34:31.343670599Z", 2024-08-18T15:45:23.000 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.000 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.000 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.000 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.000 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.000 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.000 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_O9sHsrAa3NkoKI1YNBKaQpY", 2024-08-18T15:45:23.000 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.000 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.000 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.000 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.001 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.001 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.001 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.001 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.001 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995271.347218_382.1", 2024-08-18T15:45:23.001 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.001 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:45:23.001 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:34:31.347218065Z", 2024-08-18T15:45:23.001 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.001 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.001 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_pdWHtmdUtmS11GVZZnyUudX", 2024-08-18T15:45:23.001 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.001 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.001 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.002 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_pdWHtmdUtmS11GVZZnyUudX", 2024-08-18T15:45:23.002 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.002 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.002 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.002 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.002 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.002 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.002 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.002 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.002 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995271.350113_383.1", 2024-08-18T15:45:23.002 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.002 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:45:23.002 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:34:31.350113109Z", 2024-08-18T15:45:23.003 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.003 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.003 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_pdWHtmdUtmS11GVZZnyUudX", 2024-08-18T15:45:23.003 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.003 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.003 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.003 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_pdWHtmdUtmS11GVZZnyUudX", 2024-08-18T15:45:23.003 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.003 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.003 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.003 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.003 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.003 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.004 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.004 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.004 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995462.569535_473.1", 2024-08-18T15:45:23.004 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.004 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:45:23.004 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:37:42.569535297Z", 2024-08-18T15:45:23.004 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.004 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.004 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.004 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.004 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.004 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.004 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_wpzt3Zxvq-oNY9mOoRnPKWQ", 2024-08-18T15:45:23.005 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.005 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.005 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.005 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.005 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.005 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.005 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.005 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.005 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995462.571879_474.1", 2024-08-18T15:45:23.005 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.005 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:45:23.005 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:37:42.571879550Z", 2024-08-18T15:45:23.005 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.006 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.006 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.006 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.006 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.006 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.006 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_wpzt3Zxvq-oNY9mOoRnPKWQ", 2024-08-18T15:45:23.006 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.006 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.006 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.006 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.006 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.006 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.007 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.007 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.007 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995462.572788_475.1", 2024-08-18T15:45:23.007 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.007 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-23", 2024-08-18T15:45:23.007 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:37:42.572788757Z", 2024-08-18T15:45:23.007 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.007 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.007 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.007 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.007 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.007 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.007 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_uJpVLZnIMl1lW-0wuWAfT4Y", 2024-08-18T15:45:23.008 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.008 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.008 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.008 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.008 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.008 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.008 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.008 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.008 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995462.575132_476.1", 2024-08-18T15:45:23.008 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.008 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-23", 2024-08-18T15:45:23.008 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:37:42.575132214Z", 2024-08-18T15:45:23.008 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.009 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.009 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.009 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.009 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.009 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.009 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_uJpVLZnIMl1lW-0wuWAfT4Y", 2024-08-18T15:45:23.009 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.009 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.009 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.009 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.009 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.009 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.009 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.009 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.010 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995503.967200_486.1", 2024-08-18T15:45:23.010 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.010 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:45:23.010 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:38:23.967200465Z", 2024-08-18T15:45:23.010 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.010 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.010 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_wpzt3Zxvq-oNY9mOoRnPKWQ", 2024-08-18T15:45:23.010 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.010 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.010 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.010 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_wpzt3Zxvq-oNY9mOoRnPKWQ", 2024-08-18T15:45:23.010 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.010 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.011 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.011 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.011 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.011 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.011 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.011 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.011 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995503.969777_487.1", 2024-08-18T15:45:23.011 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.011 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:45:23.011 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:38:23.969777196Z", 2024-08-18T15:45:23.011 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.011 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.011 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_wpzt3Zxvq-oNY9mOoRnPKWQ", 2024-08-18T15:45:23.012 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.012 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.012 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.012 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_wpzt3Zxvq-oNY9mOoRnPKWQ", 2024-08-18T15:45:23.012 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.012 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.012 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.012 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.012 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.012 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.012 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.012 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.012 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995565.569248_530.1", 2024-08-18T15:45:23.012 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.013 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:45:23.013 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:39:25.569248856Z", 2024-08-18T15:45:23.013 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.013 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.013 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_wpzt3Zxvq-oNY9mOoRnPKWQ", 2024-08-18T15:45:23.013 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.013 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.013 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.013 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_wpzt3Zxvq-oNY9mOoRnPKWQ", 2024-08-18T15:45:23.013 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 3 2024-08-18T15:45:23.013 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.013 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.013 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.014 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.014 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.014 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.014 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.014 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995565.571903_531.1", 2024-08-18T15:45:23.014 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.014 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:45:23.014 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:39:25.571903234Z", 2024-08-18T15:45:23.014 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.014 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.014 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_wpzt3Zxvq-oNY9mOoRnPKWQ", 2024-08-18T15:45:23.014 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.014 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.015 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.015 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_wpzt3Zxvq-oNY9mOoRnPKWQ", 2024-08-18T15:45:23.015 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 3 2024-08-18T15:45:23.015 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.015 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.015 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.015 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.015 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.015 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.015 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.015 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995627.134415_559.1", 2024-08-18T15:45:23.015 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.015 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:45:23.015 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:40:27.134415241Z", 2024-08-18T15:45:23.016 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.016 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.016 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.016 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.016 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.016 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.016 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_-bWXVuDtxeg-vFcENUxwyqY", 2024-08-18T15:45:23.016 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.016 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.016 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.016 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.016 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.016 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.017 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.017 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.017 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995627.137158_560.1", 2024-08-18T15:45:23.017 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.017 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:45:23.017 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:40:27.137158269Z", 2024-08-18T15:45:23.017 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.017 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.017 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.017 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.017 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.017 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.017 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_-bWXVuDtxeg-vFcENUxwyqY", 2024-08-18T15:45:23.018 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.018 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.018 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.018 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.018 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.018 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.018 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.018 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.018 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995627.138166_561.1", 2024-08-18T15:45:23.018 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.018 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-24", 2024-08-18T15:45:23.018 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:40:27.138166620Z", 2024-08-18T15:45:23.018 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.019 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.019 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.019 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.019 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.019 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.019 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_GqmPD6qFaIdZi7CGWg-6cQ9", 2024-08-18T15:45:23.019 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.019 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.019 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.019 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.019 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.019 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.019 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.020 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.020 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995627.140690_562.1", 2024-08-18T15:45:23.020 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.020 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-24", 2024-08-18T15:45:23.020 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:40:27.140690252Z", 2024-08-18T15:45:23.020 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.020 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.020 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.020 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.020 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.020 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.020 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_GqmPD6qFaIdZi7CGWg-6cQ9", 2024-08-18T15:45:23.020 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.021 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.021 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.021 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.021 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.021 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.021 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.021 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.021 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995627.150973_563.1", 2024-08-18T15:45:23.021 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.021 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:45:23.021 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:40:27.150973118Z", 2024-08-18T15:45:23.021 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.021 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.021 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.022 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.022 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.022 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.022 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_ZASiQO1nC1KO4fWWmGbgTG6", 2024-08-18T15:45:23.022 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.022 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.022 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.022 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.022 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.022 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.022 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.022 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.022 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995627.153561_564.1", 2024-08-18T15:45:23.023 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.023 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:45:23.023 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:40:27.153561617Z", 2024-08-18T15:45:23.023 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.023 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.023 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.023 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.023 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.023 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.023 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_ZASiQO1nC1KO4fWWmGbgTG6", 2024-08-18T15:45:23.023 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.023 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.023 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.023 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.024 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.024 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.024 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.024 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.024 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995627.154521_565.1", 2024-08-18T15:45:23.024 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.024 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-25", 2024-08-18T15:45:23.024 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:40:27.154521041Z", 2024-08-18T15:45:23.024 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.024 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.024 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.024 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.024 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.025 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.025 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_IcFaB2XBAat2mP6-7HsS5nj", 2024-08-18T15:45:23.025 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.025 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.025 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.025 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.025 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.025 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.025 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.025 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.025 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995627.156830_566.1", 2024-08-18T15:45:23.025 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.025 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-25", 2024-08-18T15:45:23.025 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:40:27.156830184Z", 2024-08-18T15:45:23.026 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.026 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.026 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.026 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.026 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.026 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.026 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_IcFaB2XBAat2mP6-7HsS5nj", 2024-08-18T15:45:23.026 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.026 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.026 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.026 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.026 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.027 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.027 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.027 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.027 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995627.173949_567.1", 2024-08-18T15:45:23.027 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.027 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:45:23.027 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:40:27.173949128Z", 2024-08-18T15:45:23.027 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.027 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.027 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.027 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.027 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.027 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.028 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_UDZD1orQ5TbJSSveyG1C6yO", 2024-08-18T15:45:23.028 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.028 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.028 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.028 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.028 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.028 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.028 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.028 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.028 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995627.176614_568.1", 2024-08-18T15:45:23.028 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.028 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:45:23.028 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:40:27.176614276Z", 2024-08-18T15:45:23.028 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.029 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.029 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.029 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.029 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.029 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.029 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_UDZD1orQ5TbJSSveyG1C6yO", 2024-08-18T15:45:23.029 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.029 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.029 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.029 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.029 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.029 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.029 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.030 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.030 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995627.177588_569.1", 2024-08-18T15:45:23.030 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.030 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-26", 2024-08-18T15:45:23.030 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:40:27.177588755Z", 2024-08-18T15:45:23.030 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.030 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.030 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.030 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.030 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.030 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.030 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_6T2erh0-8LIJN2ZhFD-d0zv", 2024-08-18T15:45:23.030 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.030 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.031 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.031 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.031 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.031 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.031 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.031 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.031 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995627.180057_570.1", 2024-08-18T15:45:23.031 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.031 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-26", 2024-08-18T15:45:23.031 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:40:27.180057289Z", 2024-08-18T15:45:23.031 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.031 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.031 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.032 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.032 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.032 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.032 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_6T2erh0-8LIJN2ZhFD-d0zv", 2024-08-18T15:45:23.032 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.032 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.032 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.032 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.032 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.032 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.032 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.032 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.032 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995627.190598_571.1", 2024-08-18T15:45:23.032 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.033 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:45:23.033 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:40:27.190598553Z", 2024-08-18T15:45:23.033 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.033 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.033 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_-bWXVuDtxeg-vFcENUxwyqY", 2024-08-18T15:45:23.033 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.033 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.033 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.033 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_-bWXVuDtxeg-vFcENUxwyqY", 2024-08-18T15:45:23.033 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.033 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.033 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.033 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.034 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.034 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.034 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.034 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.034 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995627.193264_572.1", 2024-08-18T15:45:23.034 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.034 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:45:23.034 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:40:27.193264293Z", 2024-08-18T15:45:23.034 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.034 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.034 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_-bWXVuDtxeg-vFcENUxwyqY", 2024-08-18T15:45:23.034 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.034 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.035 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.035 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_-bWXVuDtxeg-vFcENUxwyqY", 2024-08-18T15:45:23.035 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.035 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.035 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.035 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.035 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.035 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.035 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.035 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.035 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995627.196357_573.1", 2024-08-18T15:45:23.035 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.035 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:45:23.035 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:40:27.196357945Z", 2024-08-18T15:45:23.036 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.036 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.036 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_ZASiQO1nC1KO4fWWmGbgTG6", 2024-08-18T15:45:23.036 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.036 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.036 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.036 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_ZASiQO1nC1KO4fWWmGbgTG6", 2024-08-18T15:45:23.036 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.036 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.036 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.036 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.036 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.036 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.036 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.037 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.037 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995627.198715_574.1", 2024-08-18T15:45:23.037 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.037 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:45:23.037 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:40:27.198715261Z", 2024-08-18T15:45:23.037 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.037 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.037 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_ZASiQO1nC1KO4fWWmGbgTG6", 2024-08-18T15:45:23.037 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.037 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.037 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.037 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_ZASiQO1nC1KO4fWWmGbgTG6", 2024-08-18T15:45:23.037 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.038 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.038 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.038 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.038 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.038 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.038 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.038 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.038 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995627.203450_575.1", 2024-08-18T15:45:23.038 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.038 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:45:23.038 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:40:27.203450736Z", 2024-08-18T15:45:23.038 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.038 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.038 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_UDZD1orQ5TbJSSveyG1C6yO", 2024-08-18T15:45:23.039 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.039 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.039 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.039 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_UDZD1orQ5TbJSSveyG1C6yO", 2024-08-18T15:45:23.039 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.039 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.039 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.039 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.039 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.039 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.039 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.039 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.039 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995627.205732_576.1", 2024-08-18T15:45:23.040 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.040 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:45:23.040 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:40:27.205732200Z", 2024-08-18T15:45:23.040 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.040 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.040 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_UDZD1orQ5TbJSSveyG1C6yO", 2024-08-18T15:45:23.040 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.040 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.040 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.040 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_UDZD1orQ5TbJSSveyG1C6yO", 2024-08-18T15:45:23.040 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.040 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.040 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.041 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.041 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.041 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.041 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.041 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.041 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995690.547603_610.1", 2024-08-18T15:45:23.041 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.041 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:45:23.041 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:41:30.547603460Z", 2024-08-18T15:45:23.041 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.041 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.041 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.041 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.041 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.042 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.042 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_jeYug1evj8xbYPXyUA9ZAIF", 2024-08-18T15:45:23.042 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.042 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.042 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.042 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.042 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.042 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.042 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.042 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.042 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995690.550097_611.1", 2024-08-18T15:45:23.042 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.042 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:45:23.043 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:41:30.550097288Z", 2024-08-18T15:45:23.043 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.043 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.043 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.043 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.043 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.043 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.043 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_jeYug1evj8xbYPXyUA9ZAIF", 2024-08-18T15:45:23.043 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.043 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.043 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.043 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.043 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.043 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.044 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.044 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.044 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995690.551077_612.1", 2024-08-18T15:45:23.044 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.044 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-27", 2024-08-18T15:45:23.044 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:41:30.551077825Z", 2024-08-18T15:45:23.044 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.044 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.044 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.044 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.044 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.044 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.044 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_2Y1Gbxx4l53IXmgzA2xQDmh", 2024-08-18T15:45:23.044 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.045 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.045 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.045 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.045 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.045 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.045 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.045 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.045 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995690.553419_613.1", 2024-08-18T15:45:23.045 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.045 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-27", 2024-08-18T15:45:23.045 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:41:30.553419971Z", 2024-08-18T15:45:23.045 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.045 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.045 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.046 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.046 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.046 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.046 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_2Y1Gbxx4l53IXmgzA2xQDmh", 2024-08-18T15:45:23.046 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.046 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.046 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.046 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.046 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.046 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.046 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.046 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.046 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995690.562778_614.1", 2024-08-18T15:45:23.046 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.047 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:45:23.047 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:41:30.562778163Z", 2024-08-18T15:45:23.047 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.047 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.047 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.047 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.047 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.047 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.047 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_5sF5e5L5wBDjZwv2FwY2xZ2", 2024-08-18T15:45:23.047 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.047 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.047 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.048 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.048 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.048 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.048 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.048 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.048 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995690.565048_615.1", 2024-08-18T15:45:23.048 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.048 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:45:23.048 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:41:30.565048948Z", 2024-08-18T15:45:23.048 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.048 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.048 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.048 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.048 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.049 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.049 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_5sF5e5L5wBDjZwv2FwY2xZ2", 2024-08-18T15:45:23.049 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.049 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.049 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.049 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.049 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.049 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.049 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.049 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.049 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995690.566055_616.1", 2024-08-18T15:45:23.049 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.049 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-28", 2024-08-18T15:45:23.050 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:41:30.566055028Z", 2024-08-18T15:45:23.050 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.050 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.050 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.050 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.050 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.050 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.050 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_3Xacd7LBrQbPx_GZINBqEvJ", 2024-08-18T15:45:23.050 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.050 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.050 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.050 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.050 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.051 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.051 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.051 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.051 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995690.568292_617.1", 2024-08-18T15:45:23.051 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.051 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-28", 2024-08-18T15:45:23.051 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:41:30.568292915Z", 2024-08-18T15:45:23.051 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.051 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.051 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.051 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.051 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.051 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.052 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_3Xacd7LBrQbPx_GZINBqEvJ", 2024-08-18T15:45:23.052 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.052 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.052 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.052 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.052 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.052 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.052 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.052 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.052 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995690.583629_618.1", 2024-08-18T15:45:23.052 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.052 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:45:23.052 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:41:30.583629631Z", 2024-08-18T15:45:23.052 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.053 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.053 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.053 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.053 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.053 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.053 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_Ds7izdnTDTHShaa6cA2hQJO", 2024-08-18T15:45:23.053 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.053 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.053 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.053 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.053 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.053 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.053 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.054 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.054 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995690.585913_619.1", 2024-08-18T15:45:23.054 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.054 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:45:23.054 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:41:30.585913147Z", 2024-08-18T15:45:23.054 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.054 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.054 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.054 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.054 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.054 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.054 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_Ds7izdnTDTHShaa6cA2hQJO", 2024-08-18T15:45:23.054 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.055 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.055 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.055 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.055 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.055 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.055 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.055 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.055 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995690.586806_620.1", 2024-08-18T15:45:23.055 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.055 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-29", 2024-08-18T15:45:23.055 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:41:30.586806831Z", 2024-08-18T15:45:23.055 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.055 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.056 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.056 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.056 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.056 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.056 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_daU6Le-dTETKkQatmqfff9r", 2024-08-18T15:45:23.056 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.056 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.056 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.056 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.056 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.056 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.056 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.056 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.056 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995690.589012_621.1", 2024-08-18T15:45:23.057 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.057 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-29", 2024-08-18T15:45:23.057 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:41:30.589012688Z", 2024-08-18T15:45:23.057 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.057 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.057 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.057 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.057 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.057 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.057 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_daU6Le-dTETKkQatmqfff9r", 2024-08-18T15:45:23.057 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.057 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.057 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.058 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.058 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.058 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.058 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.058 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.058 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995690.598776_622.1", 2024-08-18T15:45:23.058 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.058 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:45:23.058 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:41:30.598776328Z", 2024-08-18T15:45:23.058 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.058 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.058 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_jeYug1evj8xbYPXyUA9ZAIF", 2024-08-18T15:45:23.058 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.058 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.059 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.059 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_jeYug1evj8xbYPXyUA9ZAIF", 2024-08-18T15:45:23.059 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.059 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.059 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.059 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.059 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.059 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.059 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.059 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.059 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995690.601154_623.1", 2024-08-18T15:45:23.060 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.060 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:45:23.060 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:41:30.601154145Z", 2024-08-18T15:45:23.060 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.060 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.060 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_jeYug1evj8xbYPXyUA9ZAIF", 2024-08-18T15:45:23.060 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.060 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.060 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.060 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_jeYug1evj8xbYPXyUA9ZAIF", 2024-08-18T15:45:23.060 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.060 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.061 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.061 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.061 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.061 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.061 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.061 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.061 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995690.603208_624.1", 2024-08-18T15:45:23.061 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.061 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:45:23.061 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:41:30.603208651Z", 2024-08-18T15:45:23.061 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.061 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.061 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_jeYug1evj8xbYPXyUA9ZAIF", 2024-08-18T15:45:23.061 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.062 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.062 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.062 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_jeYug1evj8xbYPXyUA9ZAIF", 2024-08-18T15:45:23.062 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 3 2024-08-18T15:45:23.062 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.062 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.062 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.062 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.062 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.062 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.062 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.062 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995690.605370_625.1", 2024-08-18T15:45:23.063 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.063 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:45:23.063 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:41:30.605370190Z", 2024-08-18T15:45:23.063 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.063 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.063 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_jeYug1evj8xbYPXyUA9ZAIF", 2024-08-18T15:45:23.063 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.063 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.063 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.063 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_jeYug1evj8xbYPXyUA9ZAIF", 2024-08-18T15:45:23.064 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 3 2024-08-18T15:45:23.064 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.064 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.064 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.064 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.064 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.064 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.064 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.064 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995690.607911_626.1", 2024-08-18T15:45:23.064 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.064 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:45:23.064 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:41:30.607911940Z", 2024-08-18T15:45:23.064 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.065 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.065 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_5sF5e5L5wBDjZwv2FwY2xZ2", 2024-08-18T15:45:23.065 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.065 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.065 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.065 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_5sF5e5L5wBDjZwv2FwY2xZ2", 2024-08-18T15:45:23.065 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.065 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.065 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.065 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.065 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.065 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.065 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.066 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.066 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995690.610104_627.1", 2024-08-18T15:45:23.066 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.066 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:45:23.066 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:41:30.610104648Z", 2024-08-18T15:45:23.066 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.066 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.066 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_5sF5e5L5wBDjZwv2FwY2xZ2", 2024-08-18T15:45:23.066 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.066 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.066 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.066 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_5sF5e5L5wBDjZwv2FwY2xZ2", 2024-08-18T15:45:23.066 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.066 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.067 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.067 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.067 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.067 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.067 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.067 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.067 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995690.614660_628.1", 2024-08-18T15:45:23.067 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.067 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:45:23.067 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:41:30.614660827Z", 2024-08-18T15:45:23.067 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.067 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.067 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_5sF5e5L5wBDjZwv2FwY2xZ2", 2024-08-18T15:45:23.068 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.068 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.068 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.068 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_5sF5e5L5wBDjZwv2FwY2xZ2", 2024-08-18T15:45:23.068 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 3 2024-08-18T15:45:23.068 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.068 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.068 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.068 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.068 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.068 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.068 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.068 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995690.616928_629.1", 2024-08-18T15:45:23.068 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.069 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:45:23.069 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:41:30.616928990Z", 2024-08-18T15:45:23.069 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.069 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.069 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_5sF5e5L5wBDjZwv2FwY2xZ2", 2024-08-18T15:45:23.069 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.069 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.069 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.069 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_5sF5e5L5wBDjZwv2FwY2xZ2", 2024-08-18T15:45:23.069 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 3 2024-08-18T15:45:23.069 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.069 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.069 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.069 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.070 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.070 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.070 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.070 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995690.621300_630.1", 2024-08-18T15:45:23.070 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.070 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:45:23.070 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:41:30.621300062Z", 2024-08-18T15:45:23.070 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.070 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.070 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_Ds7izdnTDTHShaa6cA2hQJO", 2024-08-18T15:45:23.070 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.070 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.070 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.071 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_Ds7izdnTDTHShaa6cA2hQJO", 2024-08-18T15:45:23.071 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.071 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.071 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.071 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.071 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.071 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.071 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.071 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.071 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995690.623904_631.1", 2024-08-18T15:45:23.071 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.071 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:45:23.071 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:41:30.623904678Z", 2024-08-18T15:45:23.072 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.072 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.072 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_Ds7izdnTDTHShaa6cA2hQJO", 2024-08-18T15:45:23.072 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.072 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.072 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.072 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_Ds7izdnTDTHShaa6cA2hQJO", 2024-08-18T15:45:23.072 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.072 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.072 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.072 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.072 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.072 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.072 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.073 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.073 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995690.627942_632.1", 2024-08-18T15:45:23.073 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.073 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:45:23.073 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:41:30.627942660Z", 2024-08-18T15:45:23.073 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.073 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.073 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_Ds7izdnTDTHShaa6cA2hQJO", 2024-08-18T15:45:23.073 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.073 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.073 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.073 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_Ds7izdnTDTHShaa6cA2hQJO", 2024-08-18T15:45:23.073 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 3 2024-08-18T15:45:23.074 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.074 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.074 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.074 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.074 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.074 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.074 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.074 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995690.630163_633.1", 2024-08-18T15:45:23.074 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.074 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:45:23.074 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:41:30.630163840Z", 2024-08-18T15:45:23.074 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.074 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.074 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_Ds7izdnTDTHShaa6cA2hQJO", 2024-08-18T15:45:23.075 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.075 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.075 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.075 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_Ds7izdnTDTHShaa6cA2hQJO", 2024-08-18T15:45:23.075 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 3 2024-08-18T15:45:23.075 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.075 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.075 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.075 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.075 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.075 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.075 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.075 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.521479_647.1", 2024-08-18T15:45:23.076 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.076 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-30:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.30", 2024-08-18T15:45:23.076 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.521479385Z", 2024-08-18T15:45:23.076 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.076 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.076 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.076 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.076 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.076 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.076 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_X8eWxctwhKvpxc5MGUbCdeT", 2024-08-18T15:45:23.076 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.076 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.076 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.076 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.077 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.077 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.077 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.077 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.077 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.524106_648.1", 2024-08-18T15:45:23.077 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.077 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-30:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.30", 2024-08-18T15:45:23.077 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.524106942Z", 2024-08-18T15:45:23.077 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.077 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.077 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.077 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.077 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.078 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.078 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_X8eWxctwhKvpxc5MGUbCdeT", 2024-08-18T15:45:23.078 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.078 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.078 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.078 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.078 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.078 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.078 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.078 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.078 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.525089_649.1", 2024-08-18T15:45:23.078 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.078 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-30", 2024-08-18T15:45:23.078 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.525089002Z", 2024-08-18T15:45:23.079 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.079 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.079 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.079 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.079 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.079 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.079 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_Y_twJ8vlrnjpsWlMf8A1vcl", 2024-08-18T15:45:23.079 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.079 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.079 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.079 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.079 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.079 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.080 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.080 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.080 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.527509_650.1", 2024-08-18T15:45:23.080 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.080 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-30", 2024-08-18T15:45:23.080 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.527509595Z", 2024-08-18T15:45:23.080 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.080 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.080 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.080 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.080 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.080 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.080 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_Y_twJ8vlrnjpsWlMf8A1vcl", 2024-08-18T15:45:23.080 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.081 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.081 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.081 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.081 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.081 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.081 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.081 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.081 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.537416_651.1", 2024-08-18T15:45:23.081 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.081 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-31:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.31", 2024-08-18T15:45:23.081 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.537416223Z", 2024-08-18T15:45:23.081 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.081 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.082 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.082 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.082 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.082 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.082 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_3mYRdBwrm6Dau3dYq-qztzU", 2024-08-18T15:45:23.082 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.082 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.082 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.082 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.082 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.082 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.082 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.082 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.083 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.539892_652.1", 2024-08-18T15:45:23.083 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.083 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-31:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.31", 2024-08-18T15:45:23.083 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.539892239Z", 2024-08-18T15:45:23.083 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.083 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.083 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.083 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.083 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.083 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.083 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_3mYRdBwrm6Dau3dYq-qztzU", 2024-08-18T15:45:23.083 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.083 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.083 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.084 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.084 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.084 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.084 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.084 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.084 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.541032_653.1", 2024-08-18T15:45:23.084 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.084 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-31", 2024-08-18T15:45:23.084 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.541032236Z", 2024-08-18T15:45:23.084 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.084 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.084 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.084 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.085 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.085 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.085 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_LwFQ9ExxsG8yYIT84yLWJCM", 2024-08-18T15:45:23.085 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.085 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.085 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.085 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.085 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.085 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.085 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.085 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.085 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.543337_654.1", 2024-08-18T15:45:23.085 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.085 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-31", 2024-08-18T15:45:23.086 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.543337427Z", 2024-08-18T15:45:23.086 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.086 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.086 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.086 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.086 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.086 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.086 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_LwFQ9ExxsG8yYIT84yLWJCM", 2024-08-18T15:45:23.086 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.086 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.086 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.086 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.086 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.086 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.087 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.087 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.087 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.559655_655.1", 2024-08-18T15:45:23.087 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.087 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-32:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.32", 2024-08-18T15:45:23.087 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.559655282Z", 2024-08-18T15:45:23.087 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.087 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.087 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.087 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.087 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.087 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.087 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_N4lXfxOh8y0R0tn_VtdDq--", 2024-08-18T15:45:23.088 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.088 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.088 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.088 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.088 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.088 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.088 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.088 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.088 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.562055_656.1", 2024-08-18T15:45:23.088 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.088 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-32:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.32", 2024-08-18T15:45:23.088 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.562055315Z", 2024-08-18T15:45:23.088 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.088 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.089 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.089 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.089 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.089 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.089 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_N4lXfxOh8y0R0tn_VtdDq--", 2024-08-18T15:45:23.089 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.089 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.089 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.089 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.089 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.089 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.089 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.089 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.090 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.562981_657.1", 2024-08-18T15:45:23.090 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.090 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-32", 2024-08-18T15:45:23.090 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.562981860Z", 2024-08-18T15:45:23.090 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.090 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.090 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.090 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.090 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.090 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.090 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_grj-gSxc_4fWbu_ET8qVUlc", 2024-08-18T15:45:23.090 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.090 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.090 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.091 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.091 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.091 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.091 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.091 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.091 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.565132_658.1", 2024-08-18T15:45:23.091 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.091 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-32", 2024-08-18T15:45:23.091 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.565132516Z", 2024-08-18T15:45:23.091 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.091 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.091 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.091 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.092 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.092 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.092 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_grj-gSxc_4fWbu_ET8qVUlc", 2024-08-18T15:45:23.092 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.092 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.092 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.092 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.092 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.092 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.092 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.092 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.092 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.615071_659.1", 2024-08-18T15:45:23.092 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.093 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-30:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.30", 2024-08-18T15:45:23.093 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.615071398Z", 2024-08-18T15:45:23.093 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.093 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.093 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_X8eWxctwhKvpxc5MGUbCdeT", 2024-08-18T15:45:23.093 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.093 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.093 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.093 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_X8eWxctwhKvpxc5MGUbCdeT", 2024-08-18T15:45:23.093 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.093 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.093 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.093 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.093 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.094 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.094 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.094 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.094 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.617533_660.1", 2024-08-18T15:45:23.094 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.094 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-30:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.30", 2024-08-18T15:45:23.094 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.617533647Z", 2024-08-18T15:45:23.094 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.094 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.094 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_X8eWxctwhKvpxc5MGUbCdeT", 2024-08-18T15:45:23.094 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.094 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.094 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.095 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_X8eWxctwhKvpxc5MGUbCdeT", 2024-08-18T15:45:23.095 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.095 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.095 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.095 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.095 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.095 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.095 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.095 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.095 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.622444_661.1", 2024-08-18T15:45:23.095 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.095 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-31:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.31", 2024-08-18T15:45:23.095 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.622444585Z", 2024-08-18T15:45:23.095 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.096 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.096 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_3mYRdBwrm6Dau3dYq-qztzU", 2024-08-18T15:45:23.096 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.096 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.096 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.096 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_3mYRdBwrm6Dau3dYq-qztzU", 2024-08-18T15:45:23.096 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.096 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.096 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.096 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.096 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.096 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.096 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.096 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.097 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.624798_662.1", 2024-08-18T15:45:23.097 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.097 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-31:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.31", 2024-08-18T15:45:23.097 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.624798339Z", 2024-08-18T15:45:23.097 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.097 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.097 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_3mYRdBwrm6Dau3dYq-qztzU", 2024-08-18T15:45:23.097 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.097 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.097 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.097 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_3mYRdBwrm6Dau3dYq-qztzU", 2024-08-18T15:45:23.097 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.097 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.097 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.098 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.098 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.098 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.098 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.098 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.098 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.673691_663.1", 2024-08-18T15:45:23.098 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.098 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-32:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.32", 2024-08-18T15:45:23.098 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.673691986Z", 2024-08-18T15:45:23.098 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.098 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.098 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_N4lXfxOh8y0R0tn_VtdDq--", 2024-08-18T15:45:23.098 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.099 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.099 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.099 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_N4lXfxOh8y0R0tn_VtdDq--", 2024-08-18T15:45:23.099 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.099 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.099 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.099 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.099 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.099 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.099 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.099 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.099 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.676028_664.1", 2024-08-18T15:45:23.099 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.099 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-32:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.32", 2024-08-18T15:45:23.100 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.676028814Z", 2024-08-18T15:45:23.100 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.100 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.100 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_N4lXfxOh8y0R0tn_VtdDq--", 2024-08-18T15:45:23.100 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.100 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.100 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.100 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_N4lXfxOh8y0R0tn_VtdDq--", 2024-08-18T15:45:23.100 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.100 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.100 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.100 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.100 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.100 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.100 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.101 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.101 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.684893_665.1", 2024-08-18T15:45:23.101 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.101 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-33:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.33", 2024-08-18T15:45:23.101 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.684893391Z", 2024-08-18T15:45:23.101 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.101 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.101 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.101 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.101 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.101 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.101 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_lGULOaXyl4hHx7qh454F-aa", 2024-08-18T15:45:23.101 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.101 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.102 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.102 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.102 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.102 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.102 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.102 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.102 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.687266_666.1", 2024-08-18T15:45:23.102 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.102 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-33:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.33", 2024-08-18T15:45:23.102 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.687266999Z", 2024-08-18T15:45:23.102 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.102 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.102 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.103 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.103 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.103 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.103 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_lGULOaXyl4hHx7qh454F-aa", 2024-08-18T15:45:23.103 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.103 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.103 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.103 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.103 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.103 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.103 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.103 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.103 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.688166_667.1", 2024-08-18T15:45:23.103 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.104 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-33", 2024-08-18T15:45:23.104 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.688166208Z", 2024-08-18T15:45:23.104 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.104 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.104 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.104 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.104 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.104 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.104 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_WydO77zfjR8Husc4yxYU8A8", 2024-08-18T15:45:23.104 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.104 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.104 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.104 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.104 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.105 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.105 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.105 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.105 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.690522_668.1", 2024-08-18T15:45:23.105 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.105 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-33", 2024-08-18T15:45:23.105 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.690522745Z", 2024-08-18T15:45:23.105 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.105 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.105 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.105 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.105 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.105 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.105 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_WydO77zfjR8Husc4yxYU8A8", 2024-08-18T15:45:23.106 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.106 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.106 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.106 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.106 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.106 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.106 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.106 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.106 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.699985_669.1", 2024-08-18T15:45:23.106 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.106 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-34:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.34", 2024-08-18T15:45:23.106 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.699985189Z", 2024-08-18T15:45:23.106 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.107 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.107 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.107 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.107 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.107 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.107 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_gziC3Zhjn59sc9f24iXYqez", 2024-08-18T15:45:23.107 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.107 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.107 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.107 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.107 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.107 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.107 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.107 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.108 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.702346_670.1", 2024-08-18T15:45:23.108 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.108 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-34:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.34", 2024-08-18T15:45:23.108 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.702346126Z", 2024-08-18T15:45:23.108 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.108 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.108 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.108 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.108 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.108 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.108 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_gziC3Zhjn59sc9f24iXYqez", 2024-08-18T15:45:23.108 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.108 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.109 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.109 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.109 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.109 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.109 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.109 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.109 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.703327_671.1", 2024-08-18T15:45:23.109 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.109 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-34", 2024-08-18T15:45:23.109 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.703327689Z", 2024-08-18T15:45:23.109 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.109 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.109 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.109 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.110 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.110 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.110 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_fIT8XfQf3mP3Rdkpjxd4TDK", 2024-08-18T15:45:23.110 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.110 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.110 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.110 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.110 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.110 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.110 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.110 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.110 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.705678_672.1", 2024-08-18T15:45:23.110 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.111 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-34", 2024-08-18T15:45:23.111 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.705678013Z", 2024-08-18T15:45:23.111 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.111 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.111 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.111 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.111 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.111 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.111 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_fIT8XfQf3mP3Rdkpjxd4TDK", 2024-08-18T15:45:23.111 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.111 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.111 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.111 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.111 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.112 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.112 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.112 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.112 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.722023_673.1", 2024-08-18T15:45:23.112 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.112 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-35:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.35", 2024-08-18T15:45:23.112 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.722023363Z", 2024-08-18T15:45:23.112 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.112 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.112 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.112 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.112 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.112 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.113 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_8PQDI19UyDbSfg9jXHxQK0Z", 2024-08-18T15:45:23.113 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.113 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.113 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.113 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.113 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.113 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.113 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.113 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.113 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.724501_674.1", 2024-08-18T15:45:23.113 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.113 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-35:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.35", 2024-08-18T15:45:23.113 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.724501528Z", 2024-08-18T15:45:23.114 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.114 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.114 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.114 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.114 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.114 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.114 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_8PQDI19UyDbSfg9jXHxQK0Z", 2024-08-18T15:45:23.114 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.114 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.114 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.114 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.114 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.114 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.115 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.115 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.115 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.725468_675.1", 2024-08-18T15:45:23.115 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.115 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-35", 2024-08-18T15:45:23.115 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.725468008Z", 2024-08-18T15:45:23.115 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.115 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.115 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.115 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.115 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.115 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.115 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_5pwJU04pzA5ySeDPbaf8cI9", 2024-08-18T15:45:23.116 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.116 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.116 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.116 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.116 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.116 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.116 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.116 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.116 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.728040_676.1", 2024-08-18T15:45:23.116 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.116 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-35", 2024-08-18T15:45:23.116 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.728040336Z", 2024-08-18T15:45:23.116 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.117 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.117 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.117 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.117 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.117 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.117 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_5pwJU04pzA5ySeDPbaf8cI9", 2024-08-18T15:45:23.117 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.117 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.117 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.117 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.117 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.117 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.117 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.117 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.118 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.744014_677.1", 2024-08-18T15:45:23.118 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.118 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-33:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.33", 2024-08-18T15:45:23.118 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.744014055Z", 2024-08-18T15:45:23.118 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.118 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.118 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_lGULOaXyl4hHx7qh454F-aa", 2024-08-18T15:45:23.118 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.118 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.118 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.118 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_lGULOaXyl4hHx7qh454F-aa", 2024-08-18T15:45:23.118 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.118 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.119 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.119 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.119 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.119 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.119 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.119 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.119 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.746404_678.1", 2024-08-18T15:45:23.119 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.119 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-33:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.33", 2024-08-18T15:45:23.119 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.746404003Z", 2024-08-18T15:45:23.119 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.119 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.119 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_lGULOaXyl4hHx7qh454F-aa", 2024-08-18T15:45:23.119 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.120 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.120 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.120 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_lGULOaXyl4hHx7qh454F-aa", 2024-08-18T15:45:23.120 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.120 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.120 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.120 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.120 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.120 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.120 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.120 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.120 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.751650_679.1", 2024-08-18T15:45:23.120 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.121 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-34:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.34", 2024-08-18T15:45:23.121 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.751650666Z", 2024-08-18T15:45:23.121 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.121 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.121 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_gziC3Zhjn59sc9f24iXYqez", 2024-08-18T15:45:23.121 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.121 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.121 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.121 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_gziC3Zhjn59sc9f24iXYqez", 2024-08-18T15:45:23.121 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.121 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.121 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.121 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.121 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.122 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.122 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.122 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.122 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.754074_680.1", 2024-08-18T15:45:23.122 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.122 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-34:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.34", 2024-08-18T15:45:23.122 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.754074401Z", 2024-08-18T15:45:23.122 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.122 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.122 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_gziC3Zhjn59sc9f24iXYqez", 2024-08-18T15:45:23.122 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.122 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.122 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.123 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_gziC3Zhjn59sc9f24iXYqez", 2024-08-18T15:45:23.123 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.123 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.123 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.123 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.123 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.123 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.123 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.123 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.123 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.802966_681.1", 2024-08-18T15:45:23.123 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.123 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-35:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.35", 2024-08-18T15:45:23.123 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.802966900Z", 2024-08-18T15:45:23.124 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.124 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.124 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_8PQDI19UyDbSfg9jXHxQK0Z", 2024-08-18T15:45:23.124 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.124 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.124 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.124 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_8PQDI19UyDbSfg9jXHxQK0Z", 2024-08-18T15:45:23.124 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.124 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.124 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.124 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.124 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.124 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.124 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.125 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.125 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.805637_682.1", 2024-08-18T15:45:23.125 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.125 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-35:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.35", 2024-08-18T15:45:23.125 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.805637670Z", 2024-08-18T15:45:23.125 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.125 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.125 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_8PQDI19UyDbSfg9jXHxQK0Z", 2024-08-18T15:45:23.125 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.125 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.125 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.125 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_8PQDI19UyDbSfg9jXHxQK0Z", 2024-08-18T15:45:23.125 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.126 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.126 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.126 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.126 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.126 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.126 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.126 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.126 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.862218_683.1", 2024-08-18T15:45:23.126 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.126 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-36:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.36", 2024-08-18T15:45:23.126 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.862218121Z", 2024-08-18T15:45:23.126 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.126 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.126 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.127 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.127 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.127 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.127 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_OMlfwDqiYy7MLBPapdgXZCw", 2024-08-18T15:45:23.127 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.127 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.127 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.127 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.127 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.127 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.127 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.127 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.127 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.864692_684.1", 2024-08-18T15:45:23.127 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.128 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-36:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.36", 2024-08-18T15:45:23.128 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.864692243Z", 2024-08-18T15:45:23.128 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.128 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.128 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.128 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.128 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.128 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.128 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_OMlfwDqiYy7MLBPapdgXZCw", 2024-08-18T15:45:23.128 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.128 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.128 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.128 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.128 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.129 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.129 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.129 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.129 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.865659_685.1", 2024-08-18T15:45:23.129 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.129 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-36", 2024-08-18T15:45:23.129 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.865659998Z", 2024-08-18T15:45:23.129 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.129 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.129 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.129 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.129 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.129 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.129 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_dui98arcr6Cif7iTccJdz_E", 2024-08-18T15:45:23.130 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.130 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.130 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.130 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.130 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.130 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.130 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.130 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.130 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.867847_686.1", 2024-08-18T15:45:23.130 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.130 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-36", 2024-08-18T15:45:23.130 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.867847854Z", 2024-08-18T15:45:23.130 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.131 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.131 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.131 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.131 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.131 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.131 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_dui98arcr6Cif7iTccJdz_E", 2024-08-18T15:45:23.131 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.131 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.131 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.131 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.131 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.131 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.131 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.132 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.132 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.877645_687.1", 2024-08-18T15:45:23.132 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.132 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-37:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.37", 2024-08-18T15:45:23.132 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.877645769Z", 2024-08-18T15:45:23.132 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.132 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.132 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.132 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.132 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.132 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.132 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_HFwNCilj83ivBqE5tB_PZE2", 2024-08-18T15:45:23.132 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.132 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.133 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.133 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.133 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.133 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.133 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.133 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.133 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.879910_688.1", 2024-08-18T15:45:23.133 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.133 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-37:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.37", 2024-08-18T15:45:23.133 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.879910198Z", 2024-08-18T15:45:23.133 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.133 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.133 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.133 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.134 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.134 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.134 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_HFwNCilj83ivBqE5tB_PZE2", 2024-08-18T15:45:23.134 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.134 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.134 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.134 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.134 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.134 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.134 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.134 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.134 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.880848_689.1", 2024-08-18T15:45:23.135 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.135 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-37", 2024-08-18T15:45:23.135 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.880848207Z", 2024-08-18T15:45:23.135 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.135 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.135 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.135 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.135 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.135 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.135 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_rUYGyCScmgZ3SwWHWdQlE0I", 2024-08-18T15:45:23.135 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.135 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.135 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.136 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.136 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.136 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.136 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.136 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.136 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.883023_690.1", 2024-08-18T15:45:23.136 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.136 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-37", 2024-08-18T15:45:23.136 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.883023907Z", 2024-08-18T15:45:23.136 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.136 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.136 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.136 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.137 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.137 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.137 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_rUYGyCScmgZ3SwWHWdQlE0I", 2024-08-18T15:45:23.137 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.137 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.137 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.137 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.137 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.137 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.137 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.137 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.137 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.899500_691.1", 2024-08-18T15:45:23.137 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.137 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-38:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.38", 2024-08-18T15:45:23.138 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.899500733Z", 2024-08-18T15:45:23.138 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.138 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.138 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.138 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.138 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.138 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.138 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_i-eCmhQ4NPsJf86Skaj64Wy", 2024-08-18T15:45:23.138 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.138 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.138 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.138 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.138 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.139 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.139 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.139 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.139 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.901993_692.1", 2024-08-18T15:45:23.139 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.139 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-38:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.38", 2024-08-18T15:45:23.139 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.901993132Z", 2024-08-18T15:45:23.139 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.139 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.139 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.139 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.139 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.139 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.140 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_i-eCmhQ4NPsJf86Skaj64Wy", 2024-08-18T15:45:23.140 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.140 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.140 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.140 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.140 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.140 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.140 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.140 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.140 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.903041_693.1", 2024-08-18T15:45:23.140 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.140 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-38", 2024-08-18T15:45:23.140 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.903041404Z", 2024-08-18T15:45:23.140 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.141 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.141 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.141 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.141 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.141 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.141 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_i5jx3wzjmX5pErHvMHm_pkB", 2024-08-18T15:45:23.141 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.141 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.141 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.141 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.141 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.141 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.141 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.141 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.142 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.905154_694.1", 2024-08-18T15:45:23.142 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.142 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-38", 2024-08-18T15:45:23.142 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.905154856Z", 2024-08-18T15:45:23.142 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.142 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.142 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.142 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.142 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.142 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.142 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_i5jx3wzjmX5pErHvMHm_pkB", 2024-08-18T15:45:23.142 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.142 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.142 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.143 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.143 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.143 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.143 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.143 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.143 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.915169_695.1", 2024-08-18T15:45:23.143 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.143 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-36:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.36", 2024-08-18T15:45:23.143 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.915169406Z", 2024-08-18T15:45:23.143 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.143 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.143 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_OMlfwDqiYy7MLBPapdgXZCw", 2024-08-18T15:45:23.143 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.144 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.144 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.144 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_OMlfwDqiYy7MLBPapdgXZCw", 2024-08-18T15:45:23.144 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.144 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.144 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.144 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.144 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.144 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.144 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.144 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.144 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.917556_696.1", 2024-08-18T15:45:23.144 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.144 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-36:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.36", 2024-08-18T15:45:23.145 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.917556247Z", 2024-08-18T15:45:23.145 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.145 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.145 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_OMlfwDqiYy7MLBPapdgXZCw", 2024-08-18T15:45:23.145 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.145 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.145 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.145 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_OMlfwDqiYy7MLBPapdgXZCw", 2024-08-18T15:45:23.145 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.145 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.145 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.145 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.145 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.146 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.146 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.146 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.146 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995749.970360_697.1", 2024-08-18T15:45:23.146 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.146 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-37:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.37", 2024-08-18T15:45:23.146 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:29.970360729Z", 2024-08-18T15:45:23.146 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.146 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.146 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_HFwNCilj83ivBqE5tB_PZE2", 2024-08-18T15:45:23.146 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.146 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.146 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.147 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_HFwNCilj83ivBqE5tB_PZE2", 2024-08-18T15:45:23.147 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.147 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.147 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.147 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.147 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.147 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.147 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.147 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.147 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995749.974004_698.1", 2024-08-18T15:45:23.147 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.147 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-37:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.37", 2024-08-18T15:45:23.147 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:29.974004150Z", 2024-08-18T15:45:23.147 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.148 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.148 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_HFwNCilj83ivBqE5tB_PZE2", 2024-08-18T15:45:23.148 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.148 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.148 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.148 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_HFwNCilj83ivBqE5tB_PZE2", 2024-08-18T15:45:23.148 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.148 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.148 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.148 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.148 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.148 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.148 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.148 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.149 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995751.026760_699.1", 2024-08-18T15:45:23.149 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.149 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-38:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.38", 2024-08-18T15:45:23.149 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:31.026760427Z", 2024-08-18T15:45:23.149 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.149 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.149 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_i-eCmhQ4NPsJf86Skaj64Wy", 2024-08-18T15:45:23.149 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.149 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.149 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.149 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_i-eCmhQ4NPsJf86Skaj64Wy", 2024-08-18T15:45:23.149 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.149 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.150 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.150 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.150 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.150 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.150 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.150 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.150 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995751.031154_700.1", 2024-08-18T15:45:23.150 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.150 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-38:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.38", 2024-08-18T15:45:23.150 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:31.031154082Z", 2024-08-18T15:45:23.150 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.150 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.150 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_i-eCmhQ4NPsJf86Skaj64Wy", 2024-08-18T15:45:23.151 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.151 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.151 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.151 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_i-eCmhQ4NPsJf86Skaj64Wy", 2024-08-18T15:45:23.151 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.151 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.151 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.151 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.151 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.151 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.151 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.151 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.151 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995751.043389_701.1", 2024-08-18T15:45:23.151 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.152 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:45:23.152 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:31.043389213Z", 2024-08-18T15:45:23.152 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.152 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.152 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.152 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.152 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.152 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.152 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_xmYq1VNiyUA8ztTA_d5rUm3", 2024-08-18T15:45:23.152 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.152 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.152 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.152 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.152 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.153 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.153 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.153 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.153 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995751.046426_702.1", 2024-08-18T15:45:23.153 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.153 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:45:23.153 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:31.046426883Z", 2024-08-18T15:45:23.153 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.153 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.153 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.153 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.153 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.153 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.153 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_xmYq1VNiyUA8ztTA_d5rUm3", 2024-08-18T15:45:23.154 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.154 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.154 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.154 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.154 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.154 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.154 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.154 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.154 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995751.047596_703.1", 2024-08-18T15:45:23.154 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.154 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-39", 2024-08-18T15:45:23.154 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:31.047596040Z", 2024-08-18T15:45:23.154 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.154 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.155 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.155 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.155 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.155 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.155 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "__iz5Ku-i6OWsHDvE8pjpoke", 2024-08-18T15:45:23.155 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.155 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.155 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.155 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.155 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.155 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.155 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.155 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.155 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995751.050305_704.1", 2024-08-18T15:45:23.156 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.156 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-39", 2024-08-18T15:45:23.156 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:31.050305755Z", 2024-08-18T15:45:23.156 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.156 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.156 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.156 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.156 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.156 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.156 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "__iz5Ku-i6OWsHDvE8pjpoke", 2024-08-18T15:45:23.156 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.156 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.156 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.157 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.157 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.157 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.157 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.157 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.157 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995751.063176_705.1", 2024-08-18T15:45:23.157 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.157 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:45:23.157 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:31.063176910Z", 2024-08-18T15:45:23.157 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.157 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.157 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.157 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.157 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.158 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.158 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_ejQ3MioLVUCb2QnucqqCUMJ", 2024-08-18T15:45:23.158 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.158 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.158 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.158 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.158 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.158 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.158 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.158 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.158 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995751.066133_706.1", 2024-08-18T15:45:23.158 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.158 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:45:23.158 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:31.066133710Z", 2024-08-18T15:45:23.159 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.159 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.159 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.159 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.159 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.159 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.159 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_ejQ3MioLVUCb2QnucqqCUMJ", 2024-08-18T15:45:23.159 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.159 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.159 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.159 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.159 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.159 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.159 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.160 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.160 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995751.067195_707.1", 2024-08-18T15:45:23.160 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.160 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-40", 2024-08-18T15:45:23.160 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:31.067195014Z", 2024-08-18T15:45:23.160 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.160 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.160 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.160 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.160 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.160 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.160 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_llhZcJSjwRerrd-Oaou9Sjv", 2024-08-18T15:45:23.160 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.161 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.161 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.161 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.161 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.161 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.161 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.161 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.161 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995751.069927_708.1", 2024-08-18T15:45:23.161 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.161 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-40", 2024-08-18T15:45:23.161 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:31.069927389Z", 2024-08-18T15:45:23.161 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.161 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.161 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.162 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.162 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.162 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.162 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_llhZcJSjwRerrd-Oaou9Sjv", 2024-08-18T15:45:23.162 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.162 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.162 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.162 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.162 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.162 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.162 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.162 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.162 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995751.090745_709.1", 2024-08-18T15:45:23.163 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.163 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:45:23.163 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:31.090745755Z", 2024-08-18T15:45:23.163 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.163 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.163 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.163 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.163 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.163 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.163 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_8c5popeokpV--2SpVjBST3E", 2024-08-18T15:45:23.163 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.163 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.163 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.163 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.164 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.164 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.164 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.164 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.164 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995751.093802_710.1", 2024-08-18T15:45:23.164 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.164 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:45:23.164 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:31.093802820Z", 2024-08-18T15:45:23.164 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.164 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.164 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.164 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.164 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.164 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.165 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_8c5popeokpV--2SpVjBST3E", 2024-08-18T15:45:23.165 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.165 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.165 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.165 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.165 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.165 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.165 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.165 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.165 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995751.095112_711.1", 2024-08-18T15:45:23.165 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.165 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-41", 2024-08-18T15:45:23.165 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:31.095112579Z", 2024-08-18T15:45:23.165 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.166 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.166 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.166 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.166 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.166 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.166 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_YWJNdlw8sqMdjjlIjLHAL7T", 2024-08-18T15:45:23.166 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.166 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.166 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.166 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.166 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.166 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.166 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.166 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.167 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995751.097661_712.1", 2024-08-18T15:45:23.167 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.167 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-41", 2024-08-18T15:45:23.167 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:31.097661378Z", 2024-08-18T15:45:23.167 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.167 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.167 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.167 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.167 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.167 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.167 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_YWJNdlw8sqMdjjlIjLHAL7T", 2024-08-18T15:45:23.167 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.167 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.167 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.168 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.168 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.168 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.168 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.168 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.168 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995771.988164_729.1", 2024-08-18T15:45:23.168 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.168 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-42:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.42", 2024-08-18T15:45:23.168 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:51.988164854Z", 2024-08-18T15:45:23.168 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.168 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.168 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.168 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.168 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.169 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.169 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_oGiVyEzJBYGJYJotLHjzZsK", 2024-08-18T15:45:23.169 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.169 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.169 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.169 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.169 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.169 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.169 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.169 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.169 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995771.990622_730.1", 2024-08-18T15:45:23.169 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.169 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-42:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.42", 2024-08-18T15:45:23.169 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:51.990622883Z", 2024-08-18T15:45:23.170 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.170 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.170 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.170 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.170 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.170 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.170 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_oGiVyEzJBYGJYJotLHjzZsK", 2024-08-18T15:45:23.170 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.170 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.170 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.170 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.170 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.170 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.171 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.171 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.171 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995771.991578_731.1", 2024-08-18T15:45:23.171 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.171 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-42", 2024-08-18T15:45:23.171 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:51.991578988Z", 2024-08-18T15:45:23.171 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.171 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.171 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.171 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.171 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.171 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.171 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_fG-TEUh0CxWR43RmwQg7ICF", 2024-08-18T15:45:23.171 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.172 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.172 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.172 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.172 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.172 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.172 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.172 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.172 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995771.993985_732.1", 2024-08-18T15:45:23.172 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.172 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-42", 2024-08-18T15:45:23.172 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:51.993985953Z", 2024-08-18T15:45:23.172 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.172 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.172 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.172 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.173 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.173 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.173 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_fG-TEUh0CxWR43RmwQg7ICF", 2024-08-18T15:45:23.173 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.173 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.173 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.173 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.173 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.173 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.173 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.173 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.173 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995772.004584_733.1", 2024-08-18T15:45:23.173 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.173 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-43:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.43", 2024-08-18T15:45:23.174 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:52.004584088Z", 2024-08-18T15:45:23.174 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.174 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.174 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.174 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.174 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.174 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.174 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_c-hGhbKE-b1LRVHEPUzPjib", 2024-08-18T15:45:23.174 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.174 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.174 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.174 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.174 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.174 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.175 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.175 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.175 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995772.008960_734.1", 2024-08-18T15:45:23.175 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.175 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-43:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.43", 2024-08-18T15:45:23.175 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:52.008960500Z", 2024-08-18T15:45:23.175 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.175 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.175 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.175 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.175 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.175 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.175 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_c-hGhbKE-b1LRVHEPUzPjib", 2024-08-18T15:45:23.175 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.176 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.176 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.176 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.176 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.176 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.176 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.176 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.176 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995772.010074_735.1", 2024-08-18T15:45:23.176 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.176 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-43", 2024-08-18T15:45:23.176 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:52.010074886Z", 2024-08-18T15:45:23.176 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.176 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.177 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.177 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.177 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.177 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.177 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_L-a1id3J-Bf_HtA2U8yB3t5", 2024-08-18T15:45:23.177 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.177 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.177 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.177 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.177 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.177 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.177 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.177 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.177 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995772.012225_736.1", 2024-08-18T15:45:23.178 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.178 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-43", 2024-08-18T15:45:23.178 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:52.012225687Z", 2024-08-18T15:45:23.178 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.178 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.178 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.178 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.178 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.178 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.178 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_L-a1id3J-Bf_HtA2U8yB3t5", 2024-08-18T15:45:23.178 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.178 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.178 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.179 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.179 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.179 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.179 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.179 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.179 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995772.031468_737.1", 2024-08-18T15:45:23.179 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.179 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-44:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.44", 2024-08-18T15:45:23.179 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:52.031468864Z", 2024-08-18T15:45:23.179 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.179 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.179 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.179 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.180 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.180 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.180 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_On0FEwDZQtRx6QIbTXxbmk6", 2024-08-18T15:45:23.180 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.180 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.180 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.180 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.180 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.180 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.180 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.180 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.180 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995772.034017_738.1", 2024-08-18T15:45:23.180 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.180 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-44:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.44", 2024-08-18T15:45:23.181 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:52.034017957Z", 2024-08-18T15:45:23.181 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.181 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.181 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.181 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.181 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.181 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.181 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_On0FEwDZQtRx6QIbTXxbmk6", 2024-08-18T15:45:23.181 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.181 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.181 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.181 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.181 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.181 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.182 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.182 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.182 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995772.035064_739.1", 2024-08-18T15:45:23.182 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.182 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-44", 2024-08-18T15:45:23.182 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:52.035064992Z", 2024-08-18T15:45:23.182 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.182 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.182 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.182 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.182 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.182 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.182 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_zO2kHLSbZbuAztlb-hy3PyC", 2024-08-18T15:45:23.182 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.183 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.183 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.183 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.183 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.183 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.183 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.183 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.183 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995772.038282_740.1", 2024-08-18T15:45:23.183 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.183 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-44", 2024-08-18T15:45:23.183 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:52.038282163Z", 2024-08-18T15:45:23.183 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.183 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.184 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.184 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.184 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.184 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.184 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_zO2kHLSbZbuAztlb-hy3PyC", 2024-08-18T15:45:23.184 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.184 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.184 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.184 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.184 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.184 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.184 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.184 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T15:45:23.186 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'b0d74758-1871-4454-bc0f-e3e4b6c30105', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:45:23.186 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'b0d74758-1871-4454-bc0f-e3e4b6c30105', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:45:23.186 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period b0d74758-1871-4454-bc0f-e3e4b6c30105 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:45:23.240 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:45:23.240 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:45:23.274 INFO:teuthology.orchestra.run.smithi193.stdout:[ 2024-08-18T15:45:23.274 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.275 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723994364.503712_22.1", 2024-08-18T15:45:23.275 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "user", 2024-08-18T15:45:23.275 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "rgw-multisite-test-user", 2024-08-18T15:45:23.275 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.503712511Z", 2024-08-18T15:45:23.275 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.275 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.275 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.275 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.275 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.275 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.275 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_MXHnfxZIHwQEXOVDZVD1osf", 2024-08-18T15:45:23.275 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.275 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.276 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.276 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.276 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.276 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.276 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.276 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.276 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723994364.506194_23.1", 2024-08-18T15:45:23.276 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "user", 2024-08-18T15:45:23.276 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "rgw-multisite-test-user", 2024-08-18T15:45:23.276 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194765Z", 2024-08-18T15:45:23.276 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.276 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.276 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.276 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.277 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.277 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.277 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_MXHnfxZIHwQEXOVDZVD1osf", 2024-08-18T15:45:23.277 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.277 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.277 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.277 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.277 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.277 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.277 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.277 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.277 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723994485.499738_49.1", 2024-08-18T15:45:23.277 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.278 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-1:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.1", 2024-08-18T15:45:23.278 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:25.499738805Z", 2024-08-18T15:45:23.278 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.278 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.278 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.278 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.278 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.278 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.278 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_RWKCfRB1w1J8Q2l4ptlOal-", 2024-08-18T15:45:23.278 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.278 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.278 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.278 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.279 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.279 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.279 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.279 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.279 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723994485.513055_50.1", 2024-08-18T15:45:23.279 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.279 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-1:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.1", 2024-08-18T15:45:23.279 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:25.513055821Z", 2024-08-18T15:45:23.279 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.279 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.279 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.279 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.279 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.279 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.280 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_RWKCfRB1w1J8Q2l4ptlOal-", 2024-08-18T15:45:23.280 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.280 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.280 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.280 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.280 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.280 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.280 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.280 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.280 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723994485.514799_51.1", 2024-08-18T15:45:23.280 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.280 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-1", 2024-08-18T15:45:23.280 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:25.514799747Z", 2024-08-18T15:45:23.280 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.281 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.281 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.281 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.281 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.281 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.281 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_zN4WlRRiwX-IH3XcKWBWZ7B", 2024-08-18T15:45:23.281 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.281 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.281 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.281 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.281 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.281 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.281 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.281 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.282 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723994485.517793_52.1", 2024-08-18T15:45:23.282 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.282 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-1", 2024-08-18T15:45:23.282 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:25.517793650Z", 2024-08-18T15:45:23.282 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.282 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.282 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.282 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.282 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.282 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.282 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_zN4WlRRiwX-IH3XcKWBWZ7B", 2024-08-18T15:45:23.282 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.282 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.283 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.283 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.283 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.283 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.283 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.283 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.283 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723994485.530711_53.1", 2024-08-18T15:45:23.283 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.283 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-2:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.2", 2024-08-18T15:45:23.283 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:25.530711866Z", 2024-08-18T15:45:23.283 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.283 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.283 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.283 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.284 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.284 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.284 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_kZfzVgY3qrQmzEpOy-gNh0z", 2024-08-18T15:45:23.284 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.284 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.284 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.284 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.284 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.284 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.284 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.284 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.284 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723994485.533125_54.1", 2024-08-18T15:45:23.284 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.285 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-2:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.2", 2024-08-18T15:45:23.285 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:25.533125204Z", 2024-08-18T15:45:23.285 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.285 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.285 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.285 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.285 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.285 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.285 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_kZfzVgY3qrQmzEpOy-gNh0z", 2024-08-18T15:45:23.285 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.285 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.285 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.285 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.285 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.286 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.286 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.286 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.286 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723994485.534087_55.1", 2024-08-18T15:45:23.286 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.286 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-2", 2024-08-18T15:45:23.286 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:25.534087256Z", 2024-08-18T15:45:23.286 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.286 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.286 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.286 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.286 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.286 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.286 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_4VibGFZMXb9COC_5_LGnUYq", 2024-08-18T15:45:23.287 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.287 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.287 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.287 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.287 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.287 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.287 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.287 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.287 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723994485.536327_56.1", 2024-08-18T15:45:23.287 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.287 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-2", 2024-08-18T15:45:23.287 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:25.536327238Z", 2024-08-18T15:45:23.287 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.288 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.288 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.288 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.288 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.288 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.288 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_4VibGFZMXb9COC_5_LGnUYq", 2024-08-18T15:45:23.288 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.288 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.288 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.288 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.288 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.288 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.288 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.288 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.289 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723994488.093196_57.1", 2024-08-18T15:45:23.289 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.289 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-3:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.3", 2024-08-18T15:45:23.289 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:28.093196831Z", 2024-08-18T15:45:23.289 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.289 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.289 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.289 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.289 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.289 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.289 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_7WKtlNtV_hilpzaA5UW3kBl", 2024-08-18T15:45:23.289 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.289 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.289 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.290 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.290 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.290 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.290 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.290 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.290 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723994488.096301_58.1", 2024-08-18T15:45:23.290 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.290 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-3:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.3", 2024-08-18T15:45:23.290 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:28.096301217Z", 2024-08-18T15:45:23.290 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.290 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.290 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.290 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.291 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.291 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.291 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_7WKtlNtV_hilpzaA5UW3kBl", 2024-08-18T15:45:23.291 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.291 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.291 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.291 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.291 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.291 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.291 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.291 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.291 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723994488.097328_59.1", 2024-08-18T15:45:23.291 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.291 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-3", 2024-08-18T15:45:23.292 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:28.097328364Z", 2024-08-18T15:45:23.292 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.292 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.292 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.292 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.292 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.292 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.292 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_T0NcIXjN0Rm7yrm1EC_xxAs", 2024-08-18T15:45:23.292 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.292 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.292 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.292 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.292 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.292 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.293 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.293 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.293 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723994488.100145_60.1", 2024-08-18T15:45:23.293 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.293 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-3", 2024-08-18T15:45:23.293 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:28.100145189Z", 2024-08-18T15:45:23.293 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.293 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.293 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.293 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.293 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.293 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.293 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_T0NcIXjN0Rm7yrm1EC_xxAs", 2024-08-18T15:45:23.293 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.294 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.294 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.294 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.294 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.294 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.294 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.294 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.294 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723994493.565103_61.1", 2024-08-18T15:45:23.294 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.294 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-4:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.4", 2024-08-18T15:45:23.294 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:33.565103683Z", 2024-08-18T15:45:23.294 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.294 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.295 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.295 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.295 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.295 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.295 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_NVsrONxJz03-nbFIVh9-7HH", 2024-08-18T15:45:23.295 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.295 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.295 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.295 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.295 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.295 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.295 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.295 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.295 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723994493.567515_62.1", 2024-08-18T15:45:23.296 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.296 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-4:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.4", 2024-08-18T15:45:23.296 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:33.567515090Z", 2024-08-18T15:45:23.296 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.296 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.296 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.296 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.296 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.296 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.296 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_NVsrONxJz03-nbFIVh9-7HH", 2024-08-18T15:45:23.296 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.296 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.296 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.297 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.297 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.297 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.297 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.297 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.297 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723994493.568350_63.1", 2024-08-18T15:45:23.297 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.297 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-4", 2024-08-18T15:45:23.297 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:33.568350483Z", 2024-08-18T15:45:23.297 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.297 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.297 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.297 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.297 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.298 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.298 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_5jHEePfx-6SZjjYqolXPBLX", 2024-08-18T15:45:23.298 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.298 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.298 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.298 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.298 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.298 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.298 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.298 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.298 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723994493.570568_64.1", 2024-08-18T15:45:23.298 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.298 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-4", 2024-08-18T15:45:23.299 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:33.570568110Z", 2024-08-18T15:45:23.299 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.299 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.299 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.299 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.299 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.299 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.299 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_5jHEePfx-6SZjjYqolXPBLX", 2024-08-18T15:45:23.299 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.299 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.299 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.299 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.299 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.300 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.300 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.300 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.300 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723994493.583318_65.1", 2024-08-18T15:45:23.300 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.300 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-5:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.5", 2024-08-18T15:45:23.300 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:33.583318078Z", 2024-08-18T15:45:23.300 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.300 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.300 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.300 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.300 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.300 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.300 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_gGmjq-oCwB5WheV6EKKwPBK", 2024-08-18T15:45:23.301 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.301 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.301 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.301 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.301 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.301 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.301 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.301 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.301 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723994493.586220_66.1", 2024-08-18T15:45:23.301 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.301 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-5:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.5", 2024-08-18T15:45:23.301 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:33.586220697Z", 2024-08-18T15:45:23.301 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.301 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.302 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.302 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.302 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.302 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.302 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_gGmjq-oCwB5WheV6EKKwPBK", 2024-08-18T15:45:23.302 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.302 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.302 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.302 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.302 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.302 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.302 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.302 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.303 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723994493.587090_67.1", 2024-08-18T15:45:23.303 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.303 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-5", 2024-08-18T15:45:23.303 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:33.587090828Z", 2024-08-18T15:45:23.303 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.303 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.303 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.303 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.303 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.303 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.303 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_d2Y8p1KjRuhShSVV0OrvphE", 2024-08-18T15:45:23.303 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.303 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.303 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.304 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.304 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.304 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.304 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.304 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.304 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723994493.589499_68.1", 2024-08-18T15:45:23.304 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.304 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-5", 2024-08-18T15:45:23.304 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:33.589499147Z", 2024-08-18T15:45:23.304 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.304 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.304 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.304 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.304 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.305 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.305 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_d2Y8p1KjRuhShSVV0OrvphE", 2024-08-18T15:45:23.305 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.305 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.305 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.305 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.305 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.305 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.305 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.305 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.305 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723994493.608922_69.1", 2024-08-18T15:45:23.305 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.305 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-6:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.6", 2024-08-18T15:45:23.305 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:33.608922657Z", 2024-08-18T15:45:23.306 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.306 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.306 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.306 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.306 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.306 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.306 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_ZjXDBXiyWYi6JTPxve1UMrv", 2024-08-18T15:45:23.306 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.306 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.306 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.306 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.306 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.306 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.307 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.307 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.307 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723994493.613023_70.1", 2024-08-18T15:45:23.307 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.307 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-6:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.6", 2024-08-18T15:45:23.307 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:33.613023410Z", 2024-08-18T15:45:23.307 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.307 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.307 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.307 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.307 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.307 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.307 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_ZjXDBXiyWYi6JTPxve1UMrv", 2024-08-18T15:45:23.307 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.308 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.308 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.308 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.308 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.308 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.308 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.308 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.308 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723994493.614132_71.1", 2024-08-18T15:45:23.308 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.308 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-6", 2024-08-18T15:45:23.308 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:33.614132419Z", 2024-08-18T15:45:23.308 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.308 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.308 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.309 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.309 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.309 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.309 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_ZBRWYIGyUuGJdOA0PdgsHNt", 2024-08-18T15:45:23.309 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.309 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.309 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.309 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.309 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.309 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.309 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.309 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.309 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723994493.616159_72.1", 2024-08-18T15:45:23.310 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.310 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-6", 2024-08-18T15:45:23.310 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:33.616159354Z", 2024-08-18T15:45:23.310 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.310 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.310 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.310 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.310 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.310 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.310 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_ZBRWYIGyUuGJdOA0PdgsHNt", 2024-08-18T15:45:23.310 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.310 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.310 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.310 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.311 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.311 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.311 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.311 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.311 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723994494.468832_73.1", 2024-08-18T15:45:23.311 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.311 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-7:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.7", 2024-08-18T15:45:23.311 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.468832363Z", 2024-08-18T15:45:23.311 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.311 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.311 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.311 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.311 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.312 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.312 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_qRFDABc-dwq6RziTweT5YtT", 2024-08-18T15:45:23.312 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.312 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.312 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.312 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.312 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.312 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.312 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.312 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.312 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723994494.471921_74.1", 2024-08-18T15:45:23.312 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.312 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-7:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.7", 2024-08-18T15:45:23.312 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.471921559Z", 2024-08-18T15:45:23.313 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.313 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.313 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.313 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.313 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.313 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.313 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_qRFDABc-dwq6RziTweT5YtT", 2024-08-18T15:45:23.313 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.313 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.313 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.313 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.313 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.313 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.313 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.314 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.314 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723994494.472699_75.1", 2024-08-18T15:45:23.314 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.314 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-7", 2024-08-18T15:45:23.314 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.472699971Z", 2024-08-18T15:45:23.314 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.314 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.314 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.314 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.314 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.314 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.314 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_tLmChDVmX3sImKf0993KFB1", 2024-08-18T15:45:23.314 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.314 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.315 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.315 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.315 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.315 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.315 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.315 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.315 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723994494.475169_76.1", 2024-08-18T15:45:23.315 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.315 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-7", 2024-08-18T15:45:23.315 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.475169889Z", 2024-08-18T15:45:23.315 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.315 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.315 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.316 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.316 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.316 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.316 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_tLmChDVmX3sImKf0993KFB1", 2024-08-18T15:45:23.316 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.316 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.316 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.316 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.316 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.316 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.316 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.316 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.316 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723994494.483917_77.1", 2024-08-18T15:45:23.316 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.317 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-8:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.8", 2024-08-18T15:45:23.317 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.483917175Z", 2024-08-18T15:45:23.317 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.317 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.317 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.317 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.317 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.317 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.317 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_Zi2-lA3qHEM3t_BsL4AeXRU", 2024-08-18T15:45:23.317 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.317 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.317 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.317 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.317 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.318 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.318 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.318 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.318 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723994494.487103_78.1", 2024-08-18T15:45:23.318 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.318 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-8:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.8", 2024-08-18T15:45:23.318 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.487103946Z", 2024-08-18T15:45:23.318 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.318 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.318 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.318 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.318 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.318 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.318 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_Zi2-lA3qHEM3t_BsL4AeXRU", 2024-08-18T15:45:23.319 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.319 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.319 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.319 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.319 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.319 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.319 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.319 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.319 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723994494.487887_79.1", 2024-08-18T15:45:23.319 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.319 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-8", 2024-08-18T15:45:23.319 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.487887903Z", 2024-08-18T15:45:23.319 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.320 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.320 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.320 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.320 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.320 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.320 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_7oG9nLitKlO5mrAHdrnWcCe", 2024-08-18T15:45:23.320 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.320 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.320 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.320 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.320 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.320 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.320 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.320 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.321 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723994494.489837_80.1", 2024-08-18T15:45:23.321 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.321 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-8", 2024-08-18T15:45:23.321 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.489837888Z", 2024-08-18T15:45:23.321 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.321 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.321 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.321 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.321 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.322 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.322 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_7oG9nLitKlO5mrAHdrnWcCe", 2024-08-18T15:45:23.322 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.322 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.322 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.322 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.322 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.322 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.322 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.322 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.322 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723994494.511942_81.1", 2024-08-18T15:45:23.322 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.322 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-9:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.9", 2024-08-18T15:45:23.323 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.511942269Z", 2024-08-18T15:45:23.323 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.323 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.323 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.323 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.323 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.323 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.323 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_M-lw4sdJUfr_hU7bn6pQCkY", 2024-08-18T15:45:23.323 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.323 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.323 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.323 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.323 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.323 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.324 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.324 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.324 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723994494.520062_82.1", 2024-08-18T15:45:23.324 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.324 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-9:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.9", 2024-08-18T15:45:23.324 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.520062161Z", 2024-08-18T15:45:23.324 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.324 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.324 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.324 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.324 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.324 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.324 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_M-lw4sdJUfr_hU7bn6pQCkY", 2024-08-18T15:45:23.325 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.325 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.325 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.325 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.325 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.325 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.325 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.325 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.325 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723994494.520843_83.1", 2024-08-18T15:45:23.325 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.325 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-9", 2024-08-18T15:45:23.325 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.520843453Z", 2024-08-18T15:45:23.325 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.325 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.326 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.326 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.326 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.326 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.326 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_WwzPQ6LnfxKTsseathJHhf2", 2024-08-18T15:45:23.326 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.326 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.326 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.326 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.326 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.326 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.326 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.326 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.327 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723994494.523006_84.1", 2024-08-18T15:45:23.327 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.327 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-9", 2024-08-18T15:45:23.327 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.523006825Z", 2024-08-18T15:45:23.327 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.327 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.327 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.327 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.327 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.327 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.327 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_WwzPQ6LnfxKTsseathJHhf2", 2024-08-18T15:45:23.327 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.327 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.327 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.328 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.328 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.328 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.328 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.328 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.328 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723994494.861305_85.1", 2024-08-18T15:45:23.328 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.328 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-7:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.7", 2024-08-18T15:45:23.328 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.861305566Z", 2024-08-18T15:45:23.328 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.328 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.328 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_qRFDABc-dwq6RziTweT5YtT", 2024-08-18T15:45:23.328 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.329 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.329 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.329 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_qRFDABc-dwq6RziTweT5YtT", 2024-08-18T15:45:23.329 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 2 2024-08-18T15:45:23.329 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.329 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.329 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.329 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.329 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.329 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.329 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.329 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723994494.863517_86.1", 2024-08-18T15:45:23.329 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.329 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-7:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.7", 2024-08-18T15:45:23.330 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.863517491Z", 2024-08-18T15:45:23.330 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.330 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.330 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_qRFDABc-dwq6RziTweT5YtT", 2024-08-18T15:45:23.330 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.330 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.330 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.330 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_qRFDABc-dwq6RziTweT5YtT", 2024-08-18T15:45:23.330 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 2 2024-08-18T15:45:23.330 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.330 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.330 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.330 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.331 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.331 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.331 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.331 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723994494.870867_87.1", 2024-08-18T15:45:23.331 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.331 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-7", 2024-08-18T15:45:23.331 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.870867850Z", 2024-08-18T15:45:23.331 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.331 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.331 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_tLmChDVmX3sImKf0993KFB1", 2024-08-18T15:45:23.331 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.331 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.331 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.332 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_tLmChDVmX3sImKf0993KFB1", 2024-08-18T15:45:23.332 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 2 2024-08-18T15:45:23.332 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.332 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.332 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "remove" 2024-08-18T15:45:23.332 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.332 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.332 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.332 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.332 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723994494.872883_88.1", 2024-08-18T15:45:23.332 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.332 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-7", 2024-08-18T15:45:23.332 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.872883708Z", 2024-08-18T15:45:23.332 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.333 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.333 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_tLmChDVmX3sImKf0993KFB1", 2024-08-18T15:45:23.333 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.333 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.333 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.333 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_tLmChDVmX3sImKf0993KFB1", 2024-08-18T15:45:23.333 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 2 2024-08-18T15:45:23.333 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.333 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.333 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.334 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.334 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.334 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.334 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.334 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723994494.890261_89.1", 2024-08-18T15:45:23.334 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.334 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-8:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.8", 2024-08-18T15:45:23.334 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.890261024Z", 2024-08-18T15:45:23.334 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.334 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.334 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_Zi2-lA3qHEM3t_BsL4AeXRU", 2024-08-18T15:45:23.334 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.334 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.335 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.335 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_Zi2-lA3qHEM3t_BsL4AeXRU", 2024-08-18T15:45:23.335 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 2 2024-08-18T15:45:23.335 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.335 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.335 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.335 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.335 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.335 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.335 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.335 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723994494.892455_90.1", 2024-08-18T15:45:23.335 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.335 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-8:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.8", 2024-08-18T15:45:23.336 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.892455177Z", 2024-08-18T15:45:23.336 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.336 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.336 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_Zi2-lA3qHEM3t_BsL4AeXRU", 2024-08-18T15:45:23.336 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.336 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.336 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.336 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_Zi2-lA3qHEM3t_BsL4AeXRU", 2024-08-18T15:45:23.336 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 2 2024-08-18T15:45:23.336 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.336 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.336 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.336 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.336 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.337 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.337 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.337 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723994494.899743_91.1", 2024-08-18T15:45:23.337 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.337 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-8", 2024-08-18T15:45:23.337 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.899743512Z", 2024-08-18T15:45:23.337 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.337 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.337 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_7oG9nLitKlO5mrAHdrnWcCe", 2024-08-18T15:45:23.337 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.337 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.337 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.337 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_7oG9nLitKlO5mrAHdrnWcCe", 2024-08-18T15:45:23.338 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 2 2024-08-18T15:45:23.338 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.338 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.338 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "remove" 2024-08-18T15:45:23.338 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.338 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.338 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.338 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.338 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723994494.901752_92.1", 2024-08-18T15:45:23.338 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.338 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-8", 2024-08-18T15:45:23.338 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.901752069Z", 2024-08-18T15:45:23.338 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.338 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.339 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_7oG9nLitKlO5mrAHdrnWcCe", 2024-08-18T15:45:23.339 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.339 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.339 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.339 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_7oG9nLitKlO5mrAHdrnWcCe", 2024-08-18T15:45:23.339 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 2 2024-08-18T15:45:23.339 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.339 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.339 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.339 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.339 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.339 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.339 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.340 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723994494.940403_93.1", 2024-08-18T15:45:23.340 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.340 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-9:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.9", 2024-08-18T15:45:23.340 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.940403695Z", 2024-08-18T15:45:23.340 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.340 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.340 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_M-lw4sdJUfr_hU7bn6pQCkY", 2024-08-18T15:45:23.340 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.340 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.340 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.340 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_M-lw4sdJUfr_hU7bn6pQCkY", 2024-08-18T15:45:23.340 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 2 2024-08-18T15:45:23.340 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.340 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.341 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.341 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.341 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.341 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.341 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.341 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723994494.942560_94.1", 2024-08-18T15:45:23.341 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.341 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-9:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.9", 2024-08-18T15:45:23.341 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.942560102Z", 2024-08-18T15:45:23.341 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.341 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.341 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_M-lw4sdJUfr_hU7bn6pQCkY", 2024-08-18T15:45:23.341 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.342 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.342 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.342 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_M-lw4sdJUfr_hU7bn6pQCkY", 2024-08-18T15:45:23.342 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 2 2024-08-18T15:45:23.342 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.342 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.342 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.342 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.342 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.342 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.342 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.342 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723994494.949797_95.1", 2024-08-18T15:45:23.342 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.342 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-9", 2024-08-18T15:45:23.343 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.949797571Z", 2024-08-18T15:45:23.343 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.343 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.343 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_WwzPQ6LnfxKTsseathJHhf2", 2024-08-18T15:45:23.343 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.343 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.343 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.343 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_WwzPQ6LnfxKTsseathJHhf2", 2024-08-18T15:45:23.343 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 2 2024-08-18T15:45:23.343 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.343 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.343 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "remove" 2024-08-18T15:45:23.343 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.344 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.344 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.344 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.344 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723994494.952005_96.1", 2024-08-18T15:45:23.344 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.344 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-9", 2024-08-18T15:45:23.344 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005923Z", 2024-08-18T15:45:23.344 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.344 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.344 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_WwzPQ6LnfxKTsseathJHhf2", 2024-08-18T15:45:23.344 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.344 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.344 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.344 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_WwzPQ6LnfxKTsseathJHhf2", 2024-08-18T15:45:23.345 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 2 2024-08-18T15:45:23.345 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.345 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.345 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.345 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.345 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.345 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.345 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.345 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723994505.517635_51.1", 2024-08-18T15:45:23.345 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.345 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:45:23.345 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:45.517635733Z", 2024-08-18T15:45:23.345 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.345 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.346 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.346 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.346 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.346 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.346 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "__WpsUsCter7eN_yzr7oojfP", 2024-08-18T15:45:23.346 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.346 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.346 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.346 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.346 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.346 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.346 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.346 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.347 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723994505.521539_52.1", 2024-08-18T15:45:23.347 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.347 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:45:23.347 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:45.521539522Z", 2024-08-18T15:45:23.347 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.347 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.347 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.347 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.347 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.347 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.347 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "__WpsUsCter7eN_yzr7oojfP", 2024-08-18T15:45:23.347 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.347 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.347 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.348 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.348 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.348 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.348 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.348 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.348 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723994505.522484_53.1", 2024-08-18T15:45:23.348 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.348 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-10", 2024-08-18T15:45:23.348 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:45.522484813Z", 2024-08-18T15:45:23.348 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.348 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.348 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.348 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.348 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.349 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.349 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_u-jdXqkvtd3BndfjmYUBTit", 2024-08-18T15:45:23.349 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.349 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.349 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.349 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.349 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.349 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.349 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.349 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.349 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723994505.524516_54.1", 2024-08-18T15:45:23.349 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.349 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-10", 2024-08-18T15:45:23.350 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:45.524516821Z", 2024-08-18T15:45:23.350 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.350 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.350 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.350 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.350 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.350 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.350 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_u-jdXqkvtd3BndfjmYUBTit", 2024-08-18T15:45:23.350 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.350 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.350 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.350 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.350 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.350 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.351 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.351 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.351 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723994505.535896_55.1", 2024-08-18T15:45:23.351 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.351 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:45:23.351 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:45.535896117Z", 2024-08-18T15:45:23.351 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.351 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.351 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.351 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.351 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.351 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.351 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_XktQF5h6nGNC5v3w-DrHnvQ", 2024-08-18T15:45:23.351 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.352 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.352 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.352 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.352 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.352 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.352 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.352 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.352 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723994505.539494_56.1", 2024-08-18T15:45:23.352 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.352 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:45:23.352 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:45.539494049Z", 2024-08-18T15:45:23.352 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.352 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.352 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.353 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.353 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.353 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.353 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_XktQF5h6nGNC5v3w-DrHnvQ", 2024-08-18T15:45:23.353 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.353 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.353 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.353 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.353 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.353 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.353 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.353 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.353 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723994505.540711_57.1", 2024-08-18T15:45:23.354 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.354 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-11", 2024-08-18T15:45:23.354 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:45.540711150Z", 2024-08-18T15:45:23.354 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.354 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.354 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.354 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.354 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.354 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.354 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_y9YtDg7ol2vKL0ge54w1-u0", 2024-08-18T15:45:23.354 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.354 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.354 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.354 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.355 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.355 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.355 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.355 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.355 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723994505.542807_58.1", 2024-08-18T15:45:23.355 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.355 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-11", 2024-08-18T15:45:23.355 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:45.542807482Z", 2024-08-18T15:45:23.355 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.355 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.355 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.355 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.355 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.356 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.356 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_y9YtDg7ol2vKL0ge54w1-u0", 2024-08-18T15:45:23.356 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.356 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.356 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.356 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.356 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.356 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.356 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.356 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.356 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723994505.560557_59.1", 2024-08-18T15:45:23.356 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.356 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:45:23.356 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:45.560557573Z", 2024-08-18T15:45:23.357 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.357 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.357 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.357 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.357 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.357 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.357 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_EyCJst2_YMPiGsXCrb7chb3", 2024-08-18T15:45:23.357 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.357 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.357 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.357 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.357 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.357 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.357 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.358 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.358 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723994505.563007_60.1", 2024-08-18T15:45:23.358 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.358 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:45:23.358 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:45.563007502Z", 2024-08-18T15:45:23.358 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.358 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.358 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.358 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.358 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.358 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.358 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_EyCJst2_YMPiGsXCrb7chb3", 2024-08-18T15:45:23.358 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.359 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.359 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.359 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.359 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.359 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.359 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.359 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.359 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723994505.563839_61.1", 2024-08-18T15:45:23.359 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.359 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-12", 2024-08-18T15:45:23.359 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:45.563839125Z", 2024-08-18T15:45:23.359 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.359 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.359 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.360 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.360 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.360 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.360 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_O6k9owNlHrC6zA06qCpRJJZ", 2024-08-18T15:45:23.360 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.360 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.360 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.360 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.360 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.360 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.360 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.360 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.360 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723994505.566000_62.1", 2024-08-18T15:45:23.360 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.361 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-12", 2024-08-18T15:45:23.361 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:45.566000292Z", 2024-08-18T15:45:23.361 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.361 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.361 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.361 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.361 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.361 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.361 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_O6k9owNlHrC6zA06qCpRJJZ", 2024-08-18T15:45:23.361 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.361 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.361 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.361 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.362 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.362 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.362 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.362 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.362 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723994599.644114_81.1", 2024-08-18T15:45:23.362 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.362 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:45:23.362 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:23:19.644114701Z", 2024-08-18T15:45:23.362 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.362 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.362 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.362 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.362 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.362 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.363 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_JA4U4fOsjO9zivYOJOBkWF7", 2024-08-18T15:45:23.363 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.363 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.363 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.363 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.363 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.363 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.363 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.363 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.363 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723994599.646276_82.1", 2024-08-18T15:45:23.366 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.366 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:45:23.367 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:23:19.646276235Z", 2024-08-18T15:45:23.367 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.367 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.367 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.367 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.367 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.367 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.367 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_JA4U4fOsjO9zivYOJOBkWF7", 2024-08-18T15:45:23.367 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.367 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.367 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.367 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.367 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.368 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.368 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.368 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.368 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723994599.647062_83.1", 2024-08-18T15:45:23.368 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.368 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-13", 2024-08-18T15:45:23.368 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:23:19.647062932Z", 2024-08-18T15:45:23.368 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.368 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.368 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.368 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.368 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.369 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.369 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_qqt9_2CdbPwt_U6wcKZCsvw", 2024-08-18T15:45:23.369 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.369 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.369 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.369 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.369 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.369 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.369 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.369 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.369 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723994599.649062_84.1", 2024-08-18T15:45:23.369 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.369 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-13", 2024-08-18T15:45:23.369 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:23:19.649062828Z", 2024-08-18T15:45:23.370 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.370 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.370 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.370 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.370 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.370 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.370 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_qqt9_2CdbPwt_U6wcKZCsvw", 2024-08-18T15:45:23.370 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.370 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.370 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.370 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.370 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.370 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.371 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.371 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.371 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723994599.657472_85.1", 2024-08-18T15:45:23.371 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.371 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:45:23.371 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:23:19.657472231Z", 2024-08-18T15:45:23.371 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.371 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.371 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.371 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.371 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.371 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.371 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_hVfHZOUQ7KCzUbpwOhp8fwo", 2024-08-18T15:45:23.371 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.372 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.372 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.372 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.372 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.372 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.372 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.372 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.372 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723994599.659687_86.1", 2024-08-18T15:45:23.372 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.372 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:45:23.372 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:23:19.659687294Z", 2024-08-18T15:45:23.372 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.372 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.373 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.373 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.373 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.373 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.373 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_hVfHZOUQ7KCzUbpwOhp8fwo", 2024-08-18T15:45:23.373 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.373 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.373 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.373 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.373 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.373 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.373 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.373 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.373 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723994599.660446_87.1", 2024-08-18T15:45:23.374 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.374 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-14", 2024-08-18T15:45:23.374 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:23:19.660446107Z", 2024-08-18T15:45:23.374 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.374 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.374 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.374 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.374 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.374 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.374 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_Xr1Z4jV7ZXypVMLCsCytQag", 2024-08-18T15:45:23.374 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.374 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.374 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.375 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.375 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.375 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.375 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.375 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.375 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723994599.662685_88.1", 2024-08-18T15:45:23.375 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.375 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-14", 2024-08-18T15:45:23.375 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:23:19.662685027Z", 2024-08-18T15:45:23.375 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.375 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.375 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.375 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.375 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.376 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.376 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_Xr1Z4jV7ZXypVMLCsCytQag", 2024-08-18T15:45:23.376 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.376 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.376 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.376 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.376 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.376 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.376 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.376 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.376 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723994599.682718_89.1", 2024-08-18T15:45:23.376 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.376 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:45:23.377 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:23:19.682718683Z", 2024-08-18T15:45:23.377 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.377 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.377 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.377 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.377 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.377 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.377 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_tZckUgZHpjdeXtjxrO3FDKV", 2024-08-18T15:45:23.377 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.377 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.377 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.377 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.377 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.377 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.378 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.378 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.378 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723994599.685017_90.1", 2024-08-18T15:45:23.378 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.378 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:45:23.378 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:23:19.685017287Z", 2024-08-18T15:45:23.378 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.378 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.378 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.378 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.378 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.378 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.378 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_tZckUgZHpjdeXtjxrO3FDKV", 2024-08-18T15:45:23.378 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.379 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.379 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.379 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.379 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.379 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.379 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.379 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.379 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723994599.685790_91.1", 2024-08-18T15:45:23.379 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.379 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-15", 2024-08-18T15:45:23.379 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:23:19.685790914Z", 2024-08-18T15:45:23.379 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.379 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.379 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.380 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.380 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.380 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.380 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_z4wReHTq2_y91Bl7PqzxJPS", 2024-08-18T15:45:23.380 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.380 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.380 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.380 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.380 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.380 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.380 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.380 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.380 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723994599.687927_92.1", 2024-08-18T15:45:23.380 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.381 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-15", 2024-08-18T15:45:23.381 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:23:19.687927319Z", 2024-08-18T15:45:23.381 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.381 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.381 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.381 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.381 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.381 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.381 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_z4wReHTq2_y91Bl7PqzxJPS", 2024-08-18T15:45:23.381 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.381 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.381 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.381 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.381 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.382 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.382 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.382 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.382 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723994722.684594_96.1", 2024-08-18T15:45:23.382 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.382 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:45:23.382 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:25:22.684594373Z", 2024-08-18T15:45:23.382 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.382 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.382 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.382 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.382 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.382 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.383 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_Y8QIfzmW8naLImEFza7--rg", 2024-08-18T15:45:23.383 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.383 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.383 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.383 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.383 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.383 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.383 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.383 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.383 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723994722.687709_97.1", 2024-08-18T15:45:23.383 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.383 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:45:23.383 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:25:22.687709684Z", 2024-08-18T15:45:23.383 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.384 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.384 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.384 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.384 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.384 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.384 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_Y8QIfzmW8naLImEFza7--rg", 2024-08-18T15:45:23.384 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.384 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.384 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.384 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.384 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.384 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.384 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.384 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.385 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723994722.688892_98.1", 2024-08-18T15:45:23.385 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.385 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-16", 2024-08-18T15:45:23.385 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:25:22.688892164Z", 2024-08-18T15:45:23.385 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.385 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.385 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.385 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.385 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.385 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.385 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_pogOAh2gLfyKg4IGe5K-1It", 2024-08-18T15:45:23.385 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.385 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.386 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.386 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.386 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.386 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.386 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.386 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.386 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723994722.692127_99.1", 2024-08-18T15:45:23.386 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.386 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-16", 2024-08-18T15:45:23.386 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:25:22.692127591Z", 2024-08-18T15:45:23.386 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.386 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.386 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.387 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.387 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.387 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.387 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_pogOAh2gLfyKg4IGe5K-1It", 2024-08-18T15:45:23.387 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.387 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.387 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.387 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.387 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.387 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.387 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.387 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.387 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723994722.705107_100.1", 2024-08-18T15:45:23.387 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.388 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:45:23.388 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:25:22.705107630Z", 2024-08-18T15:45:23.388 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.388 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.388 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.388 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.388 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.388 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.388 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_TizKQmOZjxDm3jCjcVr_PHs", 2024-08-18T15:45:23.388 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.388 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.388 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.388 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.388 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.389 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.389 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.389 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.389 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723994722.707889_101.1", 2024-08-18T15:45:23.389 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.389 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:45:23.389 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:25:22.707889768Z", 2024-08-18T15:45:23.389 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.389 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.389 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.389 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.389 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.389 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.390 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_TizKQmOZjxDm3jCjcVr_PHs", 2024-08-18T15:45:23.390 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.390 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.390 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.390 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.390 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.390 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.390 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.390 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.390 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723994722.708890_102.1", 2024-08-18T15:45:23.390 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.390 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-17", 2024-08-18T15:45:23.390 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:25:22.708890200Z", 2024-08-18T15:45:23.391 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.391 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.391 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.391 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.391 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.391 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.391 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_bG8oC8bXHaoSssmVUfs6qfz", 2024-08-18T15:45:23.391 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.391 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.391 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.391 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.391 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.391 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.391 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.392 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.392 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723994722.711644_103.1", 2024-08-18T15:45:23.392 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.392 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-17", 2024-08-18T15:45:23.392 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:25:22.711644822Z", 2024-08-18T15:45:23.392 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.392 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.392 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.392 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.392 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.392 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.392 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_bG8oC8bXHaoSssmVUfs6qfz", 2024-08-18T15:45:23.392 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.393 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.393 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.393 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.393 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.393 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.393 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.393 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.393 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723994722.734205_104.1", 2024-08-18T15:45:23.393 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.393 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:45:23.393 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:25:22.734205817Z", 2024-08-18T15:45:23.393 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.393 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.393 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.394 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.394 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.394 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.394 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_dLDL_K-k27vccG78KCPIIzE", 2024-08-18T15:45:23.394 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.394 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.394 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.394 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.394 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.394 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.394 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.394 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.394 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723994722.736900_105.1", 2024-08-18T15:45:23.395 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.395 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:45:23.395 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:25:22.736900697Z", 2024-08-18T15:45:23.395 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.395 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.395 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.395 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.395 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.395 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.395 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_dLDL_K-k27vccG78KCPIIzE", 2024-08-18T15:45:23.395 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.395 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.395 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.396 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.396 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.396 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.396 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.396 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.396 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723994722.737991_106.1", 2024-08-18T15:45:23.396 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.396 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-18", 2024-08-18T15:45:23.396 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:25:22.737991682Z", 2024-08-18T15:45:23.396 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.396 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.396 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.396 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.396 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.397 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.397 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_HPNnX1tYqBsOgWp8mgW7Qb0", 2024-08-18T15:45:23.397 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.397 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.397 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.397 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.397 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.397 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.397 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.397 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.397 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723994722.740810_107.1", 2024-08-18T15:45:23.397 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.397 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-18", 2024-08-18T15:45:23.397 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:25:22.740810965Z", 2024-08-18T15:45:23.398 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.398 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.398 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.398 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.398 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.398 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.398 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_HPNnX1tYqBsOgWp8mgW7Qb0", 2024-08-18T15:45:23.398 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.398 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.398 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.398 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.398 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.398 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.399 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.399 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.399 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995081.645527_272.1", 2024-08-18T15:45:23.399 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.399 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:45:23.399 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:31:21.645527834Z", 2024-08-18T15:45:23.399 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.399 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.399 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.399 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.399 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.399 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.399 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_tgDMSIYDTOiTpHaxXYDAzVY", 2024-08-18T15:45:23.399 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.400 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.400 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.400 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.400 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.400 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.400 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.400 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.400 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995081.648393_273.1", 2024-08-18T15:45:23.400 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.400 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:45:23.400 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:31:21.648393707Z", 2024-08-18T15:45:23.400 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.400 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.401 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.401 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.401 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.401 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.401 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_tgDMSIYDTOiTpHaxXYDAzVY", 2024-08-18T15:45:23.401 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.401 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.401 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.401 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.401 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.401 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.401 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.401 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.401 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995081.649470_274.1", 2024-08-18T15:45:23.402 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.402 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-19", 2024-08-18T15:45:23.402 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:31:21.649470258Z", 2024-08-18T15:45:23.402 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.402 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.402 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.402 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.402 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.402 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.402 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_K6QtNcOVkdATFYGx2kwzf4z", 2024-08-18T15:45:23.402 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.402 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.402 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.403 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.403 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.403 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.403 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.403 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.403 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995081.652067_275.1", 2024-08-18T15:45:23.403 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.403 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-19", 2024-08-18T15:45:23.403 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:31:21.652067379Z", 2024-08-18T15:45:23.403 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.403 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.403 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.403 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.404 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.404 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.404 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_K6QtNcOVkdATFYGx2kwzf4z", 2024-08-18T15:45:23.404 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.404 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.404 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.404 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.404 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.404 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.404 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.404 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.404 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995081.663969_276.1", 2024-08-18T15:45:23.404 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.404 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:45:23.405 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:31:21.663969084Z", 2024-08-18T15:45:23.405 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.405 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.405 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.405 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.405 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.405 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.405 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_Tq_y8CH-R5JTA5co_8cOHJr", 2024-08-18T15:45:23.405 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.405 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.405 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.405 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.405 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.406 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.406 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.406 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.406 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995081.666599_277.1", 2024-08-18T15:45:23.406 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.406 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:45:23.406 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:31:21.666599931Z", 2024-08-18T15:45:23.406 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.406 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.406 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.406 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.406 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.406 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.407 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_Tq_y8CH-R5JTA5co_8cOHJr", 2024-08-18T15:45:23.407 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.407 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.407 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.407 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.407 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.407 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.407 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.407 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.407 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995081.667569_278.1", 2024-08-18T15:45:23.407 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.407 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-20", 2024-08-18T15:45:23.407 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:31:21.667569296Z", 2024-08-18T15:45:23.407 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.408 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.408 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.408 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.408 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.408 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.408 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_wlqM8_vGXOeMXMyiNpDyzb0", 2024-08-18T15:45:23.408 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.408 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.408 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.408 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.408 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.408 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.408 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.408 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.409 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995081.670137_279.1", 2024-08-18T15:45:23.409 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.409 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-20", 2024-08-18T15:45:23.409 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:31:21.670137868Z", 2024-08-18T15:45:23.409 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.409 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.409 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.409 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.409 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.409 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.409 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_wlqM8_vGXOeMXMyiNpDyzb0", 2024-08-18T15:45:23.409 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.409 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.409 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.410 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.410 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.410 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.410 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.410 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.410 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995081.689780_280.1", 2024-08-18T15:45:23.410 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.410 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:45:23.410 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:31:21.689780751Z", 2024-08-18T15:45:23.410 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.410 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.410 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.410 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.411 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.411 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.411 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_tuo9kCpofi-fHLKLK3syMDP", 2024-08-18T15:45:23.411 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.411 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.411 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.411 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.411 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.411 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.411 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.411 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.411 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995081.692441_281.1", 2024-08-18T15:45:23.412 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.412 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:45:23.412 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:31:21.692441804Z", 2024-08-18T15:45:23.412 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.412 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.412 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.412 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.412 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.412 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.412 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_tuo9kCpofi-fHLKLK3syMDP", 2024-08-18T15:45:23.412 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.412 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.412 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.412 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.413 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.413 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.413 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.413 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.413 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995081.693378_282.1", 2024-08-18T15:45:23.413 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.413 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-21", 2024-08-18T15:45:23.413 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:31:21.693378470Z", 2024-08-18T15:45:23.413 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.413 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.413 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.413 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.413 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.414 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.414 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_ntCDeW1uH7FFuKrSyPKYHJm", 2024-08-18T15:45:23.414 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.414 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.414 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.414 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.414 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.414 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.414 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.414 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.414 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995081.695800_283.1", 2024-08-18T15:45:23.414 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.414 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-21", 2024-08-18T15:45:23.415 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:31:21.695800030Z", 2024-08-18T15:45:23.415 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.415 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.415 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.415 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.415 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.415 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.415 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_ntCDeW1uH7FFuKrSyPKYHJm", 2024-08-18T15:45:23.415 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.415 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.415 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.415 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.416 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.416 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.416 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.416 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.416 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995081.706620_284.1", 2024-08-18T15:45:23.416 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.416 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:45:23.416 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:31:21.706620679Z", 2024-08-18T15:45:23.416 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.416 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.416 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_tgDMSIYDTOiTpHaxXYDAzVY", 2024-08-18T15:45:23.416 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.416 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.417 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.417 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_tgDMSIYDTOiTpHaxXYDAzVY", 2024-08-18T15:45:23.417 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 2 2024-08-18T15:45:23.417 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.417 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.417 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.417 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.417 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.417 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.417 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.417 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995081.709128_285.1", 2024-08-18T15:45:23.417 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.417 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:45:23.418 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:31:21.709128917Z", 2024-08-18T15:45:23.418 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.418 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.418 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_tgDMSIYDTOiTpHaxXYDAzVY", 2024-08-18T15:45:23.418 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.418 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.418 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.418 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_tgDMSIYDTOiTpHaxXYDAzVY", 2024-08-18T15:45:23.418 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 2 2024-08-18T15:45:23.418 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.418 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.418 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.418 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.419 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.419 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.419 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.419 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995081.712458_286.1", 2024-08-18T15:45:23.419 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.419 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:45:23.419 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:31:21.712458353Z", 2024-08-18T15:45:23.419 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.419 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.419 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_Tq_y8CH-R5JTA5co_8cOHJr", 2024-08-18T15:45:23.419 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.419 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.419 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.420 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_Tq_y8CH-R5JTA5co_8cOHJr", 2024-08-18T15:45:23.420 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 2 2024-08-18T15:45:23.420 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.420 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.420 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.420 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.420 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.420 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.420 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.420 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995081.714923_287.1", 2024-08-18T15:45:23.420 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.420 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:45:23.420 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:31:21.714923303Z", 2024-08-18T15:45:23.421 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.421 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.421 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_Tq_y8CH-R5JTA5co_8cOHJr", 2024-08-18T15:45:23.421 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.421 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.421 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.421 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_Tq_y8CH-R5JTA5co_8cOHJr", 2024-08-18T15:45:23.421 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 2 2024-08-18T15:45:23.421 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.421 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.422 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.422 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.422 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.422 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.422 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.422 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995081.719853_288.1", 2024-08-18T15:45:23.422 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.422 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:45:23.422 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:31:21.719853955Z", 2024-08-18T15:45:23.422 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.422 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.422 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_tuo9kCpofi-fHLKLK3syMDP", 2024-08-18T15:45:23.422 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.423 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.423 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.423 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_tuo9kCpofi-fHLKLK3syMDP", 2024-08-18T15:45:23.423 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 2 2024-08-18T15:45:23.423 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.423 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.423 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.423 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.423 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.423 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.423 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.423 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995081.722384_289.1", 2024-08-18T15:45:23.423 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.423 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:45:23.424 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:31:21.722384196Z", 2024-08-18T15:45:23.424 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.424 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.424 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_tuo9kCpofi-fHLKLK3syMDP", 2024-08-18T15:45:23.424 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.424 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.424 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.424 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_tuo9kCpofi-fHLKLK3syMDP", 2024-08-18T15:45:23.424 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 2 2024-08-18T15:45:23.424 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.424 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.424 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.424 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.425 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.425 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.425 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.425 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995271.336261_378.1", 2024-08-18T15:45:23.425 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.425 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:45:23.425 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:34:31.336261807Z", 2024-08-18T15:45:23.425 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.425 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.425 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.425 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.425 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.425 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.426 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_pdWHtmdUtmS11GVZZnyUudX", 2024-08-18T15:45:23.426 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.426 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.426 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.426 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.426 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.426 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.426 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.426 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.426 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995271.339446_379.1", 2024-08-18T15:45:23.426 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.426 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:45:23.426 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:34:31.339446352Z", 2024-08-18T15:45:23.426 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.427 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.427 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.427 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.427 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.427 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.427 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_pdWHtmdUtmS11GVZZnyUudX", 2024-08-18T15:45:23.427 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.427 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.427 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.427 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.427 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.427 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.427 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.428 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.428 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995271.340841_380.1", 2024-08-18T15:45:23.428 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.428 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-22", 2024-08-18T15:45:23.428 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:34:31.340841665Z", 2024-08-18T15:45:23.428 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.428 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.428 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.428 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.428 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.428 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.428 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_O9sHsrAa3NkoKI1YNBKaQpY", 2024-08-18T15:45:23.428 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.428 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.429 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.429 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.429 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.429 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.429 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.429 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.429 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995271.343670_381.1", 2024-08-18T15:45:23.429 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.429 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-22", 2024-08-18T15:45:23.429 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:34:31.343670599Z", 2024-08-18T15:45:23.429 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.429 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.429 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.429 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.430 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.430 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.430 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_O9sHsrAa3NkoKI1YNBKaQpY", 2024-08-18T15:45:23.430 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.430 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.430 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.430 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.430 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.430 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.430 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.430 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.430 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995271.347218_382.1", 2024-08-18T15:45:23.430 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.431 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:45:23.431 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:34:31.347218065Z", 2024-08-18T15:45:23.431 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.431 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.431 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_pdWHtmdUtmS11GVZZnyUudX", 2024-08-18T15:45:23.431 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.431 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.431 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.431 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_pdWHtmdUtmS11GVZZnyUudX", 2024-08-18T15:45:23.431 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 2 2024-08-18T15:45:23.431 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.431 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.431 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.431 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.432 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.432 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.432 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.432 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995271.350113_383.1", 2024-08-18T15:45:23.432 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.432 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:45:23.432 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:34:31.350113109Z", 2024-08-18T15:45:23.432 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.432 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.432 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_pdWHtmdUtmS11GVZZnyUudX", 2024-08-18T15:45:23.432 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.432 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.432 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.432 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_pdWHtmdUtmS11GVZZnyUudX", 2024-08-18T15:45:23.433 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 2 2024-08-18T15:45:23.433 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.433 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.433 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.433 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.433 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.433 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.433 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.433 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995462.569535_473.1", 2024-08-18T15:45:23.433 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.433 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:45:23.433 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:37:42.569535297Z", 2024-08-18T15:45:23.434 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.434 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.434 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.434 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.434 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.434 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.434 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_wpzt3Zxvq-oNY9mOoRnPKWQ", 2024-08-18T15:45:23.434 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.434 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.434 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.434 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.434 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.434 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.435 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.435 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.435 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995462.571879_474.1", 2024-08-18T15:45:23.435 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.435 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:45:23.435 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:37:42.571879550Z", 2024-08-18T15:45:23.435 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.435 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.435 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.435 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.435 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.435 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.435 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_wpzt3Zxvq-oNY9mOoRnPKWQ", 2024-08-18T15:45:23.436 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.436 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.436 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.436 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.436 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.436 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.436 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.436 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.436 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995462.572788_475.1", 2024-08-18T15:45:23.438 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.438 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-23", 2024-08-18T15:45:23.438 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:37:42.572788757Z", 2024-08-18T15:45:23.438 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.438 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.438 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.438 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.438 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.438 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.438 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_uJpVLZnIMl1lW-0wuWAfT4Y", 2024-08-18T15:45:23.438 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.438 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.439 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.439 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.439 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.439 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.439 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.439 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.439 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995462.575132_476.1", 2024-08-18T15:45:23.439 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.439 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-23", 2024-08-18T15:45:23.439 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:37:42.575132214Z", 2024-08-18T15:45:23.439 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.439 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.439 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.439 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.440 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.440 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.440 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_uJpVLZnIMl1lW-0wuWAfT4Y", 2024-08-18T15:45:23.440 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.440 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.440 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.440 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.440 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.440 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.440 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.440 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.440 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995503.967200_486.1", 2024-08-18T15:45:23.440 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.440 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:45:23.441 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:38:23.967200465Z", 2024-08-18T15:45:23.441 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.441 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.441 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_wpzt3Zxvq-oNY9mOoRnPKWQ", 2024-08-18T15:45:23.441 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.441 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.441 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.441 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_wpzt3Zxvq-oNY9mOoRnPKWQ", 2024-08-18T15:45:23.441 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 2 2024-08-18T15:45:23.441 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.441 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.441 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.441 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.442 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.442 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.442 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.442 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995503.969777_487.1", 2024-08-18T15:45:23.442 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.442 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:45:23.442 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:38:23.969777196Z", 2024-08-18T15:45:23.442 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.442 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.442 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_wpzt3Zxvq-oNY9mOoRnPKWQ", 2024-08-18T15:45:23.442 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.442 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.442 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.442 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_wpzt3Zxvq-oNY9mOoRnPKWQ", 2024-08-18T15:45:23.443 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 2 2024-08-18T15:45:23.443 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.443 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.443 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.443 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.443 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.443 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.443 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.443 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995565.569248_530.1", 2024-08-18T15:45:23.443 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.443 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:45:23.443 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:39:25.569248856Z", 2024-08-18T15:45:23.443 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.444 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.444 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_wpzt3Zxvq-oNY9mOoRnPKWQ", 2024-08-18T15:45:23.444 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 2 2024-08-18T15:45:23.444 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.444 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.444 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_wpzt3Zxvq-oNY9mOoRnPKWQ", 2024-08-18T15:45:23.444 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 3 2024-08-18T15:45:23.444 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.444 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.444 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.444 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.444 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.444 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.445 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.445 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995565.571903_531.1", 2024-08-18T15:45:23.445 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.445 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:45:23.445 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:39:25.571903234Z", 2024-08-18T15:45:23.445 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.445 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.445 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_wpzt3Zxvq-oNY9mOoRnPKWQ", 2024-08-18T15:45:23.445 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 2 2024-08-18T15:45:23.445 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.445 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.445 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_wpzt3Zxvq-oNY9mOoRnPKWQ", 2024-08-18T15:45:23.445 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 3 2024-08-18T15:45:23.445 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.446 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.446 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.446 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.446 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.446 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.446 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.446 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995627.134415_559.1", 2024-08-18T15:45:23.446 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.446 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:45:23.446 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:40:27.134415241Z", 2024-08-18T15:45:23.446 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.446 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.446 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.447 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.447 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.447 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.447 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_-bWXVuDtxeg-vFcENUxwyqY", 2024-08-18T15:45:23.447 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.447 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.447 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.447 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.447 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.447 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.447 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.447 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.447 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995627.137158_560.1", 2024-08-18T15:45:23.448 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.448 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:45:23.448 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:40:27.137158269Z", 2024-08-18T15:45:23.448 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.448 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.448 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.448 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.448 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.448 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.448 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_-bWXVuDtxeg-vFcENUxwyqY", 2024-08-18T15:45:23.448 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.448 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.448 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.448 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.449 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.449 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.449 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.449 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.449 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995627.138166_561.1", 2024-08-18T15:45:23.449 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.449 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-24", 2024-08-18T15:45:23.449 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:40:27.138166620Z", 2024-08-18T15:45:23.449 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.449 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.449 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.449 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.449 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.449 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.450 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_GqmPD6qFaIdZi7CGWg-6cQ9", 2024-08-18T15:45:23.450 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.450 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.450 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.450 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.450 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.450 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.450 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.450 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.450 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995627.140690_562.1", 2024-08-18T15:45:23.450 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.450 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-24", 2024-08-18T15:45:23.450 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:40:27.140690252Z", 2024-08-18T15:45:23.451 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.451 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.451 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.451 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.451 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.451 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.451 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_GqmPD6qFaIdZi7CGWg-6cQ9", 2024-08-18T15:45:23.451 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.451 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.451 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.451 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.451 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.451 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.451 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.452 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.452 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995627.150973_563.1", 2024-08-18T15:45:23.452 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.452 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:45:23.452 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:40:27.150973118Z", 2024-08-18T15:45:23.452 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.452 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.452 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.452 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.452 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.452 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.452 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_ZASiQO1nC1KO4fWWmGbgTG6", 2024-08-18T15:45:23.452 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.453 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.453 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.453 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.453 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.453 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.453 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.453 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.453 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995627.153561_564.1", 2024-08-18T15:45:23.453 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.453 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:45:23.453 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:40:27.153561617Z", 2024-08-18T15:45:23.453 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.453 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.453 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.454 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.454 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.454 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.454 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_ZASiQO1nC1KO4fWWmGbgTG6", 2024-08-18T15:45:23.454 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.454 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.454 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.454 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.454 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.454 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.454 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.454 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.454 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995627.154521_565.1", 2024-08-18T15:45:23.455 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.455 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-25", 2024-08-18T15:45:23.455 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:40:27.154521041Z", 2024-08-18T15:45:23.455 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.455 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.455 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.455 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.455 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.455 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.455 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_IcFaB2XBAat2mP6-7HsS5nj", 2024-08-18T15:45:23.455 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.455 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.455 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.455 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.456 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.456 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.456 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.456 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.456 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995627.156830_566.1", 2024-08-18T15:45:23.456 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.456 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-25", 2024-08-18T15:45:23.456 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:40:27.156830184Z", 2024-08-18T15:45:23.456 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.456 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.456 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.456 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.457 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.457 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.457 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_IcFaB2XBAat2mP6-7HsS5nj", 2024-08-18T15:45:23.457 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.457 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.457 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.457 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.457 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.457 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.457 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.457 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.457 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995627.173949_567.1", 2024-08-18T15:45:23.457 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.457 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:45:23.458 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:40:27.173949128Z", 2024-08-18T15:45:23.458 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.458 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.458 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.458 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.458 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.458 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.458 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_UDZD1orQ5TbJSSveyG1C6yO", 2024-08-18T15:45:23.458 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.458 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.458 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.458 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.458 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.459 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.459 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.459 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.459 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995627.176614_568.1", 2024-08-18T15:45:23.459 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.459 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:45:23.459 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:40:27.176614276Z", 2024-08-18T15:45:23.459 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.459 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.459 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.459 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.459 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.459 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.460 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_UDZD1orQ5TbJSSveyG1C6yO", 2024-08-18T15:45:23.460 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.460 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.460 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.460 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.460 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.460 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.460 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.460 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.460 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995627.177588_569.1", 2024-08-18T15:45:23.460 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.460 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-26", 2024-08-18T15:45:23.460 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:40:27.177588755Z", 2024-08-18T15:45:23.461 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.461 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.461 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.461 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.461 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.461 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.461 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_6T2erh0-8LIJN2ZhFD-d0zv", 2024-08-18T15:45:23.461 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.461 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.461 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.461 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.461 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.461 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.462 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.462 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.462 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995627.180057_570.1", 2024-08-18T15:45:23.462 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.462 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-26", 2024-08-18T15:45:23.462 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:40:27.180057289Z", 2024-08-18T15:45:23.462 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.462 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.462 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.462 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.462 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.462 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.462 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_6T2erh0-8LIJN2ZhFD-d0zv", 2024-08-18T15:45:23.463 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.463 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.463 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.463 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.463 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.463 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.463 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.463 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.463 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995627.190598_571.1", 2024-08-18T15:45:23.463 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.463 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:45:23.463 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:40:27.190598553Z", 2024-08-18T15:45:23.463 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.463 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.464 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_-bWXVuDtxeg-vFcENUxwyqY", 2024-08-18T15:45:23.464 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.464 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.464 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.464 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_-bWXVuDtxeg-vFcENUxwyqY", 2024-08-18T15:45:23.464 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 2 2024-08-18T15:45:23.464 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.464 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.464 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.464 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.464 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.464 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.464 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.464 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995627.193264_572.1", 2024-08-18T15:45:23.465 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.465 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:45:23.465 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:40:27.193264293Z", 2024-08-18T15:45:23.465 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.465 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.465 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_-bWXVuDtxeg-vFcENUxwyqY", 2024-08-18T15:45:23.465 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.465 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.465 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.465 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_-bWXVuDtxeg-vFcENUxwyqY", 2024-08-18T15:45:23.465 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 2 2024-08-18T15:45:23.465 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.465 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.465 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.466 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.466 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.466 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.466 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.466 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995627.196357_573.1", 2024-08-18T15:45:23.466 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.466 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:45:23.466 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:40:27.196357945Z", 2024-08-18T15:45:23.466 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.466 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.466 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_ZASiQO1nC1KO4fWWmGbgTG6", 2024-08-18T15:45:23.466 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.466 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.467 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.467 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_ZASiQO1nC1KO4fWWmGbgTG6", 2024-08-18T15:45:23.467 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 2 2024-08-18T15:45:23.467 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.467 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.467 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.467 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.467 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.467 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.467 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.467 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995627.198715_574.1", 2024-08-18T15:45:23.467 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.467 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:45:23.467 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:40:27.198715261Z", 2024-08-18T15:45:23.468 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.468 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.468 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_ZASiQO1nC1KO4fWWmGbgTG6", 2024-08-18T15:45:23.468 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.468 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.468 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.468 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_ZASiQO1nC1KO4fWWmGbgTG6", 2024-08-18T15:45:23.468 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 2 2024-08-18T15:45:23.468 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.468 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.468 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.468 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.468 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.468 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.469 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.469 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995627.203450_575.1", 2024-08-18T15:45:23.469 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.469 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:45:23.469 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:40:27.203450736Z", 2024-08-18T15:45:23.469 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.469 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.469 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_UDZD1orQ5TbJSSveyG1C6yO", 2024-08-18T15:45:23.469 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.469 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.469 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.469 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_UDZD1orQ5TbJSSveyG1C6yO", 2024-08-18T15:45:23.469 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 2 2024-08-18T15:45:23.469 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.470 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.470 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.470 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.470 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.470 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.470 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.470 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995627.205732_576.1", 2024-08-18T15:45:23.470 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.470 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:45:23.470 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:40:27.205732200Z", 2024-08-18T15:45:23.470 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.470 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.470 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_UDZD1orQ5TbJSSveyG1C6yO", 2024-08-18T15:45:23.471 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.471 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.471 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.471 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_UDZD1orQ5TbJSSveyG1C6yO", 2024-08-18T15:45:23.471 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 2 2024-08-18T15:45:23.471 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.471 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.471 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.471 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.471 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.471 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.471 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.471 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995690.547603_610.1", 2024-08-18T15:45:23.471 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.472 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:45:23.472 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:41:30.547603460Z", 2024-08-18T15:45:23.472 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.472 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.472 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.472 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.472 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.472 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.472 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_jeYug1evj8xbYPXyUA9ZAIF", 2024-08-18T15:45:23.472 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.472 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.472 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.472 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.472 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.473 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.473 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.473 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.473 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995690.550097_611.1", 2024-08-18T15:45:23.473 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.473 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:45:23.473 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:41:30.550097288Z", 2024-08-18T15:45:23.473 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.473 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.473 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.473 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.473 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.473 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.473 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_jeYug1evj8xbYPXyUA9ZAIF", 2024-08-18T15:45:23.474 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.474 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.474 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.474 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.474 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.474 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.474 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.474 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.474 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995690.551077_612.1", 2024-08-18T15:45:23.474 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.474 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-27", 2024-08-18T15:45:23.474 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:41:30.551077825Z", 2024-08-18T15:45:23.474 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.475 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.475 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.475 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.475 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.475 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.475 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_2Y1Gbxx4l53IXmgzA2xQDmh", 2024-08-18T15:45:23.475 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.475 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.475 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.475 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.475 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.475 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.475 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.475 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.476 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995690.553419_613.1", 2024-08-18T15:45:23.476 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.476 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-27", 2024-08-18T15:45:23.476 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:41:30.553419971Z", 2024-08-18T15:45:23.476 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.476 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.476 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.476 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.476 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.476 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.476 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_2Y1Gbxx4l53IXmgzA2xQDmh", 2024-08-18T15:45:23.476 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.476 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.477 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.477 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.477 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.477 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.477 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.477 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.477 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995690.562778_614.1", 2024-08-18T15:45:23.477 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.477 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:45:23.477 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:41:30.562778163Z", 2024-08-18T15:45:23.477 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.477 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.477 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.477 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.478 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.478 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.478 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_5sF5e5L5wBDjZwv2FwY2xZ2", 2024-08-18T15:45:23.478 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.478 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.478 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.478 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.478 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.478 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.478 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.478 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.478 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995690.565048_615.1", 2024-08-18T15:45:23.479 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.479 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:45:23.479 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:41:30.565048948Z", 2024-08-18T15:45:23.479 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.479 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.479 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.479 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.479 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.479 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.479 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_5sF5e5L5wBDjZwv2FwY2xZ2", 2024-08-18T15:45:23.479 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.479 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.479 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.480 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.480 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.480 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.480 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.480 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.480 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995690.566055_616.1", 2024-08-18T15:45:23.480 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.480 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-28", 2024-08-18T15:45:23.480 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:41:30.566055028Z", 2024-08-18T15:45:23.480 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.480 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.480 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.480 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.480 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.481 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.481 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_3Xacd7LBrQbPx_GZINBqEvJ", 2024-08-18T15:45:23.481 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.481 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.481 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.481 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.481 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.481 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.481 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.481 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.481 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995690.568292_617.1", 2024-08-18T15:45:23.481 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.481 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-28", 2024-08-18T15:45:23.481 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:41:30.568292915Z", 2024-08-18T15:45:23.482 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.482 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.482 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.482 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.482 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.482 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.482 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_3Xacd7LBrQbPx_GZINBqEvJ", 2024-08-18T15:45:23.482 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.482 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.482 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.482 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.482 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.482 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.483 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.483 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.483 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995690.583629_618.1", 2024-08-18T15:45:23.483 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.483 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:45:23.483 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:41:30.583629631Z", 2024-08-18T15:45:23.483 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.483 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.483 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.483 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.483 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.483 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.483 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_Ds7izdnTDTHShaa6cA2hQJO", 2024-08-18T15:45:23.484 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.484 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.484 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.484 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.484 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.484 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.484 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.484 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.484 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995690.585913_619.1", 2024-08-18T15:45:23.484 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.484 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:45:23.484 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:41:30.585913147Z", 2024-08-18T15:45:23.484 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.485 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.485 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.485 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.485 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.485 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.485 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_Ds7izdnTDTHShaa6cA2hQJO", 2024-08-18T15:45:23.485 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.485 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.485 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.485 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.485 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.485 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.485 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.485 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.486 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995690.586806_620.1", 2024-08-18T15:45:23.486 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.486 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-29", 2024-08-18T15:45:23.486 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:41:30.586806831Z", 2024-08-18T15:45:23.486 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.486 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.486 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.486 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.486 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.486 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.486 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_daU6Le-dTETKkQatmqfff9r", 2024-08-18T15:45:23.486 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.486 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.487 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.487 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.487 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.487 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.487 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.487 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.487 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995690.589012_621.1", 2024-08-18T15:45:23.487 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.487 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-29", 2024-08-18T15:45:23.487 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:41:30.589012688Z", 2024-08-18T15:45:23.487 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.487 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.487 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.488 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.488 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.488 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.488 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_daU6Le-dTETKkQatmqfff9r", 2024-08-18T15:45:23.488 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.488 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.488 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.488 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.488 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.488 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.488 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.488 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.488 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995690.598776_622.1", 2024-08-18T15:45:23.488 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.489 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:45:23.489 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:41:30.598776328Z", 2024-08-18T15:45:23.489 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.489 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.489 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_jeYug1evj8xbYPXyUA9ZAIF", 2024-08-18T15:45:23.489 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.489 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.489 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.489 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_jeYug1evj8xbYPXyUA9ZAIF", 2024-08-18T15:45:23.489 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 2 2024-08-18T15:45:23.489 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.489 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.489 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.489 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.490 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.490 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.490 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.490 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995690.601154_623.1", 2024-08-18T15:45:23.490 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.490 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:45:23.490 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:41:30.601154145Z", 2024-08-18T15:45:23.490 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.490 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.490 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_jeYug1evj8xbYPXyUA9ZAIF", 2024-08-18T15:45:23.490 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.490 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.490 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.490 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_jeYug1evj8xbYPXyUA9ZAIF", 2024-08-18T15:45:23.491 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 2 2024-08-18T15:45:23.491 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.491 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.491 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.491 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.491 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.491 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.491 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.491 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995690.603208_624.1", 2024-08-18T15:45:23.491 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.491 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:45:23.491 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:41:30.603208651Z", 2024-08-18T15:45:23.491 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.491 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.492 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_jeYug1evj8xbYPXyUA9ZAIF", 2024-08-18T15:45:23.492 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 2 2024-08-18T15:45:23.492 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.492 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.492 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_jeYug1evj8xbYPXyUA9ZAIF", 2024-08-18T15:45:23.492 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 3 2024-08-18T15:45:23.492 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.492 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.492 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.492 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.492 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.492 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.492 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.493 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995690.605370_625.1", 2024-08-18T15:45:23.493 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.493 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:45:23.493 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:41:30.605370190Z", 2024-08-18T15:45:23.493 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.493 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.493 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_jeYug1evj8xbYPXyUA9ZAIF", 2024-08-18T15:45:23.493 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 2 2024-08-18T15:45:23.493 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.493 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.493 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_jeYug1evj8xbYPXyUA9ZAIF", 2024-08-18T15:45:23.493 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 3 2024-08-18T15:45:23.493 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.494 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.494 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.494 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.494 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.494 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.494 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.494 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995690.607911_626.1", 2024-08-18T15:45:23.494 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.494 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:45:23.494 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:41:30.607911940Z", 2024-08-18T15:45:23.494 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.494 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.494 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_5sF5e5L5wBDjZwv2FwY2xZ2", 2024-08-18T15:45:23.495 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.495 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.495 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.495 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_5sF5e5L5wBDjZwv2FwY2xZ2", 2024-08-18T15:45:23.495 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 2 2024-08-18T15:45:23.495 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.495 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.495 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.495 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.495 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.495 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.495 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.495 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995690.610104_627.1", 2024-08-18T15:45:23.496 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.496 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:45:23.496 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:41:30.610104648Z", 2024-08-18T15:45:23.496 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.496 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.496 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_5sF5e5L5wBDjZwv2FwY2xZ2", 2024-08-18T15:45:23.496 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.496 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.496 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.496 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_5sF5e5L5wBDjZwv2FwY2xZ2", 2024-08-18T15:45:23.496 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 2 2024-08-18T15:45:23.496 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.496 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.497 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.497 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.497 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.497 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.497 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.497 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995690.614660_628.1", 2024-08-18T15:45:23.497 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.497 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:45:23.497 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:41:30.614660827Z", 2024-08-18T15:45:23.497 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.497 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.497 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_5sF5e5L5wBDjZwv2FwY2xZ2", 2024-08-18T15:45:23.497 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 2 2024-08-18T15:45:23.498 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.498 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.498 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_5sF5e5L5wBDjZwv2FwY2xZ2", 2024-08-18T15:45:23.498 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 3 2024-08-18T15:45:23.498 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.498 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.498 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.498 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.498 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.498 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.498 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.498 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995690.616928_629.1", 2024-08-18T15:45:23.498 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.499 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:45:23.499 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:41:30.616928990Z", 2024-08-18T15:45:23.499 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.499 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.499 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_5sF5e5L5wBDjZwv2FwY2xZ2", 2024-08-18T15:45:23.499 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 2 2024-08-18T15:45:23.499 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.499 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.499 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_5sF5e5L5wBDjZwv2FwY2xZ2", 2024-08-18T15:45:23.499 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 3 2024-08-18T15:45:23.499 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.499 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.499 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.500 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.500 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.500 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.500 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.500 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995690.621300_630.1", 2024-08-18T15:45:23.500 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.500 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:45:23.500 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:41:30.621300062Z", 2024-08-18T15:45:23.500 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.500 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.500 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_Ds7izdnTDTHShaa6cA2hQJO", 2024-08-18T15:45:23.500 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.501 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.501 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.501 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_Ds7izdnTDTHShaa6cA2hQJO", 2024-08-18T15:45:23.501 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 2 2024-08-18T15:45:23.501 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.501 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.501 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.501 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.501 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.501 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.501 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.501 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995690.623904_631.1", 2024-08-18T15:45:23.501 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.501 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:45:23.502 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:41:30.623904678Z", 2024-08-18T15:45:23.502 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.502 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.502 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_Ds7izdnTDTHShaa6cA2hQJO", 2024-08-18T15:45:23.502 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.502 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.502 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.502 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_Ds7izdnTDTHShaa6cA2hQJO", 2024-08-18T15:45:23.502 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 2 2024-08-18T15:45:23.502 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.502 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.502 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.502 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.503 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.503 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.503 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.503 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995690.627942_632.1", 2024-08-18T15:45:23.503 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.503 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:45:23.503 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:41:30.627942660Z", 2024-08-18T15:45:23.503 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.503 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.503 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_Ds7izdnTDTHShaa6cA2hQJO", 2024-08-18T15:45:23.503 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 2 2024-08-18T15:45:23.503 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.503 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.504 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_Ds7izdnTDTHShaa6cA2hQJO", 2024-08-18T15:45:23.504 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 3 2024-08-18T15:45:23.504 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.504 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.504 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.504 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.504 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.504 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.504 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.504 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995690.630163_633.1", 2024-08-18T15:45:23.504 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.504 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:45:23.504 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:41:30.630163840Z", 2024-08-18T15:45:23.504 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.505 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.505 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_Ds7izdnTDTHShaa6cA2hQJO", 2024-08-18T15:45:23.505 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 2 2024-08-18T15:45:23.505 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.505 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.505 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_Ds7izdnTDTHShaa6cA2hQJO", 2024-08-18T15:45:23.505 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 3 2024-08-18T15:45:23.505 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.505 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.505 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.505 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.505 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.505 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.505 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.506 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995748.521479_647.1", 2024-08-18T15:45:23.506 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.506 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-30:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.30", 2024-08-18T15:45:23.506 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:28.521479385Z", 2024-08-18T15:45:23.506 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.506 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.506 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.506 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.506 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.506 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.506 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_X8eWxctwhKvpxc5MGUbCdeT", 2024-08-18T15:45:23.506 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.506 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.507 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.507 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.507 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.507 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.507 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.507 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.507 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995748.524106_648.1", 2024-08-18T15:45:23.507 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.507 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-30:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.30", 2024-08-18T15:45:23.507 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:28.524106942Z", 2024-08-18T15:45:23.507 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.507 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.507 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.507 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.508 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.508 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.508 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_X8eWxctwhKvpxc5MGUbCdeT", 2024-08-18T15:45:23.508 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.508 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.508 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.508 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.508 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.508 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.508 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.508 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.508 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995748.525089_649.1", 2024-08-18T15:45:23.508 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.509 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-30", 2024-08-18T15:45:23.509 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:28.525089002Z", 2024-08-18T15:45:23.509 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.509 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.509 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.509 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.509 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.509 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.509 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_Y_twJ8vlrnjpsWlMf8A1vcl", 2024-08-18T15:45:23.509 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.509 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.509 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.509 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.509 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.510 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.510 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.510 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.510 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995748.527509_650.1", 2024-08-18T15:45:23.510 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.510 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-30", 2024-08-18T15:45:23.510 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:28.527509595Z", 2024-08-18T15:45:23.510 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.510 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.510 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.510 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.510 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.510 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.511 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_Y_twJ8vlrnjpsWlMf8A1vcl", 2024-08-18T15:45:23.511 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.511 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.511 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.511 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.511 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.511 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.511 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.511 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.511 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995748.537416_651.1", 2024-08-18T15:45:23.511 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.511 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-31:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.31", 2024-08-18T15:45:23.511 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:28.537416223Z", 2024-08-18T15:45:23.511 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.512 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.512 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.512 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.512 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.512 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.512 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_3mYRdBwrm6Dau3dYq-qztzU", 2024-08-18T15:45:23.512 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.512 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.512 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.512 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.512 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.512 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.512 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.513 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.513 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995748.539892_652.1", 2024-08-18T15:45:23.513 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.513 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-31:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.31", 2024-08-18T15:45:23.513 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:28.539892239Z", 2024-08-18T15:45:23.513 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.513 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.513 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.513 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.513 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.513 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.513 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_3mYRdBwrm6Dau3dYq-qztzU", 2024-08-18T15:45:23.513 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.513 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.514 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.514 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.514 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.514 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.514 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.514 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.514 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995748.541032_653.1", 2024-08-18T15:45:23.514 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.514 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-31", 2024-08-18T15:45:23.514 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:28.541032236Z", 2024-08-18T15:45:23.514 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.514 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.514 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.514 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.515 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.515 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.515 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_LwFQ9ExxsG8yYIT84yLWJCM", 2024-08-18T15:45:23.515 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.515 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.515 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.515 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.515 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.515 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.515 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.515 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.515 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995748.543337_654.1", 2024-08-18T15:45:23.515 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.515 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-31", 2024-08-18T15:45:23.516 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:28.543337427Z", 2024-08-18T15:45:23.516 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.516 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.516 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.516 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.516 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.516 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.516 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_LwFQ9ExxsG8yYIT84yLWJCM", 2024-08-18T15:45:23.516 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.516 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.516 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.516 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.516 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.517 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.517 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.517 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.517 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995748.559655_655.1", 2024-08-18T15:45:23.517 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.517 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-32:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.32", 2024-08-18T15:45:23.517 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:28.559655282Z", 2024-08-18T15:45:23.517 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.517 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.517 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.517 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.517 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.517 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.517 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_N4lXfxOh8y0R0tn_VtdDq--", 2024-08-18T15:45:23.518 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.518 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.518 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.518 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.518 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.518 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.518 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.518 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.518 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995748.562055_656.1", 2024-08-18T15:45:23.518 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.518 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-32:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.32", 2024-08-18T15:45:23.518 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:28.562055315Z", 2024-08-18T15:45:23.518 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.519 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.519 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.519 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.519 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.519 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.519 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_N4lXfxOh8y0R0tn_VtdDq--", 2024-08-18T15:45:23.519 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.519 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.519 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.519 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.519 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.519 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.519 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.520 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.520 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995748.562981_657.1", 2024-08-18T15:45:23.520 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.520 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-32", 2024-08-18T15:45:23.520 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:28.562981860Z", 2024-08-18T15:45:23.520 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.520 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.520 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.520 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.520 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.520 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.520 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_grj-gSxc_4fWbu_ET8qVUlc", 2024-08-18T15:45:23.520 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.520 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.521 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.521 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.521 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.521 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.521 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.521 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.521 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995748.565132_658.1", 2024-08-18T15:45:23.521 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.521 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-32", 2024-08-18T15:45:23.521 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:28.565132516Z", 2024-08-18T15:45:23.521 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.521 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.522 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.522 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.522 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.522 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.522 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_grj-gSxc_4fWbu_ET8qVUlc", 2024-08-18T15:45:23.522 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.522 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.522 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.523 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.523 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.523 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.523 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.523 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.523 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995748.615071_659.1", 2024-08-18T15:45:23.523 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.523 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-30:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.30", 2024-08-18T15:45:23.523 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:28.615071398Z", 2024-08-18T15:45:23.523 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.523 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.523 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_X8eWxctwhKvpxc5MGUbCdeT", 2024-08-18T15:45:23.523 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.523 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.524 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.524 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_X8eWxctwhKvpxc5MGUbCdeT", 2024-08-18T15:45:23.524 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 2 2024-08-18T15:45:23.524 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.524 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.524 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.524 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.524 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.524 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.524 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.524 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995748.617533_660.1", 2024-08-18T15:45:23.524 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.524 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-30:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.30", 2024-08-18T15:45:23.525 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:28.617533647Z", 2024-08-18T15:45:23.525 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.525 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.525 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_X8eWxctwhKvpxc5MGUbCdeT", 2024-08-18T15:45:23.525 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.525 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.525 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.525 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_X8eWxctwhKvpxc5MGUbCdeT", 2024-08-18T15:45:23.525 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 2 2024-08-18T15:45:23.525 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.525 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.525 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.525 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.525 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.526 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.526 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.526 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995748.622444_661.1", 2024-08-18T15:45:23.526 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.526 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-31:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.31", 2024-08-18T15:45:23.526 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:28.622444585Z", 2024-08-18T15:45:23.526 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.526 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.526 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_3mYRdBwrm6Dau3dYq-qztzU", 2024-08-18T15:45:23.526 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.526 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.526 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.526 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_3mYRdBwrm6Dau3dYq-qztzU", 2024-08-18T15:45:23.527 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 2 2024-08-18T15:45:23.527 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.527 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.527 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.527 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.527 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.527 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.527 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.527 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995748.624798_662.1", 2024-08-18T15:45:23.527 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.527 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-31:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.31", 2024-08-18T15:45:23.527 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:28.624798339Z", 2024-08-18T15:45:23.528 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.528 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.528 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_3mYRdBwrm6Dau3dYq-qztzU", 2024-08-18T15:45:23.528 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.528 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.528 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.528 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_3mYRdBwrm6Dau3dYq-qztzU", 2024-08-18T15:45:23.528 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 2 2024-08-18T15:45:23.528 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.528 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.528 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.528 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.528 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.528 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.529 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.529 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995748.673691_663.1", 2024-08-18T15:45:23.529 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.529 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-32:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.32", 2024-08-18T15:45:23.529 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:28.673691986Z", 2024-08-18T15:45:23.529 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.529 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.529 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_N4lXfxOh8y0R0tn_VtdDq--", 2024-08-18T15:45:23.529 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.529 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.529 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.529 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_N4lXfxOh8y0R0tn_VtdDq--", 2024-08-18T15:45:23.529 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 2 2024-08-18T15:45:23.529 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.530 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.530 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.530 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.530 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.530 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.530 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.530 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995748.676028_664.1", 2024-08-18T15:45:23.530 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.530 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-32:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.32", 2024-08-18T15:45:23.530 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:28.676028814Z", 2024-08-18T15:45:23.530 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.530 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.530 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_N4lXfxOh8y0R0tn_VtdDq--", 2024-08-18T15:45:23.531 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.531 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.531 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.531 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_N4lXfxOh8y0R0tn_VtdDq--", 2024-08-18T15:45:23.531 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 2 2024-08-18T15:45:23.531 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.531 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.531 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.531 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.531 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.531 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.531 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.531 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995748.684893_665.1", 2024-08-18T15:45:23.532 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.532 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-33:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.33", 2024-08-18T15:45:23.532 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:28.684893391Z", 2024-08-18T15:45:23.532 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.532 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.532 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.532 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.532 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.532 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.532 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_lGULOaXyl4hHx7qh454F-aa", 2024-08-18T15:45:23.532 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.532 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.532 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.532 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.533 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.533 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.533 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.533 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.533 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995748.687266_666.1", 2024-08-18T15:45:23.533 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.533 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-33:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.33", 2024-08-18T15:45:23.533 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:28.687266999Z", 2024-08-18T15:45:23.533 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.533 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.533 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.533 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.533 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.534 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.534 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_lGULOaXyl4hHx7qh454F-aa", 2024-08-18T15:45:23.534 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.534 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.534 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.534 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.534 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.534 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.534 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.534 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.534 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995748.688166_667.1", 2024-08-18T15:45:23.534 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.534 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-33", 2024-08-18T15:45:23.535 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:28.688166208Z", 2024-08-18T15:45:23.535 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.535 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.535 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.535 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.535 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.535 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.535 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_WydO77zfjR8Husc4yxYU8A8", 2024-08-18T15:45:23.535 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.535 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.535 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.535 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.535 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.535 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.536 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.536 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.536 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995748.690522_668.1", 2024-08-18T15:45:23.536 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.536 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-33", 2024-08-18T15:45:23.536 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:28.690522745Z", 2024-08-18T15:45:23.536 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.536 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.536 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.536 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.536 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.536 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.536 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_WydO77zfjR8Husc4yxYU8A8", 2024-08-18T15:45:23.537 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.537 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.537 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.537 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.537 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.537 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.537 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.537 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.537 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995748.699985_669.1", 2024-08-18T15:45:23.537 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.537 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-34:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.34", 2024-08-18T15:45:23.537 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:28.699985189Z", 2024-08-18T15:45:23.537 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.537 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.538 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.538 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.538 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.538 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.538 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_gziC3Zhjn59sc9f24iXYqez", 2024-08-18T15:45:23.538 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.538 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.538 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.538 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.538 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.538 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.538 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.538 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.539 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995748.702346_670.1", 2024-08-18T15:45:23.539 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.539 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-34:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.34", 2024-08-18T15:45:23.539 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:28.702346126Z", 2024-08-18T15:45:23.539 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.539 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.539 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.539 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.539 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.539 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.539 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_gziC3Zhjn59sc9f24iXYqez", 2024-08-18T15:45:23.539 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.539 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.540 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.540 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.540 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.540 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.540 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.540 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.540 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995748.703327_671.1", 2024-08-18T15:45:23.540 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.540 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-34", 2024-08-18T15:45:23.540 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:28.703327689Z", 2024-08-18T15:45:23.540 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.540 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.540 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.540 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.541 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.541 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.541 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_fIT8XfQf3mP3Rdkpjxd4TDK", 2024-08-18T15:45:23.541 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.541 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.541 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.541 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.541 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.541 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.541 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.541 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.541 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995748.705678_672.1", 2024-08-18T15:45:23.541 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.541 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-34", 2024-08-18T15:45:23.542 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:28.705678013Z", 2024-08-18T15:45:23.542 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.542 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.542 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.542 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.542 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.542 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.542 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_fIT8XfQf3mP3Rdkpjxd4TDK", 2024-08-18T15:45:23.542 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.542 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.542 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.542 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.542 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.543 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.543 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.543 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.543 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995748.722023_673.1", 2024-08-18T15:45:23.543 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.543 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-35:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.35", 2024-08-18T15:45:23.543 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:28.722023363Z", 2024-08-18T15:45:23.543 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.543 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.543 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.543 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.543 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.543 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.544 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_8PQDI19UyDbSfg9jXHxQK0Z", 2024-08-18T15:45:23.544 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.544 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.544 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.544 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.544 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.544 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.544 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.544 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.544 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995748.724501_674.1", 2024-08-18T15:45:23.544 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.544 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-35:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.35", 2024-08-18T15:45:23.545 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:28.724501528Z", 2024-08-18T15:45:23.545 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.545 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.545 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.545 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.545 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.545 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.545 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_8PQDI19UyDbSfg9jXHxQK0Z", 2024-08-18T15:45:23.545 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.545 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.545 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.545 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.545 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.546 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.546 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.546 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.546 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995748.725468_675.1", 2024-08-18T15:45:23.546 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.546 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-35", 2024-08-18T15:45:23.546 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:28.725468008Z", 2024-08-18T15:45:23.546 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.546 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.546 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.546 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.546 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.546 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.547 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_5pwJU04pzA5ySeDPbaf8cI9", 2024-08-18T15:45:23.547 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.547 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.547 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.547 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.547 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.547 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.547 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.547 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.547 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995748.728040_676.1", 2024-08-18T15:45:23.547 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.547 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-35", 2024-08-18T15:45:23.547 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:28.728040336Z", 2024-08-18T15:45:23.547 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.548 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.548 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.548 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.548 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.548 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.548 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_5pwJU04pzA5ySeDPbaf8cI9", 2024-08-18T15:45:23.548 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.548 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.548 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.548 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.548 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.548 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.548 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.548 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.549 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995748.744014_677.1", 2024-08-18T15:45:23.549 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.549 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-33:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.33", 2024-08-18T15:45:23.549 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:28.744014055Z", 2024-08-18T15:45:23.549 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.549 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.549 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_lGULOaXyl4hHx7qh454F-aa", 2024-08-18T15:45:23.549 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.549 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.549 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.549 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_lGULOaXyl4hHx7qh454F-aa", 2024-08-18T15:45:23.549 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 2 2024-08-18T15:45:23.549 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.550 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.550 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.550 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.550 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.550 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.550 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.550 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995748.746404_678.1", 2024-08-18T15:45:23.550 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.550 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-33:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.33", 2024-08-18T15:45:23.550 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:28.746404003Z", 2024-08-18T15:45:23.550 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.550 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.550 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_lGULOaXyl4hHx7qh454F-aa", 2024-08-18T15:45:23.551 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.551 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.551 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.551 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_lGULOaXyl4hHx7qh454F-aa", 2024-08-18T15:45:23.551 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 2 2024-08-18T15:45:23.551 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.551 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.551 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.551 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.551 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.551 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.551 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.551 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995748.751650_679.1", 2024-08-18T15:45:23.551 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.552 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-34:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.34", 2024-08-18T15:45:23.552 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:28.751650666Z", 2024-08-18T15:45:23.552 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.552 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.552 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_gziC3Zhjn59sc9f24iXYqez", 2024-08-18T15:45:23.552 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.552 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.552 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.552 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_gziC3Zhjn59sc9f24iXYqez", 2024-08-18T15:45:23.552 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 2 2024-08-18T15:45:23.552 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.552 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.552 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.553 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.553 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.553 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.553 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.553 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995748.754074_680.1", 2024-08-18T15:45:23.553 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.553 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-34:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.34", 2024-08-18T15:45:23.553 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:28.754074401Z", 2024-08-18T15:45:23.553 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.553 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.553 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_gziC3Zhjn59sc9f24iXYqez", 2024-08-18T15:45:23.553 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.553 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.553 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.554 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_gziC3Zhjn59sc9f24iXYqez", 2024-08-18T15:45:23.554 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 2 2024-08-18T15:45:23.554 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.554 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.554 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.554 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.554 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.554 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.554 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.554 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995748.802966_681.1", 2024-08-18T15:45:23.554 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.554 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-35:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.35", 2024-08-18T15:45:23.554 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:28.802966900Z", 2024-08-18T15:45:23.555 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.555 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.555 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_8PQDI19UyDbSfg9jXHxQK0Z", 2024-08-18T15:45:23.555 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.555 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.555 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.555 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_8PQDI19UyDbSfg9jXHxQK0Z", 2024-08-18T15:45:23.555 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 2 2024-08-18T15:45:23.555 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.555 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.555 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.555 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.555 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.555 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.556 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.556 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995748.805637_682.1", 2024-08-18T15:45:23.556 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.556 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-35:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.35", 2024-08-18T15:45:23.556 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:28.805637670Z", 2024-08-18T15:45:23.556 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.556 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.556 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_8PQDI19UyDbSfg9jXHxQK0Z", 2024-08-18T15:45:23.556 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.556 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.556 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.556 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_8PQDI19UyDbSfg9jXHxQK0Z", 2024-08-18T15:45:23.556 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 2 2024-08-18T15:45:23.557 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.557 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.557 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.557 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.557 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.557 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.557 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.557 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995748.862218_683.1", 2024-08-18T15:45:23.557 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.557 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-36:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.36", 2024-08-18T15:45:23.557 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:28.862218121Z", 2024-08-18T15:45:23.557 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.557 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.557 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.558 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.558 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.558 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.558 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_OMlfwDqiYy7MLBPapdgXZCw", 2024-08-18T15:45:23.558 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.558 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.558 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.558 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.558 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.558 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.558 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.558 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.558 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995748.864692_684.1", 2024-08-18T15:45:23.559 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.559 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-36:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.36", 2024-08-18T15:45:23.559 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:28.864692243Z", 2024-08-18T15:45:23.559 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.559 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.559 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.559 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.559 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.559 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.559 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_OMlfwDqiYy7MLBPapdgXZCw", 2024-08-18T15:45:23.559 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.559 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.559 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.559 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.560 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.560 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.560 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.560 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.560 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995748.865659_685.1", 2024-08-18T15:45:23.560 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.560 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-36", 2024-08-18T15:45:23.560 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:28.865659998Z", 2024-08-18T15:45:23.560 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.560 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.560 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.560 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.560 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.561 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.561 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_dui98arcr6Cif7iTccJdz_E", 2024-08-18T15:45:23.561 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.561 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.561 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.561 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.561 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.561 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.561 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.561 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.561 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995748.867847_686.1", 2024-08-18T15:45:23.561 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.561 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-36", 2024-08-18T15:45:23.561 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:28.867847854Z", 2024-08-18T15:45:23.562 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.562 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.562 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.562 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.562 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.562 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.562 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_dui98arcr6Cif7iTccJdz_E", 2024-08-18T15:45:23.562 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.562 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.562 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.562 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.562 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.562 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.563 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.563 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.563 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995748.877645_687.1", 2024-08-18T15:45:23.563 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.563 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-37:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.37", 2024-08-18T15:45:23.563 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:28.877645769Z", 2024-08-18T15:45:23.563 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.563 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.563 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.563 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.563 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.563 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.563 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_HFwNCilj83ivBqE5tB_PZE2", 2024-08-18T15:45:23.564 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.564 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.564 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.564 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.564 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.564 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.564 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.564 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.564 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995748.879910_688.1", 2024-08-18T15:45:23.564 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.564 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-37:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.37", 2024-08-18T15:45:23.564 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:28.879910198Z", 2024-08-18T15:45:23.564 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.564 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.564 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.565 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.565 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.565 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.565 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_HFwNCilj83ivBqE5tB_PZE2", 2024-08-18T15:45:23.565 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.565 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.565 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.565 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.565 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.565 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.565 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.565 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.565 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995748.880848_689.1", 2024-08-18T15:45:23.565 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.566 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-37", 2024-08-18T15:45:23.566 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:28.880848207Z", 2024-08-18T15:45:23.566 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.566 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.566 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.566 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.566 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.566 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.566 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_rUYGyCScmgZ3SwWHWdQlE0I", 2024-08-18T15:45:23.566 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.566 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.566 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.567 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.567 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.567 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.567 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.567 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.567 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995748.883023_690.1", 2024-08-18T15:45:23.567 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.567 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-37", 2024-08-18T15:45:23.567 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:28.883023907Z", 2024-08-18T15:45:23.567 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.567 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.567 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.567 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.568 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.568 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.568 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_rUYGyCScmgZ3SwWHWdQlE0I", 2024-08-18T15:45:23.568 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.568 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.568 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.568 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.568 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.568 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.568 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.568 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.568 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995748.899500_691.1", 2024-08-18T15:45:23.568 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.569 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-38:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.38", 2024-08-18T15:45:23.569 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:28.899500733Z", 2024-08-18T15:45:23.569 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.569 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.569 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.569 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.569 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.569 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.569 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_i-eCmhQ4NPsJf86Skaj64Wy", 2024-08-18T15:45:23.569 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.569 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.569 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.569 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.570 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.570 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.570 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.570 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.570 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995748.901993_692.1", 2024-08-18T15:45:23.570 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.570 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-38:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.38", 2024-08-18T15:45:23.570 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:28.901993132Z", 2024-08-18T15:45:23.570 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.570 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.570 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.570 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.570 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.570 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.571 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_i-eCmhQ4NPsJf86Skaj64Wy", 2024-08-18T15:45:23.571 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.571 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.571 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.571 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.571 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.571 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.571 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.571 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.571 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995748.903041_693.1", 2024-08-18T15:45:23.571 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.571 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-38", 2024-08-18T15:45:23.571 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:28.903041404Z", 2024-08-18T15:45:23.572 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.572 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.572 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.572 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.572 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.572 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.572 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_i5jx3wzjmX5pErHvMHm_pkB", 2024-08-18T15:45:23.572 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.572 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.572 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.572 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.572 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.572 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.573 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.573 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.573 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995748.905154_694.1", 2024-08-18T15:45:23.573 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.573 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-38", 2024-08-18T15:45:23.573 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:28.905154856Z", 2024-08-18T15:45:23.573 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.573 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.573 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.573 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.573 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.573 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.573 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_i5jx3wzjmX5pErHvMHm_pkB", 2024-08-18T15:45:23.573 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.574 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.574 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.574 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.574 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.574 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.574 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.574 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.574 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995748.915169_695.1", 2024-08-18T15:45:23.574 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.574 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-36:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.36", 2024-08-18T15:45:23.574 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:28.915169406Z", 2024-08-18T15:45:23.574 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.574 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.575 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_OMlfwDqiYy7MLBPapdgXZCw", 2024-08-18T15:45:23.575 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.575 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.575 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.575 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_OMlfwDqiYy7MLBPapdgXZCw", 2024-08-18T15:45:23.575 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 2 2024-08-18T15:45:23.575 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.575 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.575 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.575 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.575 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.575 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.575 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.576 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995748.917556_696.1", 2024-08-18T15:45:23.576 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.576 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-36:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.36", 2024-08-18T15:45:23.576 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:28.917556247Z", 2024-08-18T15:45:23.576 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.576 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.576 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_OMlfwDqiYy7MLBPapdgXZCw", 2024-08-18T15:45:23.576 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.576 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.576 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.576 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_OMlfwDqiYy7MLBPapdgXZCw", 2024-08-18T15:45:23.576 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 2 2024-08-18T15:45:23.576 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.576 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.577 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.577 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.577 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.577 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.577 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.577 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995749.970360_697.1", 2024-08-18T15:45:23.577 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.577 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-37:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.37", 2024-08-18T15:45:23.577 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:29.970360729Z", 2024-08-18T15:45:23.577 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.577 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.577 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_HFwNCilj83ivBqE5tB_PZE2", 2024-08-18T15:45:23.577 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.578 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.578 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.578 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_HFwNCilj83ivBqE5tB_PZE2", 2024-08-18T15:45:23.578 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 2 2024-08-18T15:45:23.578 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.578 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.578 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.578 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.578 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.578 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.578 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.578 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995749.974004_698.1", 2024-08-18T15:45:23.578 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.578 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-37:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.37", 2024-08-18T15:45:23.579 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:29.974004150Z", 2024-08-18T15:45:23.579 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.579 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.579 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_HFwNCilj83ivBqE5tB_PZE2", 2024-08-18T15:45:23.579 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.579 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.579 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.579 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_HFwNCilj83ivBqE5tB_PZE2", 2024-08-18T15:45:23.579 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 2 2024-08-18T15:45:23.579 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.579 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.579 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.579 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.580 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.580 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.580 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.580 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995751.026760_699.1", 2024-08-18T15:45:23.580 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.580 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-38:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.38", 2024-08-18T15:45:23.580 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:31.026760427Z", 2024-08-18T15:45:23.580 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.580 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.580 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_i-eCmhQ4NPsJf86Skaj64Wy", 2024-08-18T15:45:23.580 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.580 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.580 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.581 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_i-eCmhQ4NPsJf86Skaj64Wy", 2024-08-18T15:45:23.581 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 2 2024-08-18T15:45:23.581 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.581 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.581 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.581 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.581 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.581 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.581 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.581 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995751.031154_700.1", 2024-08-18T15:45:23.581 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.581 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-38:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.38", 2024-08-18T15:45:23.581 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:31.031154082Z", 2024-08-18T15:45:23.582 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.582 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.582 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_i-eCmhQ4NPsJf86Skaj64Wy", 2024-08-18T15:45:23.582 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.582 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.582 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.582 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_i-eCmhQ4NPsJf86Skaj64Wy", 2024-08-18T15:45:23.582 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 2 2024-08-18T15:45:23.582 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.582 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.582 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.582 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.582 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.582 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.583 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.583 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995751.043389_701.1", 2024-08-18T15:45:23.583 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.583 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:45:23.583 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:31.043389213Z", 2024-08-18T15:45:23.583 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.583 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.583 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.583 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.583 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.583 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.583 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_xmYq1VNiyUA8ztTA_d5rUm3", 2024-08-18T15:45:23.583 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.584 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.584 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.584 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.584 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.584 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.584 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.584 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.584 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995751.046426_702.1", 2024-08-18T15:45:23.584 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.584 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:45:23.584 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:31.046426883Z", 2024-08-18T15:45:23.584 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.584 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.584 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.585 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.585 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.585 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.585 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_xmYq1VNiyUA8ztTA_d5rUm3", 2024-08-18T15:45:23.585 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.585 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.585 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.585 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.585 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.585 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.585 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.585 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.585 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995751.047596_703.1", 2024-08-18T15:45:23.585 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.586 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-39", 2024-08-18T15:45:23.586 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:31.047596040Z", 2024-08-18T15:45:23.586 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.586 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.586 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.586 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.586 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.586 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.586 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "__iz5Ku-i6OWsHDvE8pjpoke", 2024-08-18T15:45:23.586 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.586 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.586 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.587 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.587 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.587 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.587 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.587 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.587 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995751.050305_704.1", 2024-08-18T15:45:23.587 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.587 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-39", 2024-08-18T15:45:23.587 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:31.050305755Z", 2024-08-18T15:45:23.587 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.587 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.587 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.587 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.588 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.588 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.588 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "__iz5Ku-i6OWsHDvE8pjpoke", 2024-08-18T15:45:23.588 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.588 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.588 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.588 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.588 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.588 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.588 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.588 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.588 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995751.063176_705.1", 2024-08-18T15:45:23.588 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.588 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:45:23.589 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:31.063176910Z", 2024-08-18T15:45:23.589 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.589 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.589 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.589 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.589 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.589 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.589 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_ejQ3MioLVUCb2QnucqqCUMJ", 2024-08-18T15:45:23.589 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.589 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.589 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.589 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.589 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.590 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.590 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.590 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.590 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995751.066133_706.1", 2024-08-18T15:45:23.590 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.590 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:45:23.590 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:31.066133710Z", 2024-08-18T15:45:23.590 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.590 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.590 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.590 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.590 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.591 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.591 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_ejQ3MioLVUCb2QnucqqCUMJ", 2024-08-18T15:45:23.591 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.591 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.591 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.591 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.591 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.591 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.591 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.591 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.591 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995751.067195_707.1", 2024-08-18T15:45:23.591 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.591 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-40", 2024-08-18T15:45:23.592 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:31.067195014Z", 2024-08-18T15:45:23.592 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.592 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.592 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.592 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.592 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.592 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.592 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_llhZcJSjwRerrd-Oaou9Sjv", 2024-08-18T15:45:23.592 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.592 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.592 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.592 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.592 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.592 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.593 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.593 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.593 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995751.069927_708.1", 2024-08-18T15:45:23.593 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.593 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-40", 2024-08-18T15:45:23.593 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:31.069927389Z", 2024-08-18T15:45:23.593 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.593 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.593 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.593 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.593 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.593 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.593 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_llhZcJSjwRerrd-Oaou9Sjv", 2024-08-18T15:45:23.594 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.594 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.594 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.594 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.594 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.594 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.594 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.594 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.594 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995751.090745_709.1", 2024-08-18T15:45:23.594 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.594 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:45:23.594 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:31.090745755Z", 2024-08-18T15:45:23.594 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.595 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.595 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.595 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.595 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.595 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.595 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_8c5popeokpV--2SpVjBST3E", 2024-08-18T15:45:23.595 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.595 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.595 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.595 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.595 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.595 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.595 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.595 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.596 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995751.093802_710.1", 2024-08-18T15:45:23.596 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.596 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:45:23.596 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:31.093802820Z", 2024-08-18T15:45:23.596 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.596 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.596 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.596 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.596 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.596 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.596 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_8c5popeokpV--2SpVjBST3E", 2024-08-18T15:45:23.596 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.596 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.597 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.597 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.597 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.597 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.597 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.597 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.597 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995751.095112_711.1", 2024-08-18T15:45:23.597 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.597 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-41", 2024-08-18T15:45:23.597 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:31.095112579Z", 2024-08-18T15:45:23.597 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.597 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.597 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.598 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.598 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.598 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.598 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_YWJNdlw8sqMdjjlIjLHAL7T", 2024-08-18T15:45:23.598 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.598 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.598 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.598 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.598 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.598 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.598 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.598 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.598 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995751.097661_712.1", 2024-08-18T15:45:23.599 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.599 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-41", 2024-08-18T15:45:23.599 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:31.097661378Z", 2024-08-18T15:45:23.599 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.599 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.599 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.599 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.599 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.599 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.599 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_YWJNdlw8sqMdjjlIjLHAL7T", 2024-08-18T15:45:23.599 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.600 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.600 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.600 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.600 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.600 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.600 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.600 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.600 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995771.988164_729.1", 2024-08-18T15:45:23.600 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.600 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-42:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.42", 2024-08-18T15:45:23.600 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:51.988164854Z", 2024-08-18T15:45:23.600 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.600 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.601 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.601 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.601 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.601 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.601 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_oGiVyEzJBYGJYJotLHjzZsK", 2024-08-18T15:45:23.601 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.601 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.601 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.601 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.601 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.601 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.601 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.601 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.602 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995771.990622_730.1", 2024-08-18T15:45:23.602 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.602 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-42:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.42", 2024-08-18T15:45:23.602 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:51.990622883Z", 2024-08-18T15:45:23.602 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.602 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.602 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.602 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.602 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.602 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.602 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_oGiVyEzJBYGJYJotLHjzZsK", 2024-08-18T15:45:23.602 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.602 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.603 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.603 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.603 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.603 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.603 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.603 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.603 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995771.991578_731.1", 2024-08-18T15:45:23.603 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.603 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-42", 2024-08-18T15:45:23.603 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:51.991578988Z", 2024-08-18T15:45:23.603 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.603 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.603 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.603 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.604 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.604 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.604 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_fG-TEUh0CxWR43RmwQg7ICF", 2024-08-18T15:45:23.604 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.604 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.604 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.604 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.604 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.604 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.604 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.604 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.604 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995771.993985_732.1", 2024-08-18T15:45:23.604 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.604 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-42", 2024-08-18T15:45:23.605 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:51.993985953Z", 2024-08-18T15:45:23.605 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.605 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.605 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.605 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.605 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.605 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.605 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_fG-TEUh0CxWR43RmwQg7ICF", 2024-08-18T15:45:23.605 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.605 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.605 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.605 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.605 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.606 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.606 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.606 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.606 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995772.004584_733.1", 2024-08-18T15:45:23.606 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.606 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-43:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.43", 2024-08-18T15:45:23.606 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:52.004584088Z", 2024-08-18T15:45:23.606 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.606 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.606 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.606 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.606 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.606 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.607 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_c-hGhbKE-b1LRVHEPUzPjib", 2024-08-18T15:45:23.607 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.607 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.607 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.607 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.607 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.607 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.607 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.607 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.607 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995772.008960_734.1", 2024-08-18T15:45:23.607 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.607 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-43:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.43", 2024-08-18T15:45:23.607 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:52.008960500Z", 2024-08-18T15:45:23.608 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.608 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.608 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.608 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.608 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.608 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.608 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_c-hGhbKE-b1LRVHEPUzPjib", 2024-08-18T15:45:23.608 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.608 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.608 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.608 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.608 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.608 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.608 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.609 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.609 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995772.010074_735.1", 2024-08-18T15:45:23.609 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.609 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-43", 2024-08-18T15:45:23.609 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:52.010074886Z", 2024-08-18T15:45:23.609 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.609 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.609 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.609 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.609 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.609 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.609 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_L-a1id3J-Bf_HtA2U8yB3t5", 2024-08-18T15:45:23.609 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.609 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.610 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.610 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.610 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.610 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.610 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.610 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.610 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995772.012225_736.1", 2024-08-18T15:45:23.610 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.610 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-43", 2024-08-18T15:45:23.610 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:52.012225687Z", 2024-08-18T15:45:23.610 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.610 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.611 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.611 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.611 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.611 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.611 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_L-a1id3J-Bf_HtA2U8yB3t5", 2024-08-18T15:45:23.611 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.611 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.611 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.611 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.611 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.611 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.611 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.611 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.611 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995772.031468_737.1", 2024-08-18T15:45:23.612 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.612 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-44:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.44", 2024-08-18T15:45:23.612 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:52.031468864Z", 2024-08-18T15:45:23.612 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.612 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.612 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.612 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.612 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.612 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.612 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_On0FEwDZQtRx6QIbTXxbmk6", 2024-08-18T15:45:23.612 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.612 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.612 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.612 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.613 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.613 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.613 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.613 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.613 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995772.034017_738.1", 2024-08-18T15:45:23.613 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.613 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-44:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.44", 2024-08-18T15:45:23.613 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:52.034017957Z", 2024-08-18T15:45:23.613 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.613 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.613 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.613 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.614 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.614 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.614 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_On0FEwDZQtRx6QIbTXxbmk6", 2024-08-18T15:45:23.614 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.614 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.614 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.614 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.614 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.614 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.614 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.614 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.614 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995772.035064_739.1", 2024-08-18T15:45:23.614 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.614 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-44", 2024-08-18T15:45:23.615 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:52.035064992Z", 2024-08-18T15:45:23.615 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.615 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.615 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.615 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.615 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.615 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.615 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_zO2kHLSbZbuAztlb-hy3PyC", 2024-08-18T15:45:23.615 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.615 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.615 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.615 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:23.615 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.616 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.616 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.616 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:23.616 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995772.038282_740.1", 2024-08-18T15:45:23.616 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:23.616 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-44", 2024-08-18T15:45:23.616 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:52.038282163Z", 2024-08-18T15:45:23.616 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:23.616 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:23.616 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:23.616 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:23.616 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.616 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:23.616 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_zO2kHLSbZbuAztlb-hy3PyC", 2024-08-18T15:45:23.617 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:23.617 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:23.617 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:23.617 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:23.617 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.617 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.617 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:23.617 INFO:teuthology.orchestra.run.smithi193.stdout:] 2024-08-18T15:45:23.620 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'b0d74758-1871-4454-bc0f-e3e4b6c30105', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:45:23.620 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'b0d74758-1871-4454-bc0f-e3e4b6c30105', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:45:23.620 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period b0d74758-1871-4454-bc0f-e3e4b6c30105 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:45:23.667 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:45:23.668 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:45:23.698 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T15:45:23.698 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.698 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994364.503712_22.1", 2024-08-18T15:45:23.698 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "user", 2024-08-18T15:45:23.698 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "rgw-multisite-test-user", 2024-08-18T15:45:23.699 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:19:24.503712511Z", 2024-08-18T15:45:23.699 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.699 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.699 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.699 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.699 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.699 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.699 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_MXHnfxZIHwQEXOVDZVD1osf", 2024-08-18T15:45:23.699 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.699 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.699 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.699 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.699 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.700 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.700 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.700 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.700 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994364.506194_23.1", 2024-08-18T15:45:23.700 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "user", 2024-08-18T15:45:23.700 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "rgw-multisite-test-user", 2024-08-18T15:45:23.700 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:19:24.506194765Z", 2024-08-18T15:45:23.700 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.700 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.700 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.700 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.700 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.700 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.701 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_MXHnfxZIHwQEXOVDZVD1osf", 2024-08-18T15:45:23.701 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.701 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.701 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.701 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.701 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.701 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.701 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.701 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.701 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994485.499738_49.1", 2024-08-18T15:45:23.701 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.701 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-1:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.1", 2024-08-18T15:45:23.701 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:25.499738805Z", 2024-08-18T15:45:23.701 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.702 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.702 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.702 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.702 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.702 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.702 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_RWKCfRB1w1J8Q2l4ptlOal-", 2024-08-18T15:45:23.702 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.702 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.702 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.702 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.702 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.702 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.702 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.703 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.703 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994485.513055_50.1", 2024-08-18T15:45:23.703 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.703 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-1:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.1", 2024-08-18T15:45:23.703 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:25.513055821Z", 2024-08-18T15:45:23.703 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.703 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.703 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.703 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.703 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.703 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.703 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_RWKCfRB1w1J8Q2l4ptlOal-", 2024-08-18T15:45:23.704 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.704 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.704 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.704 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.704 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.704 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.704 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.704 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.704 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994485.514799_51.1", 2024-08-18T15:45:23.704 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.704 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-1", 2024-08-18T15:45:23.704 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:25.514799747Z", 2024-08-18T15:45:23.704 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.705 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.705 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.705 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.705 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.705 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.705 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_zN4WlRRiwX-IH3XcKWBWZ7B", 2024-08-18T15:45:23.705 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.705 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.705 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.705 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.705 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.705 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.705 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.706 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.706 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994485.517793_52.1", 2024-08-18T15:45:23.706 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.706 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-1", 2024-08-18T15:45:23.706 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:25.517793650Z", 2024-08-18T15:45:23.706 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.706 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.706 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.706 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.706 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.706 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.706 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_zN4WlRRiwX-IH3XcKWBWZ7B", 2024-08-18T15:45:23.706 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.707 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.707 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.707 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.707 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.707 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.707 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.707 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.707 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994485.530711_53.1", 2024-08-18T15:45:23.707 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.707 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-2:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.2", 2024-08-18T15:45:23.707 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:25.530711866Z", 2024-08-18T15:45:23.707 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.707 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.708 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.708 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.708 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.708 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.708 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_kZfzVgY3qrQmzEpOy-gNh0z", 2024-08-18T15:45:23.708 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.708 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.708 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.708 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.708 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.708 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.708 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.708 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.708 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994485.533125_54.1", 2024-08-18T15:45:23.709 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.709 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-2:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.2", 2024-08-18T15:45:23.709 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:25.533125204Z", 2024-08-18T15:45:23.709 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.709 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.709 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.709 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.709 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.709 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.709 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_kZfzVgY3qrQmzEpOy-gNh0z", 2024-08-18T15:45:23.709 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.709 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.709 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.709 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.710 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.710 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.710 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.710 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.710 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994485.534087_55.1", 2024-08-18T15:45:23.710 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.710 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-2", 2024-08-18T15:45:23.710 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:25.534087256Z", 2024-08-18T15:45:23.710 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.710 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.710 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.710 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.710 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.711 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.711 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_4VibGFZMXb9COC_5_LGnUYq", 2024-08-18T15:45:23.711 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.711 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.711 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.711 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.711 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.711 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.711 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.711 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.711 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994485.536327_56.1", 2024-08-18T15:45:23.711 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.711 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-2", 2024-08-18T15:45:23.711 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:25.536327238Z", 2024-08-18T15:45:23.712 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.712 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.712 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.712 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.712 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.712 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.712 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_4VibGFZMXb9COC_5_LGnUYq", 2024-08-18T15:45:23.712 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.712 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.712 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.712 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.712 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.712 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.713 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.713 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.713 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994488.093196_57.1", 2024-08-18T15:45:23.713 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.713 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-3:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.3", 2024-08-18T15:45:23.713 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:28.093196831Z", 2024-08-18T15:45:23.713 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.713 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.713 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.713 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.713 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.713 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.713 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_7WKtlNtV_hilpzaA5UW3kBl", 2024-08-18T15:45:23.713 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.714 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.714 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.714 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.714 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.714 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.714 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.714 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.714 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994488.096301_58.1", 2024-08-18T15:45:23.714 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.714 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-3:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.3", 2024-08-18T15:45:23.714 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:28.096301217Z", 2024-08-18T15:45:23.714 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.714 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.715 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.715 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.715 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.715 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.715 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_7WKtlNtV_hilpzaA5UW3kBl", 2024-08-18T15:45:23.715 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.715 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.715 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.715 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.715 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.715 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.715 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.715 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.715 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994488.097328_59.1", 2024-08-18T15:45:23.716 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.716 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-3", 2024-08-18T15:45:23.716 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:28.097328364Z", 2024-08-18T15:45:23.716 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.716 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.716 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.716 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.716 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.716 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.716 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_T0NcIXjN0Rm7yrm1EC_xxAs", 2024-08-18T15:45:23.716 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.716 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.716 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.716 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.717 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.717 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.717 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.717 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.717 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994488.100145_60.1", 2024-08-18T15:45:23.717 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.717 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-3", 2024-08-18T15:45:23.717 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:28.100145189Z", 2024-08-18T15:45:23.717 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.717 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.717 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.717 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.717 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.717 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.718 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_T0NcIXjN0Rm7yrm1EC_xxAs", 2024-08-18T15:45:23.718 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.718 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.718 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.718 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.718 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.718 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.718 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.718 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.718 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994493.565103_61.1", 2024-08-18T15:45:23.718 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.718 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-4:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.4", 2024-08-18T15:45:23.718 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:33.565103683Z", 2024-08-18T15:45:23.718 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.719 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.719 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.719 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.719 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.719 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.719 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_NVsrONxJz03-nbFIVh9-7HH", 2024-08-18T15:45:23.719 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.719 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.719 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.719 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.719 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.719 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.719 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.719 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.720 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994493.567515_62.1", 2024-08-18T15:45:23.720 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.720 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-4:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.4", 2024-08-18T15:45:23.720 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:33.567515090Z", 2024-08-18T15:45:23.720 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.720 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.720 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.720 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.720 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.720 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.720 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_NVsrONxJz03-nbFIVh9-7HH", 2024-08-18T15:45:23.720 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.720 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.720 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.721 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.721 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.721 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.721 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.721 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.721 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994493.568350_63.1", 2024-08-18T15:45:23.721 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.721 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-4", 2024-08-18T15:45:23.721 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:33.568350483Z", 2024-08-18T15:45:23.721 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.721 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.721 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.721 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.722 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.722 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.722 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_5jHEePfx-6SZjjYqolXPBLX", 2024-08-18T15:45:23.722 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.722 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.722 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.722 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.722 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.722 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.722 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.722 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.722 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994493.570568_64.1", 2024-08-18T15:45:23.722 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.723 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-4", 2024-08-18T15:45:23.723 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:33.570568110Z", 2024-08-18T15:45:23.723 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.723 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.723 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.723 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.723 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.723 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.723 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_5jHEePfx-6SZjjYqolXPBLX", 2024-08-18T15:45:23.723 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.723 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.723 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.723 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.723 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.724 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.724 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.724 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.724 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994493.583318_65.1", 2024-08-18T15:45:23.724 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.724 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-5:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.5", 2024-08-18T15:45:23.724 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:33.583318078Z", 2024-08-18T15:45:23.724 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.724 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.724 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.724 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.724 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.724 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.724 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_gGmjq-oCwB5WheV6EKKwPBK", 2024-08-18T15:45:23.724 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.725 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.725 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.725 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.725 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.725 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.725 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.725 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.725 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994493.586220_66.1", 2024-08-18T15:45:23.725 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.725 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-5:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.5", 2024-08-18T15:45:23.725 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:33.586220697Z", 2024-08-18T15:45:23.725 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.725 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.726 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.726 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.726 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.726 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.726 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_gGmjq-oCwB5WheV6EKKwPBK", 2024-08-18T15:45:23.726 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.726 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.726 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.726 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.726 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.726 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.726 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.726 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.726 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994493.587090_67.1", 2024-08-18T15:45:23.727 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.727 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-5", 2024-08-18T15:45:23.727 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:33.587090828Z", 2024-08-18T15:45:23.727 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.727 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.727 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.727 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.727 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.727 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.727 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_d2Y8p1KjRuhShSVV0OrvphE", 2024-08-18T15:45:23.727 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.727 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.727 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.727 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.728 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.728 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.728 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.728 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.728 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994493.589499_68.1", 2024-08-18T15:45:23.728 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.728 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-5", 2024-08-18T15:45:23.728 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:33.589499147Z", 2024-08-18T15:45:23.728 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.728 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.728 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.728 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.728 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.728 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.729 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_d2Y8p1KjRuhShSVV0OrvphE", 2024-08-18T15:45:23.729 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.729 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.729 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.729 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.729 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.729 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.729 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.729 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.729 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994493.608922_69.1", 2024-08-18T15:45:23.729 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.729 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-6:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.6", 2024-08-18T15:45:23.729 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:33.608922657Z", 2024-08-18T15:45:23.729 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.730 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.730 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.730 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.730 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.730 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.730 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_ZjXDBXiyWYi6JTPxve1UMrv", 2024-08-18T15:45:23.730 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.730 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.730 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.730 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.730 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.730 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.730 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.730 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.731 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994493.613023_70.1", 2024-08-18T15:45:23.731 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.731 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-6:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.6", 2024-08-18T15:45:23.731 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:33.613023410Z", 2024-08-18T15:45:23.731 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.731 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.731 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.731 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.731 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.731 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.731 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_ZjXDBXiyWYi6JTPxve1UMrv", 2024-08-18T15:45:23.731 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.731 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.731 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.732 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.732 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.732 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.732 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.732 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.732 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994493.614132_71.1", 2024-08-18T15:45:23.732 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.732 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-6", 2024-08-18T15:45:23.732 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:33.614132419Z", 2024-08-18T15:45:23.732 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.732 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.732 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.732 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.732 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.733 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.733 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_ZBRWYIGyUuGJdOA0PdgsHNt", 2024-08-18T15:45:23.733 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.733 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.733 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.733 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.733 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.733 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.733 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.733 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.733 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994493.616159_72.1", 2024-08-18T15:45:23.733 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.733 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-6", 2024-08-18T15:45:23.733 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:33.616159354Z", 2024-08-18T15:45:23.734 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.734 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.734 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.734 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.734 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.734 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.734 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_ZBRWYIGyUuGJdOA0PdgsHNt", 2024-08-18T15:45:23.734 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.734 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.734 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.734 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.734 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.734 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.735 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.735 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.735 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994494.468832_73.1", 2024-08-18T15:45:23.735 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.735 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-7:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.7", 2024-08-18T15:45:23.735 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:34.468832363Z", 2024-08-18T15:45:23.735 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.735 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.735 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.735 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.735 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.735 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.735 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_qRFDABc-dwq6RziTweT5YtT", 2024-08-18T15:45:23.736 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.736 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.736 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.736 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.736 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.736 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.736 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.736 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.736 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994494.471921_74.1", 2024-08-18T15:45:23.736 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.736 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-7:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.7", 2024-08-18T15:45:23.736 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:34.471921559Z", 2024-08-18T15:45:23.736 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.736 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.737 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.737 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.737 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.737 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.737 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_qRFDABc-dwq6RziTweT5YtT", 2024-08-18T15:45:23.737 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.737 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.737 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.737 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.737 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.737 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.737 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.737 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.737 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994494.472699_75.1", 2024-08-18T15:45:23.738 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.738 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-7", 2024-08-18T15:45:23.738 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:34.472699971Z", 2024-08-18T15:45:23.738 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.738 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.738 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.738 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.738 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.738 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.738 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_tLmChDVmX3sImKf0993KFB1", 2024-08-18T15:45:23.738 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.738 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.738 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.738 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.739 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.739 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.739 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.739 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.739 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994494.475169_76.1", 2024-08-18T15:45:23.739 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.739 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-7", 2024-08-18T15:45:23.739 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:34.475169889Z", 2024-08-18T15:45:23.739 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.739 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.739 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.739 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.739 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.739 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.740 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_tLmChDVmX3sImKf0993KFB1", 2024-08-18T15:45:23.740 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.740 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.740 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.740 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.740 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.740 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.740 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.740 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.740 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994494.483917_77.1", 2024-08-18T15:45:23.740 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.740 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-8:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.8", 2024-08-18T15:45:23.740 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:34.483917175Z", 2024-08-18T15:45:23.740 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.741 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.741 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.741 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.741 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.741 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.741 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_Zi2-lA3qHEM3t_BsL4AeXRU", 2024-08-18T15:45:23.741 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.741 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.741 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.741 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.741 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.741 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.741 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.741 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.741 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994494.487103_78.1", 2024-08-18T15:45:23.742 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.742 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-8:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.8", 2024-08-18T15:45:23.742 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:34.487103946Z", 2024-08-18T15:45:23.742 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.742 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.742 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.742 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.742 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.742 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.742 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_Zi2-lA3qHEM3t_BsL4AeXRU", 2024-08-18T15:45:23.742 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.742 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.742 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.743 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.743 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.743 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.743 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.743 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.743 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994494.487887_79.1", 2024-08-18T15:45:23.743 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.743 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-8", 2024-08-18T15:45:23.743 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:34.487887903Z", 2024-08-18T15:45:23.743 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.743 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.743 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.743 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.744 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.744 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.744 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_7oG9nLitKlO5mrAHdrnWcCe", 2024-08-18T15:45:23.744 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.744 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.744 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.744 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.744 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.744 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.744 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.744 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.744 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994494.489837_80.1", 2024-08-18T15:45:23.744 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.744 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-8", 2024-08-18T15:45:23.745 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:34.489837888Z", 2024-08-18T15:45:23.745 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.745 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.745 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.745 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.745 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.745 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.745 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_7oG9nLitKlO5mrAHdrnWcCe", 2024-08-18T15:45:23.745 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.745 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.745 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.745 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.745 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.745 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.746 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.746 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.746 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994494.511942_81.1", 2024-08-18T15:45:23.746 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.746 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-9:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.9", 2024-08-18T15:45:23.746 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:34.511942269Z", 2024-08-18T15:45:23.746 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.746 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.746 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.746 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.746 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.746 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.746 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_M-lw4sdJUfr_hU7bn6pQCkY", 2024-08-18T15:45:23.747 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.747 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.747 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.747 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.747 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.747 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.747 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.747 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.747 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994494.520062_82.1", 2024-08-18T15:45:23.747 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.747 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-9:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.9", 2024-08-18T15:45:23.747 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:34.520062161Z", 2024-08-18T15:45:23.747 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.747 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.748 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.748 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.748 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.748 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.748 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_M-lw4sdJUfr_hU7bn6pQCkY", 2024-08-18T15:45:23.748 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.748 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.748 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.748 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.748 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.748 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.748 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.748 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.748 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994494.520843_83.1", 2024-08-18T15:45:23.749 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.749 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-9", 2024-08-18T15:45:23.749 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:34.520843453Z", 2024-08-18T15:45:23.749 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.749 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.749 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.749 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.749 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.749 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.749 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_WwzPQ6LnfxKTsseathJHhf2", 2024-08-18T15:45:23.749 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.749 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.749 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.749 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.749 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.750 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.750 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.750 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.750 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994494.523006_84.1", 2024-08-18T15:45:23.750 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.750 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-9", 2024-08-18T15:45:23.750 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:34.523006825Z", 2024-08-18T15:45:23.750 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.750 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.750 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.750 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.750 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.750 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.751 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_WwzPQ6LnfxKTsseathJHhf2", 2024-08-18T15:45:23.751 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.751 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.751 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.751 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.751 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.751 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.751 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.751 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.751 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994494.861305_85.1", 2024-08-18T15:45:23.751 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.751 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-7:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.7", 2024-08-18T15:45:23.751 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:34.861305566Z", 2024-08-18T15:45:23.751 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.752 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.752 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_qRFDABc-dwq6RziTweT5YtT", 2024-08-18T15:45:23.752 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.752 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.752 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.752 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_qRFDABc-dwq6RziTweT5YtT", 2024-08-18T15:45:23.752 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.752 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.752 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.752 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.752 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.752 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.752 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.752 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.753 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994494.863517_86.1", 2024-08-18T15:45:23.753 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.753 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-7:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.7", 2024-08-18T15:45:23.753 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:34.863517491Z", 2024-08-18T15:45:23.753 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.753 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.753 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_qRFDABc-dwq6RziTweT5YtT", 2024-08-18T15:45:23.753 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.753 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.753 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.753 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_qRFDABc-dwq6RziTweT5YtT", 2024-08-18T15:45:23.753 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.753 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.753 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.754 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.754 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.754 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.754 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.754 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.754 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994494.870867_87.1", 2024-08-18T15:45:23.754 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.754 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-7", 2024-08-18T15:45:23.754 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:34.870867850Z", 2024-08-18T15:45:23.754 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.754 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.754 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_tLmChDVmX3sImKf0993KFB1", 2024-08-18T15:45:23.754 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.755 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.755 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.755 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_tLmChDVmX3sImKf0993KFB1", 2024-08-18T15:45:23.755 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.755 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.755 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.755 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "remove" 2024-08-18T15:45:23.755 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.755 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.755 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.755 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.755 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994494.872883_88.1", 2024-08-18T15:45:23.755 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.755 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-7", 2024-08-18T15:45:23.755 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:34.872883708Z", 2024-08-18T15:45:23.756 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.756 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.756 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_tLmChDVmX3sImKf0993KFB1", 2024-08-18T15:45:23.756 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.756 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.756 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.756 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_tLmChDVmX3sImKf0993KFB1", 2024-08-18T15:45:23.756 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.756 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.756 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.756 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.756 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.756 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.756 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.757 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.757 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994494.890261_89.1", 2024-08-18T15:45:23.757 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.757 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-8:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.8", 2024-08-18T15:45:23.757 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:34.890261024Z", 2024-08-18T15:45:23.757 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.757 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.757 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_Zi2-lA3qHEM3t_BsL4AeXRU", 2024-08-18T15:45:23.757 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.757 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.757 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.757 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_Zi2-lA3qHEM3t_BsL4AeXRU", 2024-08-18T15:45:23.757 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.758 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.758 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.758 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.758 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.758 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.758 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.758 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.758 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994494.892455_90.1", 2024-08-18T15:45:23.758 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.758 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-8:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.8", 2024-08-18T15:45:23.758 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:34.892455177Z", 2024-08-18T15:45:23.758 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.758 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.758 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_Zi2-lA3qHEM3t_BsL4AeXRU", 2024-08-18T15:45:23.759 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.759 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.759 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.759 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_Zi2-lA3qHEM3t_BsL4AeXRU", 2024-08-18T15:45:23.759 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.759 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.759 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.759 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.759 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.759 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.759 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.759 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.759 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994494.899743_91.1", 2024-08-18T15:45:23.759 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.760 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-8", 2024-08-18T15:45:23.760 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:34.899743512Z", 2024-08-18T15:45:23.760 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.760 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.760 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_7oG9nLitKlO5mrAHdrnWcCe", 2024-08-18T15:45:23.760 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.760 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.760 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.760 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_7oG9nLitKlO5mrAHdrnWcCe", 2024-08-18T15:45:23.760 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.760 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.760 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.760 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "remove" 2024-08-18T15:45:23.761 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.761 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.761 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.761 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.761 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994494.901752_92.1", 2024-08-18T15:45:23.761 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.761 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-8", 2024-08-18T15:45:23.761 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:34.901752069Z", 2024-08-18T15:45:23.761 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.761 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.761 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_7oG9nLitKlO5mrAHdrnWcCe", 2024-08-18T15:45:23.761 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.761 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.761 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.762 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_7oG9nLitKlO5mrAHdrnWcCe", 2024-08-18T15:45:23.762 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.762 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.762 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.762 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.762 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.762 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.762 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.762 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.762 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994494.940403_93.1", 2024-08-18T15:45:23.762 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.762 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-9:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.9", 2024-08-18T15:45:23.762 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:34.940403695Z", 2024-08-18T15:45:23.763 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.763 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.763 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_M-lw4sdJUfr_hU7bn6pQCkY", 2024-08-18T15:45:23.763 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.763 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.763 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.763 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_M-lw4sdJUfr_hU7bn6pQCkY", 2024-08-18T15:45:23.763 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.763 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.763 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.764 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.764 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.764 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.764 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.764 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.764 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994494.942560_94.1", 2024-08-18T15:45:23.764 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.764 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-9:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.9", 2024-08-18T15:45:23.764 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:34.942560102Z", 2024-08-18T15:45:23.764 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.764 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.764 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_M-lw4sdJUfr_hU7bn6pQCkY", 2024-08-18T15:45:23.764 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.765 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.765 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.765 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_M-lw4sdJUfr_hU7bn6pQCkY", 2024-08-18T15:45:23.765 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.765 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.765 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.765 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.765 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.765 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.765 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.765 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.765 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994494.949797_95.1", 2024-08-18T15:45:23.765 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.766 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-9", 2024-08-18T15:45:23.766 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:34.949797571Z", 2024-08-18T15:45:23.766 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.766 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.766 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_WwzPQ6LnfxKTsseathJHhf2", 2024-08-18T15:45:23.766 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.766 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.766 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.766 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_WwzPQ6LnfxKTsseathJHhf2", 2024-08-18T15:45:23.766 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.766 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.766 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.766 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "remove" 2024-08-18T15:45:23.767 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.767 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.767 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.767 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.767 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994494.952005_96.1", 2024-08-18T15:45:23.767 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.767 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-9", 2024-08-18T15:45:23.767 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:34.952005923Z", 2024-08-18T15:45:23.767 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.767 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.767 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_WwzPQ6LnfxKTsseathJHhf2", 2024-08-18T15:45:23.767 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.767 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.768 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.768 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_WwzPQ6LnfxKTsseathJHhf2", 2024-08-18T15:45:23.768 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.768 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.768 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.768 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.768 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.768 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.768 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.768 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.768 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994505.517635_51.1", 2024-08-18T15:45:23.768 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.768 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:45:23.769 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:45.517635733Z", 2024-08-18T15:45:23.769 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.769 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.769 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.769 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.769 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.769 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.769 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "__WpsUsCter7eN_yzr7oojfP", 2024-08-18T15:45:23.769 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.769 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.769 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.769 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.769 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.769 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.770 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.770 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.770 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994505.521539_52.1", 2024-08-18T15:45:23.770 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.770 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:45:23.770 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:45.521539522Z", 2024-08-18T15:45:23.770 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.770 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.770 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.770 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.770 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.770 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.770 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "__WpsUsCter7eN_yzr7oojfP", 2024-08-18T15:45:23.770 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.771 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.771 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.771 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.771 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.771 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.771 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.771 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.771 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994505.522484_53.1", 2024-08-18T15:45:23.771 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.771 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-10", 2024-08-18T15:45:23.771 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:45.522484813Z", 2024-08-18T15:45:23.771 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.771 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.771 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.772 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.772 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.772 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.772 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_u-jdXqkvtd3BndfjmYUBTit", 2024-08-18T15:45:23.772 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.772 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.772 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.772 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.772 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.772 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.772 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.772 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.772 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994505.524516_54.1", 2024-08-18T15:45:23.772 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.773 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-10", 2024-08-18T15:45:23.773 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:45.524516821Z", 2024-08-18T15:45:23.773 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.773 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.773 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.773 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.773 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.773 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.773 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_u-jdXqkvtd3BndfjmYUBTit", 2024-08-18T15:45:23.773 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.773 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.773 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.773 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.773 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.774 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.774 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.774 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.774 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994505.535896_55.1", 2024-08-18T15:45:23.774 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.774 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:45:23.774 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:45.535896117Z", 2024-08-18T15:45:23.774 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.774 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.774 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.774 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.774 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.774 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.774 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_XktQF5h6nGNC5v3w-DrHnvQ", 2024-08-18T15:45:23.775 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.775 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.775 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.775 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.775 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.775 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.775 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.775 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.775 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994505.539494_56.1", 2024-08-18T15:45:23.775 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.778 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:45:23.779 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:45.539494049Z", 2024-08-18T15:45:23.779 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.779 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.779 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.779 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.779 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.779 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.779 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_XktQF5h6nGNC5v3w-DrHnvQ", 2024-08-18T15:45:23.779 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.779 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.779 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.779 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.779 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.780 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.780 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.780 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.780 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994505.540711_57.1", 2024-08-18T15:45:23.780 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.780 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-11", 2024-08-18T15:45:23.780 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:45.540711150Z", 2024-08-18T15:45:23.780 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.780 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.780 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.780 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.780 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.781 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.781 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_y9YtDg7ol2vKL0ge54w1-u0", 2024-08-18T15:45:23.781 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.781 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.781 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.781 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.781 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.781 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.781 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.781 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.781 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994505.542807_58.1", 2024-08-18T15:45:23.781 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.781 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-11", 2024-08-18T15:45:23.782 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:45.542807482Z", 2024-08-18T15:45:23.782 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.782 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.782 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.782 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.782 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.782 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.782 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_y9YtDg7ol2vKL0ge54w1-u0", 2024-08-18T15:45:23.782 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.782 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.782 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.782 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.782 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.783 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.783 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.783 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.783 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994505.560557_59.1", 2024-08-18T15:45:23.783 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.783 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:45:23.783 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:45.560557573Z", 2024-08-18T15:45:23.783 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.783 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.783 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.783 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.783 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.783 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.784 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_EyCJst2_YMPiGsXCrb7chb3", 2024-08-18T15:45:23.784 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.784 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.784 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.784 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.784 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.784 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.784 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.784 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.784 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994505.563007_60.1", 2024-08-18T15:45:23.784 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.784 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:45:23.784 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:45.563007502Z", 2024-08-18T15:45:23.785 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.785 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.785 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.785 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.785 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.785 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.785 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_EyCJst2_YMPiGsXCrb7chb3", 2024-08-18T15:45:23.785 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.785 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.785 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.785 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.785 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.785 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.785 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.786 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.786 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994505.563839_61.1", 2024-08-18T15:45:23.786 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.786 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-12", 2024-08-18T15:45:23.786 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:45.563839125Z", 2024-08-18T15:45:23.786 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.786 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.786 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.786 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.786 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.786 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.786 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_O6k9owNlHrC6zA06qCpRJJZ", 2024-08-18T15:45:23.786 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.787 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.787 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.787 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.787 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.787 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.787 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.787 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.787 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994505.566000_62.1", 2024-08-18T15:45:23.787 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.787 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-12", 2024-08-18T15:45:23.787 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:45.566000292Z", 2024-08-18T15:45:23.787 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.787 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.788 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.788 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.788 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.788 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.788 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_O6k9owNlHrC6zA06qCpRJJZ", 2024-08-18T15:45:23.788 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.788 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.788 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.788 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.788 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.788 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.788 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.788 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.789 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994599.644114_81.1", 2024-08-18T15:45:23.789 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.789 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:45:23.789 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:23:19.644114701Z", 2024-08-18T15:45:23.789 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.789 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.789 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.789 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.789 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.789 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.789 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_JA4U4fOsjO9zivYOJOBkWF7", 2024-08-18T15:45:23.789 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.789 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.789 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.790 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.790 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.790 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.790 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.790 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.790 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994599.646276_82.1", 2024-08-18T15:45:23.790 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.790 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:45:23.790 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:23:19.646276235Z", 2024-08-18T15:45:23.790 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.790 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.790 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.790 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.791 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.791 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.791 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_JA4U4fOsjO9zivYOJOBkWF7", 2024-08-18T15:45:23.791 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.791 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.791 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.791 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.791 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.791 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.791 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.791 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.791 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994599.647062_83.1", 2024-08-18T15:45:23.791 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.791 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-13", 2024-08-18T15:45:23.792 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:23:19.647062932Z", 2024-08-18T15:45:23.792 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.792 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.792 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.792 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.792 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.792 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.792 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_qqt9_2CdbPwt_U6wcKZCsvw", 2024-08-18T15:45:23.792 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.792 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.792 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.792 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.792 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.792 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.793 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.793 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.793 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994599.649062_84.1", 2024-08-18T15:45:23.793 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.793 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-13", 2024-08-18T15:45:23.793 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:23:19.649062828Z", 2024-08-18T15:45:23.793 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.793 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.793 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.793 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.793 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.793 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.793 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_qqt9_2CdbPwt_U6wcKZCsvw", 2024-08-18T15:45:23.793 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.794 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.794 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.794 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.794 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.794 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.794 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.794 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.794 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994599.657472_85.1", 2024-08-18T15:45:23.794 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.794 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:45:23.794 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:23:19.657472231Z", 2024-08-18T15:45:23.794 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.794 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.795 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.795 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.795 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.795 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.795 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_hVfHZOUQ7KCzUbpwOhp8fwo", 2024-08-18T15:45:23.795 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.795 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.795 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.795 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.795 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.795 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.795 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.795 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.795 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994599.659687_86.1", 2024-08-18T15:45:23.796 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.796 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:45:23.796 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:23:19.659687294Z", 2024-08-18T15:45:23.796 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.796 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.796 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.796 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.796 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.796 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.796 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_hVfHZOUQ7KCzUbpwOhp8fwo", 2024-08-18T15:45:23.796 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.796 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.796 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.797 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.797 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.797 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.797 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.797 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.797 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994599.660446_87.1", 2024-08-18T15:45:23.797 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.797 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-14", 2024-08-18T15:45:23.797 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:23:19.660446107Z", 2024-08-18T15:45:23.797 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.797 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.797 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.797 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.797 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.798 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.798 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_Xr1Z4jV7ZXypVMLCsCytQag", 2024-08-18T15:45:23.798 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.798 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.798 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.798 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.798 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.798 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.798 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.798 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.798 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994599.662685_88.1", 2024-08-18T15:45:23.798 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.798 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-14", 2024-08-18T15:45:23.799 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:23:19.662685027Z", 2024-08-18T15:45:23.799 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.799 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.799 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.799 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.799 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.799 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.799 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_Xr1Z4jV7ZXypVMLCsCytQag", 2024-08-18T15:45:23.799 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.799 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.799 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.799 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.799 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.799 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.800 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.800 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.800 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994599.682718_89.1", 2024-08-18T15:45:23.800 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.800 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:45:23.800 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:23:19.682718683Z", 2024-08-18T15:45:23.800 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.800 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.800 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.800 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.800 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.800 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.800 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_tZckUgZHpjdeXtjxrO3FDKV", 2024-08-18T15:45:23.800 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.801 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.801 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.801 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.801 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.801 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.801 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.801 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.801 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994599.685017_90.1", 2024-08-18T15:45:23.801 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.801 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:45:23.801 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:23:19.685017287Z", 2024-08-18T15:45:23.801 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.802 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.802 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.802 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.802 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.802 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.802 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_tZckUgZHpjdeXtjxrO3FDKV", 2024-08-18T15:45:23.802 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.802 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.802 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.802 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.802 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.802 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.803 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.803 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.803 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994599.685790_91.1", 2024-08-18T15:45:23.803 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.803 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-15", 2024-08-18T15:45:23.803 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:23:19.685790914Z", 2024-08-18T15:45:23.803 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.803 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.803 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.803 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.803 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.803 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.803 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_z4wReHTq2_y91Bl7PqzxJPS", 2024-08-18T15:45:23.803 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.804 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.804 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.804 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.804 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.804 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.804 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.804 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.804 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994599.687927_92.1", 2024-08-18T15:45:23.804 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.804 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-15", 2024-08-18T15:45:23.804 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:23:19.687927319Z", 2024-08-18T15:45:23.804 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.804 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.804 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.805 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.805 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.805 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.805 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_z4wReHTq2_y91Bl7PqzxJPS", 2024-08-18T15:45:23.805 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.805 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.805 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.805 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.805 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.805 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.805 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.805 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.805 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994722.684594_96.1", 2024-08-18T15:45:23.805 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.806 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:45:23.806 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:25:22.684594373Z", 2024-08-18T15:45:23.806 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.806 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.806 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.806 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.806 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.806 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.806 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_Y8QIfzmW8naLImEFza7--rg", 2024-08-18T15:45:23.806 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.806 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.806 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.806 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.807 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.807 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.807 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.807 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.807 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994722.687709_97.1", 2024-08-18T15:45:23.807 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.807 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:45:23.807 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:25:22.687709684Z", 2024-08-18T15:45:23.807 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.807 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.807 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.807 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.807 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.808 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.808 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_Y8QIfzmW8naLImEFza7--rg", 2024-08-18T15:45:23.808 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.808 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.808 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.808 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.808 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.808 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.808 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.808 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.808 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994722.688892_98.1", 2024-08-18T15:45:23.808 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.808 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-16", 2024-08-18T15:45:23.808 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:25:22.688892164Z", 2024-08-18T15:45:23.809 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.809 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.809 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.809 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.809 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.809 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.809 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_pogOAh2gLfyKg4IGe5K-1It", 2024-08-18T15:45:23.809 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.809 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.809 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.809 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.809 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.809 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.810 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.810 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.810 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994722.692127_99.1", 2024-08-18T15:45:23.810 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.810 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-16", 2024-08-18T15:45:23.810 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:25:22.692127591Z", 2024-08-18T15:45:23.810 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.810 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.810 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.810 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.810 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.810 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.810 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_pogOAh2gLfyKg4IGe5K-1It", 2024-08-18T15:45:23.810 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.811 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.811 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.811 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.811 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.811 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.811 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.811 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.811 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994722.705107_100.1", 2024-08-18T15:45:23.811 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.811 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:45:23.811 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:25:22.705107630Z", 2024-08-18T15:45:23.811 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.811 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.812 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.812 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.812 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.812 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.812 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_TizKQmOZjxDm3jCjcVr_PHs", 2024-08-18T15:45:23.812 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.812 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.812 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.812 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.812 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.812 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.812 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.812 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.812 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994722.707889_101.1", 2024-08-18T15:45:23.813 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.813 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:45:23.813 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:25:22.707889768Z", 2024-08-18T15:45:23.813 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.813 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.813 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.813 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.813 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.813 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.813 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_TizKQmOZjxDm3jCjcVr_PHs", 2024-08-18T15:45:23.813 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.813 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.813 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.813 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.814 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.814 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.814 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.814 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.814 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994722.708890_102.1", 2024-08-18T15:45:23.814 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.814 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-17", 2024-08-18T15:45:23.814 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:25:22.708890200Z", 2024-08-18T15:45:23.814 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.814 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.814 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.814 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.814 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.815 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.815 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_bG8oC8bXHaoSssmVUfs6qfz", 2024-08-18T15:45:23.815 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.815 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.815 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.815 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.815 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.815 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.815 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.815 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.815 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994722.711644_103.1", 2024-08-18T15:45:23.815 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.815 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-17", 2024-08-18T15:45:23.815 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:25:22.711644822Z", 2024-08-18T15:45:23.816 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.816 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.816 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.816 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.816 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.816 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.816 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_bG8oC8bXHaoSssmVUfs6qfz", 2024-08-18T15:45:23.816 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.816 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.816 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.816 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.816 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.816 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.816 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.817 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.817 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994722.734205_104.1", 2024-08-18T15:45:23.817 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.817 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:45:23.817 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:25:22.734205817Z", 2024-08-18T15:45:23.817 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.817 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.817 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.817 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.817 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.817 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.817 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_dLDL_K-k27vccG78KCPIIzE", 2024-08-18T15:45:23.817 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.818 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.818 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.818 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.818 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.818 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.818 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.818 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.818 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994722.736900_105.1", 2024-08-18T15:45:23.818 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.818 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:45:23.818 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:25:22.736900697Z", 2024-08-18T15:45:23.818 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.818 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.818 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.819 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.819 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.819 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.819 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_dLDL_K-k27vccG78KCPIIzE", 2024-08-18T15:45:23.819 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.819 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.819 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.819 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.819 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.819 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.819 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.819 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.819 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994722.737991_106.1", 2024-08-18T15:45:23.819 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.820 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-18", 2024-08-18T15:45:23.820 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:25:22.737991682Z", 2024-08-18T15:45:23.820 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.820 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.820 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.820 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.820 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.820 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.820 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_HPNnX1tYqBsOgWp8mgW7Qb0", 2024-08-18T15:45:23.820 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.820 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.820 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.820 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.820 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.821 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.821 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.821 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.821 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723994722.740810_107.1", 2024-08-18T15:45:23.821 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.821 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-18", 2024-08-18T15:45:23.821 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:25:22.740810965Z", 2024-08-18T15:45:23.821 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.821 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.821 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.821 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.821 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.821 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.822 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_HPNnX1tYqBsOgWp8mgW7Qb0", 2024-08-18T15:45:23.822 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.822 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.822 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.822 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.822 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.822 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.822 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.822 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.822 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995081.645527_272.1", 2024-08-18T15:45:23.822 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.822 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:45:23.822 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:31:21.645527834Z", 2024-08-18T15:45:23.822 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.823 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.823 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.823 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.823 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.823 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.823 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_tgDMSIYDTOiTpHaxXYDAzVY", 2024-08-18T15:45:23.823 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.823 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.823 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.823 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.823 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.823 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.823 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.824 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.824 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995081.648393_273.1", 2024-08-18T15:45:23.824 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.824 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:45:23.824 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:31:21.648393707Z", 2024-08-18T15:45:23.824 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.824 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.824 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.824 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.824 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.824 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.824 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_tgDMSIYDTOiTpHaxXYDAzVY", 2024-08-18T15:45:23.824 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.824 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.825 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.825 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.825 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.825 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.825 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.825 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.825 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995081.649470_274.1", 2024-08-18T15:45:23.825 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.825 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-19", 2024-08-18T15:45:23.825 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:31:21.649470258Z", 2024-08-18T15:45:23.825 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.825 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.825 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.825 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.826 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.826 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.826 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_K6QtNcOVkdATFYGx2kwzf4z", 2024-08-18T15:45:23.826 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.826 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.826 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.826 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.826 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.826 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.826 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.826 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.826 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995081.652067_275.1", 2024-08-18T15:45:23.826 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.827 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-19", 2024-08-18T15:45:23.827 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:31:21.652067379Z", 2024-08-18T15:45:23.827 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.827 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.827 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.827 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.827 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.827 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.827 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_K6QtNcOVkdATFYGx2kwzf4z", 2024-08-18T15:45:23.827 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.827 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.827 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.827 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.827 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.828 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.828 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.828 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.828 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995081.663969_276.1", 2024-08-18T15:45:23.828 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.828 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:45:23.828 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:31:21.663969084Z", 2024-08-18T15:45:23.828 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.828 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.828 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.828 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.828 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.828 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.828 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_Tq_y8CH-R5JTA5co_8cOHJr", 2024-08-18T15:45:23.829 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.829 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.829 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.829 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.829 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.829 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.829 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.829 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.829 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995081.666599_277.1", 2024-08-18T15:45:23.829 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.829 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:45:23.829 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:31:21.666599931Z", 2024-08-18T15:45:23.829 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.830 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.830 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.830 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.830 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.830 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.830 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_Tq_y8CH-R5JTA5co_8cOHJr", 2024-08-18T15:45:23.830 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.830 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.830 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.830 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.830 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.830 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.830 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.830 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.831 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995081.667569_278.1", 2024-08-18T15:45:23.831 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.831 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-20", 2024-08-18T15:45:23.831 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:31:21.667569296Z", 2024-08-18T15:45:23.831 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.831 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.831 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.831 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.831 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.831 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.831 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_wlqM8_vGXOeMXMyiNpDyzb0", 2024-08-18T15:45:23.831 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.831 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.831 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.832 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.832 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.832 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.832 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.832 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.832 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995081.670137_279.1", 2024-08-18T15:45:23.832 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.832 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-20", 2024-08-18T15:45:23.832 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:31:21.670137868Z", 2024-08-18T15:45:23.832 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.832 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.832 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.832 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.832 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.833 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.833 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_wlqM8_vGXOeMXMyiNpDyzb0", 2024-08-18T15:45:23.833 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.833 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.833 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.833 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.833 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.833 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.833 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.833 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.833 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995081.689780_280.1", 2024-08-18T15:45:23.833 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.833 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:45:23.833 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:31:21.689780751Z", 2024-08-18T15:45:23.834 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.834 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.834 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.834 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.834 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.834 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.834 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_tuo9kCpofi-fHLKLK3syMDP", 2024-08-18T15:45:23.834 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.834 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.834 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.834 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.834 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.834 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.835 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.835 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.835 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995081.692441_281.1", 2024-08-18T15:45:23.835 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.835 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:45:23.835 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:31:21.692441804Z", 2024-08-18T15:45:23.835 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.835 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.835 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.835 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.835 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.835 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.835 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_tuo9kCpofi-fHLKLK3syMDP", 2024-08-18T15:45:23.835 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.835 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.836 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.836 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.836 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.836 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.836 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.836 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.836 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995081.693378_282.1", 2024-08-18T15:45:23.836 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.836 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-21", 2024-08-18T15:45:23.836 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:31:21.693378470Z", 2024-08-18T15:45:23.836 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.836 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.836 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.837 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.837 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.837 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.837 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_ntCDeW1uH7FFuKrSyPKYHJm", 2024-08-18T15:45:23.837 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.837 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.837 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.837 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.837 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.837 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.837 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.837 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.837 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995081.695800_283.1", 2024-08-18T15:45:23.837 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.837 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-21", 2024-08-18T15:45:23.838 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:31:21.695800030Z", 2024-08-18T15:45:23.838 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.838 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.838 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.838 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.838 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.838 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.838 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_ntCDeW1uH7FFuKrSyPKYHJm", 2024-08-18T15:45:23.838 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.838 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.838 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.838 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.838 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.839 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.839 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.839 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.839 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995081.706620_284.1", 2024-08-18T15:45:23.839 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.839 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:45:23.839 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:31:21.706620679Z", 2024-08-18T15:45:23.839 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.839 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.839 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_tgDMSIYDTOiTpHaxXYDAzVY", 2024-08-18T15:45:23.839 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.839 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.839 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.839 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_tgDMSIYDTOiTpHaxXYDAzVY", 2024-08-18T15:45:23.840 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.840 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.840 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.840 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.840 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.840 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.840 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.840 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.840 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995081.709128_285.1", 2024-08-18T15:45:23.840 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.840 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:45:23.840 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:31:21.709128917Z", 2024-08-18T15:45:23.840 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.840 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.841 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_tgDMSIYDTOiTpHaxXYDAzVY", 2024-08-18T15:45:23.841 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.841 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.841 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.841 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_tgDMSIYDTOiTpHaxXYDAzVY", 2024-08-18T15:45:23.841 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.841 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.841 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.841 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.841 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.841 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.841 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.841 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.842 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995081.712458_286.1", 2024-08-18T15:45:23.842 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.842 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:45:23.842 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:31:21.712458353Z", 2024-08-18T15:45:23.842 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.842 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.842 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_Tq_y8CH-R5JTA5co_8cOHJr", 2024-08-18T15:45:23.842 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.842 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.842 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.842 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_Tq_y8CH-R5JTA5co_8cOHJr", 2024-08-18T15:45:23.842 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.842 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.843 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.843 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.843 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.843 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.843 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.843 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.843 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995081.714923_287.1", 2024-08-18T15:45:23.843 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.843 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:45:23.843 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:31:21.714923303Z", 2024-08-18T15:45:23.843 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.843 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.843 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_Tq_y8CH-R5JTA5co_8cOHJr", 2024-08-18T15:45:23.843 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.844 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.844 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.844 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_Tq_y8CH-R5JTA5co_8cOHJr", 2024-08-18T15:45:23.844 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.844 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.844 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.844 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.844 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.844 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.844 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.844 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.844 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995081.719853_288.1", 2024-08-18T15:45:23.844 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.844 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:45:23.845 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:31:21.719853955Z", 2024-08-18T15:45:23.845 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.845 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.845 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_tuo9kCpofi-fHLKLK3syMDP", 2024-08-18T15:45:23.845 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.845 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.845 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.845 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_tuo9kCpofi-fHLKLK3syMDP", 2024-08-18T15:45:23.845 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.845 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.845 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.845 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.846 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.846 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.846 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.846 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.846 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995081.722384_289.1", 2024-08-18T15:45:23.846 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.846 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:45:23.846 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:31:21.722384196Z", 2024-08-18T15:45:23.846 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.846 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.846 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_tuo9kCpofi-fHLKLK3syMDP", 2024-08-18T15:45:23.846 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.846 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.846 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.849 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_tuo9kCpofi-fHLKLK3syMDP", 2024-08-18T15:45:23.849 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.849 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.849 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.849 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.849 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.850 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.850 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.850 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.850 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995271.336261_378.1", 2024-08-18T15:45:23.850 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.850 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:45:23.850 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:34:31.336261807Z", 2024-08-18T15:45:23.850 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.850 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.850 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.850 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.850 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.850 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.851 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_pdWHtmdUtmS11GVZZnyUudX", 2024-08-18T15:45:23.851 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.851 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.851 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.851 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.851 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.851 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.851 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.851 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.851 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995271.339446_379.1", 2024-08-18T15:45:23.851 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.851 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:45:23.851 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:34:31.339446352Z", 2024-08-18T15:45:23.852 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.852 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.852 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.852 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.852 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.852 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.852 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_pdWHtmdUtmS11GVZZnyUudX", 2024-08-18T15:45:23.852 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.852 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.852 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.852 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.852 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.852 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.852 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.853 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.853 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995271.340841_380.1", 2024-08-18T15:45:23.853 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.853 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-22", 2024-08-18T15:45:23.853 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:34:31.340841665Z", 2024-08-18T15:45:23.853 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.853 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.853 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.853 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.853 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.853 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.853 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_O9sHsrAa3NkoKI1YNBKaQpY", 2024-08-18T15:45:23.853 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.853 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.854 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.854 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.854 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.854 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.854 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.854 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.854 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995271.343670_381.1", 2024-08-18T15:45:23.854 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.854 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-22", 2024-08-18T15:45:23.854 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:34:31.343670599Z", 2024-08-18T15:45:23.854 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.854 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.854 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.855 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.855 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.855 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.855 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_O9sHsrAa3NkoKI1YNBKaQpY", 2024-08-18T15:45:23.855 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.855 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.855 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.855 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.855 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.855 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.855 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.855 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.855 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995271.347218_382.1", 2024-08-18T15:45:23.855 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.856 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:45:23.856 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:34:31.347218065Z", 2024-08-18T15:45:23.856 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.856 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.856 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_pdWHtmdUtmS11GVZZnyUudX", 2024-08-18T15:45:23.856 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.856 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.856 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.856 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_pdWHtmdUtmS11GVZZnyUudX", 2024-08-18T15:45:23.856 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.856 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.856 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.856 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.856 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.857 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.857 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.857 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.857 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995271.350113_383.1", 2024-08-18T15:45:23.857 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.857 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:45:23.857 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:34:31.350113109Z", 2024-08-18T15:45:23.857 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.857 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.857 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_pdWHtmdUtmS11GVZZnyUudX", 2024-08-18T15:45:23.857 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.857 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.857 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.857 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_pdWHtmdUtmS11GVZZnyUudX", 2024-08-18T15:45:23.858 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.858 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.858 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.858 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.858 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.858 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.858 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.858 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.858 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995462.569535_473.1", 2024-08-18T15:45:23.858 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.858 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:45:23.858 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:37:42.569535297Z", 2024-08-18T15:45:23.858 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.859 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.859 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.859 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.859 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.859 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.859 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_wpzt3Zxvq-oNY9mOoRnPKWQ", 2024-08-18T15:45:23.859 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.859 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.859 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.859 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.859 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.859 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.859 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.860 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.860 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995462.571879_474.1", 2024-08-18T15:45:23.860 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.860 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:45:23.860 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:37:42.571879550Z", 2024-08-18T15:45:23.860 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.860 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.860 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.860 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.860 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.860 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.860 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_wpzt3Zxvq-oNY9mOoRnPKWQ", 2024-08-18T15:45:23.860 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.860 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.861 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.861 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.861 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.861 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.861 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.861 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.861 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995462.572788_475.1", 2024-08-18T15:45:23.861 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.861 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-23", 2024-08-18T15:45:23.861 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:37:42.572788757Z", 2024-08-18T15:45:23.861 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.861 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.861 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.861 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.862 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.862 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.862 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_uJpVLZnIMl1lW-0wuWAfT4Y", 2024-08-18T15:45:23.862 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.862 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.862 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.862 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.862 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.862 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.862 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.862 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.862 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995462.575132_476.1", 2024-08-18T15:45:23.862 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.863 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-23", 2024-08-18T15:45:23.863 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:37:42.575132214Z", 2024-08-18T15:45:23.863 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.863 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.863 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.863 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.863 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.863 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.863 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_uJpVLZnIMl1lW-0wuWAfT4Y", 2024-08-18T15:45:23.864 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.864 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.864 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.864 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.864 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.864 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.864 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.864 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.864 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995503.967200_486.1", 2024-08-18T15:45:23.864 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.864 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:45:23.864 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:38:23.967200465Z", 2024-08-18T15:45:23.864 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.865 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.865 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_wpzt3Zxvq-oNY9mOoRnPKWQ", 2024-08-18T15:45:23.865 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.865 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.865 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.865 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_wpzt3Zxvq-oNY9mOoRnPKWQ", 2024-08-18T15:45:23.865 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.865 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.865 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.865 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.865 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.865 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.865 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.865 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.866 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995503.969777_487.1", 2024-08-18T15:45:23.866 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.866 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:45:23.866 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:38:23.969777196Z", 2024-08-18T15:45:23.866 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.866 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.866 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_wpzt3Zxvq-oNY9mOoRnPKWQ", 2024-08-18T15:45:23.866 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.866 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.866 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.866 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_wpzt3Zxvq-oNY9mOoRnPKWQ", 2024-08-18T15:45:23.866 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.866 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.867 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.867 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.867 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.867 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.867 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.867 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.867 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995565.569248_530.1", 2024-08-18T15:45:23.867 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.867 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:45:23.867 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:39:25.569248856Z", 2024-08-18T15:45:23.867 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.867 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.867 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_wpzt3Zxvq-oNY9mOoRnPKWQ", 2024-08-18T15:45:23.868 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.868 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.868 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.868 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_wpzt3Zxvq-oNY9mOoRnPKWQ", 2024-08-18T15:45:23.868 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 3 2024-08-18T15:45:23.868 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.868 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.868 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.868 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.868 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.868 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.868 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.868 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995565.571903_531.1", 2024-08-18T15:45:23.868 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.869 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:45:23.869 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:39:25.571903234Z", 2024-08-18T15:45:23.869 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.869 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.869 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_wpzt3Zxvq-oNY9mOoRnPKWQ", 2024-08-18T15:45:23.869 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.869 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.869 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.869 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_wpzt3Zxvq-oNY9mOoRnPKWQ", 2024-08-18T15:45:23.869 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 3 2024-08-18T15:45:23.869 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.869 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.869 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.870 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.870 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.870 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.870 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.870 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995627.134415_559.1", 2024-08-18T15:45:23.870 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.870 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:45:23.870 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:40:27.134415241Z", 2024-08-18T15:45:23.870 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.870 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.870 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.870 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.870 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.871 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.871 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_-bWXVuDtxeg-vFcENUxwyqY", 2024-08-18T15:45:23.871 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.871 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.871 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.871 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.871 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.871 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.871 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.871 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.871 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995627.137158_560.1", 2024-08-18T15:45:23.871 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.871 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:45:23.871 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:40:27.137158269Z", 2024-08-18T15:45:23.872 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.872 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.872 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.872 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.872 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.872 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.872 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_-bWXVuDtxeg-vFcENUxwyqY", 2024-08-18T15:45:23.872 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.872 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.872 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.872 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.872 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.872 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.872 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.873 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.873 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995627.138166_561.1", 2024-08-18T15:45:23.873 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.873 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-24", 2024-08-18T15:45:23.873 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:40:27.138166620Z", 2024-08-18T15:45:23.873 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.873 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.873 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.873 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.873 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.873 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.873 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_GqmPD6qFaIdZi7CGWg-6cQ9", 2024-08-18T15:45:23.873 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.873 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.874 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.874 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.874 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.874 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.874 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.874 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.874 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995627.140690_562.1", 2024-08-18T15:45:23.874 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.874 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-24", 2024-08-18T15:45:23.874 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:40:27.140690252Z", 2024-08-18T15:45:23.874 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.874 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.874 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.875 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.875 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.875 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.875 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_GqmPD6qFaIdZi7CGWg-6cQ9", 2024-08-18T15:45:23.875 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.875 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.875 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.875 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.875 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.875 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.875 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.875 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.875 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995627.150973_563.1", 2024-08-18T15:45:23.875 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.876 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:45:23.876 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:40:27.150973118Z", 2024-08-18T15:45:23.876 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.876 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.876 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.876 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.876 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.876 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.876 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_ZASiQO1nC1KO4fWWmGbgTG6", 2024-08-18T15:45:23.876 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.876 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.876 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.876 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.876 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.877 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.877 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.877 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.877 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995627.153561_564.1", 2024-08-18T15:45:23.877 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.877 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:45:23.877 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:40:27.153561617Z", 2024-08-18T15:45:23.877 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.877 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.877 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.877 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.877 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.877 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.877 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_ZASiQO1nC1KO4fWWmGbgTG6", 2024-08-18T15:45:23.878 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.878 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.878 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.878 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.878 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.878 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.878 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.878 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.878 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995627.154521_565.1", 2024-08-18T15:45:23.878 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.878 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-25", 2024-08-18T15:45:23.878 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:40:27.154521041Z", 2024-08-18T15:45:23.878 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.878 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.879 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.879 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.879 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.879 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.879 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_IcFaB2XBAat2mP6-7HsS5nj", 2024-08-18T15:45:23.879 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.879 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.879 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.879 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.879 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.879 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.879 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.879 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.879 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995627.156830_566.1", 2024-08-18T15:45:23.880 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.880 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-25", 2024-08-18T15:45:23.880 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:40:27.156830184Z", 2024-08-18T15:45:23.880 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.880 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.880 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.880 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.880 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.880 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.880 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_IcFaB2XBAat2mP6-7HsS5nj", 2024-08-18T15:45:23.880 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.880 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.880 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.880 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.881 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.881 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.881 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.881 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.881 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995627.173949_567.1", 2024-08-18T15:45:23.881 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.881 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:45:23.881 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:40:27.173949128Z", 2024-08-18T15:45:23.881 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.881 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.881 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.881 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.881 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.881 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.882 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_UDZD1orQ5TbJSSveyG1C6yO", 2024-08-18T15:45:23.882 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.882 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.882 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.882 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.882 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.882 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.882 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.882 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.882 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995627.176614_568.1", 2024-08-18T15:45:23.882 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.882 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:45:23.882 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:40:27.176614276Z", 2024-08-18T15:45:23.883 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.883 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.883 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.883 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.883 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.883 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.883 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_UDZD1orQ5TbJSSveyG1C6yO", 2024-08-18T15:45:23.883 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.883 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.883 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.883 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.883 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.883 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.883 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.884 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.884 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995627.177588_569.1", 2024-08-18T15:45:23.884 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.884 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-26", 2024-08-18T15:45:23.884 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:40:27.177588755Z", 2024-08-18T15:45:23.884 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.884 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.884 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.884 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.884 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.884 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.884 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_6T2erh0-8LIJN2ZhFD-d0zv", 2024-08-18T15:45:23.884 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.884 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.885 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.885 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.885 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.885 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.885 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.885 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.885 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995627.180057_570.1", 2024-08-18T15:45:23.885 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.885 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-26", 2024-08-18T15:45:23.885 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:40:27.180057289Z", 2024-08-18T15:45:23.885 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.885 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.885 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.885 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.886 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.886 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.886 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_6T2erh0-8LIJN2ZhFD-d0zv", 2024-08-18T15:45:23.886 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.886 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.886 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.886 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.886 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.886 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.886 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.886 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.886 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995627.190598_571.1", 2024-08-18T15:45:23.886 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.886 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:45:23.887 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:40:27.190598553Z", 2024-08-18T15:45:23.887 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.887 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.887 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_-bWXVuDtxeg-vFcENUxwyqY", 2024-08-18T15:45:23.887 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.887 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.887 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.887 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_-bWXVuDtxeg-vFcENUxwyqY", 2024-08-18T15:45:23.887 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.887 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.887 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.887 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.887 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.887 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.888 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.888 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.888 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995627.193264_572.1", 2024-08-18T15:45:23.888 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.888 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:45:23.888 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:40:27.193264293Z", 2024-08-18T15:45:23.888 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.888 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.888 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_-bWXVuDtxeg-vFcENUxwyqY", 2024-08-18T15:45:23.888 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.888 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.888 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.888 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_-bWXVuDtxeg-vFcENUxwyqY", 2024-08-18T15:45:23.889 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.889 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.889 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.889 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.889 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.889 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.889 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.889 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.889 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995627.196357_573.1", 2024-08-18T15:45:23.889 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.889 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:45:23.889 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:40:27.196357945Z", 2024-08-18T15:45:23.889 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.889 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.890 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_ZASiQO1nC1KO4fWWmGbgTG6", 2024-08-18T15:45:23.890 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.890 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.890 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.890 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_ZASiQO1nC1KO4fWWmGbgTG6", 2024-08-18T15:45:23.890 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.890 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.890 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.890 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.890 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.890 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.890 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.890 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.890 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995627.198715_574.1", 2024-08-18T15:45:23.891 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.891 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:45:23.891 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:40:27.198715261Z", 2024-08-18T15:45:23.891 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.891 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.891 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_ZASiQO1nC1KO4fWWmGbgTG6", 2024-08-18T15:45:23.891 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.891 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.891 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.891 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_ZASiQO1nC1KO4fWWmGbgTG6", 2024-08-18T15:45:23.891 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.891 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.891 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.892 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.892 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.892 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.892 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.892 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.892 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995627.203450_575.1", 2024-08-18T15:45:23.892 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.892 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:45:23.892 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:40:27.203450736Z", 2024-08-18T15:45:23.892 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.892 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.892 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_UDZD1orQ5TbJSSveyG1C6yO", 2024-08-18T15:45:23.892 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.892 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.893 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.893 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_UDZD1orQ5TbJSSveyG1C6yO", 2024-08-18T15:45:23.893 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.893 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.893 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.893 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.893 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.893 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.893 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.893 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.893 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995627.205732_576.1", 2024-08-18T15:45:23.893 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.893 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:45:23.893 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:40:27.205732200Z", 2024-08-18T15:45:23.894 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.894 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.894 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_UDZD1orQ5TbJSSveyG1C6yO", 2024-08-18T15:45:23.894 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.894 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.894 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.894 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_UDZD1orQ5TbJSSveyG1C6yO", 2024-08-18T15:45:23.894 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.894 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.894 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.894 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.894 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.894 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.895 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.895 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.895 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995690.547603_610.1", 2024-08-18T15:45:23.895 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.895 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:45:23.895 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:41:30.547603460Z", 2024-08-18T15:45:23.895 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.895 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.895 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.895 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.895 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.895 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.895 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_jeYug1evj8xbYPXyUA9ZAIF", 2024-08-18T15:45:23.895 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.896 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.896 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.896 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.896 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.896 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.896 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.896 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.896 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995690.550097_611.1", 2024-08-18T15:45:23.896 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.896 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:45:23.896 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:41:30.550097288Z", 2024-08-18T15:45:23.896 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.896 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.896 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.897 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.897 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.897 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.897 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_jeYug1evj8xbYPXyUA9ZAIF", 2024-08-18T15:45:23.897 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.897 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.897 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.897 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.897 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.897 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.897 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.897 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.897 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995690.551077_612.1", 2024-08-18T15:45:23.897 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.898 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-27", 2024-08-18T15:45:23.898 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:41:30.551077825Z", 2024-08-18T15:45:23.898 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.898 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.898 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.898 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.898 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.898 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.898 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_2Y1Gbxx4l53IXmgzA2xQDmh", 2024-08-18T15:45:23.898 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.898 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.898 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.898 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.898 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.899 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.899 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.899 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.899 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995690.553419_613.1", 2024-08-18T15:45:23.899 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.899 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-27", 2024-08-18T15:45:23.899 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:41:30.553419971Z", 2024-08-18T15:45:23.899 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.899 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.899 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.899 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.899 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.899 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.900 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_2Y1Gbxx4l53IXmgzA2xQDmh", 2024-08-18T15:45:23.900 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.900 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.900 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.900 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.900 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.900 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.900 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.900 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.900 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995690.562778_614.1", 2024-08-18T15:45:23.900 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.900 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:45:23.900 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:41:30.562778163Z", 2024-08-18T15:45:23.900 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.901 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.901 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.901 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.901 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.901 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.901 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_5sF5e5L5wBDjZwv2FwY2xZ2", 2024-08-18T15:45:23.901 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.901 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.901 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.901 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.901 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.901 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.901 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.901 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.902 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995690.565048_615.1", 2024-08-18T15:45:23.902 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.902 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:45:23.902 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:41:30.565048948Z", 2024-08-18T15:45:23.902 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.902 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.902 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.902 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.902 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.902 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.902 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_5sF5e5L5wBDjZwv2FwY2xZ2", 2024-08-18T15:45:23.902 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.903 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.903 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.903 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.903 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.903 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.903 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.903 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.903 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995690.566055_616.1", 2024-08-18T15:45:23.903 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.903 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-28", 2024-08-18T15:45:23.903 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:41:30.566055028Z", 2024-08-18T15:45:23.903 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.903 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.903 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.904 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.904 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.904 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.904 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_3Xacd7LBrQbPx_GZINBqEvJ", 2024-08-18T15:45:23.904 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.904 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.904 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.904 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.904 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.904 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.904 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.904 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.904 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995690.568292_617.1", 2024-08-18T15:45:23.905 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.905 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-28", 2024-08-18T15:45:23.905 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:41:30.568292915Z", 2024-08-18T15:45:23.905 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.905 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.905 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.905 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.905 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.905 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.905 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_3Xacd7LBrQbPx_GZINBqEvJ", 2024-08-18T15:45:23.905 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.905 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.905 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.906 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.906 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.906 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.906 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.906 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.906 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995690.583629_618.1", 2024-08-18T15:45:23.906 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.906 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:45:23.906 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:41:30.583629631Z", 2024-08-18T15:45:23.906 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.906 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.906 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.906 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.907 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.907 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.907 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_Ds7izdnTDTHShaa6cA2hQJO", 2024-08-18T15:45:23.907 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.907 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.907 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.907 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.907 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.907 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.907 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.907 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.907 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995690.585913_619.1", 2024-08-18T15:45:23.907 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.907 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:45:23.908 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:41:30.585913147Z", 2024-08-18T15:45:23.908 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.908 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.908 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.908 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.908 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.908 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.908 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_Ds7izdnTDTHShaa6cA2hQJO", 2024-08-18T15:45:23.908 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.908 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.908 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.908 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.908 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.908 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.909 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.909 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.909 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995690.586806_620.1", 2024-08-18T15:45:23.909 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.909 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-29", 2024-08-18T15:45:23.909 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:41:30.586806831Z", 2024-08-18T15:45:23.909 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.909 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.909 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.909 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.909 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.909 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.909 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_daU6Le-dTETKkQatmqfff9r", 2024-08-18T15:45:23.909 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.910 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.910 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.910 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.910 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.910 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.910 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.910 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.910 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995690.589012_621.1", 2024-08-18T15:45:23.910 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.910 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-29", 2024-08-18T15:45:23.910 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:41:30.589012688Z", 2024-08-18T15:45:23.910 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.910 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.911 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.911 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.911 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.911 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.911 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_daU6Le-dTETKkQatmqfff9r", 2024-08-18T15:45:23.911 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.911 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.911 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.911 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.911 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.911 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.911 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.911 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.911 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995690.598776_622.1", 2024-08-18T15:45:23.912 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.912 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:45:23.912 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:41:30.598776328Z", 2024-08-18T15:45:23.912 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.912 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.912 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_jeYug1evj8xbYPXyUA9ZAIF", 2024-08-18T15:45:23.912 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.912 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.912 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.912 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_jeYug1evj8xbYPXyUA9ZAIF", 2024-08-18T15:45:23.912 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.912 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.912 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.913 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.913 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.913 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.913 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.913 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.913 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995690.601154_623.1", 2024-08-18T15:45:23.913 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.913 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:45:23.913 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:41:30.601154145Z", 2024-08-18T15:45:23.913 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.913 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.913 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_jeYug1evj8xbYPXyUA9ZAIF", 2024-08-18T15:45:23.913 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.913 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.914 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.914 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_jeYug1evj8xbYPXyUA9ZAIF", 2024-08-18T15:45:23.914 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.914 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.914 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.914 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.914 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.914 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.914 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.914 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.914 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995690.603208_624.1", 2024-08-18T15:45:23.914 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.914 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:45:23.915 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:41:30.603208651Z", 2024-08-18T15:45:23.915 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.915 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.915 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_jeYug1evj8xbYPXyUA9ZAIF", 2024-08-18T15:45:23.915 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.915 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.915 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.915 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_jeYug1evj8xbYPXyUA9ZAIF", 2024-08-18T15:45:23.915 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 3 2024-08-18T15:45:23.915 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.915 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.915 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.915 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.915 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.916 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.916 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.916 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995690.605370_625.1", 2024-08-18T15:45:23.916 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.916 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:45:23.916 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:41:30.605370190Z", 2024-08-18T15:45:23.916 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.916 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.916 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_jeYug1evj8xbYPXyUA9ZAIF", 2024-08-18T15:45:23.916 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.916 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.916 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.916 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_jeYug1evj8xbYPXyUA9ZAIF", 2024-08-18T15:45:23.916 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 3 2024-08-18T15:45:23.917 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.917 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.917 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.917 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.917 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.917 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.917 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.917 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995690.607911_626.1", 2024-08-18T15:45:23.917 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.917 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:45:23.917 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:41:30.607911940Z", 2024-08-18T15:45:23.917 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.917 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.918 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_5sF5e5L5wBDjZwv2FwY2xZ2", 2024-08-18T15:45:23.918 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.918 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.918 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.918 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_5sF5e5L5wBDjZwv2FwY2xZ2", 2024-08-18T15:45:23.918 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.918 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.918 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.918 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.918 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.918 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.918 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.918 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.918 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995690.610104_627.1", 2024-08-18T15:45:23.919 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.919 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:45:23.919 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:41:30.610104648Z", 2024-08-18T15:45:23.919 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.919 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.919 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_5sF5e5L5wBDjZwv2FwY2xZ2", 2024-08-18T15:45:23.919 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.919 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.919 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.919 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_5sF5e5L5wBDjZwv2FwY2xZ2", 2024-08-18T15:45:23.919 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.919 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.919 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.920 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.920 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.920 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.920 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.920 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.920 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995690.614660_628.1", 2024-08-18T15:45:23.920 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.920 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:45:23.920 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:41:30.614660827Z", 2024-08-18T15:45:23.920 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.920 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.920 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_5sF5e5L5wBDjZwv2FwY2xZ2", 2024-08-18T15:45:23.920 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.920 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.921 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.921 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_5sF5e5L5wBDjZwv2FwY2xZ2", 2024-08-18T15:45:23.921 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 3 2024-08-18T15:45:23.921 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.921 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.921 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.921 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.921 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.921 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.921 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.921 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995690.616928_629.1", 2024-08-18T15:45:23.921 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.921 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:45:23.921 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:41:30.616928990Z", 2024-08-18T15:45:23.922 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.922 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.922 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_5sF5e5L5wBDjZwv2FwY2xZ2", 2024-08-18T15:45:23.922 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.922 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.922 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.922 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_5sF5e5L5wBDjZwv2FwY2xZ2", 2024-08-18T15:45:23.922 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 3 2024-08-18T15:45:23.922 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.922 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.922 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.922 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.923 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.923 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.923 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.923 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995690.621300_630.1", 2024-08-18T15:45:23.923 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.923 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:45:23.923 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:41:30.621300062Z", 2024-08-18T15:45:23.923 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.923 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.923 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_Ds7izdnTDTHShaa6cA2hQJO", 2024-08-18T15:45:23.923 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.923 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.923 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.923 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_Ds7izdnTDTHShaa6cA2hQJO", 2024-08-18T15:45:23.924 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.924 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.924 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.924 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.924 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.924 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.924 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.924 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.924 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995690.623904_631.1", 2024-08-18T15:45:23.924 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.924 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:45:23.924 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:41:30.623904678Z", 2024-08-18T15:45:23.924 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.925 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.925 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_Ds7izdnTDTHShaa6cA2hQJO", 2024-08-18T15:45:23.925 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.925 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.925 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.925 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_Ds7izdnTDTHShaa6cA2hQJO", 2024-08-18T15:45:23.925 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.925 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.925 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.925 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.925 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.925 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.925 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.925 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.925 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995690.627942_632.1", 2024-08-18T15:45:23.926 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.926 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:45:23.926 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:41:30.627942660Z", 2024-08-18T15:45:23.926 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.926 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.926 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_Ds7izdnTDTHShaa6cA2hQJO", 2024-08-18T15:45:23.926 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.926 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.926 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.926 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_Ds7izdnTDTHShaa6cA2hQJO", 2024-08-18T15:45:23.926 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 3 2024-08-18T15:45:23.926 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.926 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.927 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.927 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.927 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.927 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.927 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.927 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995690.630163_633.1", 2024-08-18T15:45:23.927 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.927 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:45:23.927 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:41:30.630163840Z", 2024-08-18T15:45:23.927 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.927 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.927 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_Ds7izdnTDTHShaa6cA2hQJO", 2024-08-18T15:45:23.927 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.928 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.928 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.928 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_Ds7izdnTDTHShaa6cA2hQJO", 2024-08-18T15:45:23.928 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 3 2024-08-18T15:45:23.928 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.928 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.928 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.928 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.928 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.928 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.928 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.928 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.521479_647.1", 2024-08-18T15:45:23.928 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.928 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-30:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.30", 2024-08-18T15:45:23.929 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.521479385Z", 2024-08-18T15:45:23.929 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.929 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.929 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.929 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.929 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.929 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.929 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_X8eWxctwhKvpxc5MGUbCdeT", 2024-08-18T15:45:23.929 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.929 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.929 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.929 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.929 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.930 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.930 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.930 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.930 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.524106_648.1", 2024-08-18T15:45:23.930 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.930 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-30:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.30", 2024-08-18T15:45:23.930 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.524106942Z", 2024-08-18T15:45:23.930 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.930 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.930 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.930 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.930 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.930 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.931 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_X8eWxctwhKvpxc5MGUbCdeT", 2024-08-18T15:45:23.931 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.931 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.931 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.931 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.931 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.931 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.931 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.931 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.931 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.525089_649.1", 2024-08-18T15:45:23.931 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.931 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-30", 2024-08-18T15:45:23.931 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.525089002Z", 2024-08-18T15:45:23.931 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.932 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.932 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.932 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.932 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.932 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.932 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_Y_twJ8vlrnjpsWlMf8A1vcl", 2024-08-18T15:45:23.932 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.932 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.932 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.932 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.932 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.932 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.932 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.932 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.933 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.527509_650.1", 2024-08-18T15:45:23.933 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.933 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-30", 2024-08-18T15:45:23.933 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.527509595Z", 2024-08-18T15:45:23.933 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.933 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.933 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.933 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.933 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.933 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.933 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_Y_twJ8vlrnjpsWlMf8A1vcl", 2024-08-18T15:45:23.933 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.933 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.934 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.934 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.934 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.934 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.934 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.934 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.934 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.537416_651.1", 2024-08-18T15:45:23.934 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.934 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-31:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.31", 2024-08-18T15:45:23.934 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.537416223Z", 2024-08-18T15:45:23.934 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.934 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.934 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.935 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.935 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.935 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.935 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_3mYRdBwrm6Dau3dYq-qztzU", 2024-08-18T15:45:23.935 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.935 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.935 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.935 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.935 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.935 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.935 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.935 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.935 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.539892_652.1", 2024-08-18T15:45:23.935 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.936 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-31:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.31", 2024-08-18T15:45:23.936 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.539892239Z", 2024-08-18T15:45:23.936 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.936 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.936 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.936 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.936 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.936 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.936 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_3mYRdBwrm6Dau3dYq-qztzU", 2024-08-18T15:45:23.936 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.936 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.936 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.936 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.936 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.937 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.937 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.937 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.937 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.541032_653.1", 2024-08-18T15:45:23.937 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.937 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-31", 2024-08-18T15:45:23.937 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.541032236Z", 2024-08-18T15:45:23.937 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.937 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.937 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.937 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.937 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.937 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.938 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_LwFQ9ExxsG8yYIT84yLWJCM", 2024-08-18T15:45:23.938 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.938 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.938 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.938 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.938 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.938 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.938 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.938 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.938 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.543337_654.1", 2024-08-18T15:45:23.938 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.938 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-31", 2024-08-18T15:45:23.938 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.543337427Z", 2024-08-18T15:45:23.939 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.939 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.939 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.939 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.939 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.939 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.939 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_LwFQ9ExxsG8yYIT84yLWJCM", 2024-08-18T15:45:23.939 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.939 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.939 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.939 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.939 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.939 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.939 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.940 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.940 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.559655_655.1", 2024-08-18T15:45:23.940 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.940 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-32:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.32", 2024-08-18T15:45:23.940 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.559655282Z", 2024-08-18T15:45:23.940 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.940 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.940 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.940 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.940 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.940 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.940 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_N4lXfxOh8y0R0tn_VtdDq--", 2024-08-18T15:45:23.940 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.940 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.941 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.941 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.941 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.941 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.941 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.941 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.941 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.562055_656.1", 2024-08-18T15:45:23.941 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.941 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-32:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.32", 2024-08-18T15:45:23.941 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.562055315Z", 2024-08-18T15:45:23.941 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.941 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.941 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.941 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.942 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.942 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.942 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_N4lXfxOh8y0R0tn_VtdDq--", 2024-08-18T15:45:23.942 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.942 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.942 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.942 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.942 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.942 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.942 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.942 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.942 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.562981_657.1", 2024-08-18T15:45:23.942 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.943 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-32", 2024-08-18T15:45:23.943 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.562981860Z", 2024-08-18T15:45:23.943 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.943 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.943 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.943 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.943 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.943 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.943 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_grj-gSxc_4fWbu_ET8qVUlc", 2024-08-18T15:45:23.943 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.943 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.943 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.943 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.944 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.944 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.944 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.944 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.944 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.565132_658.1", 2024-08-18T15:45:23.944 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.944 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-32", 2024-08-18T15:45:23.944 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.565132516Z", 2024-08-18T15:45:23.944 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.944 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.944 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.944 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.944 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.944 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.945 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_grj-gSxc_4fWbu_ET8qVUlc", 2024-08-18T15:45:23.945 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.945 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.945 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.945 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.945 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.945 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.945 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.945 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.945 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.615071_659.1", 2024-08-18T15:45:23.945 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.945 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-30:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.30", 2024-08-18T15:45:23.945 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.615071398Z", 2024-08-18T15:45:23.946 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.946 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.946 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_X8eWxctwhKvpxc5MGUbCdeT", 2024-08-18T15:45:23.946 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.946 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.946 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.946 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_X8eWxctwhKvpxc5MGUbCdeT", 2024-08-18T15:45:23.946 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.946 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.946 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.946 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.946 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.946 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.947 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.947 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.947 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.617533_660.1", 2024-08-18T15:45:23.947 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.947 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-30:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.30", 2024-08-18T15:45:23.947 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.617533647Z", 2024-08-18T15:45:23.947 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.947 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.947 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_X8eWxctwhKvpxc5MGUbCdeT", 2024-08-18T15:45:23.947 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.947 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.947 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.947 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_X8eWxctwhKvpxc5MGUbCdeT", 2024-08-18T15:45:23.947 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.948 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.948 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.948 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.948 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.948 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.948 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.948 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.948 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.622444_661.1", 2024-08-18T15:45:23.948 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.948 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-31:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.31", 2024-08-18T15:45:23.948 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.622444585Z", 2024-08-18T15:45:23.948 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.948 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.948 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_3mYRdBwrm6Dau3dYq-qztzU", 2024-08-18T15:45:23.949 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.949 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.949 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.949 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_3mYRdBwrm6Dau3dYq-qztzU", 2024-08-18T15:45:23.949 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.949 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.949 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.949 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.949 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.949 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.949 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.949 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.949 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.624798_662.1", 2024-08-18T15:45:23.949 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.950 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-31:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.31", 2024-08-18T15:45:23.950 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.624798339Z", 2024-08-18T15:45:23.950 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.950 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.950 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_3mYRdBwrm6Dau3dYq-qztzU", 2024-08-18T15:45:23.950 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.950 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.950 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.950 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_3mYRdBwrm6Dau3dYq-qztzU", 2024-08-18T15:45:23.950 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.950 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.950 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.951 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.951 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.951 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.951 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.951 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.951 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.673691_663.1", 2024-08-18T15:45:23.951 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.951 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-32:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.32", 2024-08-18T15:45:23.951 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.673691986Z", 2024-08-18T15:45:23.951 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.951 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.951 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_N4lXfxOh8y0R0tn_VtdDq--", 2024-08-18T15:45:23.951 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.951 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.952 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.952 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_N4lXfxOh8y0R0tn_VtdDq--", 2024-08-18T15:45:23.952 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.952 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.952 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.952 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.952 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.952 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.952 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.952 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.952 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.676028_664.1", 2024-08-18T15:45:23.952 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.952 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-32:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.32", 2024-08-18T15:45:23.952 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.676028814Z", 2024-08-18T15:45:23.952 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.953 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.953 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_N4lXfxOh8y0R0tn_VtdDq--", 2024-08-18T15:45:23.953 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.953 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.953 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.953 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_N4lXfxOh8y0R0tn_VtdDq--", 2024-08-18T15:45:23.953 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.953 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.953 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.953 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.953 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.953 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.953 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.953 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.954 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.684893_665.1", 2024-08-18T15:45:23.954 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.954 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-33:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.33", 2024-08-18T15:45:23.954 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.684893391Z", 2024-08-18T15:45:23.954 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.954 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.954 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.954 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.954 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.954 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.954 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_lGULOaXyl4hHx7qh454F-aa", 2024-08-18T15:45:23.954 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.955 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.955 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.955 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.955 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.955 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.955 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.955 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.955 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.687266_666.1", 2024-08-18T15:45:23.955 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.955 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-33:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.33", 2024-08-18T15:45:23.955 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.687266999Z", 2024-08-18T15:45:23.955 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.955 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.955 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.956 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.956 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.956 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.956 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_lGULOaXyl4hHx7qh454F-aa", 2024-08-18T15:45:23.956 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.956 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.956 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.956 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.956 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.956 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.956 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.956 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.956 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.688166_667.1", 2024-08-18T15:45:23.956 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.957 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-33", 2024-08-18T15:45:23.957 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.688166208Z", 2024-08-18T15:45:23.957 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.957 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.957 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.957 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.957 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.957 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.957 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_WydO77zfjR8Husc4yxYU8A8", 2024-08-18T15:45:23.957 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.957 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.957 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.957 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.958 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.958 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.958 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.958 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.958 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.690522_668.1", 2024-08-18T15:45:23.958 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.958 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-33", 2024-08-18T15:45:23.958 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.690522745Z", 2024-08-18T15:45:23.958 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.958 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.958 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.958 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.958 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.959 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.959 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_WydO77zfjR8Husc4yxYU8A8", 2024-08-18T15:45:23.959 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.959 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.959 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.959 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.959 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.959 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.959 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.959 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.959 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.699985_669.1", 2024-08-18T15:45:23.959 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.959 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-34:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.34", 2024-08-18T15:45:23.959 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.699985189Z", 2024-08-18T15:45:23.960 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.960 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.960 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.960 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.960 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.960 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.960 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_gziC3Zhjn59sc9f24iXYqez", 2024-08-18T15:45:23.960 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.960 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.960 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.960 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.960 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.960 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.960 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.961 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.961 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.702346_670.1", 2024-08-18T15:45:23.961 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.961 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-34:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.34", 2024-08-18T15:45:23.961 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.702346126Z", 2024-08-18T15:45:23.961 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.961 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.961 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.961 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.961 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.961 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.961 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_gziC3Zhjn59sc9f24iXYqez", 2024-08-18T15:45:23.961 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.962 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.962 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.962 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.962 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.962 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.962 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.962 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.962 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.703327_671.1", 2024-08-18T15:45:23.962 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.962 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-34", 2024-08-18T15:45:23.962 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.703327689Z", 2024-08-18T15:45:23.962 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.962 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.963 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.963 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.963 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.963 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.963 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_fIT8XfQf3mP3Rdkpjxd4TDK", 2024-08-18T15:45:23.963 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.963 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.963 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.963 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.963 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.963 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.963 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.963 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.964 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.705678_672.1", 2024-08-18T15:45:23.964 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.964 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-34", 2024-08-18T15:45:23.964 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.705678013Z", 2024-08-18T15:45:23.964 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.964 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.964 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.964 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.964 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.964 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.964 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_fIT8XfQf3mP3Rdkpjxd4TDK", 2024-08-18T15:45:23.965 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.965 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.965 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.965 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.965 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.965 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.965 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.965 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.965 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.722023_673.1", 2024-08-18T15:45:23.965 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.965 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-35:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.35", 2024-08-18T15:45:23.965 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.722023363Z", 2024-08-18T15:45:23.965 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.966 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.966 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.966 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.966 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.966 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.966 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_8PQDI19UyDbSfg9jXHxQK0Z", 2024-08-18T15:45:23.966 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.966 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.966 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.966 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.966 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.966 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.966 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.967 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.967 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.724501_674.1", 2024-08-18T15:45:23.967 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.967 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-35:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.35", 2024-08-18T15:45:23.967 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.724501528Z", 2024-08-18T15:45:23.967 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.967 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.967 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.967 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.967 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.967 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.967 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_8PQDI19UyDbSfg9jXHxQK0Z", 2024-08-18T15:45:23.967 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.967 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.968 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.968 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.968 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.968 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.968 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.968 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.968 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.725468_675.1", 2024-08-18T15:45:23.968 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.968 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-35", 2024-08-18T15:45:23.968 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.725468008Z", 2024-08-18T15:45:23.968 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.968 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.968 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.969 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.969 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.969 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.969 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_5pwJU04pzA5ySeDPbaf8cI9", 2024-08-18T15:45:23.969 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.969 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.969 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.969 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.969 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.969 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.969 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.969 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.969 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.728040_676.1", 2024-08-18T15:45:23.969 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.970 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-35", 2024-08-18T15:45:23.970 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.728040336Z", 2024-08-18T15:45:23.970 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.970 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.970 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.970 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.970 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.970 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.970 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_5pwJU04pzA5ySeDPbaf8cI9", 2024-08-18T15:45:23.970 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.970 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.970 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.970 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.971 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.971 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.971 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.971 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.971 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.744014_677.1", 2024-08-18T15:45:23.971 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.971 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-33:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.33", 2024-08-18T15:45:23.971 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.744014055Z", 2024-08-18T15:45:23.971 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.971 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.971 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_lGULOaXyl4hHx7qh454F-aa", 2024-08-18T15:45:23.971 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.971 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.971 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.972 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_lGULOaXyl4hHx7qh454F-aa", 2024-08-18T15:45:23.972 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.972 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.972 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.972 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.972 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.972 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.972 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.972 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.972 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.746404_678.1", 2024-08-18T15:45:23.972 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.972 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-33:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.33", 2024-08-18T15:45:23.972 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.746404003Z", 2024-08-18T15:45:23.972 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.973 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.973 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_lGULOaXyl4hHx7qh454F-aa", 2024-08-18T15:45:23.973 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.973 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.973 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.973 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_lGULOaXyl4hHx7qh454F-aa", 2024-08-18T15:45:23.973 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.973 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.973 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.973 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.973 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.973 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.973 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.974 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.974 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.751650_679.1", 2024-08-18T15:45:23.974 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.974 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-34:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.34", 2024-08-18T15:45:23.974 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.751650666Z", 2024-08-18T15:45:23.974 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.974 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.974 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_gziC3Zhjn59sc9f24iXYqez", 2024-08-18T15:45:23.974 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.974 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.974 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.974 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_gziC3Zhjn59sc9f24iXYqez", 2024-08-18T15:45:23.974 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.975 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.975 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.975 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.975 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.975 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.975 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.975 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.975 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.754074_680.1", 2024-08-18T15:45:23.975 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.975 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-34:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.34", 2024-08-18T15:45:23.975 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.754074401Z", 2024-08-18T15:45:23.975 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.975 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.975 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_gziC3Zhjn59sc9f24iXYqez", 2024-08-18T15:45:23.975 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.976 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.976 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.976 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_gziC3Zhjn59sc9f24iXYqez", 2024-08-18T15:45:23.976 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.976 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.976 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.976 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.976 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.976 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.976 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.976 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.976 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.802966_681.1", 2024-08-18T15:45:23.976 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.977 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-35:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.35", 2024-08-18T15:45:23.977 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.802966900Z", 2024-08-18T15:45:23.977 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.977 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.977 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_8PQDI19UyDbSfg9jXHxQK0Z", 2024-08-18T15:45:23.977 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.977 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.977 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.977 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_8PQDI19UyDbSfg9jXHxQK0Z", 2024-08-18T15:45:23.977 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.977 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.977 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.977 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.978 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.978 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.978 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.978 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.978 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.805637_682.1", 2024-08-18T15:45:23.978 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.978 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-35:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.35", 2024-08-18T15:45:23.978 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.805637670Z", 2024-08-18T15:45:23.978 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.978 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.978 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_8PQDI19UyDbSfg9jXHxQK0Z", 2024-08-18T15:45:23.978 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.978 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.979 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.979 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_8PQDI19UyDbSfg9jXHxQK0Z", 2024-08-18T15:45:23.979 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.979 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.979 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.979 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.979 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.979 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.979 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.979 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.979 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.862218_683.1", 2024-08-18T15:45:23.979 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.979 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-36:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.36", 2024-08-18T15:45:23.980 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.862218121Z", 2024-08-18T15:45:23.980 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.980 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.980 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.980 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.980 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.980 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.980 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_OMlfwDqiYy7MLBPapdgXZCw", 2024-08-18T15:45:23.980 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.980 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.980 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.980 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.980 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.980 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.981 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.981 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.981 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.864692_684.1", 2024-08-18T15:45:23.981 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.981 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-36:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.36", 2024-08-18T15:45:23.981 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.864692243Z", 2024-08-18T15:45:23.981 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.981 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.981 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.981 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.981 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.981 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.981 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_OMlfwDqiYy7MLBPapdgXZCw", 2024-08-18T15:45:23.982 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.982 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.982 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.982 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.982 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.982 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.982 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.982 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.982 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.865659_685.1", 2024-08-18T15:45:23.982 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.982 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-36", 2024-08-18T15:45:23.982 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.865659998Z", 2024-08-18T15:45:23.982 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.982 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.983 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.983 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.983 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.983 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.983 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_dui98arcr6Cif7iTccJdz_E", 2024-08-18T15:45:23.983 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.983 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.983 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.983 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.983 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.983 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.983 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.983 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.984 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.867847_686.1", 2024-08-18T15:45:23.984 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.984 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-36", 2024-08-18T15:45:23.984 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.867847854Z", 2024-08-18T15:45:23.984 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.984 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.984 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.984 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.984 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.984 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.984 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_dui98arcr6Cif7iTccJdz_E", 2024-08-18T15:45:23.984 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.984 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.984 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.985 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.985 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.985 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.985 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.985 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.985 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.877645_687.1", 2024-08-18T15:45:23.985 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.985 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-37:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.37", 2024-08-18T15:45:23.985 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.877645769Z", 2024-08-18T15:45:23.985 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.985 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.985 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.985 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.985 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.986 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.986 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_HFwNCilj83ivBqE5tB_PZE2", 2024-08-18T15:45:23.986 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.986 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.986 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.986 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.986 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.986 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.986 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.986 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.986 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.879910_688.1", 2024-08-18T15:45:23.986 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.986 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-37:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.37", 2024-08-18T15:45:23.987 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.879910198Z", 2024-08-18T15:45:23.987 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.987 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.987 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.987 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.987 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.987 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.987 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_HFwNCilj83ivBqE5tB_PZE2", 2024-08-18T15:45:23.987 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.987 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.987 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.987 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.987 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.987 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.988 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.988 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.988 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.880848_689.1", 2024-08-18T15:45:23.988 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.988 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-37", 2024-08-18T15:45:23.988 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.880848207Z", 2024-08-18T15:45:23.988 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.988 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.988 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.988 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.988 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.988 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.988 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_rUYGyCScmgZ3SwWHWdQlE0I", 2024-08-18T15:45:23.989 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.989 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.989 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.989 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.989 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.989 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.989 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.989 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.989 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.883023_690.1", 2024-08-18T15:45:23.989 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.989 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-37", 2024-08-18T15:45:23.989 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.883023907Z", 2024-08-18T15:45:23.989 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.989 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.990 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.990 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.990 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.990 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.990 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_rUYGyCScmgZ3SwWHWdQlE0I", 2024-08-18T15:45:23.990 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.990 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.990 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.990 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.990 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.990 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.990 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.990 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.990 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.899500_691.1", 2024-08-18T15:45:23.991 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.991 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-38:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.38", 2024-08-18T15:45:23.991 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.899500733Z", 2024-08-18T15:45:23.991 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.991 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.991 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.991 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.991 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.991 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.991 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_i-eCmhQ4NPsJf86Skaj64Wy", 2024-08-18T15:45:23.991 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.991 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.991 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.992 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.992 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.992 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.992 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.992 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.992 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.901993_692.1", 2024-08-18T15:45:23.992 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.992 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-38:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.38", 2024-08-18T15:45:23.992 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.901993132Z", 2024-08-18T15:45:23.992 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.992 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.992 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.992 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.992 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.993 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.993 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_i-eCmhQ4NPsJf86Skaj64Wy", 2024-08-18T15:45:23.993 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.993 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.993 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.993 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.993 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.993 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.993 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.993 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.993 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.903041_693.1", 2024-08-18T15:45:23.993 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.993 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-38", 2024-08-18T15:45:23.994 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.903041404Z", 2024-08-18T15:45:23.994 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.994 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.994 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.994 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.994 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.994 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.994 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_i5jx3wzjmX5pErHvMHm_pkB", 2024-08-18T15:45:23.994 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.994 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.994 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.994 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.994 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.994 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.995 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.995 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.995 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.905154_694.1", 2024-08-18T15:45:23.995 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:23.995 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-38", 2024-08-18T15:45:23.995 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.905154856Z", 2024-08-18T15:45:23.995 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.995 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.995 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:23.995 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:23.995 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.995 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.995 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_i5jx3wzjmX5pErHvMHm_pkB", 2024-08-18T15:45:23.996 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.996 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.996 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.996 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.996 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.996 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.996 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.996 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.996 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.915169_695.1", 2024-08-18T15:45:23.996 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.996 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-36:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.36", 2024-08-18T15:45:23.996 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.915169406Z", 2024-08-18T15:45:23.996 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.996 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.997 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_OMlfwDqiYy7MLBPapdgXZCw", 2024-08-18T15:45:23.997 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.997 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.997 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.997 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_OMlfwDqiYy7MLBPapdgXZCw", 2024-08-18T15:45:23.997 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.997 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.997 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.997 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:23.997 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.997 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.997 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.997 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.997 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995748.917556_696.1", 2024-08-18T15:45:23.998 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.998 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-36:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.36", 2024-08-18T15:45:23.998 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:28.917556247Z", 2024-08-18T15:45:23.998 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.998 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.998 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_OMlfwDqiYy7MLBPapdgXZCw", 2024-08-18T15:45:23.998 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:23.998 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.998 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:23.998 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_OMlfwDqiYy7MLBPapdgXZCw", 2024-08-18T15:45:23.998 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:23.998 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.998 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:23.999 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:23.999 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.999 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:23.999 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:23.999 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:23.999 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995749.970360_697.1", 2024-08-18T15:45:23.999 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:23.999 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-37:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.37", 2024-08-18T15:45:23.999 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:29.970360729Z", 2024-08-18T15:45:23.999 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:23.999 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:23.999 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_HFwNCilj83ivBqE5tB_PZE2", 2024-08-18T15:45:23.999 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:24.000 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.000 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:24.000 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_HFwNCilj83ivBqE5tB_PZE2", 2024-08-18T15:45:24.000 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:24.000 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.000 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:24.000 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:24.000 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.000 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.000 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.000 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:24.000 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995749.974004_698.1", 2024-08-18T15:45:24.000 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:24.000 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-37:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.37", 2024-08-18T15:45:24.001 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:29.974004150Z", 2024-08-18T15:45:24.001 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:24.001 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:24.001 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_HFwNCilj83ivBqE5tB_PZE2", 2024-08-18T15:45:24.001 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:24.001 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.001 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:24.001 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_HFwNCilj83ivBqE5tB_PZE2", 2024-08-18T15:45:24.001 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:24.001 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.001 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:24.001 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:24.001 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.001 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.002 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.002 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:24.002 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995751.026760_699.1", 2024-08-18T15:45:24.002 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:24.002 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-38:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.38", 2024-08-18T15:45:24.002 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:31.026760427Z", 2024-08-18T15:45:24.002 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:24.002 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:24.002 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_i-eCmhQ4NPsJf86Skaj64Wy", 2024-08-18T15:45:24.002 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:24.002 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.002 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:24.002 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_i-eCmhQ4NPsJf86Skaj64Wy", 2024-08-18T15:45:24.002 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:24.003 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.003 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:24.003 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:24.003 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.003 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.003 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.003 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:24.003 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995751.031154_700.1", 2024-08-18T15:45:24.003 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:24.003 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-38:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.38", 2024-08-18T15:45:24.003 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:31.031154082Z", 2024-08-18T15:45:24.003 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:24.003 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:24.004 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_i-eCmhQ4NPsJf86Skaj64Wy", 2024-08-18T15:45:24.004 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:24.004 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.004 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:24.004 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_i-eCmhQ4NPsJf86Skaj64Wy", 2024-08-18T15:45:24.004 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 2 2024-08-18T15:45:24.004 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.004 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:24.004 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:24.004 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.004 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.004 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.004 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:24.004 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995751.043389_701.1", 2024-08-18T15:45:24.005 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:24.005 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:45:24.005 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:31.043389213Z", 2024-08-18T15:45:24.005 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:24.005 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:24.005 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:24.005 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:24.005 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.005 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:24.005 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_xmYq1VNiyUA8ztTA_d5rUm3", 2024-08-18T15:45:24.005 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:24.005 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.005 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:24.005 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:24.006 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.006 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.006 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.006 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:24.006 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995751.046426_702.1", 2024-08-18T15:45:24.006 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:24.006 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:45:24.006 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:31.046426883Z", 2024-08-18T15:45:24.006 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:24.006 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:24.006 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:24.006 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:24.006 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.006 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:24.007 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_xmYq1VNiyUA8ztTA_d5rUm3", 2024-08-18T15:45:24.007 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:24.007 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.007 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:24.007 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:24.007 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.007 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.007 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.007 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:24.007 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995751.047596_703.1", 2024-08-18T15:45:24.007 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:24.007 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-39", 2024-08-18T15:45:24.007 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:31.047596040Z", 2024-08-18T15:45:24.007 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:24.008 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:24.008 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:24.008 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:24.008 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.008 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:24.008 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "__iz5Ku-i6OWsHDvE8pjpoke", 2024-08-18T15:45:24.008 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:24.008 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.008 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:24.008 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:24.008 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.008 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.008 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.008 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:24.009 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995751.050305_704.1", 2024-08-18T15:45:24.009 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:24.009 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-39", 2024-08-18T15:45:24.009 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:31.050305755Z", 2024-08-18T15:45:24.009 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:24.009 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:24.009 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:24.009 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:24.009 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.009 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:24.009 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "__iz5Ku-i6OWsHDvE8pjpoke", 2024-08-18T15:45:24.009 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:24.009 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.009 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:24.010 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:24.010 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.010 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.010 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.010 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:24.010 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995751.063176_705.1", 2024-08-18T15:45:24.010 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:24.010 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:45:24.010 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:31.063176910Z", 2024-08-18T15:45:24.010 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:24.010 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:24.010 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:24.010 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:24.010 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.011 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:24.011 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_ejQ3MioLVUCb2QnucqqCUMJ", 2024-08-18T15:45:24.011 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:24.011 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.011 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:24.011 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:24.011 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.011 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.011 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.011 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:24.011 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995751.066133_706.1", 2024-08-18T15:45:24.011 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:24.011 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:45:24.012 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:31.066133710Z", 2024-08-18T15:45:24.012 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:24.012 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:24.012 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:24.012 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:24.012 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.012 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:24.012 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_ejQ3MioLVUCb2QnucqqCUMJ", 2024-08-18T15:45:24.012 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:24.012 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.012 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:24.012 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:24.012 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.012 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.013 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.013 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:24.013 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995751.067195_707.1", 2024-08-18T15:45:24.013 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:24.013 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-40", 2024-08-18T15:45:24.013 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:31.067195014Z", 2024-08-18T15:45:24.013 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:24.013 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:24.013 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:24.013 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:24.013 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.013 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:24.013 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_llhZcJSjwRerrd-Oaou9Sjv", 2024-08-18T15:45:24.013 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:24.014 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.014 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:24.014 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:24.014 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.014 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.014 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.014 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:24.014 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995751.069927_708.1", 2024-08-18T15:45:24.014 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:24.014 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-40", 2024-08-18T15:45:24.014 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:31.069927389Z", 2024-08-18T15:45:24.014 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:24.014 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:24.015 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:24.015 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:24.015 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.015 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:24.015 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_llhZcJSjwRerrd-Oaou9Sjv", 2024-08-18T15:45:24.015 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:24.015 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.015 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:24.015 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:24.015 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.015 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.015 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.015 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:24.015 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995751.090745_709.1", 2024-08-18T15:45:24.015 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:24.016 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:45:24.016 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:31.090745755Z", 2024-08-18T15:45:24.016 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:24.016 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:24.016 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:24.016 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:24.016 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.016 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:24.016 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_8c5popeokpV--2SpVjBST3E", 2024-08-18T15:45:24.016 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:24.016 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.016 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:24.016 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:24.016 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.017 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.017 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.017 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:24.017 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995751.093802_710.1", 2024-08-18T15:45:24.017 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:24.017 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:45:24.017 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:31.093802820Z", 2024-08-18T15:45:24.017 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:24.017 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:24.017 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:24.017 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:24.017 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.017 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:24.017 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_8c5popeokpV--2SpVjBST3E", 2024-08-18T15:45:24.018 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:24.018 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.018 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:24.018 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:24.018 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.018 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.018 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.018 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:24.018 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995751.095112_711.1", 2024-08-18T15:45:24.018 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:24.018 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-41", 2024-08-18T15:45:24.018 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:31.095112579Z", 2024-08-18T15:45:24.018 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:24.019 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:24.019 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:24.019 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:24.019 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.019 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:24.019 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_YWJNdlw8sqMdjjlIjLHAL7T", 2024-08-18T15:45:24.019 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:24.019 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.019 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:24.019 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:24.019 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.019 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.019 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.019 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:24.020 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995751.097661_712.1", 2024-08-18T15:45:24.020 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:24.020 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-41", 2024-08-18T15:45:24.020 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:31.097661378Z", 2024-08-18T15:45:24.020 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:24.020 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:24.020 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:24.020 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:24.020 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.020 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:24.020 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_YWJNdlw8sqMdjjlIjLHAL7T", 2024-08-18T15:45:24.020 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:24.020 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.020 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:24.021 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:24.021 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.021 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.021 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.021 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:24.021 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995771.988164_729.1", 2024-08-18T15:45:24.021 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:24.021 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-42:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.42", 2024-08-18T15:45:24.021 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:51.988164854Z", 2024-08-18T15:45:24.021 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:24.021 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:24.021 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:24.021 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:24.021 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.022 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:24.022 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_oGiVyEzJBYGJYJotLHjzZsK", 2024-08-18T15:45:24.022 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:24.022 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.022 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:24.022 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:24.022 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.022 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.022 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.022 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:24.022 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995771.990622_730.1", 2024-08-18T15:45:24.022 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:24.022 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-42:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.42", 2024-08-18T15:45:24.022 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:51.990622883Z", 2024-08-18T15:45:24.023 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:24.023 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:24.023 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:24.023 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:24.023 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.023 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:24.023 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_oGiVyEzJBYGJYJotLHjzZsK", 2024-08-18T15:45:24.023 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:24.023 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.023 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:24.023 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:24.023 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.023 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.024 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.024 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:24.024 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995771.991578_731.1", 2024-08-18T15:45:24.024 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:24.024 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-42", 2024-08-18T15:45:24.024 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:51.991578988Z", 2024-08-18T15:45:24.024 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:24.024 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:24.024 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:24.024 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:24.024 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.024 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:24.024 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_fG-TEUh0CxWR43RmwQg7ICF", 2024-08-18T15:45:24.024 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:24.025 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.025 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:24.025 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:24.025 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.025 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.025 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.025 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:24.025 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995771.993985_732.1", 2024-08-18T15:45:24.025 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:24.025 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-42", 2024-08-18T15:45:24.025 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:51.993985953Z", 2024-08-18T15:45:24.025 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:24.025 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:24.025 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:24.026 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:24.026 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.026 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:24.026 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_fG-TEUh0CxWR43RmwQg7ICF", 2024-08-18T15:45:24.026 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:24.026 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.026 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:24.026 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:24.026 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.026 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.026 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.026 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:24.026 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995772.004584_733.1", 2024-08-18T15:45:24.026 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:24.027 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-43:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.43", 2024-08-18T15:45:24.027 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:52.004584088Z", 2024-08-18T15:45:24.027 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:24.027 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:24.027 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:24.027 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:24.027 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.027 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:24.027 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_c-hGhbKE-b1LRVHEPUzPjib", 2024-08-18T15:45:24.027 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:24.027 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.027 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:24.027 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:24.028 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.028 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.028 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.028 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:24.028 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995772.008960_734.1", 2024-08-18T15:45:24.028 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:24.028 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-43:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.43", 2024-08-18T15:45:24.028 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:52.008960500Z", 2024-08-18T15:45:24.028 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:24.028 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:24.028 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:24.028 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:24.028 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.028 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:24.029 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_c-hGhbKE-b1LRVHEPUzPjib", 2024-08-18T15:45:24.029 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:24.029 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.029 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:24.029 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:24.029 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.029 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.029 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.029 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:24.029 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995772.010074_735.1", 2024-08-18T15:45:24.029 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:24.029 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-43", 2024-08-18T15:45:24.029 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:52.010074886Z", 2024-08-18T15:45:24.030 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:24.030 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:24.030 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:24.030 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:24.030 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.030 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:24.030 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_L-a1id3J-Bf_HtA2U8yB3t5", 2024-08-18T15:45:24.030 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:24.030 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.030 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:24.030 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:24.030 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.030 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.031 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.031 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:24.031 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995772.012225_736.1", 2024-08-18T15:45:24.031 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:24.031 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-43", 2024-08-18T15:45:24.031 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:52.012225687Z", 2024-08-18T15:45:24.031 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:24.031 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:24.031 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:24.031 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:24.031 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.031 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:24.031 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_L-a1id3J-Bf_HtA2U8yB3t5", 2024-08-18T15:45:24.031 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:24.032 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.032 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:24.032 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:24.032 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.032 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.032 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.032 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:24.032 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995772.031468_737.1", 2024-08-18T15:45:24.032 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:24.032 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-44:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.44", 2024-08-18T15:45:24.032 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:52.031468864Z", 2024-08-18T15:45:24.032 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:24.032 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:24.033 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:24.033 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:24.033 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.033 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:24.033 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_On0FEwDZQtRx6QIbTXxbmk6", 2024-08-18T15:45:24.033 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:24.033 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.033 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:24.033 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:24.033 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.033 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.033 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.033 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:24.034 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995772.034017_738.1", 2024-08-18T15:45:24.034 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:24.034 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-44:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.44", 2024-08-18T15:45:24.034 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:52.034017957Z", 2024-08-18T15:45:24.034 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:24.034 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:24.034 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:24.034 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:24.034 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.034 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:24.034 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_On0FEwDZQtRx6QIbTXxbmk6", 2024-08-18T15:45:24.034 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:24.034 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.035 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:24.035 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:24.035 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.035 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.035 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.035 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:24.035 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995772.035064_739.1", 2024-08-18T15:45:24.035 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:24.035 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-44", 2024-08-18T15:45:24.035 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:52.035064992Z", 2024-08-18T15:45:24.035 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:24.035 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:24.035 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:24.035 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:24.036 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.036 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:24.036 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_zO2kHLSbZbuAztlb-hy3PyC", 2024-08-18T15:45:24.036 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:24.036 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.036 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:24.036 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:24.036 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.036 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.036 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.036 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:24.036 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995772.038282_740.1", 2024-08-18T15:45:24.036 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:24.037 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-44", 2024-08-18T15:45:24.037 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:52.038282163Z", 2024-08-18T15:45:24.037 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:24.037 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:24.037 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:24.037 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:24.037 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.037 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:24.037 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_zO2kHLSbZbuAztlb-hy3PyC", 2024-08-18T15:45:24.037 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:24.037 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.037 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:24.037 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:24.037 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.038 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.038 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.038 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T15:45:24.090 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '64309209-7567-46e5-8881-65783d84cf3c', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:45:24.091 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '64309209-7567-46e5-8881-65783d84cf3c', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:45:24.091 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 64309209-7567-46e5-8881-65783d84cf3c --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:45:24.138 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:45:24.138 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:45:24.168 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T15:45:24.169 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:24.169 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995848.895198_180.1", 2024-08-18T15:45:24.169 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:24.169 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-45:10090ccf-aad6-416d-9775-df8b7b27345a.5181.1", 2024-08-18T15:45:24.169 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:44:08.895198309Z", 2024-08-18T15:45:24.169 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:24.169 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:24.169 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:24.169 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:24.170 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.170 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:24.170 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_6PVK0GlJ0EocWsIkwIHUQja", 2024-08-18T15:45:24.170 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:24.170 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.170 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:24.170 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:24.170 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.170 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.170 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.170 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:24.171 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995848.898448_181.1", 2024-08-18T15:45:24.171 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:24.171 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-45:10090ccf-aad6-416d-9775-df8b7b27345a.5181.1", 2024-08-18T15:45:24.171 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:44:08.898448461Z", 2024-08-18T15:45:24.171 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:24.171 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:24.171 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:24.171 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:24.171 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.171 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:24.171 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_6PVK0GlJ0EocWsIkwIHUQja", 2024-08-18T15:45:24.171 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:24.171 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.172 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:24.172 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:24.172 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.172 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.172 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.172 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:24.172 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995848.899565_182.1", 2024-08-18T15:45:24.172 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:24.172 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-45", 2024-08-18T15:45:24.172 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:44:08.899565177Z", 2024-08-18T15:45:24.172 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:24.172 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:24.172 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:24.172 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:24.173 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.173 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:24.173 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_pgb4jYZ1RI-iMrOEaxNHWBT", 2024-08-18T15:45:24.173 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:24.173 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.173 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:24.173 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:24.173 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.173 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.173 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.173 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:24.173 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995848.902372_183.1", 2024-08-18T15:45:24.173 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:24.173 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-45", 2024-08-18T15:45:24.174 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:44:08.902372001Z", 2024-08-18T15:45:24.174 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:24.174 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:24.174 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:24.174 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:24.174 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.174 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:24.174 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_pgb4jYZ1RI-iMrOEaxNHWBT", 2024-08-18T15:45:24.174 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:24.174 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.175 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:24.175 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:24.175 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.175 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.175 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.175 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:24.175 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995848.923600_184.1", 2024-08-18T15:45:24.176 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:24.176 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-46:10090ccf-aad6-416d-9775-df8b7b27345a.5181.2", 2024-08-18T15:45:24.177 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:44:08.923600131Z", 2024-08-18T15:45:24.177 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:24.177 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:24.177 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:24.177 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:24.177 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.177 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:24.177 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_A5TS04I6RYCCfDae8aKKtI7", 2024-08-18T15:45:24.177 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:24.177 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.178 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:24.178 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:24.178 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.178 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.178 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.178 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:24.178 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995848.926669_185.1", 2024-08-18T15:45:24.178 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:24.178 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-46:10090ccf-aad6-416d-9775-df8b7b27345a.5181.2", 2024-08-18T15:45:24.178 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:44:08.926669459Z", 2024-08-18T15:45:24.178 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:24.178 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:24.178 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:24.179 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:24.179 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.179 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:24.179 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_A5TS04I6RYCCfDae8aKKtI7", 2024-08-18T15:45:24.179 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:24.179 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.179 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:24.179 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:24.179 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.179 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.179 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.179 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:24.179 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995848.927747_186.1", 2024-08-18T15:45:24.179 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:24.180 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-46", 2024-08-18T15:45:24.180 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:44:08.927747783Z", 2024-08-18T15:45:24.180 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:24.180 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:24.180 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:24.180 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:24.180 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.180 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:24.180 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_wxbKicm28WImmgDM0_RhnfU", 2024-08-18T15:45:24.180 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:24.180 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.180 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:24.180 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:24.180 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.181 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.181 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.181 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:24.181 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995848.930494_187.1", 2024-08-18T15:45:24.181 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:24.181 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-46", 2024-08-18T15:45:24.181 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:44:08.930494922Z", 2024-08-18T15:45:24.181 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:24.182 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:24.182 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:24.182 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:24.182 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.182 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:24.182 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_wxbKicm28WImmgDM0_RhnfU", 2024-08-18T15:45:24.182 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:24.182 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.182 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:24.183 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:24.183 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.183 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.183 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.184 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T15:45:24.184 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '64309209-7567-46e5-8881-65783d84cf3c', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:45:24.184 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '64309209-7567-46e5-8881-65783d84cf3c', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:45:24.185 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 64309209-7567-46e5-8881-65783d84cf3c --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:45:24.239 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:45:24.239 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:45:24.273 INFO:teuthology.orchestra.run.smithi193.stdout:[ 2024-08-18T15:45:24.273 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:24.273 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995848.895198_180.1", 2024-08-18T15:45:24.273 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:24.273 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-45:10090ccf-aad6-416d-9775-df8b7b27345a.5181.1", 2024-08-18T15:45:24.273 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:44:08.895198309Z", 2024-08-18T15:45:24.273 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:24.273 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:24.273 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:24.273 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:24.274 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:24.274 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:24.274 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_6PVK0GlJ0EocWsIkwIHUQja", 2024-08-18T15:45:24.274 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:24.274 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:24.274 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:24.274 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:24.274 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:24.274 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:24.274 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:24.274 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:24.274 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995848.898448_181.1", 2024-08-18T15:45:24.275 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:24.275 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-45:10090ccf-aad6-416d-9775-df8b7b27345a.5181.1", 2024-08-18T15:45:24.275 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:44:08.898448461Z", 2024-08-18T15:45:24.275 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:24.275 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:24.275 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:24.275 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:24.275 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:24.275 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:24.275 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_6PVK0GlJ0EocWsIkwIHUQja", 2024-08-18T15:45:24.275 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:24.275 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:24.275 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:24.276 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:24.276 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:24.276 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:24.276 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:24.276 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:24.276 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995848.899565_182.1", 2024-08-18T15:45:24.276 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:24.276 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-45", 2024-08-18T15:45:24.276 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:44:08.899565177Z", 2024-08-18T15:45:24.276 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:24.276 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:24.276 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:24.276 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:24.276 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:24.277 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:24.277 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_pgb4jYZ1RI-iMrOEaxNHWBT", 2024-08-18T15:45:24.277 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:24.277 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:24.277 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:24.277 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:24.277 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:24.277 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:24.277 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:24.277 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:24.277 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995848.902372_183.1", 2024-08-18T15:45:24.277 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:24.277 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-45", 2024-08-18T15:45:24.277 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:44:08.902372001Z", 2024-08-18T15:45:24.278 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:24.278 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:24.278 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:24.278 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:24.278 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:24.278 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:24.278 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_pgb4jYZ1RI-iMrOEaxNHWBT", 2024-08-18T15:45:24.278 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:24.278 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:24.278 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:24.278 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:24.278 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:24.278 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:24.279 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:24.279 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:24.279 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995848.923600_184.1", 2024-08-18T15:45:24.279 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:24.279 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-46:10090ccf-aad6-416d-9775-df8b7b27345a.5181.2", 2024-08-18T15:45:24.279 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:44:08.923600131Z", 2024-08-18T15:45:24.279 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:24.279 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:24.279 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:24.279 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:24.279 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:24.279 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:24.279 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_A5TS04I6RYCCfDae8aKKtI7", 2024-08-18T15:45:24.279 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:24.280 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:24.280 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:24.280 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:24.280 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:24.280 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:24.280 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:24.280 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:24.280 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995848.926669_185.1", 2024-08-18T15:45:24.280 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:24.280 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-46:10090ccf-aad6-416d-9775-df8b7b27345a.5181.2", 2024-08-18T15:45:24.280 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:44:08.926669459Z", 2024-08-18T15:45:24.280 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:24.280 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:24.281 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:24.281 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:24.281 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:24.281 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:24.281 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_A5TS04I6RYCCfDae8aKKtI7", 2024-08-18T15:45:24.281 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:24.281 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:24.281 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:24.281 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:24.281 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:24.281 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:24.282 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:24.282 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:24.282 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995848.927747_186.1", 2024-08-18T15:45:24.282 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:24.282 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-46", 2024-08-18T15:45:24.282 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:44:08.927747783Z", 2024-08-18T15:45:24.282 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:24.282 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:24.282 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:24.282 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:24.282 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:24.283 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:24.283 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_wxbKicm28WImmgDM0_RhnfU", 2024-08-18T15:45:24.283 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:24.283 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:24.283 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:24.283 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:24.283 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:24.283 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:24.283 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:24.283 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:24.283 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995848.930494_187.1", 2024-08-18T15:45:24.283 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:24.283 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-46", 2024-08-18T15:45:24.284 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:44:08.930494922Z", 2024-08-18T15:45:24.284 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:24.284 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:24.284 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:24.284 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:24.284 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:24.284 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:24.285 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_wxbKicm28WImmgDM0_RhnfU", 2024-08-18T15:45:24.285 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:24.285 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:24.285 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:24.285 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:24.286 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:24.286 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:24.286 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:24.286 INFO:teuthology.orchestra.run.smithi193.stdout:] 2024-08-18T15:45:24.287 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '64309209-7567-46e5-8881-65783d84cf3c', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:45:24.288 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '64309209-7567-46e5-8881-65783d84cf3c', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:45:24.288 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 64309209-7567-46e5-8881-65783d84cf3c --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:45:24.334 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:45:24.334 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:45:24.364 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T15:45:24.364 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:24.364 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995848.895198_180.1", 2024-08-18T15:45:24.364 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:24.364 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-45:10090ccf-aad6-416d-9775-df8b7b27345a.5181.1", 2024-08-18T15:45:24.364 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:44:08.895198309Z", 2024-08-18T15:45:24.364 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:24.364 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:24.364 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:24.364 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:24.364 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.365 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:24.365 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_6PVK0GlJ0EocWsIkwIHUQja", 2024-08-18T15:45:24.365 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:24.365 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.365 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:24.365 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:24.365 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.365 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.365 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.365 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:24.365 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995848.898448_181.1", 2024-08-18T15:45:24.365 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:24.365 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-45:10090ccf-aad6-416d-9775-df8b7b27345a.5181.1", 2024-08-18T15:45:24.366 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:44:08.898448461Z", 2024-08-18T15:45:24.366 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:24.366 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:24.366 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:24.366 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:24.366 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.366 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:24.366 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_6PVK0GlJ0EocWsIkwIHUQja", 2024-08-18T15:45:24.366 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:24.366 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.366 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:24.366 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:24.367 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.367 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.367 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.367 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:24.367 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995848.899565_182.1", 2024-08-18T15:45:24.367 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:24.367 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-45", 2024-08-18T15:45:24.367 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:44:08.899565177Z", 2024-08-18T15:45:24.367 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:24.367 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:24.367 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:24.367 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:24.367 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.368 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:24.368 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_pgb4jYZ1RI-iMrOEaxNHWBT", 2024-08-18T15:45:24.368 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:24.368 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.368 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:24.368 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:24.368 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.368 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.368 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.368 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:24.368 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995848.902372_183.1", 2024-08-18T15:45:24.368 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:24.368 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-45", 2024-08-18T15:45:24.369 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:44:08.902372001Z", 2024-08-18T15:45:24.369 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:24.369 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:24.369 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:24.369 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:24.369 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.369 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:24.369 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_pgb4jYZ1RI-iMrOEaxNHWBT", 2024-08-18T15:45:24.369 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:24.369 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.369 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:24.369 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:24.369 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.370 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.370 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.370 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:24.370 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995848.923600_184.1", 2024-08-18T15:45:24.370 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:24.370 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-46:10090ccf-aad6-416d-9775-df8b7b27345a.5181.2", 2024-08-18T15:45:24.370 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:44:08.923600131Z", 2024-08-18T15:45:24.370 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:24.370 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:24.370 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:24.370 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:24.370 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.370 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:24.371 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_A5TS04I6RYCCfDae8aKKtI7", 2024-08-18T15:45:24.371 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:24.371 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.371 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:24.371 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:24.371 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.371 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.371 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.371 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:24.371 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995848.926669_185.1", 2024-08-18T15:45:24.371 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:24.371 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-46:10090ccf-aad6-416d-9775-df8b7b27345a.5181.2", 2024-08-18T15:45:24.371 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:44:08.926669459Z", 2024-08-18T15:45:24.372 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:24.372 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:24.372 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:24.372 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:24.372 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.372 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:24.372 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_A5TS04I6RYCCfDae8aKKtI7", 2024-08-18T15:45:24.372 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:24.372 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.372 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:24.372 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:24.372 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.372 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.372 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.373 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:24.373 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995848.927747_186.1", 2024-08-18T15:45:24.373 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:24.373 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-46", 2024-08-18T15:45:24.373 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:44:08.927747783Z", 2024-08-18T15:45:24.373 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:24.373 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:24.373 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:24.373 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:24.373 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.373 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:24.373 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_wxbKicm28WImmgDM0_RhnfU", 2024-08-18T15:45:24.373 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:24.374 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.374 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:24.374 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:24.374 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.374 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.374 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.374 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:24.374 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995848.930494_187.1", 2024-08-18T15:45:24.374 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:24.374 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-46", 2024-08-18T15:45:24.374 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:44:08.930494922Z", 2024-08-18T15:45:24.374 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:24.374 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:24.375 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:24.375 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:24.375 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.375 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:24.375 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_wxbKicm28WImmgDM0_RhnfU", 2024-08-18T15:45:24.375 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:24.375 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.375 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:24.376 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:24.376 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.376 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.376 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.376 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T15:45:24.377 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '01050e30-179b-43be-bdbb-3bb23cf37ae8', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:45:24.377 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '01050e30-179b-43be-bdbb-3bb23cf37ae8', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:45:24.377 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 01050e30-179b-43be-bdbb-3bb23cf37ae8 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:45:24.422 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:45:24.423 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:45:24.452 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T15:45:24.453 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:24.453 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995914.451454_248.1", 2024-08-18T15:45:24.453 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:24.453 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-47:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.1", 2024-08-18T15:45:24.453 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:45:14.451454387Z", 2024-08-18T15:45:24.453 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:24.453 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:24.453 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:24.453 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:24.453 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.453 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:24.453 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_u930aIHTanezFJHJvQLhoPf", 2024-08-18T15:45:24.454 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:24.454 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.454 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:24.454 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:24.454 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.454 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.454 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.454 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:24.454 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995914.454645_249.1", 2024-08-18T15:45:24.454 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:24.454 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-47:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.1", 2024-08-18T15:45:24.454 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:45:14.454645743Z", 2024-08-18T15:45:24.455 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:24.455 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:24.455 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:24.455 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:24.455 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.455 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:24.455 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_u930aIHTanezFJHJvQLhoPf", 2024-08-18T15:45:24.455 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:24.455 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.455 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:24.455 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:24.455 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.455 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.456 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.456 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:24.456 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995914.455787_250.1", 2024-08-18T15:45:24.456 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:24.456 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-47", 2024-08-18T15:45:24.456 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:45:14.455787078Z", 2024-08-18T15:45:24.456 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:24.456 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:24.456 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:24.456 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:24.456 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.456 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:24.457 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_llmulx2a1WfhYTvdzqRxi0j", 2024-08-18T15:45:24.457 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:24.457 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.457 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:24.457 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:24.457 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.457 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.457 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.457 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:24.457 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995914.458819_251.1", 2024-08-18T15:45:24.457 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:24.457 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-47", 2024-08-18T15:45:24.457 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:45:14.458819520Z", 2024-08-18T15:45:24.458 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:24.458 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:24.458 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:24.458 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:24.458 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.458 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:24.458 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_llmulx2a1WfhYTvdzqRxi0j", 2024-08-18T15:45:24.458 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:24.458 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.458 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:24.458 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:24.458 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.458 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.459 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.459 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:24.459 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995914.477398_252.1", 2024-08-18T15:45:24.459 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:24.459 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-48:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.2", 2024-08-18T15:45:24.459 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:45:14.477398035Z", 2024-08-18T15:45:24.459 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:24.459 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:24.459 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:24.459 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:24.459 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.459 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:24.460 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_uNiDh2QOdGKaDDLvOsKAWg-", 2024-08-18T15:45:24.460 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:24.460 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.460 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:24.460 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:24.460 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.460 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.460 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.460 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:24.460 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995914.480590_253.1", 2024-08-18T15:45:24.460 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:24.460 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-48:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.2", 2024-08-18T15:45:24.460 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:45:14.480590533Z", 2024-08-18T15:45:24.460 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:24.461 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:24.461 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:24.461 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:24.461 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.461 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:24.461 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_uNiDh2QOdGKaDDLvOsKAWg-", 2024-08-18T15:45:24.461 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:24.461 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.461 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:24.461 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:24.461 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.461 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.461 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.462 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:24.462 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995914.481810_254.1", 2024-08-18T15:45:24.462 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:24.462 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-48", 2024-08-18T15:45:24.462 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:45:14.481810281Z", 2024-08-18T15:45:24.462 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:24.462 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:24.462 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:24.462 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:24.462 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.462 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:24.462 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_IL4cXsr2vfmg_MPq8aMuhvE", 2024-08-18T15:45:24.462 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:24.463 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.463 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:24.463 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:24.463 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.463 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.463 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.463 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:24.463 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995914.484839_255.1", 2024-08-18T15:45:24.463 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:24.463 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-48", 2024-08-18T15:45:24.463 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:45:14.484839991Z", 2024-08-18T15:45:24.463 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:24.463 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:24.463 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:24.464 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:24.464 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.464 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:24.464 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_IL4cXsr2vfmg_MPq8aMuhvE", 2024-08-18T15:45:24.464 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:24.464 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.464 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:24.465 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:24.465 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.465 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.465 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.465 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T15:45:24.466 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '01050e30-179b-43be-bdbb-3bb23cf37ae8', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:45:24.466 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '01050e30-179b-43be-bdbb-3bb23cf37ae8', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:45:24.466 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 01050e30-179b-43be-bdbb-3bb23cf37ae8 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:45:24.515 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:45:24.515 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:45:24.546 INFO:teuthology.orchestra.run.smithi193.stdout:[ 2024-08-18T15:45:24.547 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:24.547 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995914.451454_248.1", 2024-08-18T15:45:24.547 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:24.547 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-47:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.1", 2024-08-18T15:45:24.547 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:45:14.451454387Z", 2024-08-18T15:45:24.547 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:24.547 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:24.547 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:24.547 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:24.547 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:24.547 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:24.547 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_u930aIHTanezFJHJvQLhoPf", 2024-08-18T15:45:24.548 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:24.548 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:24.548 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:24.548 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:24.548 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:24.548 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:24.548 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:24.548 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:24.548 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995914.454645_249.1", 2024-08-18T15:45:24.548 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:24.548 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-47:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.1", 2024-08-18T15:45:24.548 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:45:14.454645743Z", 2024-08-18T15:45:24.548 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:24.549 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:24.549 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:24.549 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:24.549 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:24.549 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:24.549 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_u930aIHTanezFJHJvQLhoPf", 2024-08-18T15:45:24.549 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:24.549 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:24.549 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:24.549 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:24.549 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:24.549 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:24.549 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:24.550 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:24.550 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995914.455787_250.1", 2024-08-18T15:45:24.550 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:24.550 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-47", 2024-08-18T15:45:24.550 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:45:14.455787078Z", 2024-08-18T15:45:24.550 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:24.550 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:24.550 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:24.550 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:24.550 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:24.550 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:24.550 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_llmulx2a1WfhYTvdzqRxi0j", 2024-08-18T15:45:24.550 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:24.551 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:24.551 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:24.551 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:24.551 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:24.551 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:24.551 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:24.551 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:24.551 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995914.458819_251.1", 2024-08-18T15:45:24.551 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:24.551 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-47", 2024-08-18T15:45:24.551 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:45:14.458819520Z", 2024-08-18T15:45:24.551 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:24.551 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:24.551 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:24.552 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:24.552 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:24.552 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:24.552 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_llmulx2a1WfhYTvdzqRxi0j", 2024-08-18T15:45:24.552 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:24.552 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:24.552 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:24.552 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:24.553 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:24.553 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:24.553 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:24.553 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:24.553 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995914.477398_252.1", 2024-08-18T15:45:24.553 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:24.553 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-48:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.2", 2024-08-18T15:45:24.553 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:45:14.477398035Z", 2024-08-18T15:45:24.553 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:24.553 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:24.553 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:24.553 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:24.553 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:24.553 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:24.554 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_uNiDh2QOdGKaDDLvOsKAWg-", 2024-08-18T15:45:24.554 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:24.554 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:24.554 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:24.554 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:24.554 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:24.554 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:24.554 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:24.554 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:24.554 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995914.480590_253.1", 2024-08-18T15:45:24.554 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket.instance", 2024-08-18T15:45:24.554 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-48:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.2", 2024-08-18T15:45:24.554 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:45:14.480590533Z", 2024-08-18T15:45:24.555 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:24.555 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:24.555 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:24.555 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:24.555 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:24.555 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:24.555 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_uNiDh2QOdGKaDDLvOsKAWg-", 2024-08-18T15:45:24.555 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:24.555 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:24.555 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:24.555 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:24.555 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:24.555 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:24.556 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:24.556 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:24.556 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995914.481810_254.1", 2024-08-18T15:45:24.556 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:24.556 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-48", 2024-08-18T15:45:24.556 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:45:14.481810281Z", 2024-08-18T15:45:24.556 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:24.556 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:24.556 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:24.556 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:24.556 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:24.556 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:24.556 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_IL4cXsr2vfmg_MPq8aMuhvE", 2024-08-18T15:45:24.556 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:24.557 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:24.557 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:24.557 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "write" 2024-08-18T15:45:24.557 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:24.557 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:24.557 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:24.557 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:24.557 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "1_1723995914.484839_255.1", 2024-08-18T15:45:24.557 INFO:teuthology.orchestra.run.smithi193.stdout: "section": "bucket", 2024-08-18T15:45:24.557 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "kibgmb-48", 2024-08-18T15:45:24.557 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:45:14.484839991Z", 2024-08-18T15:45:24.557 INFO:teuthology.orchestra.run.smithi193.stdout: "data": { 2024-08-18T15:45:24.557 INFO:teuthology.orchestra.run.smithi193.stdout: "read_version": { 2024-08-18T15:45:24.558 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "", 2024-08-18T15:45:24.558 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 0 2024-08-18T15:45:24.558 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:24.558 INFO:teuthology.orchestra.run.smithi193.stdout: "write_version": { 2024-08-18T15:45:24.558 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_IL4cXsr2vfmg_MPq8aMuhvE", 2024-08-18T15:45:24.558 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": 1 2024-08-18T15:45:24.558 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:24.558 INFO:teuthology.orchestra.run.smithi193.stdout: "status": { 2024-08-18T15:45:24.559 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "complete" 2024-08-18T15:45:24.559 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:24.559 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:24.559 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:24.559 INFO:teuthology.orchestra.run.smithi193.stdout:] 2024-08-18T15:45:24.560 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '01050e30-179b-43be-bdbb-3bb23cf37ae8', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:45:24.560 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '01050e30-179b-43be-bdbb-3bb23cf37ae8', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:45:24.560 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 01050e30-179b-43be-bdbb-3bb23cf37ae8 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:45:24.606 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:45:24.607 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:45:24.638 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T15:45:24.638 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:24.638 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995914.451454_248.1", 2024-08-18T15:45:24.638 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:24.638 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-47:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.1", 2024-08-18T15:45:24.638 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:45:14.451454387Z", 2024-08-18T15:45:24.638 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:24.638 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:24.638 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:24.638 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:24.638 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.639 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:24.639 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_u930aIHTanezFJHJvQLhoPf", 2024-08-18T15:45:24.639 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:24.639 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.639 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:24.639 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:24.639 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.639 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.639 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.640 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:24.640 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995914.454645_249.1", 2024-08-18T15:45:24.640 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:24.640 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-47:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.1", 2024-08-18T15:45:24.640 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:45:14.454645743Z", 2024-08-18T15:45:24.640 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:24.640 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:24.640 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:24.640 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:24.640 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.640 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:24.640 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_u930aIHTanezFJHJvQLhoPf", 2024-08-18T15:45:24.640 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:24.640 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.641 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:24.641 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:24.641 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.641 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.641 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.641 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:24.641 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995914.455787_250.1", 2024-08-18T15:45:24.641 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:24.641 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-47", 2024-08-18T15:45:24.641 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:45:14.455787078Z", 2024-08-18T15:45:24.641 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:24.641 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:24.641 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:24.641 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:24.642 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.642 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:24.642 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_llmulx2a1WfhYTvdzqRxi0j", 2024-08-18T15:45:24.642 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:24.642 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.642 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:24.642 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:24.643 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.643 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.643 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.643 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:24.643 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995914.458819_251.1", 2024-08-18T15:45:24.643 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:24.644 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-47", 2024-08-18T15:45:24.644 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:45:14.458819520Z", 2024-08-18T15:45:24.644 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:24.644 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:24.644 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:24.644 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:24.644 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.644 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:24.644 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_llmulx2a1WfhYTvdzqRxi0j", 2024-08-18T15:45:24.644 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:24.644 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.644 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:24.644 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:24.644 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.645 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.645 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.645 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:24.645 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995914.477398_252.1", 2024-08-18T15:45:24.645 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:24.645 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-48:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.2", 2024-08-18T15:45:24.645 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:45:14.477398035Z", 2024-08-18T15:45:24.645 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:24.646 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:24.646 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:24.646 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:24.646 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.646 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:24.646 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_uNiDh2QOdGKaDDLvOsKAWg-", 2024-08-18T15:45:24.646 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:24.646 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.646 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:24.646 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:24.647 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.647 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.647 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.647 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:24.647 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995914.480590_253.1", 2024-08-18T15:45:24.647 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket.instance", 2024-08-18T15:45:24.647 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-48:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.2", 2024-08-18T15:45:24.647 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:45:14.480590533Z", 2024-08-18T15:45:24.647 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:24.647 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:24.647 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:24.647 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:24.647 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.647 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:24.648 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_uNiDh2QOdGKaDDLvOsKAWg-", 2024-08-18T15:45:24.648 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:24.648 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.648 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:24.648 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:24.648 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.648 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.648 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.648 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:24.648 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995914.481810_254.1", 2024-08-18T15:45:24.648 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:24.648 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-48", 2024-08-18T15:45:24.649 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:45:14.481810281Z", 2024-08-18T15:45:24.649 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:24.649 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:24.649 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:24.649 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:24.649 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.649 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:24.649 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_IL4cXsr2vfmg_MPq8aMuhvE", 2024-08-18T15:45:24.649 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:24.649 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.650 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:24.650 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "write" 2024-08-18T15:45:24.650 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.650 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.650 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.650 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:24.650 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "1_1723995914.484839_255.1", 2024-08-18T15:45:24.650 INFO:teuthology.orchestra.run.smithi134.stdout: "section": "bucket", 2024-08-18T15:45:24.650 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "kibgmb-48", 2024-08-18T15:45:24.650 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:45:14.484839991Z", 2024-08-18T15:45:24.650 INFO:teuthology.orchestra.run.smithi134.stdout: "data": { 2024-08-18T15:45:24.650 INFO:teuthology.orchestra.run.smithi134.stdout: "read_version": { 2024-08-18T15:45:24.650 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "", 2024-08-18T15:45:24.651 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 0 2024-08-18T15:45:24.651 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.651 INFO:teuthology.orchestra.run.smithi134.stdout: "write_version": { 2024-08-18T15:45:24.651 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_IL4cXsr2vfmg_MPq8aMuhvE", 2024-08-18T15:45:24.651 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": 1 2024-08-18T15:45:24.651 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:24.651 INFO:teuthology.orchestra.run.smithi134.stdout: "status": { 2024-08-18T15:45:24.652 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "complete" 2024-08-18T15:45:24.652 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.652 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.652 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:24.652 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T15:45:24.653 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:45:24.653 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:45:24.653 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T15:45:24.696 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:45:24.696 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:45:24.754 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:45:24.754 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:45:24.754 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog autotrim --cluster c2 --debug-rgw 0 --debug-ms 0 2024-08-18T15:45:24.798 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:45:24.798 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:45:24.943 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:45:24.943 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:45:24.943 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T15:45:24.987 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:45:24.987 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:45:25.033 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'b0d74758-1871-4454-bc0f-e3e4b6c30105', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:45:25.033 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'b0d74758-1871-4454-bc0f-e3e4b6c30105', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:45:25.034 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period b0d74758-1871-4454-bc0f-e3e4b6c30105 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:45:25.120 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:45:25.121 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:45:25.160 INFO:teuthology.orchestra.run.smithi134.stdout:[] 2024-08-18T15:45:25.162 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'b0d74758-1871-4454-bc0f-e3e4b6c30105', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:45:25.162 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'b0d74758-1871-4454-bc0f-e3e4b6c30105', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:45:25.162 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period b0d74758-1871-4454-bc0f-e3e4b6c30105 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:45:25.209 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:45:25.210 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:45:25.251 INFO:teuthology.orchestra.run.smithi193.stdout:[] 2024-08-18T15:45:25.253 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'b0d74758-1871-4454-bc0f-e3e4b6c30105', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:45:25.253 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'b0d74758-1871-4454-bc0f-e3e4b6c30105', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:45:25.253 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period b0d74758-1871-4454-bc0f-e3e4b6c30105 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:45:25.299 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:45:25.299 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:45:25.339 INFO:teuthology.orchestra.run.smithi134.stdout:[] 2024-08-18T15:45:25.341 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '64309209-7567-46e5-8881-65783d84cf3c', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:45:25.341 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '64309209-7567-46e5-8881-65783d84cf3c', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:45:25.342 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 64309209-7567-46e5-8881-65783d84cf3c --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:45:25.386 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:45:25.387 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:45:25.426 INFO:teuthology.orchestra.run.smithi134.stdout:[] 2024-08-18T15:45:25.428 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '64309209-7567-46e5-8881-65783d84cf3c', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:45:25.428 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '64309209-7567-46e5-8881-65783d84cf3c', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:45:25.428 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 64309209-7567-46e5-8881-65783d84cf3c --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:45:25.474 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:45:25.475 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:45:25.516 INFO:teuthology.orchestra.run.smithi193.stdout:[] 2024-08-18T15:45:25.518 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '64309209-7567-46e5-8881-65783d84cf3c', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:45:25.518 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '64309209-7567-46e5-8881-65783d84cf3c', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:45:25.518 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 64309209-7567-46e5-8881-65783d84cf3c --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:45:25.564 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:45:25.565 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:45:25.604 INFO:teuthology.orchestra.run.smithi134.stdout:[] 2024-08-18T15:45:25.606 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '01050e30-179b-43be-bdbb-3bb23cf37ae8', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:45:25.606 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '01050e30-179b-43be-bdbb-3bb23cf37ae8', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:45:25.606 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 01050e30-179b-43be-bdbb-3bb23cf37ae8 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:45:25.650 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:45:25.651 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:45:25.695 INFO:teuthology.orchestra.run.smithi134.stdout:[] 2024-08-18T15:45:25.695 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '01050e30-179b-43be-bdbb-3bb23cf37ae8', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:45:25.695 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '01050e30-179b-43be-bdbb-3bb23cf37ae8', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:45:25.695 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 01050e30-179b-43be-bdbb-3bb23cf37ae8 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:45:25.742 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:45:25.742 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:45:25.783 INFO:teuthology.orchestra.run.smithi193.stdout:[] 2024-08-18T15:45:25.784 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '01050e30-179b-43be-bdbb-3bb23cf37ae8', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:45:25.785 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '01050e30-179b-43be-bdbb-3bb23cf37ae8', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:45:25.785 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 01050e30-179b-43be-bdbb-3bb23cf37ae8 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:45:25.831 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:45:25.831 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:45:25.871 INFO:teuthology.orchestra.run.smithi134.stdout:[] 2024-08-18T15:45:25.873 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multi_period_incremental_sync ... ok 2024-08-18T15:45:25.873 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=kibgmb-49 2024-08-18T15:45:25.887 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=kibgmb-50 2024-08-18T15:45:25.910 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=kibgmb-51 2024-08-18T15:45:25.999 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:45:25.999 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:45:25.999 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:45:26.049 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:45:26.049 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:45:26.079 INFO:teuthology.orchestra.run.smithi134.stderr:No --period given, using current period=01050e30-179b-43be-bdbb-3bb23cf37ae8 2024-08-18T15:45:26.091 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T15:45:26.091 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:26.092 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T15:45:26.092 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T15:45:26.092 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:26.092 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:26.092 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T15:45:26.092 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T15:45:26.092 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:26.093 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:26.093 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T15:45:26.093 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T15:45:26.093 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:26.093 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:26.093 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723995925.925075_273.1", 2024-08-18T15:45:26.093 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:45:25.925075Z" 2024-08-18T15:45:26.093 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:26.093 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T15:45:26.094 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-08-18T15:45:26.094 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:45:26.094 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:45:26.095 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:45:26.144 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:45:26.144 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:45:26.172 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:45:26.172+0000 7f1c3240f880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:45:26.173 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:45:26.173+0000 7f1c3240f880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:45:26.185 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:45:26.185 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:45:26.185 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:45:26.185 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:45:26.185 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:45:26.185 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T15:45:26.186 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:45:26.186 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:26.186 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:45:26.186 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:26.186 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:45:26.186 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:45:26.186 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:45:26.186 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:45:26.186 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:45:26.186 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:45:26.186 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:45:26.186 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:45:26.186 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:45:26.187 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:26.187 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:26.187 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:26.187 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:45:26.187 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:45:26.187 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:45:26.187 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:45:26.187 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:45:26.187 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:45:26.187 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:45:26.187 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:45:26.187 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:45:26.187 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:26.187 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:26.188 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:26.188 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:45:26.188 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:45:26.188 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:45:26.188 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:45:26.188 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:45:26.188 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:45:26.188 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:45:26.188 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:45:26.188 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:45:26.188 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:26.188 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:26.188 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:26.188 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:45:26.189 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:45:26.189 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:45:26.189 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723995925.880653_261.1", 2024-08-18T15:45:26.189 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:45:26.189 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:45:26.189 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:45:26.189 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:45:25.880653Z", 2024-08-18T15:45:26.189 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:45:26.189 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:26.189 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:26.189 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:45:26.189 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:26.189 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:45:26.189 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:45:26.189 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:45:26.190 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:26.190 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:45:26Z" 2024-08-18T15:45:26.190 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:45:26.191 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1723995925.925075_273.1 target=1_1723995925.880653_261.1 2024-08-18T15:45:31.191 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:45:31.192 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:45:31.192 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:45:31.247 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:45:31.247 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:45:31.279 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:45:31.278+0000 7faf8ff42880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:45:31.280 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:45:31.279+0000 7faf8ff42880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:45:31.293 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:45:31.293 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:45:31.294 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:45:31.294 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:45:31.294 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:45:31.294 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T15:45:31.294 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:45:31.294 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:31.294 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:45:31.294 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:31.294 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:45:31.294 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:45:31.294 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:45:31.294 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:45:31.295 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:45:31.295 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:45:31.295 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:45:31.295 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:45:31.295 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:45:31.295 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:31.295 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:31.295 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:31.295 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:45:31.295 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:45:31.295 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:45:31.295 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:45:31.295 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:45:31.296 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:45:31.296 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:45:31.296 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:45:31.296 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:45:31.296 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:31.296 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:31.296 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:31.296 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:45:31.296 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:45:31.296 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:45:31.296 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:45:31.296 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:45:31.296 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:45:31.297 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:45:31.297 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:45:31.297 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:45:31.297 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:31.297 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:31.297 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:31.297 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:45:31.297 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:45:31.297 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:45:31.297 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723995925.925075_273.1", 2024-08-18T15:45:31.297 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:45:31.297 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:45:31.297 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:45:31.298 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:45:25.925075Z", 2024-08-18T15:45:31.298 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:45:31.298 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:31.298 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:31.298 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:45:31.298 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:31.298 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:45:31.298 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:45:31.298 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:45:31.298 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:31.298 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:45:31Z" 2024-08-18T15:45:31.298 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:45:31.299 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-08-18T15:45:31.299 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-08-18T15:45:31.300 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:45:31.300 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:45:31.300 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:45:31.355 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:45:31.355 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:45:31.386 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:45:31.386+0000 7f99b6c0f880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:45:31.387 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:45:31.387+0000 7f99b6c0f880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:45:31.400 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T15:45:31.400 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T15:45:31.400 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T15:45:31.400 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T15:45:31.400 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T15:45:31.400 INFO:teuthology.orchestra.run.smithi134.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T15:45:31.400 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T15:45:31.400 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:31.401 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T15:45:31.401 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:31.401 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T15:45:31.401 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:45:31.401 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:45:31.401 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:45:31.401 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:45:31.401 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 1, 2024-08-18T15:45:31.401 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:45:31.401 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:45:31.401 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:45:31.401 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:31.401 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:31.402 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:31.402 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T15:45:31.402 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:45:31.402 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:45:31.402 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T15:45:31.402 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:45:31.402 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:45:31.402 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:45:31.402 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T15:45:31.402 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:45:31.402 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:31.402 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:31.402 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:31.403 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T15:45:31.403 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:45:31.403 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:45:31.403 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:45:31.403 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:45:31.403 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:45:31.403 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:45:31.403 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:45:31.403 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:45:31.403 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:31.403 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:31.403 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:31.403 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T15:45:31.403 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:45:31.404 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:45:31.404 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723995925.925075_273.1", 2024-08-18T15:45:31.404 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:45:31.404 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:45:31.404 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:45:31.404 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:45:25.925075Z", 2024-08-18T15:45:31.404 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T15:45:31.404 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:31.404 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:31.405 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T15:45:31.405 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:31.405 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T15:45:31.405 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 1, 2024-08-18T15:45:31.405 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 1 2024-08-18T15:45:31.405 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:31.405 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T15:45:31Z" 2024-08-18T15:45:31.405 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T15:45:31.406 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-08-18T15:45:31.406 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:45:31.406 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:45:31.406 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:45:31.454 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:45:31.454 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:45:31.501 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T15:45:31.501 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:31.501 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000003622", 2024-08-18T15:45:31.501 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:42:57.659462Z" 2024-08-18T15:45:31.501 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:31.502 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:31.502 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000004037", 2024-08-18T15:45:31.502 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:32:51.378244Z" 2024-08-18T15:45:31.502 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:31.502 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:31.502 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000007421", 2024-08-18T15:45:31.502 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:45:25.940487Z" 2024-08-18T15:45:31.502 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:31.502 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:31.502 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000008126", 2024-08-18T15:45:31.502 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:35:25.388487Z" 2024-08-18T15:45:31.502 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:31.503 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T15:45:31.503 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-08-18T15:45:31.504 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:45:31.504 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:45:31.504 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:45:31.558 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:45:31.558 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:45:31.603 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:45:31.603 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:45:31.603 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:45:31.603 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:45:31.603 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:45:31.603 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 8935070562734594544 2024-08-18T15:45:31.603 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:31.603 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:45:31.604 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:31.604 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:45:31.604 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:45:31.604 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T15:45:31.604 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000003622", 2024-08-18T15:45:31.604 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:45:31.604 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:45:31.604 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:45:31.604 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:57.659462Z" 2024-08-18T15:45:31.604 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:31.604 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:31.604 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:31.604 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:45:31.605 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:45:31.605 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T15:45:31.605 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000004037", 2024-08-18T15:45:31.605 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:45:31.605 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:45:31.605 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:45:31.605 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:32:51.378244Z" 2024-08-18T15:45:31.605 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:31.605 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:31.605 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:31.605 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:45:31.605 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:45:31.605 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T15:45:31.606 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000007280", 2024-08-18T15:45:31.606 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:45:31.606 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:45:31.606 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:45:31.606 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:01.406989Z" 2024-08-18T15:45:31.606 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:31.606 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:31.606 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:31.606 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:45:31.606 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:45:31.606 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T15:45:31.606 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000008126", 2024-08-18T15:45:31.606 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:45:31.607 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:45:31.607 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:45:31.607 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:35:25.388487Z" 2024-08-18T15:45:31.607 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:31.607 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:31.607 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:45:31.607 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:31.607 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:45:31.607 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 0, 2024-08-18T15:45:31.607 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 0 2024-08-18T15:45:31.607 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:31.607 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:45:31Z" 2024-08-18T15:45:31.608 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:45:31.609 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=2 master=00000000000000000000:00000000000000007421 target=00000000000000000000:00000000000000007280 2024-08-18T15:45:36.609 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:45:36.610 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:45:36.610 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:45:36.669 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:45:36.669 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:45:36.716 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:45:36.717 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:45:36.717 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:45:36.717 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:45:36.717 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:45:36.717 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 8935070562734594544 2024-08-18T15:45:36.717 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:36.718 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:45:36.718 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:36.718 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:45:36.718 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:45:36.718 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T15:45:36.718 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000003622", 2024-08-18T15:45:36.718 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:45:36.718 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:45:36.719 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:45:36.719 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:57.659462Z" 2024-08-18T15:45:36.719 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:36.719 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:36.719 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:36.719 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:45:36.719 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:45:36.719 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T15:45:36.720 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000004037", 2024-08-18T15:45:36.720 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:45:36.720 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:45:36.720 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:45:36.720 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:32:51.378244Z" 2024-08-18T15:45:36.720 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:36.720 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:36.720 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:36.720 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:45:36.720 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:45:36.721 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T15:45:36.721 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000007280", 2024-08-18T15:45:36.721 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:45:36.721 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:45:36.721 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:45:36.721 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:01.406989Z" 2024-08-18T15:45:36.721 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:36.721 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:36.721 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:36.721 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:45:36.721 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:45:36.721 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T15:45:36.721 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000008126", 2024-08-18T15:45:36.722 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:45:36.722 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:45:36.722 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:45:36.722 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:35:25.388487Z" 2024-08-18T15:45:36.722 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:36.722 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:36.722 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:45:36.722 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:36.722 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:45:36.722 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 0, 2024-08-18T15:45:36.722 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 0 2024-08-18T15:45:36.722 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:36.722 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:45:36Z" 2024-08-18T15:45:36.723 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:45:36.724 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=2 master=00000000000000000000:00000000000000007421 target=00000000000000000000:00000000000000007280 2024-08-18T15:45:41.725 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:45:41.725 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:45:41.725 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:45:41.781 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:45:41.782 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:45:41.829 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:45:41.829 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:45:41.829 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:45:41.829 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:45:41.829 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:45:41.829 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 8935070562734594544 2024-08-18T15:45:41.829 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:41.830 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:45:41.830 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:41.830 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:45:41.830 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:45:41.830 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T15:45:41.830 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000003622", 2024-08-18T15:45:41.830 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:45:41.830 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:45:41.830 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:45:41.830 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:57.659462Z" 2024-08-18T15:45:41.830 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:41.830 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:41.830 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:41.831 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:45:41.831 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:45:41.831 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T15:45:41.831 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000004037", 2024-08-18T15:45:41.831 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:45:41.831 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:45:41.831 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:45:41.831 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:32:51.378244Z" 2024-08-18T15:45:41.831 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:41.832 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:41.832 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:41.832 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:45:41.832 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:45:41.832 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T15:45:41.832 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000007280", 2024-08-18T15:45:41.832 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:45:41.832 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:45:41.832 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:45:41.832 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:01.406989Z" 2024-08-18T15:45:41.832 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:41.832 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:41.832 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:41.833 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:45:41.833 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:45:41.833 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T15:45:41.833 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000008126", 2024-08-18T15:45:41.833 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:45:41.833 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:45:41.833 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:45:41.833 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:35:25.388487Z" 2024-08-18T15:45:41.833 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:41.833 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:41.833 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:45:41.833 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:41.833 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:45:41.834 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 0, 2024-08-18T15:45:41.834 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 0 2024-08-18T15:45:41.834 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:41.834 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:45:41Z" 2024-08-18T15:45:41.834 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:45:41.835 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=2 master=00000000000000000000:00000000000000007421 target=00000000000000000000:00000000000000007280 2024-08-18T15:45:46.836 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:45:46.837 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:45:46.837 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:45:46.894 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:45:46.894 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:45:46.940 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:45:46.940 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:45:46.941 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:45:46.941 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:45:46.941 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:45:46.941 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 8935070562734594544 2024-08-18T15:45:46.941 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:46.941 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:45:46.941 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:46.941 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:45:46.942 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:45:46.942 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T15:45:46.942 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000003622", 2024-08-18T15:45:46.942 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:45:46.942 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:45:46.942 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:45:46.942 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:42:57.659462Z" 2024-08-18T15:45:46.942 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:46.942 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:46.942 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:46.942 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:45:46.942 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:45:46.942 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T15:45:46.942 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000004037", 2024-08-18T15:45:46.943 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:45:46.943 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:45:46.943 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:45:46.943 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:32:51.378244Z" 2024-08-18T15:45:46.943 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:46.943 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:46.943 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:46.943 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:45:46.943 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:45:46.943 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T15:45:46.943 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000007561", 2024-08-18T15:45:46.943 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:45:46.943 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:45:46.944 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:45:46.944 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:45:34.735914Z" 2024-08-18T15:45:46.944 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:46.944 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:46.944 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:46.944 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:45:46.944 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:45:46.944 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T15:45:46.944 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000008126", 2024-08-18T15:45:46.944 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:45:46.944 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:45:46.944 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:45:46.944 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:35:25.388487Z" 2024-08-18T15:45:46.944 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:46.945 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:46.945 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:45:46.945 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:46.945 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:45:46.945 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 0, 2024-08-18T15:45:46.945 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 0 2024-08-18T15:45:46.945 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:46.945 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:45:46Z" 2024-08-18T15:45:46.946 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:45:46.947 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-08-18T15:45:46.947 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:45:46.947 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:45:46.947 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:45:47.002 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:45:47.002 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:45:47.053 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T15:45:47.054 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:47.054 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000003622", 2024-08-18T15:45:47.054 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:42:57.659462Z" 2024-08-18T15:45:47.054 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:47.054 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:47.054 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000004037", 2024-08-18T15:45:47.054 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:32:51.378244Z" 2024-08-18T15:45:47.054 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:47.054 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:47.054 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000007701", 2024-08-18T15:45:47.055 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:45:45.805061Z" 2024-08-18T15:45:47.055 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:47.055 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:47.055 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000008126", 2024-08-18T15:45:47.055 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:35:25.388487Z" 2024-08-18T15:45:47.055 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:47.055 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T15:45:47.056 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2024-08-18T15:45:47.056 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:45:47.056 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:45:47.057 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:45:47.106 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:45:47.107 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:45:47.147 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T15:45:47.148 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T15:45:47.148 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T15:45:47.148 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T15:45:47.148 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T15:45:47.148 INFO:teuthology.orchestra.run.smithi134.stdout: "instance_id": 6908254841848037526 2024-08-18T15:45:47.148 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:47.148 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T15:45:47.148 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:47.148 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T15:45:47.148 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:45:47.148 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T15:45:47.148 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000003622", 2024-08-18T15:45:47.149 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:45:47.149 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:45:47.149 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:45:47.149 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:57.659462Z" 2024-08-18T15:45:47.149 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:47.149 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:47.149 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:47.149 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T15:45:47.149 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:45:47.149 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T15:45:47.149 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000004037", 2024-08-18T15:45:47.149 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:45:47.149 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:45:47.149 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:45:47.150 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:32:51.378244Z" 2024-08-18T15:45:47.150 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:47.150 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:47.150 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:47.150 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T15:45:47.150 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:45:47.150 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T15:45:47.150 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000007421", 2024-08-18T15:45:47.150 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:45:47.150 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:45:47.150 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:45:47.150 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:45:25.940487Z" 2024-08-18T15:45:47.150 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:47.150 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:47.151 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:47.151 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T15:45:47.151 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:45:47.151 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T15:45:47.151 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000008126", 2024-08-18T15:45:47.151 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:45:47.151 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:45:47.151 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:45:47.151 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:35:25.388487Z" 2024-08-18T15:45:47.151 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:47.151 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:47.151 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T15:45:47.151 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:47.151 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T15:45:47.151 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 0, 2024-08-18T15:45:47.152 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 0 2024-08-18T15:45:47.152 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:47.152 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T15:45:47Z" 2024-08-18T15:45:47.152 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T15:45:47.153 WARNING:rgw_multi.tests:data of zone test-zone3 behind zone test-zone1: shard=2 master=00000000000000000000:00000000000000007701 target=00000000000000000000:00000000000000007421 2024-08-18T15:45:52.154 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:45:52.154 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:45:52.155 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:45:52.213 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:45:52.213 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:45:52.258 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T15:45:52.258 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T15:45:52.258 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T15:45:52.259 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T15:45:52.259 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T15:45:52.259 INFO:teuthology.orchestra.run.smithi134.stdout: "instance_id": 6908254841848037526 2024-08-18T15:45:52.259 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:52.259 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T15:45:52.259 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:52.259 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T15:45:52.259 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:45:52.259 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T15:45:52.259 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000003622", 2024-08-18T15:45:52.259 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:45:52.259 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:45:52.259 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:45:52.260 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:57.659462Z" 2024-08-18T15:45:52.260 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:52.260 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:52.260 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:52.260 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T15:45:52.260 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:45:52.260 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T15:45:52.260 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000004037", 2024-08-18T15:45:52.260 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:45:52.260 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:45:52.260 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:45:52.260 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:32:51.378244Z" 2024-08-18T15:45:52.260 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:52.261 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:52.261 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:52.261 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T15:45:52.261 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:45:52.261 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T15:45:52.261 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000007421", 2024-08-18T15:45:52.261 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:45:52.261 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:45:52.261 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:45:52.261 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:45:25.940487Z" 2024-08-18T15:45:52.261 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:52.261 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:52.261 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:52.262 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T15:45:52.262 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:45:52.262 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T15:45:52.262 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000008126", 2024-08-18T15:45:52.262 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:45:52.262 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:45:52.262 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:45:52.262 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:35:25.388487Z" 2024-08-18T15:45:52.262 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:52.262 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:52.262 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T15:45:52.262 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:52.262 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T15:45:52.263 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 0, 2024-08-18T15:45:52.263 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 0 2024-08-18T15:45:52.263 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:52.263 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T15:45:52Z" 2024-08-18T15:45:52.263 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T15:45:52.264 WARNING:rgw_multi.tests:data of zone test-zone3 behind zone test-zone1: shard=2 master=00000000000000000000:00000000000000007701 target=00000000000000000000:00000000000000007421 2024-08-18T15:45:57.265 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:45:57.265 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:45:57.266 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:45:57.322 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:45:57.322 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:45:57.367 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T15:45:57.367 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T15:45:57.367 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T15:45:57.368 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T15:45:57.368 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T15:45:57.368 INFO:teuthology.orchestra.run.smithi134.stdout: "instance_id": 6908254841848037526 2024-08-18T15:45:57.368 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:57.368 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T15:45:57.368 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:57.368 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T15:45:57.368 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:45:57.368 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T15:45:57.368 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000003622", 2024-08-18T15:45:57.368 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:45:57.368 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:45:57.368 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:45:57.369 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:57.659462Z" 2024-08-18T15:45:57.369 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:57.369 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:57.369 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:57.369 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T15:45:57.369 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:45:57.369 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T15:45:57.369 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000004037", 2024-08-18T15:45:57.369 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:45:57.369 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:45:57.369 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:45:57.369 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:32:51.378244Z" 2024-08-18T15:45:57.369 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:57.370 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:57.370 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:57.370 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T15:45:57.370 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:45:57.370 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T15:45:57.370 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000007701", 2024-08-18T15:45:57.370 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:45:57.370 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:45:57.370 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:45:57.370 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:45:45.805061Z" 2024-08-18T15:45:57.370 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:57.370 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:57.370 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:57.371 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T15:45:57.371 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:45:57.371 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T15:45:57.371 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000008126", 2024-08-18T15:45:57.371 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:45:57.371 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:45:57.371 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:45:57.371 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:35:25.388487Z" 2024-08-18T15:45:57.371 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:57.371 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:57.371 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T15:45:57.371 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:57.371 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T15:45:57.371 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 0, 2024-08-18T15:45:57.372 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 0 2024-08-18T15:45:57.372 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:57.372 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T15:45:57Z" 2024-08-18T15:45:57.372 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T15:45:57.373 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2024-08-18T15:45:57.373 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:45:57.373 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:45:57.373 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:45:57.428 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:45:57.428 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:45:57.479 INFO:teuthology.orchestra.run.smithi193.stdout:[ 2024-08-18T15:45:57.479 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:57.479 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000003340", 2024-08-18T15:45:57.479 INFO:teuthology.orchestra.run.smithi193.stdout: "last_update": "2024-08-18T15:42:59.670104Z" 2024-08-18T15:45:57.479 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:57.480 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:57.480 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000004178", 2024-08-18T15:45:57.480 INFO:teuthology.orchestra.run.smithi193.stdout: "last_update": "2024-08-18T15:33:17.304450Z" 2024-08-18T15:45:57.480 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:57.480 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:57.480 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000007278", 2024-08-18T15:45:57.480 INFO:teuthology.orchestra.run.smithi193.stdout: "last_update": "2024-08-18T15:45:54.846666Z" 2024-08-18T15:45:57.480 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:45:57.480 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:45:57.480 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000007139", 2024-08-18T15:45:57.480 INFO:teuthology.orchestra.run.smithi193.stdout: "last_update": "2024-08-18T15:35:43.061910Z" 2024-08-18T15:45:57.480 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:45:57.480 INFO:teuthology.orchestra.run.smithi193.stdout:] 2024-08-18T15:45:57.481 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2024-08-18T15:45:57.482 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:45:57.482 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:45:57.482 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:45:57.533 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:45:57.533 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:45:57.575 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T15:45:57.576 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T15:45:57.576 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T15:45:57.576 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T15:45:57.576 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T15:45:57.576 INFO:teuthology.orchestra.run.smithi134.stdout: "instance_id": 15510818030922206205 2024-08-18T15:45:57.576 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:57.576 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T15:45:57.576 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:57.576 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T15:45:57.576 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:45:57.576 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T15:45:57.577 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000003340", 2024-08-18T15:45:57.577 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:45:57.577 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:45:57.577 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:45:57.577 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:59.670104Z" 2024-08-18T15:45:57.577 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:57.577 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:57.577 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:57.577 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T15:45:57.577 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:45:57.577 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T15:45:57.577 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000004178", 2024-08-18T15:45:57.577 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:45:57.577 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:45:57.578 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:45:57.578 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:33:17.304450Z" 2024-08-18T15:45:57.578 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:57.578 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:57.578 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:57.578 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T15:45:57.578 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:45:57.578 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T15:45:57.578 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000007138", 2024-08-18T15:45:57.578 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:45:57.578 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:45:57.579 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:45:57.579 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:45:34.893067Z" 2024-08-18T15:45:57.579 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:57.579 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:57.579 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:45:57.579 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T15:45:57.579 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:45:57.579 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T15:45:57.579 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000007139", 2024-08-18T15:45:57.579 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:45:57.579 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:45:57.579 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:45:57.579 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:35:43.061910Z" 2024-08-18T15:45:57.579 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:57.580 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:45:57.580 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T15:45:57.580 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:57.580 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T15:45:57.580 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 0, 2024-08-18T15:45:57.580 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 0 2024-08-18T15:45:57.580 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:45:57.580 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T15:45:57Z" 2024-08-18T15:45:57.580 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T15:45:57.581 WARNING:rgw_multi.tests:data of zone test-zone1 behind zone test-zone2: shard=2 master=00000000000000000000:00000000000000007278 target=00000000000000000000:00000000000000007138 2024-08-18T15:46:02.581 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:46:02.582 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:46:02.582 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:46:02.639 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:46:02.639 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:46:02.686 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T15:46:02.686 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T15:46:02.686 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T15:46:02.686 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T15:46:02.686 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T15:46:02.687 INFO:teuthology.orchestra.run.smithi134.stdout: "instance_id": 15510818030922206205 2024-08-18T15:46:02.687 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:46:02.687 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T15:46:02.687 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:46:02.687 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T15:46:02.687 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:46:02.687 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T15:46:02.687 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000003340", 2024-08-18T15:46:02.687 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:46:02.687 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:46:02.688 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:46:02.688 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:59.670104Z" 2024-08-18T15:46:02.688 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:46:02.688 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:46:02.688 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:46:02.688 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T15:46:02.688 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:46:02.688 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T15:46:02.688 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000004178", 2024-08-18T15:46:02.688 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:46:02.688 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:46:02.688 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:46:02.688 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:33:17.304450Z" 2024-08-18T15:46:02.689 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:46:02.689 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:46:02.689 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:46:02.689 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T15:46:02.689 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:46:02.689 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T15:46:02.689 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000007138", 2024-08-18T15:46:02.689 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:46:02.689 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:46:02.689 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:46:02.689 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:45:34.893067Z" 2024-08-18T15:46:02.690 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:46:02.690 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:46:02.690 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:46:02.690 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T15:46:02.690 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:46:02.690 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T15:46:02.690 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000007139", 2024-08-18T15:46:02.690 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:46:02.690 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:46:02.690 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:46:02.690 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:35:43.061910Z" 2024-08-18T15:46:02.691 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:46:02.691 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:46:02.691 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T15:46:02.691 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:46:02.691 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T15:46:02.691 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 0, 2024-08-18T15:46:02.691 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 0 2024-08-18T15:46:02.691 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:46:02.691 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T15:46:02Z" 2024-08-18T15:46:02.691 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T15:46:02.692 WARNING:rgw_multi.tests:data of zone test-zone1 behind zone test-zone2: shard=2 master=00000000000000000000:00000000000000007278 target=00000000000000000000:00000000000000007138 2024-08-18T15:46:07.693 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:46:07.693 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:46:07.693 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:46:07.750 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:46:07.750 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:46:07.797 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T15:46:07.797 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T15:46:07.797 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T15:46:07.798 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T15:46:07.798 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T15:46:07.798 INFO:teuthology.orchestra.run.smithi134.stdout: "instance_id": 15510818030922206205 2024-08-18T15:46:07.798 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:46:07.798 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T15:46:07.798 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:46:07.798 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T15:46:07.798 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:46:07.798 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T15:46:07.798 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000003340", 2024-08-18T15:46:07.799 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:46:07.799 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:46:07.799 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:46:07.799 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:59.670104Z" 2024-08-18T15:46:07.799 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:46:07.799 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:46:07.799 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:46:07.799 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T15:46:07.799 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:46:07.799 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T15:46:07.799 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000004178", 2024-08-18T15:46:07.800 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:46:07.800 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:46:07.800 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:46:07.800 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:33:17.304450Z" 2024-08-18T15:46:07.800 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:46:07.800 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:46:07.800 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:46:07.800 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T15:46:07.800 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:46:07.800 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T15:46:07.801 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000007278", 2024-08-18T15:46:07.801 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:46:07.801 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:46:07.801 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:46:07.801 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:45:54.846666Z" 2024-08-18T15:46:07.801 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:46:07.801 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:46:07.801 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:46:07.801 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T15:46:07.801 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:46:07.801 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T15:46:07.801 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000007139", 2024-08-18T15:46:07.801 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:46:07.802 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:46:07.802 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:46:07.802 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:35:43.061910Z" 2024-08-18T15:46:07.802 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:46:07.802 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:46:07.802 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T15:46:07.802 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:46:07.802 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T15:46:07.802 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 0, 2024-08-18T15:46:07.802 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 0 2024-08-18T15:46:07.802 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:46:07.802 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T15:46:07Z" 2024-08-18T15:46:07.802 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T15:46:07.804 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2024-08-18T15:46:07.805 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:46:07.805 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:46:07.805 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:46:07.859 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:46:07.859 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:46:07.911 INFO:teuthology.orchestra.run.smithi193.stdout:[ 2024-08-18T15:46:07.912 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:46:07.912 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000003340", 2024-08-18T15:46:07.912 INFO:teuthology.orchestra.run.smithi193.stdout: "last_update": "2024-08-18T15:42:59.670104Z" 2024-08-18T15:46:07.912 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:46:07.912 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:46:07.912 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000004178", 2024-08-18T15:46:07.912 INFO:teuthology.orchestra.run.smithi193.stdout: "last_update": "2024-08-18T15:33:17.304450Z" 2024-08-18T15:46:07.912 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:46:07.912 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:46:07.912 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000007278", 2024-08-18T15:46:07.913 INFO:teuthology.orchestra.run.smithi193.stdout: "last_update": "2024-08-18T15:45:54.846666Z" 2024-08-18T15:46:07.913 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:46:07.913 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:46:07.913 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000007139", 2024-08-18T15:46:07.913 INFO:teuthology.orchestra.run.smithi193.stdout: "last_update": "2024-08-18T15:35:43.061910Z" 2024-08-18T15:46:07.913 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:46:07.913 INFO:teuthology.orchestra.run.smithi193.stdout:] 2024-08-18T15:46:07.914 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2024-08-18T15:46:07.914 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:46:07.914 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:46:07.914 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:46:07.965 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:46:07.965 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:46:08.006 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T15:46:08.006 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T15:46:08.006 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T15:46:08.007 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T15:46:08.007 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T15:46:08.007 INFO:teuthology.orchestra.run.smithi134.stdout: "instance_id": 9282142518545740118 2024-08-18T15:46:08.007 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:46:08.007 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T15:46:08.007 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:46:08.007 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T15:46:08.007 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:46:08.008 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T15:46:08.008 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000003340", 2024-08-18T15:46:08.008 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:46:08.008 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:46:08.008 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:46:08.008 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:59.670104Z" 2024-08-18T15:46:08.008 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:46:08.008 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:46:08.008 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:46:08.008 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T15:46:08.008 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:46:08.008 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T15:46:08.008 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000004178", 2024-08-18T15:46:08.008 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:46:08.008 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:46:08.009 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:46:08.009 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:33:17.304450Z" 2024-08-18T15:46:08.009 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:46:08.009 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:46:08.009 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:46:08.009 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T15:46:08.009 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:46:08.009 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T15:46:08.009 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000007138", 2024-08-18T15:46:08.009 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:46:08.009 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:46:08.009 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:46:08.009 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:45:34.893067Z" 2024-08-18T15:46:08.009 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:46:08.009 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:46:08.010 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:46:08.010 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T15:46:08.010 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:46:08.010 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T15:46:08.010 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000007139", 2024-08-18T15:46:08.010 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:46:08.010 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:46:08.010 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:46:08.010 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:35:43.061910Z" 2024-08-18T15:46:08.010 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:46:08.010 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:46:08.010 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T15:46:08.010 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:46:08.011 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T15:46:08.011 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 0, 2024-08-18T15:46:08.011 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 0 2024-08-18T15:46:08.011 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:46:08.011 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T15:46:07Z" 2024-08-18T15:46:08.011 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T15:46:08.012 WARNING:rgw_multi.tests:data of zone test-zone3 behind zone test-zone2: shard=2 master=00000000000000000000:00000000000000007278 target=00000000000000000000:00000000000000007138 2024-08-18T15:46:13.013 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:46:13.013 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:46:13.013 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:46:13.071 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:46:13.072 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:46:13.120 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T15:46:13.120 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T15:46:13.121 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T15:46:13.121 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T15:46:13.121 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T15:46:13.121 INFO:teuthology.orchestra.run.smithi134.stdout: "instance_id": 9282142518545740118 2024-08-18T15:46:13.121 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:46:13.121 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T15:46:13.121 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:46:13.121 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T15:46:13.121 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:46:13.121 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T15:46:13.122 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000003340", 2024-08-18T15:46:13.122 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:46:13.122 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:46:13.122 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:46:13.122 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:59.670104Z" 2024-08-18T15:46:13.122 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:46:13.122 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:46:13.122 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:46:13.122 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T15:46:13.122 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:46:13.123 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T15:46:13.123 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000004178", 2024-08-18T15:46:13.123 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:46:13.123 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:46:13.123 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:46:13.123 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:33:17.304450Z" 2024-08-18T15:46:13.123 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:46:13.123 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:46:13.123 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:46:13.123 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T15:46:13.123 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:46:13.124 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T15:46:13.124 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000007138", 2024-08-18T15:46:13.124 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:46:13.124 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:46:13.124 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:46:13.124 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:45:34.893067Z" 2024-08-18T15:46:13.124 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:46:13.124 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:46:13.124 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:46:13.124 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T15:46:13.125 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:46:13.125 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T15:46:13.125 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000007139", 2024-08-18T15:46:13.125 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:46:13.125 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:46:13.125 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:46:13.125 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:35:43.061910Z" 2024-08-18T15:46:13.125 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:46:13.125 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:46:13.125 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T15:46:13.125 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:46:13.125 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T15:46:13.126 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 0, 2024-08-18T15:46:13.126 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 0 2024-08-18T15:46:13.126 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:46:13.126 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T15:46:13Z" 2024-08-18T15:46:13.126 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T15:46:13.127 WARNING:rgw_multi.tests:data of zone test-zone3 behind zone test-zone2: shard=2 master=00000000000000000000:00000000000000007278 target=00000000000000000000:00000000000000007138 2024-08-18T15:46:18.127 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:46:18.128 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:46:18.128 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:46:18.185 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:46:18.185 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:46:18.233 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T15:46:18.233 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T15:46:18.233 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T15:46:18.233 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T15:46:18.233 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T15:46:18.233 INFO:teuthology.orchestra.run.smithi134.stdout: "instance_id": 9282142518545740118 2024-08-18T15:46:18.234 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:46:18.234 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T15:46:18.234 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:46:18.234 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T15:46:18.234 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:46:18.234 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T15:46:18.234 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000003340", 2024-08-18T15:46:18.234 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:46:18.234 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:46:18.234 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:46:18.234 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:42:59.670104Z" 2024-08-18T15:46:18.234 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:46:18.234 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:46:18.235 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:46:18.235 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T15:46:18.235 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:46:18.235 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T15:46:18.235 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000004178", 2024-08-18T15:46:18.235 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:46:18.235 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:46:18.235 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:46:18.235 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:33:17.304450Z" 2024-08-18T15:46:18.235 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:46:18.235 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:46:18.235 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:46:18.235 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T15:46:18.236 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:46:18.236 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T15:46:18.236 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000007278", 2024-08-18T15:46:18.236 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:46:18.236 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:46:18.236 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:46:18.236 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:45:54.846666Z" 2024-08-18T15:46:18.236 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:46:18.236 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:46:18.236 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:46:18.236 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T15:46:18.236 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:46:18.236 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T15:46:18.237 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000007139", 2024-08-18T15:46:18.237 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:46:18.237 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:46:18.237 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:46:18.237 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:35:43.061910Z" 2024-08-18T15:46:18.237 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:46:18.237 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:46:18.237 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T15:46:18.237 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:46:18.237 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T15:46:18.237 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 0, 2024-08-18T15:46:18.237 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 0 2024-08-18T15:46:18.237 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:46:18.237 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T15:46:18Z" 2024-08-18T15:46:18.238 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T15:46:18.239 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2024-08-18T15:46:18.239 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:46:18.239 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:46:18.239 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:46:18.286 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:46:18.286 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:46:18.332 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T15:46:18.332 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:46:18.332 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000003482", 2024-08-18T15:46:18.333 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:45:36.611332Z" 2024-08-18T15:46:18.333 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:46:18.333 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:46:18.333 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000004460", 2024-08-18T15:46:18.333 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:32:57.456234Z" 2024-08-18T15:46:18.333 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:46:18.333 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:46:18.333 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000007560", 2024-08-18T15:46:18.333 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:45:34.813897Z" 2024-08-18T15:46:18.334 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:46:18.334 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:46:18.334 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000008408", 2024-08-18T15:46:18.334 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:35:09.464574Z" 2024-08-18T15:46:18.334 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:46:18.334 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T15:46:18.335 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2024-08-18T15:46:18.335 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:46:18.335 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:46:18.336 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:46:18.384 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:46:18.384 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:46:18.426 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T15:46:18.426 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T15:46:18.427 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T15:46:18.427 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T15:46:18.427 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T15:46:18.427 INFO:teuthology.orchestra.run.smithi134.stdout: "instance_id": 16929744281786545893 2024-08-18T15:46:18.427 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:46:18.427 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T15:46:18.427 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:46:18.427 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T15:46:18.427 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:46:18.427 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T15:46:18.427 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000003482", 2024-08-18T15:46:18.428 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:46:18.428 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:46:18.428 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:46:18.428 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:45:36.611332Z" 2024-08-18T15:46:18.428 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:46:18.428 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:46:18.428 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:46:18.428 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T15:46:18.428 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:46:18.428 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T15:46:18.428 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000004460", 2024-08-18T15:46:18.429 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:46:18.429 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:46:18.429 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:46:18.429 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:32:57.456234Z" 2024-08-18T15:46:18.429 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:46:18.429 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:46:18.429 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:46:18.429 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T15:46:18.429 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:46:18.429 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T15:46:18.429 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000007560", 2024-08-18T15:46:18.430 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:46:18.430 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:46:18.430 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:46:18.430 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:45:34.813897Z" 2024-08-18T15:46:18.430 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:46:18.430 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:46:18.430 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:46:18.430 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T15:46:18.430 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:46:18.430 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T15:46:18.430 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000008408", 2024-08-18T15:46:18.430 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:46:18.431 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:46:18.431 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:46:18.431 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:35:09.464574Z" 2024-08-18T15:46:18.431 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:46:18.431 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:46:18.431 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T15:46:18.431 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:46:18.431 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T15:46:18.431 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 0, 2024-08-18T15:46:18.431 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 0 2024-08-18T15:46:18.431 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:46:18.431 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T15:46:18Z" 2024-08-18T15:46:18.432 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T15:46:18.433 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2024-08-18T15:46:18.433 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:46:18.433 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:46:18.433 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:46:18.479 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:46:18.479 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:46:18.525 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T15:46:18.526 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:46:18.526 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000003482", 2024-08-18T15:46:18.526 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:45:36.611332Z" 2024-08-18T15:46:18.526 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:46:18.526 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:46:18.526 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000004460", 2024-08-18T15:46:18.526 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:32:57.456234Z" 2024-08-18T15:46:18.526 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:46:18.526 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:46:18.526 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000007560", 2024-08-18T15:46:18.527 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:45:34.813897Z" 2024-08-18T15:46:18.527 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:46:18.527 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:46:18.527 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000008408", 2024-08-18T15:46:18.527 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:35:09.464574Z" 2024-08-18T15:46:18.527 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:46:18.527 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T15:46:18.528 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2024-08-18T15:46:18.528 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:46:18.528 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:46:18.528 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:46:18.585 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:46:18.585 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:46:18.631 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:46:18.632 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:46:18.632 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:46:18.632 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:46:18.632 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:46:18.632 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17055221296585213178 2024-08-18T15:46:18.632 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:46:18.632 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:46:18.632 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:46:18.632 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:46:18.632 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:46:18.632 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T15:46:18.632 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000003482", 2024-08-18T15:46:18.633 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:46:18.633 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:46:18.633 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:46:18.633 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:45:36.611332Z" 2024-08-18T15:46:18.633 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:46:18.633 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:46:18.633 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:46:18.633 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:46:18.633 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:46:18.633 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T15:46:18.634 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000004460", 2024-08-18T15:46:18.634 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:46:18.634 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:46:18.634 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:46:18.634 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:32:57.456234Z" 2024-08-18T15:46:18.634 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:46:18.634 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:46:18.634 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:46:18.634 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:46:18.634 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:46:18.634 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T15:46:18.635 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000007560", 2024-08-18T15:46:18.635 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:46:18.635 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:46:18.635 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:46:18.635 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:45:34.813897Z" 2024-08-18T15:46:18.635 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:46:18.635 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:46:18.635 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:46:18.635 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:46:18.635 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:46:18.635 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T15:46:18.635 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000008408", 2024-08-18T15:46:18.635 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:46:18.636 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:46:18.636 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:46:18.636 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:35:09.464574Z" 2024-08-18T15:46:18.636 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:46:18.636 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:46:18.636 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:46:18.636 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:46:18.636 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:46:18.636 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 0, 2024-08-18T15:46:18.636 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 0 2024-08-18T15:46:18.636 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:46:18.636 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:46:18Z" 2024-08-18T15:46:18.636 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:46:18.637 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2024-08-18T15:46:18.638 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:46:18.638 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:46:18.638 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:46:18.682 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:46:18.682 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:46:18.729 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T15:46:18.729 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:46:18.729 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000003622", 2024-08-18T15:46:18.729 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:42:57.659462Z" 2024-08-18T15:46:18.729 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:46:18.730 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:46:18.730 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000004037", 2024-08-18T15:46:18.730 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:32:51.378244Z" 2024-08-18T15:46:18.730 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:46:18.730 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:46:18.730 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000007701", 2024-08-18T15:46:18.730 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:45:45.805061Z" 2024-08-18T15:46:18.730 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:46:18.730 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:46:18.730 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000008126", 2024-08-18T15:46:18.730 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:35:25.388487Z" 2024-08-18T15:46:18.730 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:46:18.731 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T15:46:18.731 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:46:18.732 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:46:18.732 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T15:46:18.814 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:46:18.814 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:46:18.866 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '0', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:46:18.866 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '0', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:46:18.866 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 0 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:46:18.953 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:46:18.953 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:46:18.995 INFO:teuthology.orchestra.run.smithi134.stdout:[] 2024-08-18T15:46:18.997 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '1', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:46:18.997 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '1', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:46:18.997 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 1 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:46:19.044 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:46:19.044 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:46:19.087 INFO:teuthology.orchestra.run.smithi134.stdout:[] 2024-08-18T15:46:19.089 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '2', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:46:19.089 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '2', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:46:19.089 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 2 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:46:19.136 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:46:19.136 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:46:19.179 INFO:teuthology.orchestra.run.smithi134.stdout:[] 2024-08-18T15:46:19.180 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '3', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:46:19.180 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '3', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:46:19.181 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 3 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:46:19.228 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:46:19.228 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:46:19.269 INFO:teuthology.orchestra.run.smithi134.stdout:[] 2024-08-18T15:46:19.271 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:46:19.271 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:46:19.271 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:46:19.322 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:46:19.322 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:46:19.370 INFO:teuthology.orchestra.run.smithi193.stdout:[ 2024-08-18T15:46:19.370 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:46:19.370 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000003340", 2024-08-18T15:46:19.371 INFO:teuthology.orchestra.run.smithi193.stdout: "last_update": "2024-08-18T15:42:59.670104Z" 2024-08-18T15:46:19.371 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:46:19.371 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:46:19.371 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000004178", 2024-08-18T15:46:19.371 INFO:teuthology.orchestra.run.smithi193.stdout: "last_update": "2024-08-18T15:33:17.304450Z" 2024-08-18T15:46:19.371 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:46:19.371 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:46:19.371 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000007278", 2024-08-18T15:46:19.371 INFO:teuthology.orchestra.run.smithi193.stdout: "last_update": "2024-08-18T15:45:54.846666Z" 2024-08-18T15:46:19.371 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:46:19.371 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:46:19.371 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000007139", 2024-08-18T15:46:19.371 INFO:teuthology.orchestra.run.smithi193.stdout: "last_update": "2024-08-18T15:35:43.061910Z" 2024-08-18T15:46:19.372 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:46:19.372 INFO:teuthology.orchestra.run.smithi193.stdout:] 2024-08-18T15:46:19.373 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:46:19.373 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:46:19.373 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog autotrim --cluster c2 --debug-rgw 0 --debug-ms 0 2024-08-18T15:46:19.416 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:46:19.416 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:46:19.468 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'list', '--shard-id', '0', '--max-entries', '1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:46:19.468 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'list', '--shard-id', '0', '--max-entries', '1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:46:19.468 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog list --shard-id 0 --max-entries 1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:46:19.556 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:46:19.556 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:46:19.598 INFO:teuthology.orchestra.run.smithi193.stdout:[] 2024-08-18T15:46:19.600 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'list', '--shard-id', '1', '--max-entries', '1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:46:19.600 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'list', '--shard-id', '1', '--max-entries', '1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:46:19.600 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog list --shard-id 1 --max-entries 1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:46:19.647 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:46:19.647 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:46:19.689 INFO:teuthology.orchestra.run.smithi193.stdout:[] 2024-08-18T15:46:19.691 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'list', '--shard-id', '2', '--max-entries', '1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:46:19.691 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'list', '--shard-id', '2', '--max-entries', '1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:46:19.691 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog list --shard-id 2 --max-entries 1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:46:19.738 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:46:19.738 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:46:19.781 INFO:teuthology.orchestra.run.smithi193.stdout:[] 2024-08-18T15:46:19.783 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'list', '--shard-id', '3', '--max-entries', '1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:46:19.783 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'list', '--shard-id', '3', '--max-entries', '1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:46:19.783 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog list --shard-id 3 --max-entries 1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:46:19.830 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:46:19.830 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:46:19.874 INFO:teuthology.orchestra.run.smithi193.stdout:[] 2024-08-18T15:46:19.876 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:46:19.876 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:46:19.876 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:46:19.921 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:46:19.921 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:46:19.970 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T15:46:19.971 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:46:19.971 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000003622", 2024-08-18T15:46:19.971 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:42:57.659462Z" 2024-08-18T15:46:19.971 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:46:19.971 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:46:19.971 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000004037", 2024-08-18T15:46:19.971 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:32:51.378244Z" 2024-08-18T15:46:19.971 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:46:19.971 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:46:19.971 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000007701", 2024-08-18T15:46:19.971 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:45:45.805061Z" 2024-08-18T15:46:19.972 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:46:19.972 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:46:19.972 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000008126", 2024-08-18T15:46:19.972 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:35:25.388487Z" 2024-08-18T15:46:19.972 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:46:19.972 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T15:46:19.973 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:46:19.973 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:46:19.973 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T15:46:20.017 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:46:20.018 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:46:20.070 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '0', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:46:20.070 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '0', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:46:20.070 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 0 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:46:20.118 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:46:20.118 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:46:20.160 INFO:teuthology.orchestra.run.smithi134.stdout:[] 2024-08-18T15:46:20.161 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '1', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:46:20.162 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '1', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:46:20.162 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 1 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:46:20.209 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:46:20.209 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:46:20.251 INFO:teuthology.orchestra.run.smithi134.stdout:[] 2024-08-18T15:46:20.253 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '2', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:46:20.253 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '2', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:46:20.253 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 2 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:46:20.299 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:46:20.300 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:46:20.341 INFO:teuthology.orchestra.run.smithi134.stdout:[] 2024-08-18T15:46:20.343 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '3', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:46:20.343 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '3', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:46:20.343 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 3 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:46:20.390 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:46:20.390 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:46:20.432 INFO:teuthology.orchestra.run.smithi134.stdout:[] 2024-08-18T15:46:20.435 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_datalog_autotrim ... ok 2024-08-18T15:46:20.436 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:46:20.436 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:46:20.436 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --sync-from-all=false --cluster c2 --debug-rgw 0 --debug-ms 0 2024-08-18T15:46:20.484 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:46:20.484 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:46:20.504 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:46:20.505 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "10090ccf-aad6-416d-9775-df8b7b27345a", 2024-08-18T15:46:20.505 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "test-zone2", 2024-08-18T15:46:20.505 INFO:teuthology.orchestra.run.smithi193.stdout: "domain_root": "test-zone2.rgw.meta:root", 2024-08-18T15:46:20.505 INFO:teuthology.orchestra.run.smithi193.stdout: "control_pool": "test-zone2.rgw.control", 2024-08-18T15:46:20.505 INFO:teuthology.orchestra.run.smithi193.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2024-08-18T15:46:20.505 INFO:teuthology.orchestra.run.smithi193.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2024-08-18T15:46:20.505 INFO:teuthology.orchestra.run.smithi193.stdout: "log_pool": "test-zone2.rgw.log", 2024-08-18T15:46:20.505 INFO:teuthology.orchestra.run.smithi193.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2024-08-18T15:46:20.505 INFO:teuthology.orchestra.run.smithi193.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2024-08-18T15:46:20.505 INFO:teuthology.orchestra.run.smithi193.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2024-08-18T15:46:20.505 INFO:teuthology.orchestra.run.smithi193.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2024-08-18T15:46:20.505 INFO:teuthology.orchestra.run.smithi193.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2024-08-18T15:46:20.505 INFO:teuthology.orchestra.run.smithi193.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2024-08-18T15:46:20.506 INFO:teuthology.orchestra.run.smithi193.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2024-08-18T15:46:20.506 INFO:teuthology.orchestra.run.smithi193.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2024-08-18T15:46:20.506 INFO:teuthology.orchestra.run.smithi193.stdout: "otp_pool": "test-zone2.rgw.otp", 2024-08-18T15:46:20.506 INFO:teuthology.orchestra.run.smithi193.stdout: "system_key": { 2024-08-18T15:46:20.506 INFO:teuthology.orchestra.run.smithi193.stdout: "access_key": "MRX9HX7RYBDXO1UM", 2024-08-18T15:46:20.506 INFO:teuthology.orchestra.run.smithi193.stdout: "secret_key": "hGUsvhUdpGZAYWjQwu3StBaYXJPhuCEj" 2024-08-18T15:46:20.506 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:46:20.506 INFO:teuthology.orchestra.run.smithi193.stdout: "placement_pools": [ 2024-08-18T15:46:20.506 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:46:20.506 INFO:teuthology.orchestra.run.smithi193.stdout: "key": "default-placement", 2024-08-18T15:46:20.506 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:46:20.506 INFO:teuthology.orchestra.run.smithi193.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2024-08-18T15:46:20.506 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_classes": { 2024-08-18T15:46:20.507 INFO:teuthology.orchestra.run.smithi193.stdout: "STANDARD": { 2024-08-18T15:46:20.507 INFO:teuthology.orchestra.run.smithi193.stdout: "data_pool": "test-zone2.rgw.buckets.data", 2024-08-18T15:46:20.507 INFO:teuthology.orchestra.run.smithi193.stdout: "compression_type": "random" 2024-08-18T15:46:20.507 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:46:20.507 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:46:20.507 INFO:teuthology.orchestra.run.smithi193.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2024-08-18T15:46:20.507 INFO:teuthology.orchestra.run.smithi193.stdout: "index_type": 0, 2024-08-18T15:46:20.507 INFO:teuthology.orchestra.run.smithi193.stdout: "inline_data": true 2024-08-18T15:46:20.507 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:46:20.507 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:46:20.507 INFO:teuthology.orchestra.run.smithi193.stdout: ], 2024-08-18T15:46:20.507 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_id": "2827c6f0-bf0d-4836-944c-9e7fcf22457b", 2024-08-18T15:46:20.507 INFO:teuthology.orchestra.run.smithi193.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2024-08-18T15:46:20.507 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:46:20.508 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:46:20.508 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:46:20.509 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2024-08-18T15:46:20.555 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:46:20.555 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:46:20.597 INFO:teuthology.orchestra.run.smithi193.stderr:Sending period to new master zone 96bf8502-2f39-4d53-8196-3378c28da3b4 2024-08-18T15:46:20.616 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:46:20.616 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T15:46:20.616 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 2, 2024-08-18T15:46:20.616 INFO:teuthology.orchestra.run.smithi193.stdout: "predecessor_uuid": "64309209-7567-46e5-8881-65783d84cf3c", 2024-08-18T15:46:20.616 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": [ 2024-08-18T15:46:20.616 INFO:teuthology.orchestra.run.smithi193.stdout: "", 2024-08-18T15:46:20.616 INFO:teuthology.orchestra.run.smithi193.stdout: "", 2024-08-18T15:46:20.616 INFO:teuthology.orchestra.run.smithi193.stdout: "", 2024-08-18T15:46:20.616 INFO:teuthology.orchestra.run.smithi193.stdout: "1_1723995848.930494_187.1" 2024-08-18T15:46:20.616 INFO:teuthology.orchestra.run.smithi193.stdout: ], 2024-08-18T15:46:20.616 INFO:teuthology.orchestra.run.smithi193.stdout: "period_map": { 2024-08-18T15:46:20.616 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T15:46:20.617 INFO:teuthology.orchestra.run.smithi193.stdout: "zonegroups": [ 2024-08-18T15:46:20.617 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:46:20.617 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "test-zonegroup", 2024-08-18T15:46:20.617 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "test-zonegroup", 2024-08-18T15:46:20.617 INFO:teuthology.orchestra.run.smithi193.stdout: "api_name": "", 2024-08-18T15:46:20.617 INFO:teuthology.orchestra.run.smithi193.stdout: "is_master": true, 2024-08-18T15:46:20.617 INFO:teuthology.orchestra.run.smithi193.stdout: "endpoints": [ 2024-08-18T15:46:20.617 INFO:teuthology.orchestra.run.smithi193.stdout: "http://smithi134.front.sepia.ceph.com:8000" 2024-08-18T15:46:20.617 INFO:teuthology.orchestra.run.smithi193.stdout: ], 2024-08-18T15:46:20.617 INFO:teuthology.orchestra.run.smithi193.stdout: "hostnames": [], 2024-08-18T15:46:20.617 INFO:teuthology.orchestra.run.smithi193.stdout: "hostnames_s3website": [], 2024-08-18T15:46:20.617 INFO:teuthology.orchestra.run.smithi193.stdout: "master_zone": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:46:20.617 INFO:teuthology.orchestra.run.smithi193.stdout: "zones": [ 2024-08-18T15:46:20.617 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:46:20.618 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T15:46:20.618 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "test-zone3", 2024-08-18T15:46:20.618 INFO:teuthology.orchestra.run.smithi193.stdout: "endpoints": [ 2024-08-18T15:46:20.618 INFO:teuthology.orchestra.run.smithi193.stdout: "http://smithi134.front.sepia.ceph.com:8001" 2024-08-18T15:46:20.618 INFO:teuthology.orchestra.run.smithi193.stdout: ], 2024-08-18T15:46:20.618 INFO:teuthology.orchestra.run.smithi193.stdout: "log_meta": false, 2024-08-18T15:46:20.618 INFO:teuthology.orchestra.run.smithi193.stdout: "log_data": true, 2024-08-18T15:46:20.618 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket_index_max_shards": 11, 2024-08-18T15:46:20.618 INFO:teuthology.orchestra.run.smithi193.stdout: "read_only": false, 2024-08-18T15:46:20.618 INFO:teuthology.orchestra.run.smithi193.stdout: "tier_type": "", 2024-08-18T15:46:20.618 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_from_all": true, 2024-08-18T15:46:20.618 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_from": [], 2024-08-18T15:46:20.618 INFO:teuthology.orchestra.run.smithi193.stdout: "redirect_zone": "", 2024-08-18T15:46:20.619 INFO:teuthology.orchestra.run.smithi193.stdout: "supported_features": [ 2024-08-18T15:46:20.619 INFO:teuthology.orchestra.run.smithi193.stdout: "compress-encrypted", 2024-08-18T15:46:20.619 INFO:teuthology.orchestra.run.smithi193.stdout: "resharding" 2024-08-18T15:46:20.619 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:46:20.619 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:46:20.619 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:46:20.619 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "10090ccf-aad6-416d-9775-df8b7b27345a", 2024-08-18T15:46:20.619 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "test-zone2", 2024-08-18T15:46:20.619 INFO:teuthology.orchestra.run.smithi193.stdout: "endpoints": [ 2024-08-18T15:46:20.619 INFO:teuthology.orchestra.run.smithi193.stdout: "http://smithi193.front.sepia.ceph.com:8000" 2024-08-18T15:46:20.619 INFO:teuthology.orchestra.run.smithi193.stdout: ], 2024-08-18T15:46:20.619 INFO:teuthology.orchestra.run.smithi193.stdout: "log_meta": false, 2024-08-18T15:46:20.619 INFO:teuthology.orchestra.run.smithi193.stdout: "log_data": true, 2024-08-18T15:46:20.619 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket_index_max_shards": 11, 2024-08-18T15:46:20.620 INFO:teuthology.orchestra.run.smithi193.stdout: "read_only": false, 2024-08-18T15:46:20.620 INFO:teuthology.orchestra.run.smithi193.stdout: "tier_type": "", 2024-08-18T15:46:20.620 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_from_all": false, 2024-08-18T15:46:20.620 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_from": [], 2024-08-18T15:46:20.620 INFO:teuthology.orchestra.run.smithi193.stdout: "redirect_zone": "", 2024-08-18T15:46:20.620 INFO:teuthology.orchestra.run.smithi193.stdout: "supported_features": [ 2024-08-18T15:46:20.620 INFO:teuthology.orchestra.run.smithi193.stdout: "compress-encrypted", 2024-08-18T15:46:20.620 INFO:teuthology.orchestra.run.smithi193.stdout: "resharding" 2024-08-18T15:46:20.620 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:46:20.620 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:46:20.620 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:46:20.620 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:46:20.620 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "test-zone1", 2024-08-18T15:46:20.620 INFO:teuthology.orchestra.run.smithi193.stdout: "endpoints": [ 2024-08-18T15:46:20.621 INFO:teuthology.orchestra.run.smithi193.stdout: "http://smithi134.front.sepia.ceph.com:8000" 2024-08-18T15:46:20.621 INFO:teuthology.orchestra.run.smithi193.stdout: ], 2024-08-18T15:46:20.621 INFO:teuthology.orchestra.run.smithi193.stdout: "log_meta": false, 2024-08-18T15:46:20.621 INFO:teuthology.orchestra.run.smithi193.stdout: "log_data": true, 2024-08-18T15:46:20.621 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket_index_max_shards": 11, 2024-08-18T15:46:20.621 INFO:teuthology.orchestra.run.smithi193.stdout: "read_only": false, 2024-08-18T15:46:20.621 INFO:teuthology.orchestra.run.smithi193.stdout: "tier_type": "", 2024-08-18T15:46:20.621 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_from_all": true, 2024-08-18T15:46:20.621 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_from": [], 2024-08-18T15:46:20.621 INFO:teuthology.orchestra.run.smithi193.stdout: "redirect_zone": "", 2024-08-18T15:46:20.621 INFO:teuthology.orchestra.run.smithi193.stdout: "supported_features": [ 2024-08-18T15:46:20.621 INFO:teuthology.orchestra.run.smithi193.stdout: "compress-encrypted", 2024-08-18T15:46:20.621 INFO:teuthology.orchestra.run.smithi193.stdout: "resharding" 2024-08-18T15:46:20.621 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:46:20.622 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:46:20.622 INFO:teuthology.orchestra.run.smithi193.stdout: ], 2024-08-18T15:46:20.622 INFO:teuthology.orchestra.run.smithi193.stdout: "placement_targets": [ 2024-08-18T15:46:20.622 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:46:20.622 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "default-placement", 2024-08-18T15:46:20.622 INFO:teuthology.orchestra.run.smithi193.stdout: "tags": [], 2024-08-18T15:46:20.622 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_classes": [ 2024-08-18T15:46:20.622 INFO:teuthology.orchestra.run.smithi193.stdout: "STANDARD" 2024-08-18T15:46:20.622 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:46:20.622 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:46:20.622 INFO:teuthology.orchestra.run.smithi193.stdout: ], 2024-08-18T15:46:20.622 INFO:teuthology.orchestra.run.smithi193.stdout: "default_placement": "default-placement", 2024-08-18T15:46:20.622 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_id": "2827c6f0-bf0d-4836-944c-9e7fcf22457b", 2024-08-18T15:46:20.623 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_policy": { 2024-08-18T15:46:20.623 INFO:teuthology.orchestra.run.smithi193.stdout: "groups": [] 2024-08-18T15:46:20.623 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:46:20.623 INFO:teuthology.orchestra.run.smithi193.stdout: "enabled_features": [ 2024-08-18T15:46:20.623 INFO:teuthology.orchestra.run.smithi193.stdout: "resharding" 2024-08-18T15:46:20.623 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:46:20.623 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:46:20.623 INFO:teuthology.orchestra.run.smithi193.stdout: ], 2024-08-18T15:46:20.623 INFO:teuthology.orchestra.run.smithi193.stdout: "short_zone_ids": [ 2024-08-18T15:46:20.623 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:46:20.623 INFO:teuthology.orchestra.run.smithi193.stdout: "key": "018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T15:46:20.623 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 462036791 2024-08-18T15:46:20.623 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:46:20.623 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:46:20.628 INFO:teuthology.orchestra.run.smithi193.stdout: "key": "10090ccf-aad6-416d-9775-df8b7b27345a", 2024-08-18T15:46:20.628 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 874289624 2024-08-18T15:46:20.628 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:46:20.628 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:46:20.628 INFO:teuthology.orchestra.run.smithi193.stdout: "key": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:46:20.629 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 1721399886 2024-08-18T15:46:20.629 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:46:20.629 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:46:20.629 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:46:20.629 INFO:teuthology.orchestra.run.smithi193.stdout: "master_zonegroup": "test-zonegroup", 2024-08-18T15:46:20.629 INFO:teuthology.orchestra.run.smithi193.stdout: "master_zone": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:46:20.629 INFO:teuthology.orchestra.run.smithi193.stdout: "period_config": { 2024-08-18T15:46:20.629 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket_quota": { 2024-08-18T15:46:20.629 INFO:teuthology.orchestra.run.smithi193.stdout: "enabled": false, 2024-08-18T15:46:20.629 INFO:teuthology.orchestra.run.smithi193.stdout: "check_on_raw": false, 2024-08-18T15:46:20.629 INFO:teuthology.orchestra.run.smithi193.stdout: "max_size": -1, 2024-08-18T15:46:20.629 INFO:teuthology.orchestra.run.smithi193.stdout: "max_size_kb": 0, 2024-08-18T15:46:20.630 INFO:teuthology.orchestra.run.smithi193.stdout: "max_objects": -1 2024-08-18T15:46:20.630 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:46:20.630 INFO:teuthology.orchestra.run.smithi193.stdout: "user_quota": { 2024-08-18T15:46:20.630 INFO:teuthology.orchestra.run.smithi193.stdout: "enabled": false, 2024-08-18T15:46:20.630 INFO:teuthology.orchestra.run.smithi193.stdout: "check_on_raw": false, 2024-08-18T15:46:20.630 INFO:teuthology.orchestra.run.smithi193.stdout: "max_size": -1, 2024-08-18T15:46:20.630 INFO:teuthology.orchestra.run.smithi193.stdout: "max_size_kb": 0, 2024-08-18T15:46:20.630 INFO:teuthology.orchestra.run.smithi193.stdout: "max_objects": -1 2024-08-18T15:46:20.630 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:46:20.630 INFO:teuthology.orchestra.run.smithi193.stdout: "user_ratelimit": { 2024-08-18T15:46:20.630 INFO:teuthology.orchestra.run.smithi193.stdout: "max_read_ops": 0, 2024-08-18T15:46:20.631 INFO:teuthology.orchestra.run.smithi193.stdout: "max_write_ops": 0, 2024-08-18T15:46:20.631 INFO:teuthology.orchestra.run.smithi193.stdout: "max_read_bytes": 0, 2024-08-18T15:46:20.631 INFO:teuthology.orchestra.run.smithi193.stdout: "max_write_bytes": 0, 2024-08-18T15:46:20.631 INFO:teuthology.orchestra.run.smithi193.stdout: "enabled": false 2024-08-18T15:46:20.631 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:46:20.631 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket_ratelimit": { 2024-08-18T15:46:20.631 INFO:teuthology.orchestra.run.smithi193.stdout: "max_read_ops": 0, 2024-08-18T15:46:20.631 INFO:teuthology.orchestra.run.smithi193.stdout: "max_write_ops": 0, 2024-08-18T15:46:20.631 INFO:teuthology.orchestra.run.smithi193.stdout: "max_read_bytes": 0, 2024-08-18T15:46:20.631 INFO:teuthology.orchestra.run.smithi193.stdout: "max_write_bytes": 0, 2024-08-18T15:46:20.631 INFO:teuthology.orchestra.run.smithi193.stdout: "enabled": false 2024-08-18T15:46:20.632 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:46:20.632 INFO:teuthology.orchestra.run.smithi193.stdout: "anonymous_ratelimit": { 2024-08-18T15:46:20.632 INFO:teuthology.orchestra.run.smithi193.stdout: "max_read_ops": 0, 2024-08-18T15:46:20.632 INFO:teuthology.orchestra.run.smithi193.stdout: "max_write_ops": 0, 2024-08-18T15:46:20.632 INFO:teuthology.orchestra.run.smithi193.stdout: "max_read_bytes": 0, 2024-08-18T15:46:20.632 INFO:teuthology.orchestra.run.smithi193.stdout: "max_write_bytes": 0, 2024-08-18T15:46:20.632 INFO:teuthology.orchestra.run.smithi193.stdout: "enabled": false 2024-08-18T15:46:20.632 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:46:20.632 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:46:20.632 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_id": "2827c6f0-bf0d-4836-944c-9e7fcf22457b", 2024-08-18T15:46:20.632 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_name": "test-realm", 2024-08-18T15:46:20.632 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:46:20.633 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:46:20.634 INFO:rgw_multi.tests:Set sync_from_all flag on zone test-zone2 to false 2024-08-18T15:47:20.634 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=kibgmb-52 2024-08-18T15:47:20.666 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:47:20.667 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:47:20.667 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:47:20.719 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:47:20.719 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:47:20.751 INFO:teuthology.orchestra.run.smithi134.stderr:No --period given, using current period=01050e30-179b-43be-bdbb-3bb23cf37ae8 2024-08-18T15:47:20.764 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T15:47:20.764 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:47:20.765 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T15:47:20.765 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T15:47:20.765 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:47:20.765 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:47:20.765 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T15:47:20.765 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T15:47:20.765 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:47:20.765 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:47:20.765 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T15:47:20.765 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T15:47:20.765 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:47:20.766 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:47:20.766 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723996040.655168_301.1", 2024-08-18T15:47:20.766 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:47:20.655168Z" 2024-08-18T15:47:20.766 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:47:20.766 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T15:47:20.767 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-08-18T15:47:20.767 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:47:20.767 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:47:20.767 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:47:20.823 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:47:20.823 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:47:20.856 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:47:20.855+0000 7fdd45734880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:47:20.856 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:47:20.856+0000 7fdd45734880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:47:20.870 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:47:20.870 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:47:20.870 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:47:20.870 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:47:20.870 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:47:20.870 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T15:47:20.870 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:47:20.870 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:47:20.871 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:47:20.871 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:47:20.871 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:47:20.871 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:47:20.871 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:47:20.871 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:47:20.871 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:47:20.871 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:47:20.871 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:47:20.871 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:47:20.871 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:47:20.871 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:47:20.872 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:47:20.872 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:47:20.872 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:47:20.872 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:47:20.872 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:47:20.872 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:47:20.872 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:47:20.872 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:47:20.872 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:47:20.872 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:47:20.872 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:47:20.872 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:47:20.873 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:47:20.873 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:47:20.873 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:47:20.873 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:47:20.873 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:47:20.873 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:47:20.873 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:47:20.873 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:47:20.873 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:47:20.873 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:47:20.873 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:47:20.873 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:47:20.873 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:47:20.874 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:47:20.874 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:47:20.874 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:47:20.874 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:47:20.874 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723995925.925075_273.1", 2024-08-18T15:47:20.874 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:47:20.874 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:47:20.874 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:47:20.874 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:45:25.925075Z", 2024-08-18T15:47:20.874 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:47:20.874 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:47:20.874 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:47:20.874 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:47:20.875 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:47:20.875 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:47:20.875 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:47:20.875 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:47:20.875 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:47:20.875 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:47:20Z" 2024-08-18T15:47:20.875 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:47:20.876 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1723996040.655168_301.1 target=1_1723995925.925075_273.1 2024-08-18T15:47:25.877 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:47:25.877 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:47:25.877 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:47:25.933 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:47:25.933 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:47:25.966 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:47:25.965+0000 7f85abc13880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:47:25.967 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:47:25.966+0000 7f85abc13880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:47:25.983 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:47:25.983 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:47:25.983 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:47:25.983 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:47:25.983 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:47:25.983 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T15:47:25.983 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:47:25.983 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:47:25.983 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:47:25.984 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:47:25.984 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:47:25.984 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:47:25.984 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:47:25.984 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:47:25.984 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:47:25.984 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:47:25.984 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:47:25.984 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:47:25.984 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:47:25.984 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:47:25.984 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:47:25.984 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:47:25.984 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:47:25.985 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:47:25.985 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:47:25.985 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:47:25.985 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:47:25.985 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:47:25.985 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:47:25.985 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:47:25.985 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:47:25.985 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:47:25.985 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:47:25.985 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:47:25.985 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:47:25.985 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:47:25.985 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:47:25.986 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:47:25.986 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:47:25.986 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:47:25.986 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:47:25.986 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:47:25.986 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:47:25.986 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:47:25.986 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:47:25.986 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:47:25.986 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:47:25.986 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:47:25.986 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:47:25.987 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723995925.925075_273.1", 2024-08-18T15:47:25.987 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:47:25.987 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:47:25.987 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:47:25.987 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:45:25.925075Z", 2024-08-18T15:47:25.987 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:47:25.987 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:47:25.987 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:47:25.987 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:47:25.987 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:47:25.987 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:47:25.987 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:47:25.987 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:47:25.987 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:47:25.988 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:47:25Z" 2024-08-18T15:47:25.988 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:47:25.989 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1723996040.655168_301.1 target=1_1723995925.925075_273.1 2024-08-18T15:47:30.989 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:47:30.990 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:47:30.990 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:47:31.045 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:47:31.046 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:47:31.078 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:47:31.077+0000 7fc0222bb880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:47:31.078 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:47:31.078+0000 7fc0222bb880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:47:31.092 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:47:31.092 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:47:31.092 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:47:31.092 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:47:31.092 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:47:31.092 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T15:47:31.093 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:47:31.093 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:47:31.093 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:47:31.093 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:47:31.093 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:47:31.093 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:47:31.093 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:47:31.093 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:47:31.093 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:47:31.093 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:47:31.093 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:47:31.093 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:47:31.094 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:47:31.094 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:47:31.094 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:47:31.094 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:47:31.094 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:47:31.094 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:47:31.094 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:47:31.094 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:47:31.094 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:47:31.094 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:47:31.094 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:47:31.095 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:47:31.095 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:47:31.095 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:47:31.095 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:47:31.095 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:47:31.095 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:47:31.095 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:47:31.095 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:47:31.095 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:47:31.095 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:47:31.095 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:47:31.095 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:47:31.095 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:47:31.096 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:47:31.096 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:47:31.096 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:47:31.096 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:47:31.096 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:47:31.096 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:47:31.096 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:47:31.096 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723995925.925075_273.1", 2024-08-18T15:47:31.096 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:47:31.096 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:47:31.096 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:47:31.096 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:45:25.925075Z", 2024-08-18T15:47:31.096 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:47:31.096 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:47:31.097 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:47:31.097 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:47:31.097 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:47:31.097 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:47:31.097 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:47:31.097 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:47:31.097 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:47:31.097 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:47:31Z" 2024-08-18T15:47:31.097 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:47:31.098 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1723996040.655168_301.1 target=1_1723995925.925075_273.1 2024-08-18T15:47:36.099 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:47:36.099 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:47:36.099 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:47:36.157 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:47:36.158 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:47:36.189 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:47:36.189+0000 7f0c43b19880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:47:36.190 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:47:36.190+0000 7f0c43b19880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:47:36.204 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:47:36.204 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:47:36.204 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:47:36.204 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:47:36.204 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:47:36.204 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T15:47:36.204 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:47:36.205 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:47:36.205 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:47:36.205 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:47:36.205 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:47:36.205 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:47:36.205 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:47:36.205 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:47:36.205 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:47:36.205 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:47:36.205 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:47:36.205 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:47:36.205 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:47:36.205 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:47:36.206 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:47:36.206 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:47:36.206 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:47:36.206 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:47:36.206 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:47:36.206 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:47:36.206 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:47:36.206 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:47:36.206 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:47:36.206 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:47:36.207 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:47:36.207 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:47:36.207 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:47:36.207 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:47:36.207 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:47:36.207 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:47:36.207 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:47:36.207 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:47:36.207 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:47:36.208 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:47:36.208 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:47:36.208 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:47:36.208 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:47:36.208 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:47:36.208 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:47:36.208 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:47:36.208 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:47:36.208 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:47:36.209 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:47:36.209 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723995925.925075_273.1", 2024-08-18T15:47:36.209 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:47:36.209 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:47:36.209 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:47:36.209 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:45:25.925075Z", 2024-08-18T15:47:36.209 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:47:36.209 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:47:36.210 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:47:36.210 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:47:36.210 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:47:36.210 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:47:36.210 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:47:36.210 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:47:36.210 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:47:36.210 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:47:36Z" 2024-08-18T15:47:36.211 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:47:36.212 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1723996040.655168_301.1 target=1_1723995925.925075_273.1 2024-08-18T15:47:41.213 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:47:41.213 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:47:41.213 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:47:41.269 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:47:41.269 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:47:41.301 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:47:41.300+0000 7fbf54d36880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:47:41.302 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:47:41.301+0000 7fbf54d36880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:47:41.315 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:47:41.315 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:47:41.316 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:47:41.316 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:47:41.316 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:47:41.316 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T15:47:41.316 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:47:41.316 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:47:41.316 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:47:41.316 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:47:41.316 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:47:41.316 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:47:41.316 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:47:41.316 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:47:41.317 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:47:41.317 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:47:41.317 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:47:41.317 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:47:41.317 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:47:41.317 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:47:41.317 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:47:41.317 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:47:41.317 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:47:41.317 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:47:41.317 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:47:41.317 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:47:41.317 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:47:41.318 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:47:41.318 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:47:41.318 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:47:41.318 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:47:41.318 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:47:41.318 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:47:41.318 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:47:41.318 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:47:41.318 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:47:41.318 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:47:41.318 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:47:41.318 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:47:41.319 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:47:41.319 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:47:41.319 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:47:41.319 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:47:41.319 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:47:41.319 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:47:41.319 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:47:41.319 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:47:41.319 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:47:41.319 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:47:41.319 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723995925.925075_273.1", 2024-08-18T15:47:41.319 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:47:41.319 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:47:41.320 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:47:41.320 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:45:25.925075Z", 2024-08-18T15:47:41.320 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:47:41.320 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:47:41.320 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:47:41.320 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:47:41.320 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:47:41.320 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:47:41.320 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:47:41.320 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:47:41.320 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:47:41.320 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:47:41Z" 2024-08-18T15:47:41.320 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:47:41.321 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1723996040.655168_301.1 target=1_1723995925.925075_273.1 2024-08-18T15:47:46.322 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:47:46.322 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:47:46.323 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:47:46.378 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:47:46.378 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:47:46.410 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:47:46.410+0000 7f8499d42880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:47:46.411 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:47:46.411+0000 7f8499d42880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:47:46.424 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:47:46.425 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:47:46.425 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:47:46.425 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:47:46.425 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:47:46.425 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T15:47:46.425 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:47:46.425 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:47:46.425 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:47:46.425 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:47:46.425 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:47:46.425 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:47:46.425 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:47:46.426 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:47:46.426 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:47:46.426 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:47:46.426 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:47:46.426 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:47:46.426 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:47:46.426 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:47:46.426 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:47:46.426 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:47:46.426 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:47:46.426 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:47:46.427 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:47:46.427 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:47:46.427 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:47:46.427 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:47:46.427 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:47:46.427 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:47:46.427 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:47:46.427 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:47:46.427 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:47:46.427 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:47:46.427 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:47:46.428 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:47:46.428 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:47:46.428 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:47:46.428 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:47:46.428 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:47:46.428 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:47:46.428 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:47:46.428 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:47:46.428 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:47:46.428 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:47:46.428 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:47:46.428 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:47:46.428 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:47:46.428 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:47:46.429 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723995925.925075_273.1", 2024-08-18T15:47:46.429 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:47:46.429 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:47:46.429 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:47:46.429 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:45:25.925075Z", 2024-08-18T15:47:46.429 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:47:46.429 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:47:46.429 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:47:46.429 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:47:46.429 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:47:46.429 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:47:46.429 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:47:46.429 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:47:46.429 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:47:46.429 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:47:46Z" 2024-08-18T15:47:46.430 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:47:46.431 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1723996040.655168_301.1 target=1_1723995925.925075_273.1 2024-08-18T15:47:51.432 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:47:51.432 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:47:51.432 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:47:51.488 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:47:51.488 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:47:51.520 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:47:51.520+0000 7f9734610880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:47:51.521 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:47:51.520+0000 7f9734610880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:47:51.534 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:47:51.534 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:47:51.534 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:47:51.535 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:47:51.535 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:47:51.535 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T15:47:51.535 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:47:51.535 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:47:51.535 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:47:51.535 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:47:51.535 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:47:51.535 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:47:51.535 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:47:51.535 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:47:51.535 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:47:51.536 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:47:51.536 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:47:51.536 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:47:51.536 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:47:51.536 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:47:51.536 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:47:51.536 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:47:51.536 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:47:51.536 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:47:51.536 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:47:51.536 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:47:51.536 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:47:51.537 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:47:51.537 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:47:51.537 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:47:51.537 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:47:51.537 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:47:51.537 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:47:51.537 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:47:51.537 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:47:51.537 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:47:51.537 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:47:51.537 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:47:51.537 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:47:51.537 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:47:51.538 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:47:51.538 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:47:51.538 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:47:51.538 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:47:51.538 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:47:51.538 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:47:51.538 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:47:51.538 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:47:51.538 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:47:51.538 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723995925.925075_273.1", 2024-08-18T15:47:51.539 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:47:51.539 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:47:51.539 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:47:51.539 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:45:25.925075Z", 2024-08-18T15:47:51.539 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:47:51.539 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:47:51.539 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:47:51.539 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:47:51.539 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:47:51.539 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:47:51.539 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:47:51.539 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:47:51.539 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:47:51.540 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:47:51Z" 2024-08-18T15:47:51.540 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:47:51.541 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1723996040.655168_301.1 target=1_1723995925.925075_273.1 2024-08-18T15:47:56.541 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:47:56.542 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:47:56.542 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:47:56.597 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:47:56.597 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:47:56.629 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:47:56.628+0000 7f4f6cc13880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:47:56.630 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:47:56.629+0000 7f4f6cc13880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:47:56.643 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:47:56.643 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:47:56.643 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:47:56.643 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:47:56.643 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:47:56.644 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T15:47:56.644 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:47:56.644 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:47:56.644 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:47:56.644 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:47:56.644 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:47:56.644 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:47:56.644 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:47:56.644 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:47:56.644 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:47:56.644 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:47:56.644 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:47:56.644 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:47:56.645 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:47:56.645 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:47:56.645 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:47:56.645 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:47:56.645 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:47:56.645 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:47:56.645 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:47:56.645 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:47:56.645 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:47:56.645 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:47:56.645 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:47:56.645 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:47:56.645 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:47:56.645 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:47:56.646 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:47:56.646 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:47:56.646 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:47:56.646 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:47:56.646 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:47:56.646 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:47:56.646 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:47:56.646 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:47:56.646 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:47:56.646 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:47:56.646 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:47:56.646 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:47:56.647 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:47:56.647 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:47:56.647 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:47:56.647 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:47:56.647 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:47:56.647 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723995925.925075_273.1", 2024-08-18T15:47:56.647 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:47:56.647 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:47:56.647 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:47:56.647 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:45:25.925075Z", 2024-08-18T15:47:56.647 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:47:56.647 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:47:56.647 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:47:56.647 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:47:56.648 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:47:56.648 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:47:56.648 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:47:56.648 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:47:56.648 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:47:56.648 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:47:56Z" 2024-08-18T15:47:56.648 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:47:56.649 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1723996040.655168_301.1 target=1_1723995925.925075_273.1 2024-08-18T15:48:01.650 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:48:01.650 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:48:01.650 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:48:01.705 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:48:01.705 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:48:01.737 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:48:01.737+0000 7f8bc9e0f880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:48:01.738 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:48:01.738+0000 7f8bc9e0f880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:48:01.752 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:48:01.752 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:48:01.752 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:48:01.752 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:48:01.752 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:48:01.752 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T15:48:01.752 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:48:01.752 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:48:01.752 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:48:01.752 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:48:01.752 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:48:01.753 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:48:01.753 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:48:01.753 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:48:01.753 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:48:01.753 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:48:01.753 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:48:01.753 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:48:01.753 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:48:01.753 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:48:01.753 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:48:01.753 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:48:01.753 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:48:01.753 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:48:01.754 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:48:01.754 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:48:01.754 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:48:01.754 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:48:01.754 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:48:01.754 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:48:01.754 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:48:01.754 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:48:01.754 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:48:01.754 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:48:01.754 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:48:01.754 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:48:01.754 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:48:01.755 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:48:01.755 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:48:01.755 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:48:01.755 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:48:01.755 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:48:01.755 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:48:01.755 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:48:01.755 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:48:01.755 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:48:01.755 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:48:01.755 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:48:01.755 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:48:01.755 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723995925.925075_273.1", 2024-08-18T15:48:01.756 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:48:01.756 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:48:01.756 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:48:01.756 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:45:25.925075Z", 2024-08-18T15:48:01.756 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:48:01.756 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:48:01.756 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:48:01.756 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:48:01.756 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:48:01.756 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:48:01.756 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:48:01.756 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:48:01.756 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:48:01.757 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:48:01Z" 2024-08-18T15:48:01.757 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:48:01.758 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1723996040.655168_301.1 target=1_1723995925.925075_273.1 2024-08-18T15:48:06.758 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:48:06.758 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:48:06.758 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:48:06.813 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:48:06.813 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:48:06.847 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:48:06.847+0000 7f4b8aa13880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:48:06.848 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:48:06.847+0000 7f4b8aa13880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:48:06.862 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:48:06.862 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:48:06.862 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:48:06.862 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:48:06.862 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:48:06.862 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T15:48:06.863 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:48:06.863 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:48:06.863 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:48:06.863 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:48:06.863 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:48:06.863 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:48:06.863 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:48:06.863 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:48:06.863 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:48:06.863 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:48:06.863 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:48:06.863 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:48:06.863 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:48:06.863 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:48:06.864 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:48:06.864 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:48:06.864 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:48:06.864 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:48:06.864 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:48:06.864 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:48:06.864 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:48:06.864 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:48:06.864 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:48:06.864 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:48:06.864 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:48:06.864 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:48:06.864 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:48:06.865 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:48:06.865 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:48:06.865 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:48:06.865 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:48:06.865 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:48:06.865 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:48:06.865 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:48:06.865 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:48:06.865 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:48:06.865 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:48:06.865 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:48:06.865 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:48:06.865 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:48:06.865 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:48:06.866 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:48:06.866 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:48:06.866 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723995925.925075_273.1", 2024-08-18T15:48:06.866 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:48:06.866 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:48:06.866 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:48:06.866 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:45:25.925075Z", 2024-08-18T15:48:06.866 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:48:06.866 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:48:06.866 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:48:06.867 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:48:06.867 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:48:06.867 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:48:06.867 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:48:06.867 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:48:06.867 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:48:06.867 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:48:06Z" 2024-08-18T15:48:06.867 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:48:06.868 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1723996040.655168_301.1 target=1_1723995925.925075_273.1 2024-08-18T15:48:11.869 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:48:11.870 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:48:11.870 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:48:11.927 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:48:11.927 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:48:11.960 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:48:11.959+0000 7f7618824880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:48:11.961 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:48:11.960+0000 7f7618824880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:48:11.974 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:48:11.974 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:48:11.974 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:48:11.975 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:48:11.975 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:48:11.975 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T15:48:11.975 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:48:11.975 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:48:11.975 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:48:11.975 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:48:11.975 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:48:11.975 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:48:11.975 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:48:11.975 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:48:11.975 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:48:11.975 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:48:11.975 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:48:11.976 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:48:11.976 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:48:11.976 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:48:11.976 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:48:11.976 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:48:11.976 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:48:11.976 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:48:11.976 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:48:11.976 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:48:11.976 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:48:11.976 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:48:11.976 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:48:11.976 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:48:11.976 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:48:11.977 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:48:11.977 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:48:11.977 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:48:11.977 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:48:11.977 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:48:11.977 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:48:11.977 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:48:11.977 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:48:11.977 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:48:11.977 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:48:11.977 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:48:11.977 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:48:11.977 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:48:11.978 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:48:11.978 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:48:11.978 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:48:11.978 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:48:11.978 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:48:11.978 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723995925.925075_273.1", 2024-08-18T15:48:11.978 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:48:11.978 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:48:11.978 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:48:11.978 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:45:25.925075Z", 2024-08-18T15:48:11.978 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:48:11.978 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:48:11.978 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:48:11.978 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:48:11.979 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:48:11.979 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:48:11.979 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:48:11.979 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:48:11.979 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:48:11.979 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:48:11Z" 2024-08-18T15:48:11.979 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:48:11.980 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1723996040.655168_301.1 target=1_1723995925.925075_273.1 2024-08-18T15:48:16.981 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:48:16.982 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:48:16.982 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:48:17.037 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:48:17.037 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:48:17.069 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:48:17.069+0000 7efd8b8bb880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:48:17.070 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:48:17.070+0000 7efd8b8bb880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:48:17.084 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:48:17.084 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:48:17.084 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:48:17.084 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:48:17.084 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:48:17.084 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T15:48:17.084 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:48:17.084 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:48:17.084 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:48:17.084 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:48:17.084 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:48:17.085 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:48:17.085 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:48:17.085 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:48:17.085 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:48:17.085 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:48:17.085 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:48:17.085 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:48:17.085 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:48:17.085 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:48:17.085 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:48:17.085 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:48:17.085 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:48:17.085 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:48:17.085 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:48:17.086 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:48:17.086 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:48:17.086 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:48:17.086 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:48:17.086 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:48:17.086 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:48:17.086 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:48:17.086 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:48:17.086 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:48:17.086 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:48:17.086 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:48:17.086 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:48:17.087 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:48:17.087 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:48:17.087 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:48:17.087 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:48:17.087 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:48:17.087 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:48:17.087 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:48:17.087 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:48:17.087 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:48:17.087 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:48:17.087 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:48:17.087 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:48:17.087 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723995925.925075_273.1", 2024-08-18T15:48:17.088 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:48:17.088 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:48:17.088 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:48:17.088 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:45:25.925075Z", 2024-08-18T15:48:17.088 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:48:17.088 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:48:17.088 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:48:17.088 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:48:17.088 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:48:17.088 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:48:17.088 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:48:17.088 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:48:17.088 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:48:17.088 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:48:17Z" 2024-08-18T15:48:17.089 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:48:17.090 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1723996040.655168_301.1 target=1_1723995925.925075_273.1 2024-08-18T15:48:22.091 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:48:22.091 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:48:22.091 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:48:22.140 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:48:22.140 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:48:22.169 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:48:22.168+0000 7f077f03f880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:48:22.169 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:48:22.169+0000 7f077f03f880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:48:22.182 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:48:22.182 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:48:22.182 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:48:22.182 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:48:22.182 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:48:22.182 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T15:48:22.182 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:48:22.182 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:48:22.182 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:48:22.183 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:48:22.183 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:48:22.183 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:48:22.183 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:48:22.183 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:48:22.183 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:48:22.183 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:48:22.183 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:48:22.183 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:48:22.183 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:48:22.183 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:48:22.183 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:48:22.183 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:48:22.184 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:48:22.184 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:48:22.184 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:48:22.184 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:48:22.184 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:48:22.184 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:48:22.184 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:48:22.184 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:48:22.184 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:48:22.184 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:48:22.184 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:48:22.184 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:48:22.184 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:48:22.184 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:48:22.185 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:48:22.185 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:48:22.185 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:48:22.185 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:48:22.185 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:48:22.185 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:48:22.185 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:48:22.185 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:48:22.185 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:48:22.185 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:48:22.185 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:48:22.185 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:48:22.185 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:48:22.186 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723996040.655168_301.1", 2024-08-18T15:48:22.186 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:48:22.186 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:48:22.186 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:48:22.186 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:47:20.655168Z", 2024-08-18T15:48:22.186 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:48:22.186 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:48:22.186 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:48:22.186 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:48:22.186 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:48:22.186 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:48:22.186 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:48:22.186 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:48:22.186 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:48:22.187 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:48:22Z" 2024-08-18T15:48:22.187 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:48:22.188 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-08-18T15:48:22.188 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-08-18T15:48:22.188 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:48:22.188 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:48:22.188 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:48:22.235 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:48:22.236 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:48:22.264 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:48:22.263+0000 7f7a99a86880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:48:22.265 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:48:22.264+0000 7f7a99a86880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:48:22.276 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T15:48:22.276 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T15:48:22.277 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T15:48:22.277 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T15:48:22.277 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T15:48:22.277 INFO:teuthology.orchestra.run.smithi134.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T15:48:22.277 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T15:48:22.278 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:48:22.278 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T15:48:22.278 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:48:22.278 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T15:48:22.278 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:48:22.278 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:48:22.278 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:48:22.278 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:48:22.279 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 1, 2024-08-18T15:48:22.279 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:48:22.279 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:48:22.279 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:48:22.279 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:48:22.280 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:48:22.280 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:48:22.280 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T15:48:22.280 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:48:22.280 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:48:22.281 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T15:48:22.281 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:48:22.281 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:48:22.281 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:48:22.281 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T15:48:22.281 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:48:22.282 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:48:22.282 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:48:22.282 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:48:22.282 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T15:48:22.282 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:48:22.283 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:48:22.283 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:48:22.283 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:48:22.283 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:48:22.283 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:48:22.283 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:48:22.283 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:48:22.284 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:48:22.284 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:48:22.284 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:48:22.284 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T15:48:22.284 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:48:22.284 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:48:22.284 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723996040.655168_301.1", 2024-08-18T15:48:22.284 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:48:22.284 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:48:22.284 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:48:22.285 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:47:20.655168Z", 2024-08-18T15:48:22.285 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T15:48:22.285 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:48:22.285 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:48:22.285 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T15:48:22.286 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:48:22.286 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T15:48:22.286 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 1, 2024-08-18T15:48:22.286 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 1 2024-08-18T15:48:22.286 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:48:22.286 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T15:48:22Z" 2024-08-18T15:48:22.286 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T15:48:22.287 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-08-18T15:48:22.296 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=96bf8502-2f39-4d53-8196-3378c28da3b4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:48:22.296 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=96bf8502-2f39-4d53-8196-3378c28da3b4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:48:22.296 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --redirect-zone=96bf8502-2f39-4d53-8196-3378c28da3b4 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-08-18T15:48:22.343 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:48:22.343 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:48:22.363 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:48:22.363 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "10090ccf-aad6-416d-9775-df8b7b27345a", 2024-08-18T15:48:22.363 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "test-zone2", 2024-08-18T15:48:22.363 INFO:teuthology.orchestra.run.smithi193.stdout: "domain_root": "test-zone2.rgw.meta:root", 2024-08-18T15:48:22.363 INFO:teuthology.orchestra.run.smithi193.stdout: "control_pool": "test-zone2.rgw.control", 2024-08-18T15:48:22.363 INFO:teuthology.orchestra.run.smithi193.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2024-08-18T15:48:22.363 INFO:teuthology.orchestra.run.smithi193.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2024-08-18T15:48:22.364 INFO:teuthology.orchestra.run.smithi193.stdout: "log_pool": "test-zone2.rgw.log", 2024-08-18T15:48:22.364 INFO:teuthology.orchestra.run.smithi193.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2024-08-18T15:48:22.364 INFO:teuthology.orchestra.run.smithi193.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2024-08-18T15:48:22.364 INFO:teuthology.orchestra.run.smithi193.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2024-08-18T15:48:22.364 INFO:teuthology.orchestra.run.smithi193.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2024-08-18T15:48:22.364 INFO:teuthology.orchestra.run.smithi193.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2024-08-18T15:48:22.364 INFO:teuthology.orchestra.run.smithi193.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2024-08-18T15:48:22.364 INFO:teuthology.orchestra.run.smithi193.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2024-08-18T15:48:22.364 INFO:teuthology.orchestra.run.smithi193.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2024-08-18T15:48:22.364 INFO:teuthology.orchestra.run.smithi193.stdout: "otp_pool": "test-zone2.rgw.otp", 2024-08-18T15:48:22.365 INFO:teuthology.orchestra.run.smithi193.stdout: "system_key": { 2024-08-18T15:48:22.365 INFO:teuthology.orchestra.run.smithi193.stdout: "access_key": "MRX9HX7RYBDXO1UM", 2024-08-18T15:48:22.365 INFO:teuthology.orchestra.run.smithi193.stdout: "secret_key": "hGUsvhUdpGZAYWjQwu3StBaYXJPhuCEj" 2024-08-18T15:48:22.365 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:48:22.365 INFO:teuthology.orchestra.run.smithi193.stdout: "placement_pools": [ 2024-08-18T15:48:22.365 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:48:22.365 INFO:teuthology.orchestra.run.smithi193.stdout: "key": "default-placement", 2024-08-18T15:48:22.365 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:48:22.365 INFO:teuthology.orchestra.run.smithi193.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2024-08-18T15:48:22.366 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_classes": { 2024-08-18T15:48:22.366 INFO:teuthology.orchestra.run.smithi193.stdout: "STANDARD": { 2024-08-18T15:48:22.366 INFO:teuthology.orchestra.run.smithi193.stdout: "data_pool": "test-zone2.rgw.buckets.data", 2024-08-18T15:48:22.366 INFO:teuthology.orchestra.run.smithi193.stdout: "compression_type": "random" 2024-08-18T15:48:22.366 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:48:22.366 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:48:22.366 INFO:teuthology.orchestra.run.smithi193.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2024-08-18T15:48:22.366 INFO:teuthology.orchestra.run.smithi193.stdout: "index_type": 0, 2024-08-18T15:48:22.367 INFO:teuthology.orchestra.run.smithi193.stdout: "inline_data": true 2024-08-18T15:48:22.367 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:48:22.367 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:48:22.367 INFO:teuthology.orchestra.run.smithi193.stdout: ], 2024-08-18T15:48:22.367 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_id": "2827c6f0-bf0d-4836-944c-9e7fcf22457b", 2024-08-18T15:48:22.367 INFO:teuthology.orchestra.run.smithi193.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2024-08-18T15:48:22.367 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:48:22.368 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:48:22.369 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:48:22.369 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2024-08-18T15:48:22.415 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:48:22.415 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:48:22.455 INFO:teuthology.orchestra.run.smithi193.stderr:Sending period to new master zone 96bf8502-2f39-4d53-8196-3378c28da3b4 2024-08-18T15:48:22.473 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:48:22.474 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T15:48:22.474 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 3, 2024-08-18T15:48:22.474 INFO:teuthology.orchestra.run.smithi193.stdout: "predecessor_uuid": "64309209-7567-46e5-8881-65783d84cf3c", 2024-08-18T15:48:22.474 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": [ 2024-08-18T15:48:22.474 INFO:teuthology.orchestra.run.smithi193.stdout: "", 2024-08-18T15:48:22.474 INFO:teuthology.orchestra.run.smithi193.stdout: "", 2024-08-18T15:48:22.474 INFO:teuthology.orchestra.run.smithi193.stdout: "", 2024-08-18T15:48:22.474 INFO:teuthology.orchestra.run.smithi193.stdout: "1_1723995848.930494_187.1" 2024-08-18T15:48:22.474 INFO:teuthology.orchestra.run.smithi193.stdout: ], 2024-08-18T15:48:22.474 INFO:teuthology.orchestra.run.smithi193.stdout: "period_map": { 2024-08-18T15:48:22.474 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T15:48:22.475 INFO:teuthology.orchestra.run.smithi193.stdout: "zonegroups": [ 2024-08-18T15:48:22.475 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:48:22.475 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "test-zonegroup", 2024-08-18T15:48:22.475 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "test-zonegroup", 2024-08-18T15:48:22.475 INFO:teuthology.orchestra.run.smithi193.stdout: "api_name": "", 2024-08-18T15:48:22.475 INFO:teuthology.orchestra.run.smithi193.stdout: "is_master": true, 2024-08-18T15:48:22.475 INFO:teuthology.orchestra.run.smithi193.stdout: "endpoints": [ 2024-08-18T15:48:22.475 INFO:teuthology.orchestra.run.smithi193.stdout: "http://smithi134.front.sepia.ceph.com:8000" 2024-08-18T15:48:22.475 INFO:teuthology.orchestra.run.smithi193.stdout: ], 2024-08-18T15:48:22.475 INFO:teuthology.orchestra.run.smithi193.stdout: "hostnames": [], 2024-08-18T15:48:22.475 INFO:teuthology.orchestra.run.smithi193.stdout: "hostnames_s3website": [], 2024-08-18T15:48:22.475 INFO:teuthology.orchestra.run.smithi193.stdout: "master_zone": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:48:22.476 INFO:teuthology.orchestra.run.smithi193.stdout: "zones": [ 2024-08-18T15:48:22.476 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:48:22.476 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T15:48:22.476 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "test-zone3", 2024-08-18T15:48:22.476 INFO:teuthology.orchestra.run.smithi193.stdout: "endpoints": [ 2024-08-18T15:48:22.476 INFO:teuthology.orchestra.run.smithi193.stdout: "http://smithi134.front.sepia.ceph.com:8001" 2024-08-18T15:48:22.476 INFO:teuthology.orchestra.run.smithi193.stdout: ], 2024-08-18T15:48:22.476 INFO:teuthology.orchestra.run.smithi193.stdout: "log_meta": false, 2024-08-18T15:48:22.476 INFO:teuthology.orchestra.run.smithi193.stdout: "log_data": true, 2024-08-18T15:48:22.476 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket_index_max_shards": 11, 2024-08-18T15:48:22.476 INFO:teuthology.orchestra.run.smithi193.stdout: "read_only": false, 2024-08-18T15:48:22.476 INFO:teuthology.orchestra.run.smithi193.stdout: "tier_type": "", 2024-08-18T15:48:22.476 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_from_all": true, 2024-08-18T15:48:22.477 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_from": [], 2024-08-18T15:48:22.477 INFO:teuthology.orchestra.run.smithi193.stdout: "redirect_zone": "", 2024-08-18T15:48:22.477 INFO:teuthology.orchestra.run.smithi193.stdout: "supported_features": [ 2024-08-18T15:48:22.477 INFO:teuthology.orchestra.run.smithi193.stdout: "compress-encrypted", 2024-08-18T15:48:22.477 INFO:teuthology.orchestra.run.smithi193.stdout: "resharding" 2024-08-18T15:48:22.477 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:48:22.477 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:48:22.477 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:48:22.477 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "10090ccf-aad6-416d-9775-df8b7b27345a", 2024-08-18T15:48:22.477 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "test-zone2", 2024-08-18T15:48:22.477 INFO:teuthology.orchestra.run.smithi193.stdout: "endpoints": [ 2024-08-18T15:48:22.477 INFO:teuthology.orchestra.run.smithi193.stdout: "http://smithi193.front.sepia.ceph.com:8000" 2024-08-18T15:48:22.477 INFO:teuthology.orchestra.run.smithi193.stdout: ], 2024-08-18T15:48:22.478 INFO:teuthology.orchestra.run.smithi193.stdout: "log_meta": false, 2024-08-18T15:48:22.478 INFO:teuthology.orchestra.run.smithi193.stdout: "log_data": true, 2024-08-18T15:48:22.478 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket_index_max_shards": 11, 2024-08-18T15:48:22.478 INFO:teuthology.orchestra.run.smithi193.stdout: "read_only": false, 2024-08-18T15:48:22.478 INFO:teuthology.orchestra.run.smithi193.stdout: "tier_type": "", 2024-08-18T15:48:22.478 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_from_all": false, 2024-08-18T15:48:22.478 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_from": [], 2024-08-18T15:48:22.478 INFO:teuthology.orchestra.run.smithi193.stdout: "redirect_zone": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:48:22.478 INFO:teuthology.orchestra.run.smithi193.stdout: "supported_features": [ 2024-08-18T15:48:22.478 INFO:teuthology.orchestra.run.smithi193.stdout: "compress-encrypted", 2024-08-18T15:48:22.478 INFO:teuthology.orchestra.run.smithi193.stdout: "resharding" 2024-08-18T15:48:22.478 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:48:22.478 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:48:22.479 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:48:22.479 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:48:22.479 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "test-zone1", 2024-08-18T15:48:22.479 INFO:teuthology.orchestra.run.smithi193.stdout: "endpoints": [ 2024-08-18T15:48:22.479 INFO:teuthology.orchestra.run.smithi193.stdout: "http://smithi134.front.sepia.ceph.com:8000" 2024-08-18T15:48:22.479 INFO:teuthology.orchestra.run.smithi193.stdout: ], 2024-08-18T15:48:22.479 INFO:teuthology.orchestra.run.smithi193.stdout: "log_meta": false, 2024-08-18T15:48:22.479 INFO:teuthology.orchestra.run.smithi193.stdout: "log_data": true, 2024-08-18T15:48:22.479 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket_index_max_shards": 11, 2024-08-18T15:48:22.479 INFO:teuthology.orchestra.run.smithi193.stdout: "read_only": false, 2024-08-18T15:48:22.479 INFO:teuthology.orchestra.run.smithi193.stdout: "tier_type": "", 2024-08-18T15:48:22.479 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_from_all": true, 2024-08-18T15:48:22.479 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_from": [], 2024-08-18T15:48:22.480 INFO:teuthology.orchestra.run.smithi193.stdout: "redirect_zone": "", 2024-08-18T15:48:22.480 INFO:teuthology.orchestra.run.smithi193.stdout: "supported_features": [ 2024-08-18T15:48:22.480 INFO:teuthology.orchestra.run.smithi193.stdout: "compress-encrypted", 2024-08-18T15:48:22.480 INFO:teuthology.orchestra.run.smithi193.stdout: "resharding" 2024-08-18T15:48:22.480 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:48:22.480 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:48:22.480 INFO:teuthology.orchestra.run.smithi193.stdout: ], 2024-08-18T15:48:22.480 INFO:teuthology.orchestra.run.smithi193.stdout: "placement_targets": [ 2024-08-18T15:48:22.480 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:48:22.480 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "default-placement", 2024-08-18T15:48:22.480 INFO:teuthology.orchestra.run.smithi193.stdout: "tags": [], 2024-08-18T15:48:22.480 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_classes": [ 2024-08-18T15:48:22.480 INFO:teuthology.orchestra.run.smithi193.stdout: "STANDARD" 2024-08-18T15:48:22.480 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:48:22.481 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:48:22.481 INFO:teuthology.orchestra.run.smithi193.stdout: ], 2024-08-18T15:48:22.481 INFO:teuthology.orchestra.run.smithi193.stdout: "default_placement": "default-placement", 2024-08-18T15:48:22.481 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_id": "2827c6f0-bf0d-4836-944c-9e7fcf22457b", 2024-08-18T15:48:22.481 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_policy": { 2024-08-18T15:48:22.481 INFO:teuthology.orchestra.run.smithi193.stdout: "groups": [] 2024-08-18T15:48:22.481 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:48:22.481 INFO:teuthology.orchestra.run.smithi193.stdout: "enabled_features": [ 2024-08-18T15:48:22.481 INFO:teuthology.orchestra.run.smithi193.stdout: "resharding" 2024-08-18T15:48:22.481 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:48:22.481 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:48:22.481 INFO:teuthology.orchestra.run.smithi193.stdout: ], 2024-08-18T15:48:22.481 INFO:teuthology.orchestra.run.smithi193.stdout: "short_zone_ids": [ 2024-08-18T15:48:22.482 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:48:22.482 INFO:teuthology.orchestra.run.smithi193.stdout: "key": "018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T15:48:22.482 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 462036791 2024-08-18T15:48:22.482 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:48:22.485 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:48:22.485 INFO:teuthology.orchestra.run.smithi193.stdout: "key": "10090ccf-aad6-416d-9775-df8b7b27345a", 2024-08-18T15:48:22.485 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 874289624 2024-08-18T15:48:22.485 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:48:22.486 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:48:22.486 INFO:teuthology.orchestra.run.smithi193.stdout: "key": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:48:22.486 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 1721399886 2024-08-18T15:48:22.486 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:48:22.486 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:48:22.486 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:48:22.486 INFO:teuthology.orchestra.run.smithi193.stdout: "master_zonegroup": "test-zonegroup", 2024-08-18T15:48:22.486 INFO:teuthology.orchestra.run.smithi193.stdout: "master_zone": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:48:22.486 INFO:teuthology.orchestra.run.smithi193.stdout: "period_config": { 2024-08-18T15:48:22.486 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket_quota": { 2024-08-18T15:48:22.486 INFO:teuthology.orchestra.run.smithi193.stdout: "enabled": false, 2024-08-18T15:48:22.486 INFO:teuthology.orchestra.run.smithi193.stdout: "check_on_raw": false, 2024-08-18T15:48:22.486 INFO:teuthology.orchestra.run.smithi193.stdout: "max_size": -1, 2024-08-18T15:48:22.487 INFO:teuthology.orchestra.run.smithi193.stdout: "max_size_kb": 0, 2024-08-18T15:48:22.487 INFO:teuthology.orchestra.run.smithi193.stdout: "max_objects": -1 2024-08-18T15:48:22.487 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:48:22.487 INFO:teuthology.orchestra.run.smithi193.stdout: "user_quota": { 2024-08-18T15:48:22.487 INFO:teuthology.orchestra.run.smithi193.stdout: "enabled": false, 2024-08-18T15:48:22.487 INFO:teuthology.orchestra.run.smithi193.stdout: "check_on_raw": false, 2024-08-18T15:48:22.487 INFO:teuthology.orchestra.run.smithi193.stdout: "max_size": -1, 2024-08-18T15:48:22.487 INFO:teuthology.orchestra.run.smithi193.stdout: "max_size_kb": 0, 2024-08-18T15:48:22.487 INFO:teuthology.orchestra.run.smithi193.stdout: "max_objects": -1 2024-08-18T15:48:22.487 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:48:22.487 INFO:teuthology.orchestra.run.smithi193.stdout: "user_ratelimit": { 2024-08-18T15:48:22.487 INFO:teuthology.orchestra.run.smithi193.stdout: "max_read_ops": 0, 2024-08-18T15:48:22.487 INFO:teuthology.orchestra.run.smithi193.stdout: "max_write_ops": 0, 2024-08-18T15:48:22.488 INFO:teuthology.orchestra.run.smithi193.stdout: "max_read_bytes": 0, 2024-08-18T15:48:22.488 INFO:teuthology.orchestra.run.smithi193.stdout: "max_write_bytes": 0, 2024-08-18T15:48:22.488 INFO:teuthology.orchestra.run.smithi193.stdout: "enabled": false 2024-08-18T15:48:22.488 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:48:22.488 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket_ratelimit": { 2024-08-18T15:48:22.488 INFO:teuthology.orchestra.run.smithi193.stdout: "max_read_ops": 0, 2024-08-18T15:48:22.488 INFO:teuthology.orchestra.run.smithi193.stdout: "max_write_ops": 0, 2024-08-18T15:48:22.488 INFO:teuthology.orchestra.run.smithi193.stdout: "max_read_bytes": 0, 2024-08-18T15:48:22.488 INFO:teuthology.orchestra.run.smithi193.stdout: "max_write_bytes": 0, 2024-08-18T15:48:22.488 INFO:teuthology.orchestra.run.smithi193.stdout: "enabled": false 2024-08-18T15:48:22.488 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:48:22.488 INFO:teuthology.orchestra.run.smithi193.stdout: "anonymous_ratelimit": { 2024-08-18T15:48:22.488 INFO:teuthology.orchestra.run.smithi193.stdout: "max_read_ops": 0, 2024-08-18T15:48:22.488 INFO:teuthology.orchestra.run.smithi193.stdout: "max_write_ops": 0, 2024-08-18T15:48:22.488 INFO:teuthology.orchestra.run.smithi193.stdout: "max_read_bytes": 0, 2024-08-18T15:48:22.489 INFO:teuthology.orchestra.run.smithi193.stdout: "max_write_bytes": 0, 2024-08-18T15:48:22.489 INFO:teuthology.orchestra.run.smithi193.stdout: "enabled": false 2024-08-18T15:48:22.489 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:48:22.489 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:48:22.489 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_id": "2827c6f0-bf0d-4836-944c-9e7fcf22457b", 2024-08-18T15:48:22.489 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_name": "test-realm", 2024-08-18T15:48:22.489 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:48:22.489 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:48:22.490 INFO:rgw_multi.tests:Set redirect_zone zone test-zone2 to "96bf8502-2f39-4d53-8196-3378c28da3b4" 2024-08-18T15:49:22.602 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=true', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:49:22.602 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=true', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:49:22.603 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --sync-from-all=true --cluster c2 --debug-rgw 0 --debug-ms 0 2024-08-18T15:49:22.659 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:49:22.659 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:49:22.682 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:49:22.683 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "10090ccf-aad6-416d-9775-df8b7b27345a", 2024-08-18T15:49:22.683 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "test-zone2", 2024-08-18T15:49:22.683 INFO:teuthology.orchestra.run.smithi193.stdout: "domain_root": "test-zone2.rgw.meta:root", 2024-08-18T15:49:22.683 INFO:teuthology.orchestra.run.smithi193.stdout: "control_pool": "test-zone2.rgw.control", 2024-08-18T15:49:22.683 INFO:teuthology.orchestra.run.smithi193.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2024-08-18T15:49:22.683 INFO:teuthology.orchestra.run.smithi193.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2024-08-18T15:49:22.683 INFO:teuthology.orchestra.run.smithi193.stdout: "log_pool": "test-zone2.rgw.log", 2024-08-18T15:49:22.683 INFO:teuthology.orchestra.run.smithi193.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2024-08-18T15:49:22.683 INFO:teuthology.orchestra.run.smithi193.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2024-08-18T15:49:22.683 INFO:teuthology.orchestra.run.smithi193.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2024-08-18T15:49:22.683 INFO:teuthology.orchestra.run.smithi193.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2024-08-18T15:49:22.684 INFO:teuthology.orchestra.run.smithi193.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2024-08-18T15:49:22.684 INFO:teuthology.orchestra.run.smithi193.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2024-08-18T15:49:22.684 INFO:teuthology.orchestra.run.smithi193.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2024-08-18T15:49:22.684 INFO:teuthology.orchestra.run.smithi193.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2024-08-18T15:49:22.684 INFO:teuthology.orchestra.run.smithi193.stdout: "otp_pool": "test-zone2.rgw.otp", 2024-08-18T15:49:22.684 INFO:teuthology.orchestra.run.smithi193.stdout: "system_key": { 2024-08-18T15:49:22.684 INFO:teuthology.orchestra.run.smithi193.stdout: "access_key": "MRX9HX7RYBDXO1UM", 2024-08-18T15:49:22.684 INFO:teuthology.orchestra.run.smithi193.stdout: "secret_key": "hGUsvhUdpGZAYWjQwu3StBaYXJPhuCEj" 2024-08-18T15:49:22.684 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:49:22.684 INFO:teuthology.orchestra.run.smithi193.stdout: "placement_pools": [ 2024-08-18T15:49:22.684 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:49:22.684 INFO:teuthology.orchestra.run.smithi193.stdout: "key": "default-placement", 2024-08-18T15:49:22.684 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:49:22.685 INFO:teuthology.orchestra.run.smithi193.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2024-08-18T15:49:22.685 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_classes": { 2024-08-18T15:49:22.685 INFO:teuthology.orchestra.run.smithi193.stdout: "STANDARD": { 2024-08-18T15:49:22.685 INFO:teuthology.orchestra.run.smithi193.stdout: "data_pool": "test-zone2.rgw.buckets.data", 2024-08-18T15:49:22.685 INFO:teuthology.orchestra.run.smithi193.stdout: "compression_type": "random" 2024-08-18T15:49:22.685 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:49:22.685 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:49:22.685 INFO:teuthology.orchestra.run.smithi193.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2024-08-18T15:49:22.685 INFO:teuthology.orchestra.run.smithi193.stdout: "index_type": 0, 2024-08-18T15:49:22.685 INFO:teuthology.orchestra.run.smithi193.stdout: "inline_data": true 2024-08-18T15:49:22.685 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:49:22.685 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:49:22.685 INFO:teuthology.orchestra.run.smithi193.stdout: ], 2024-08-18T15:49:22.685 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_id": "2827c6f0-bf0d-4836-944c-9e7fcf22457b", 2024-08-18T15:49:22.686 INFO:teuthology.orchestra.run.smithi193.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2024-08-18T15:49:22.686 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:49:22.687 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:49:22.687 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:49:22.687 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2024-08-18T15:49:22.735 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:49:22.735 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:49:22.774 INFO:teuthology.orchestra.run.smithi193.stderr:Sending period to new master zone 96bf8502-2f39-4d53-8196-3378c28da3b4 2024-08-18T15:49:22.796 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:49:22.796 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T15:49:22.796 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 4, 2024-08-18T15:49:22.796 INFO:teuthology.orchestra.run.smithi193.stdout: "predecessor_uuid": "64309209-7567-46e5-8881-65783d84cf3c", 2024-08-18T15:49:22.796 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": [ 2024-08-18T15:49:22.796 INFO:teuthology.orchestra.run.smithi193.stdout: "", 2024-08-18T15:49:22.797 INFO:teuthology.orchestra.run.smithi193.stdout: "", 2024-08-18T15:49:22.797 INFO:teuthology.orchestra.run.smithi193.stdout: "", 2024-08-18T15:49:22.797 INFO:teuthology.orchestra.run.smithi193.stdout: "1_1723995848.930494_187.1" 2024-08-18T15:49:22.797 INFO:teuthology.orchestra.run.smithi193.stdout: ], 2024-08-18T15:49:22.797 INFO:teuthology.orchestra.run.smithi193.stdout: "period_map": { 2024-08-18T15:49:22.797 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T15:49:22.797 INFO:teuthology.orchestra.run.smithi193.stdout: "zonegroups": [ 2024-08-18T15:49:22.797 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:49:22.797 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "test-zonegroup", 2024-08-18T15:49:22.797 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "test-zonegroup", 2024-08-18T15:49:22.797 INFO:teuthology.orchestra.run.smithi193.stdout: "api_name": "", 2024-08-18T15:49:22.797 INFO:teuthology.orchestra.run.smithi193.stdout: "is_master": true, 2024-08-18T15:49:22.797 INFO:teuthology.orchestra.run.smithi193.stdout: "endpoints": [ 2024-08-18T15:49:22.798 INFO:teuthology.orchestra.run.smithi193.stdout: "http://smithi134.front.sepia.ceph.com:8000" 2024-08-18T15:49:22.798 INFO:teuthology.orchestra.run.smithi193.stdout: ], 2024-08-18T15:49:22.798 INFO:teuthology.orchestra.run.smithi193.stdout: "hostnames": [], 2024-08-18T15:49:22.798 INFO:teuthology.orchestra.run.smithi193.stdout: "hostnames_s3website": [], 2024-08-18T15:49:22.798 INFO:teuthology.orchestra.run.smithi193.stdout: "master_zone": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:49:22.798 INFO:teuthology.orchestra.run.smithi193.stdout: "zones": [ 2024-08-18T15:49:22.798 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:49:22.798 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T15:49:22.798 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "test-zone3", 2024-08-18T15:49:22.798 INFO:teuthology.orchestra.run.smithi193.stdout: "endpoints": [ 2024-08-18T15:49:22.798 INFO:teuthology.orchestra.run.smithi193.stdout: "http://smithi134.front.sepia.ceph.com:8001" 2024-08-18T15:49:22.799 INFO:teuthology.orchestra.run.smithi193.stdout: ], 2024-08-18T15:49:22.799 INFO:teuthology.orchestra.run.smithi193.stdout: "log_meta": false, 2024-08-18T15:49:22.799 INFO:teuthology.orchestra.run.smithi193.stdout: "log_data": true, 2024-08-18T15:49:22.799 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket_index_max_shards": 11, 2024-08-18T15:49:22.799 INFO:teuthology.orchestra.run.smithi193.stdout: "read_only": false, 2024-08-18T15:49:22.799 INFO:teuthology.orchestra.run.smithi193.stdout: "tier_type": "", 2024-08-18T15:49:22.799 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_from_all": true, 2024-08-18T15:49:22.799 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_from": [], 2024-08-18T15:49:22.799 INFO:teuthology.orchestra.run.smithi193.stdout: "redirect_zone": "", 2024-08-18T15:49:22.799 INFO:teuthology.orchestra.run.smithi193.stdout: "supported_features": [ 2024-08-18T15:49:22.799 INFO:teuthology.orchestra.run.smithi193.stdout: "compress-encrypted", 2024-08-18T15:49:22.799 INFO:teuthology.orchestra.run.smithi193.stdout: "resharding" 2024-08-18T15:49:22.799 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:49:22.799 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:49:22.800 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:49:22.800 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "10090ccf-aad6-416d-9775-df8b7b27345a", 2024-08-18T15:49:22.800 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "test-zone2", 2024-08-18T15:49:22.800 INFO:teuthology.orchestra.run.smithi193.stdout: "endpoints": [ 2024-08-18T15:49:22.800 INFO:teuthology.orchestra.run.smithi193.stdout: "http://smithi193.front.sepia.ceph.com:8000" 2024-08-18T15:49:22.800 INFO:teuthology.orchestra.run.smithi193.stdout: ], 2024-08-18T15:49:22.800 INFO:teuthology.orchestra.run.smithi193.stdout: "log_meta": false, 2024-08-18T15:49:22.800 INFO:teuthology.orchestra.run.smithi193.stdout: "log_data": true, 2024-08-18T15:49:22.800 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket_index_max_shards": 11, 2024-08-18T15:49:22.800 INFO:teuthology.orchestra.run.smithi193.stdout: "read_only": false, 2024-08-18T15:49:22.800 INFO:teuthology.orchestra.run.smithi193.stdout: "tier_type": "", 2024-08-18T15:49:22.800 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_from_all": true, 2024-08-18T15:49:22.800 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_from": [], 2024-08-18T15:49:22.801 INFO:teuthology.orchestra.run.smithi193.stdout: "redirect_zone": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:49:22.801 INFO:teuthology.orchestra.run.smithi193.stdout: "supported_features": [ 2024-08-18T15:49:22.801 INFO:teuthology.orchestra.run.smithi193.stdout: "compress-encrypted", 2024-08-18T15:49:22.801 INFO:teuthology.orchestra.run.smithi193.stdout: "resharding" 2024-08-18T15:49:22.801 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:49:22.801 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:49:22.801 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:49:22.801 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:49:22.801 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "test-zone1", 2024-08-18T15:49:22.801 INFO:teuthology.orchestra.run.smithi193.stdout: "endpoints": [ 2024-08-18T15:49:22.801 INFO:teuthology.orchestra.run.smithi193.stdout: "http://smithi134.front.sepia.ceph.com:8000" 2024-08-18T15:49:22.801 INFO:teuthology.orchestra.run.smithi193.stdout: ], 2024-08-18T15:49:22.801 INFO:teuthology.orchestra.run.smithi193.stdout: "log_meta": false, 2024-08-18T15:49:22.801 INFO:teuthology.orchestra.run.smithi193.stdout: "log_data": true, 2024-08-18T15:49:22.802 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket_index_max_shards": 11, 2024-08-18T15:49:22.802 INFO:teuthology.orchestra.run.smithi193.stdout: "read_only": false, 2024-08-18T15:49:22.802 INFO:teuthology.orchestra.run.smithi193.stdout: "tier_type": "", 2024-08-18T15:49:22.802 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_from_all": true, 2024-08-18T15:49:22.802 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_from": [], 2024-08-18T15:49:22.802 INFO:teuthology.orchestra.run.smithi193.stdout: "redirect_zone": "", 2024-08-18T15:49:22.802 INFO:teuthology.orchestra.run.smithi193.stdout: "supported_features": [ 2024-08-18T15:49:22.802 INFO:teuthology.orchestra.run.smithi193.stdout: "compress-encrypted", 2024-08-18T15:49:22.802 INFO:teuthology.orchestra.run.smithi193.stdout: "resharding" 2024-08-18T15:49:22.802 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:49:22.803 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:49:22.803 INFO:teuthology.orchestra.run.smithi193.stdout: ], 2024-08-18T15:49:22.803 INFO:teuthology.orchestra.run.smithi193.stdout: "placement_targets": [ 2024-08-18T15:49:22.803 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:49:22.803 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "default-placement", 2024-08-18T15:49:22.803 INFO:teuthology.orchestra.run.smithi193.stdout: "tags": [], 2024-08-18T15:49:22.803 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_classes": [ 2024-08-18T15:49:22.803 INFO:teuthology.orchestra.run.smithi193.stdout: "STANDARD" 2024-08-18T15:49:22.803 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:49:22.803 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:49:22.803 INFO:teuthology.orchestra.run.smithi193.stdout: ], 2024-08-18T15:49:22.803 INFO:teuthology.orchestra.run.smithi193.stdout: "default_placement": "default-placement", 2024-08-18T15:49:22.804 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_id": "2827c6f0-bf0d-4836-944c-9e7fcf22457b", 2024-08-18T15:49:22.804 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_policy": { 2024-08-18T15:49:22.804 INFO:teuthology.orchestra.run.smithi193.stdout: "groups": [] 2024-08-18T15:49:22.804 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:49:22.804 INFO:teuthology.orchestra.run.smithi193.stdout: "enabled_features": [ 2024-08-18T15:49:22.804 INFO:teuthology.orchestra.run.smithi193.stdout: "resharding" 2024-08-18T15:49:22.804 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:49:22.804 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:49:22.804 INFO:teuthology.orchestra.run.smithi193.stdout: ], 2024-08-18T15:49:22.804 INFO:teuthology.orchestra.run.smithi193.stdout: "short_zone_ids": [ 2024-08-18T15:49:22.804 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:49:22.804 INFO:teuthology.orchestra.run.smithi193.stdout: "key": "018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T15:49:22.804 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 462036791 2024-08-18T15:49:22.805 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:49:22.809 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:49:22.810 INFO:teuthology.orchestra.run.smithi193.stdout: "key": "10090ccf-aad6-416d-9775-df8b7b27345a", 2024-08-18T15:49:22.810 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 874289624 2024-08-18T15:49:22.810 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:49:22.810 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:49:22.811 INFO:teuthology.orchestra.run.smithi193.stdout: "key": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:49:22.811 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 1721399886 2024-08-18T15:49:22.811 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:49:22.811 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:49:22.811 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:49:22.811 INFO:teuthology.orchestra.run.smithi193.stdout: "master_zonegroup": "test-zonegroup", 2024-08-18T15:49:22.811 INFO:teuthology.orchestra.run.smithi193.stdout: "master_zone": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:49:22.811 INFO:teuthology.orchestra.run.smithi193.stdout: "period_config": { 2024-08-18T15:49:22.811 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket_quota": { 2024-08-18T15:49:22.811 INFO:teuthology.orchestra.run.smithi193.stdout: "enabled": false, 2024-08-18T15:49:22.811 INFO:teuthology.orchestra.run.smithi193.stdout: "check_on_raw": false, 2024-08-18T15:49:22.812 INFO:teuthology.orchestra.run.smithi193.stdout: "max_size": -1, 2024-08-18T15:49:22.812 INFO:teuthology.orchestra.run.smithi193.stdout: "max_size_kb": 0, 2024-08-18T15:49:22.812 INFO:teuthology.orchestra.run.smithi193.stdout: "max_objects": -1 2024-08-18T15:49:22.812 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:49:22.812 INFO:teuthology.orchestra.run.smithi193.stdout: "user_quota": { 2024-08-18T15:49:22.812 INFO:teuthology.orchestra.run.smithi193.stdout: "enabled": false, 2024-08-18T15:49:22.812 INFO:teuthology.orchestra.run.smithi193.stdout: "check_on_raw": false, 2024-08-18T15:49:22.812 INFO:teuthology.orchestra.run.smithi193.stdout: "max_size": -1, 2024-08-18T15:49:22.812 INFO:teuthology.orchestra.run.smithi193.stdout: "max_size_kb": 0, 2024-08-18T15:49:22.813 INFO:teuthology.orchestra.run.smithi193.stdout: "max_objects": -1 2024-08-18T15:49:22.813 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:49:22.813 INFO:teuthology.orchestra.run.smithi193.stdout: "user_ratelimit": { 2024-08-18T15:49:22.813 INFO:teuthology.orchestra.run.smithi193.stdout: "max_read_ops": 0, 2024-08-18T15:49:22.813 INFO:teuthology.orchestra.run.smithi193.stdout: "max_write_ops": 0, 2024-08-18T15:49:22.813 INFO:teuthology.orchestra.run.smithi193.stdout: "max_read_bytes": 0, 2024-08-18T15:49:22.813 INFO:teuthology.orchestra.run.smithi193.stdout: "max_write_bytes": 0, 2024-08-18T15:49:22.813 INFO:teuthology.orchestra.run.smithi193.stdout: "enabled": false 2024-08-18T15:49:22.813 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:49:22.813 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket_ratelimit": { 2024-08-18T15:49:22.813 INFO:teuthology.orchestra.run.smithi193.stdout: "max_read_ops": 0, 2024-08-18T15:49:22.813 INFO:teuthology.orchestra.run.smithi193.stdout: "max_write_ops": 0, 2024-08-18T15:49:22.814 INFO:teuthology.orchestra.run.smithi193.stdout: "max_read_bytes": 0, 2024-08-18T15:49:22.814 INFO:teuthology.orchestra.run.smithi193.stdout: "max_write_bytes": 0, 2024-08-18T15:49:22.814 INFO:teuthology.orchestra.run.smithi193.stdout: "enabled": false 2024-08-18T15:49:22.814 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:49:22.814 INFO:teuthology.orchestra.run.smithi193.stdout: "anonymous_ratelimit": { 2024-08-18T15:49:22.814 INFO:teuthology.orchestra.run.smithi193.stdout: "max_read_ops": 0, 2024-08-18T15:49:22.814 INFO:teuthology.orchestra.run.smithi193.stdout: "max_write_ops": 0, 2024-08-18T15:49:22.815 INFO:teuthology.orchestra.run.smithi193.stdout: "max_read_bytes": 0, 2024-08-18T15:49:22.815 INFO:teuthology.orchestra.run.smithi193.stdout: "max_write_bytes": 0, 2024-08-18T15:49:22.815 INFO:teuthology.orchestra.run.smithi193.stdout: "enabled": false 2024-08-18T15:49:22.815 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:49:22.815 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:49:22.815 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_id": "2827c6f0-bf0d-4836-944c-9e7fcf22457b", 2024-08-18T15:49:22.815 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_name": "test-realm", 2024-08-18T15:49:22.815 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:49:22.815 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:49:22.818 INFO:rgw_multi.tests:Set sync_from_all flag on zone test-zone2 to true 2024-08-18T15:50:22.819 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:50:22.820 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:50:22.820 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --redirect-zone= --cluster c2 --debug-rgw 0 --debug-ms 0 2024-08-18T15:50:22.876 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:50:22.876 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:50:22.898 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:50:22.899 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "10090ccf-aad6-416d-9775-df8b7b27345a", 2024-08-18T15:50:22.899 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "test-zone2", 2024-08-18T15:50:22.899 INFO:teuthology.orchestra.run.smithi193.stdout: "domain_root": "test-zone2.rgw.meta:root", 2024-08-18T15:50:22.899 INFO:teuthology.orchestra.run.smithi193.stdout: "control_pool": "test-zone2.rgw.control", 2024-08-18T15:50:22.899 INFO:teuthology.orchestra.run.smithi193.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2024-08-18T15:50:22.899 INFO:teuthology.orchestra.run.smithi193.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2024-08-18T15:50:22.899 INFO:teuthology.orchestra.run.smithi193.stdout: "log_pool": "test-zone2.rgw.log", 2024-08-18T15:50:22.899 INFO:teuthology.orchestra.run.smithi193.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2024-08-18T15:50:22.899 INFO:teuthology.orchestra.run.smithi193.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2024-08-18T15:50:22.899 INFO:teuthology.orchestra.run.smithi193.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2024-08-18T15:50:22.899 INFO:teuthology.orchestra.run.smithi193.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2024-08-18T15:50:22.900 INFO:teuthology.orchestra.run.smithi193.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2024-08-18T15:50:22.900 INFO:teuthology.orchestra.run.smithi193.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2024-08-18T15:50:22.900 INFO:teuthology.orchestra.run.smithi193.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2024-08-18T15:50:22.900 INFO:teuthology.orchestra.run.smithi193.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2024-08-18T15:50:22.900 INFO:teuthology.orchestra.run.smithi193.stdout: "otp_pool": "test-zone2.rgw.otp", 2024-08-18T15:50:22.900 INFO:teuthology.orchestra.run.smithi193.stdout: "system_key": { 2024-08-18T15:50:22.900 INFO:teuthology.orchestra.run.smithi193.stdout: "access_key": "MRX9HX7RYBDXO1UM", 2024-08-18T15:50:22.900 INFO:teuthology.orchestra.run.smithi193.stdout: "secret_key": "hGUsvhUdpGZAYWjQwu3StBaYXJPhuCEj" 2024-08-18T15:50:22.900 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:50:22.900 INFO:teuthology.orchestra.run.smithi193.stdout: "placement_pools": [ 2024-08-18T15:50:22.900 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:50:22.900 INFO:teuthology.orchestra.run.smithi193.stdout: "key": "default-placement", 2024-08-18T15:50:22.900 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:50:22.900 INFO:teuthology.orchestra.run.smithi193.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2024-08-18T15:50:22.901 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_classes": { 2024-08-18T15:50:22.901 INFO:teuthology.orchestra.run.smithi193.stdout: "STANDARD": { 2024-08-18T15:50:22.901 INFO:teuthology.orchestra.run.smithi193.stdout: "data_pool": "test-zone2.rgw.buckets.data", 2024-08-18T15:50:22.901 INFO:teuthology.orchestra.run.smithi193.stdout: "compression_type": "random" 2024-08-18T15:50:22.901 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:50:22.901 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:50:22.901 INFO:teuthology.orchestra.run.smithi193.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2024-08-18T15:50:22.901 INFO:teuthology.orchestra.run.smithi193.stdout: "index_type": 0, 2024-08-18T15:50:22.901 INFO:teuthology.orchestra.run.smithi193.stdout: "inline_data": true 2024-08-18T15:50:22.901 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:50:22.901 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:50:22.901 INFO:teuthology.orchestra.run.smithi193.stdout: ], 2024-08-18T15:50:22.902 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_id": "2827c6f0-bf0d-4836-944c-9e7fcf22457b", 2024-08-18T15:50:22.902 INFO:teuthology.orchestra.run.smithi193.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2024-08-18T15:50:22.902 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:50:22.903 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:50:22.903 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:50:22.903 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2024-08-18T15:50:22.952 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:50:22.952 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:50:22.999 INFO:teuthology.orchestra.run.smithi193.stderr:Sending period to new master zone 96bf8502-2f39-4d53-8196-3378c28da3b4 2024-08-18T15:50:23.024 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:50:23.025 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T15:50:23.025 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 5, 2024-08-18T15:50:23.025 INFO:teuthology.orchestra.run.smithi193.stdout: "predecessor_uuid": "64309209-7567-46e5-8881-65783d84cf3c", 2024-08-18T15:50:23.025 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": [ 2024-08-18T15:50:23.025 INFO:teuthology.orchestra.run.smithi193.stdout: "", 2024-08-18T15:50:23.025 INFO:teuthology.orchestra.run.smithi193.stdout: "", 2024-08-18T15:50:23.025 INFO:teuthology.orchestra.run.smithi193.stdout: "", 2024-08-18T15:50:23.025 INFO:teuthology.orchestra.run.smithi193.stdout: "1_1723995848.930494_187.1" 2024-08-18T15:50:23.025 INFO:teuthology.orchestra.run.smithi193.stdout: ], 2024-08-18T15:50:23.025 INFO:teuthology.orchestra.run.smithi193.stdout: "period_map": { 2024-08-18T15:50:23.025 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T15:50:23.025 INFO:teuthology.orchestra.run.smithi193.stdout: "zonegroups": [ 2024-08-18T15:50:23.025 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:50:23.025 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "test-zonegroup", 2024-08-18T15:50:23.026 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "test-zonegroup", 2024-08-18T15:50:23.026 INFO:teuthology.orchestra.run.smithi193.stdout: "api_name": "", 2024-08-18T15:50:23.026 INFO:teuthology.orchestra.run.smithi193.stdout: "is_master": true, 2024-08-18T15:50:23.026 INFO:teuthology.orchestra.run.smithi193.stdout: "endpoints": [ 2024-08-18T15:50:23.026 INFO:teuthology.orchestra.run.smithi193.stdout: "http://smithi134.front.sepia.ceph.com:8000" 2024-08-18T15:50:23.026 INFO:teuthology.orchestra.run.smithi193.stdout: ], 2024-08-18T15:50:23.026 INFO:teuthology.orchestra.run.smithi193.stdout: "hostnames": [], 2024-08-18T15:50:23.026 INFO:teuthology.orchestra.run.smithi193.stdout: "hostnames_s3website": [], 2024-08-18T15:50:23.026 INFO:teuthology.orchestra.run.smithi193.stdout: "master_zone": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:50:23.026 INFO:teuthology.orchestra.run.smithi193.stdout: "zones": [ 2024-08-18T15:50:23.026 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:50:23.026 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T15:50:23.026 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "test-zone3", 2024-08-18T15:50:23.027 INFO:teuthology.orchestra.run.smithi193.stdout: "endpoints": [ 2024-08-18T15:50:23.027 INFO:teuthology.orchestra.run.smithi193.stdout: "http://smithi134.front.sepia.ceph.com:8001" 2024-08-18T15:50:23.027 INFO:teuthology.orchestra.run.smithi193.stdout: ], 2024-08-18T15:50:23.027 INFO:teuthology.orchestra.run.smithi193.stdout: "log_meta": false, 2024-08-18T15:50:23.027 INFO:teuthology.orchestra.run.smithi193.stdout: "log_data": true, 2024-08-18T15:50:23.027 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket_index_max_shards": 11, 2024-08-18T15:50:23.027 INFO:teuthology.orchestra.run.smithi193.stdout: "read_only": false, 2024-08-18T15:50:23.027 INFO:teuthology.orchestra.run.smithi193.stdout: "tier_type": "", 2024-08-18T15:50:23.027 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_from_all": true, 2024-08-18T15:50:23.027 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_from": [], 2024-08-18T15:50:23.027 INFO:teuthology.orchestra.run.smithi193.stdout: "redirect_zone": "", 2024-08-18T15:50:23.027 INFO:teuthology.orchestra.run.smithi193.stdout: "supported_features": [ 2024-08-18T15:50:23.027 INFO:teuthology.orchestra.run.smithi193.stdout: "compress-encrypted", 2024-08-18T15:50:23.028 INFO:teuthology.orchestra.run.smithi193.stdout: "resharding" 2024-08-18T15:50:23.028 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:50:23.028 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:50:23.028 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:50:23.028 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "10090ccf-aad6-416d-9775-df8b7b27345a", 2024-08-18T15:50:23.028 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "test-zone2", 2024-08-18T15:50:23.028 INFO:teuthology.orchestra.run.smithi193.stdout: "endpoints": [ 2024-08-18T15:50:23.028 INFO:teuthology.orchestra.run.smithi193.stdout: "http://smithi193.front.sepia.ceph.com:8000" 2024-08-18T15:50:23.028 INFO:teuthology.orchestra.run.smithi193.stdout: ], 2024-08-18T15:50:23.028 INFO:teuthology.orchestra.run.smithi193.stdout: "log_meta": false, 2024-08-18T15:50:23.028 INFO:teuthology.orchestra.run.smithi193.stdout: "log_data": true, 2024-08-18T15:50:23.028 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket_index_max_shards": 11, 2024-08-18T15:50:23.029 INFO:teuthology.orchestra.run.smithi193.stdout: "read_only": false, 2024-08-18T15:50:23.029 INFO:teuthology.orchestra.run.smithi193.stdout: "tier_type": "", 2024-08-18T15:50:23.029 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_from_all": true, 2024-08-18T15:50:23.029 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_from": [], 2024-08-18T15:50:23.029 INFO:teuthology.orchestra.run.smithi193.stdout: "redirect_zone": "", 2024-08-18T15:50:23.029 INFO:teuthology.orchestra.run.smithi193.stdout: "supported_features": [ 2024-08-18T15:50:23.029 INFO:teuthology.orchestra.run.smithi193.stdout: "compress-encrypted", 2024-08-18T15:50:23.029 INFO:teuthology.orchestra.run.smithi193.stdout: "resharding" 2024-08-18T15:50:23.029 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:50:23.029 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:50:23.029 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:50:23.029 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:50:23.030 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "test-zone1", 2024-08-18T15:50:23.030 INFO:teuthology.orchestra.run.smithi193.stdout: "endpoints": [ 2024-08-18T15:50:23.030 INFO:teuthology.orchestra.run.smithi193.stdout: "http://smithi134.front.sepia.ceph.com:8000" 2024-08-18T15:50:23.030 INFO:teuthology.orchestra.run.smithi193.stdout: ], 2024-08-18T15:50:23.030 INFO:teuthology.orchestra.run.smithi193.stdout: "log_meta": false, 2024-08-18T15:50:23.030 INFO:teuthology.orchestra.run.smithi193.stdout: "log_data": true, 2024-08-18T15:50:23.030 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket_index_max_shards": 11, 2024-08-18T15:50:23.030 INFO:teuthology.orchestra.run.smithi193.stdout: "read_only": false, 2024-08-18T15:50:23.030 INFO:teuthology.orchestra.run.smithi193.stdout: "tier_type": "", 2024-08-18T15:50:23.030 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_from_all": true, 2024-08-18T15:50:23.030 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_from": [], 2024-08-18T15:50:23.030 INFO:teuthology.orchestra.run.smithi193.stdout: "redirect_zone": "", 2024-08-18T15:50:23.030 INFO:teuthology.orchestra.run.smithi193.stdout: "supported_features": [ 2024-08-18T15:50:23.031 INFO:teuthology.orchestra.run.smithi193.stdout: "compress-encrypted", 2024-08-18T15:50:23.031 INFO:teuthology.orchestra.run.smithi193.stdout: "resharding" 2024-08-18T15:50:23.031 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:50:23.031 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:50:23.031 INFO:teuthology.orchestra.run.smithi193.stdout: ], 2024-08-18T15:50:23.031 INFO:teuthology.orchestra.run.smithi193.stdout: "placement_targets": [ 2024-08-18T15:50:23.031 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:50:23.031 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "default-placement", 2024-08-18T15:50:23.031 INFO:teuthology.orchestra.run.smithi193.stdout: "tags": [], 2024-08-18T15:50:23.031 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_classes": [ 2024-08-18T15:50:23.031 INFO:teuthology.orchestra.run.smithi193.stdout: "STANDARD" 2024-08-18T15:50:23.031 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:50:23.031 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:50:23.031 INFO:teuthology.orchestra.run.smithi193.stdout: ], 2024-08-18T15:50:23.032 INFO:teuthology.orchestra.run.smithi193.stdout: "default_placement": "default-placement", 2024-08-18T15:50:23.032 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_id": "2827c6f0-bf0d-4836-944c-9e7fcf22457b", 2024-08-18T15:50:23.032 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_policy": { 2024-08-18T15:50:23.032 INFO:teuthology.orchestra.run.smithi193.stdout: "groups": [] 2024-08-18T15:50:23.032 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:50:23.032 INFO:teuthology.orchestra.run.smithi193.stdout: "enabled_features": [ 2024-08-18T15:50:23.032 INFO:teuthology.orchestra.run.smithi193.stdout: "resharding" 2024-08-18T15:50:23.032 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:50:23.032 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:50:23.032 INFO:teuthology.orchestra.run.smithi193.stdout: ], 2024-08-18T15:50:23.032 INFO:teuthology.orchestra.run.smithi193.stdout: "short_zone_ids": [ 2024-08-18T15:50:23.032 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:50:23.032 INFO:teuthology.orchestra.run.smithi193.stdout: "key": "018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T15:50:23.033 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 462036791 2024-08-18T15:50:23.033 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:50:23.033 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:50:23.036 INFO:teuthology.orchestra.run.smithi193.stdout: "key": "10090ccf-aad6-416d-9775-df8b7b27345a", 2024-08-18T15:50:23.037 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 874289624 2024-08-18T15:50:23.037 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:50:23.037 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:50:23.037 INFO:teuthology.orchestra.run.smithi193.stdout: "key": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:50:23.037 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 1721399886 2024-08-18T15:50:23.037 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:50:23.037 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:50:23.037 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:50:23.038 INFO:teuthology.orchestra.run.smithi193.stdout: "master_zonegroup": "test-zonegroup", 2024-08-18T15:50:23.038 INFO:teuthology.orchestra.run.smithi193.stdout: "master_zone": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:50:23.038 INFO:teuthology.orchestra.run.smithi193.stdout: "period_config": { 2024-08-18T15:50:23.038 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket_quota": { 2024-08-18T15:50:23.038 INFO:teuthology.orchestra.run.smithi193.stdout: "enabled": false, 2024-08-18T15:50:23.038 INFO:teuthology.orchestra.run.smithi193.stdout: "check_on_raw": false, 2024-08-18T15:50:23.038 INFO:teuthology.orchestra.run.smithi193.stdout: "max_size": -1, 2024-08-18T15:50:23.039 INFO:teuthology.orchestra.run.smithi193.stdout: "max_size_kb": 0, 2024-08-18T15:50:23.039 INFO:teuthology.orchestra.run.smithi193.stdout: "max_objects": -1 2024-08-18T15:50:23.039 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:50:23.039 INFO:teuthology.orchestra.run.smithi193.stdout: "user_quota": { 2024-08-18T15:50:23.039 INFO:teuthology.orchestra.run.smithi193.stdout: "enabled": false, 2024-08-18T15:50:23.039 INFO:teuthology.orchestra.run.smithi193.stdout: "check_on_raw": false, 2024-08-18T15:50:23.039 INFO:teuthology.orchestra.run.smithi193.stdout: "max_size": -1, 2024-08-18T15:50:23.039 INFO:teuthology.orchestra.run.smithi193.stdout: "max_size_kb": 0, 2024-08-18T15:50:23.040 INFO:teuthology.orchestra.run.smithi193.stdout: "max_objects": -1 2024-08-18T15:50:23.040 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:50:23.040 INFO:teuthology.orchestra.run.smithi193.stdout: "user_ratelimit": { 2024-08-18T15:50:23.040 INFO:teuthology.orchestra.run.smithi193.stdout: "max_read_ops": 0, 2024-08-18T15:50:23.040 INFO:teuthology.orchestra.run.smithi193.stdout: "max_write_ops": 0, 2024-08-18T15:50:23.040 INFO:teuthology.orchestra.run.smithi193.stdout: "max_read_bytes": 0, 2024-08-18T15:50:23.040 INFO:teuthology.orchestra.run.smithi193.stdout: "max_write_bytes": 0, 2024-08-18T15:50:23.040 INFO:teuthology.orchestra.run.smithi193.stdout: "enabled": false 2024-08-18T15:50:23.041 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:50:23.041 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket_ratelimit": { 2024-08-18T15:50:23.041 INFO:teuthology.orchestra.run.smithi193.stdout: "max_read_ops": 0, 2024-08-18T15:50:23.041 INFO:teuthology.orchestra.run.smithi193.stdout: "max_write_ops": 0, 2024-08-18T15:50:23.041 INFO:teuthology.orchestra.run.smithi193.stdout: "max_read_bytes": 0, 2024-08-18T15:50:23.041 INFO:teuthology.orchestra.run.smithi193.stdout: "max_write_bytes": 0, 2024-08-18T15:50:23.041 INFO:teuthology.orchestra.run.smithi193.stdout: "enabled": false 2024-08-18T15:50:23.041 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:50:23.042 INFO:teuthology.orchestra.run.smithi193.stdout: "anonymous_ratelimit": { 2024-08-18T15:50:23.042 INFO:teuthology.orchestra.run.smithi193.stdout: "max_read_ops": 0, 2024-08-18T15:50:23.042 INFO:teuthology.orchestra.run.smithi193.stdout: "max_write_ops": 0, 2024-08-18T15:50:23.042 INFO:teuthology.orchestra.run.smithi193.stdout: "max_read_bytes": 0, 2024-08-18T15:50:23.042 INFO:teuthology.orchestra.run.smithi193.stdout: "max_write_bytes": 0, 2024-08-18T15:50:23.042 INFO:teuthology.orchestra.run.smithi193.stdout: "enabled": false 2024-08-18T15:50:23.042 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:50:23.043 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:50:23.043 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_id": "2827c6f0-bf0d-4836-944c-9e7fcf22457b", 2024-08-18T15:50:23.043 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_name": "test-realm", 2024-08-18T15:50:23.043 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:50:23.043 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:50:23.044 INFO:rgw_multi.tests:Set redirect_zone zone test-zone2 to "" 2024-08-18T15:51:23.046 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multi_zone_redirect ... ok 2024-08-18T15:51:23.048 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:51:23.048 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:51:23.048 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:51:23.104 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:51:23.104 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:51:23.138 INFO:teuthology.orchestra.run.smithi134.stderr:No --period given, using current period=01050e30-179b-43be-bdbb-3bb23cf37ae8 2024-08-18T15:51:23.153 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T15:51:23.153 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:51:23.153 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T15:51:23.154 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T15:51:23.154 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:51:23.154 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:51:23.154 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T15:51:23.154 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T15:51:23.155 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:51:23.155 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:51:23.155 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T15:51:23.155 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T15:51:23.155 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:51:23.155 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:51:23.156 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723996040.655168_301.1", 2024-08-18T15:51:23.156 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:47:20.655168Z" 2024-08-18T15:51:23.156 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:51:23.156 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T15:51:23.158 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-08-18T15:51:23.158 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:51:23.159 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:51:23.159 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:51:23.215 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:51:23.215 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:51:23.249 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:51:23.248+0000 7fdc1bb20880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:51:23.249 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:51:23.249+0000 7fdc1bb20880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:51:23.265 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:51:23.266 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:51:23.266 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:51:23.266 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:51:23.266 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:51:23.266 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T15:51:23.266 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:51:23.266 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:51:23.267 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:51:23.267 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:51:23.267 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:51:23.267 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:51:23.267 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:51:23.267 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:51:23.267 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:51:23.268 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:51:23.268 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:51:23.268 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:51:23.268 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:51:23.268 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:51:23.268 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:51:23.268 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:51:23.268 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:51:23.269 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:51:23.269 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:51:23.269 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:51:23.269 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:51:23.269 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:51:23.269 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:51:23.269 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:51:23.269 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:51:23.269 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:51:23.269 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:51:23.269 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:51:23.269 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:51:23.270 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:51:23.270 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:51:23.270 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:51:23.270 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:51:23.270 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:51:23.270 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:51:23.270 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:51:23.270 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:51:23.270 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:51:23.271 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:51:23.271 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:51:23.271 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:51:23.271 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:51:23.271 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:51:23.271 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723996040.655168_301.1", 2024-08-18T15:51:23.272 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:51:23.272 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:51:23.272 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:51:23.272 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:47:20.655168Z", 2024-08-18T15:51:23.272 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:51:23.273 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:51:23.273 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:51:23.273 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:51:23.273 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:51:23.273 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:51:23.273 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:51:23.273 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:51:23.274 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:51:23.274 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:51:23Z" 2024-08-18T15:51:23.274 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:51:23.275 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-08-18T15:51:23.276 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-08-18T15:51:23.276 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:51:23.276 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:51:23.276 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:51:23.329 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:51:23.330 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:51:23.412 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:51:23.412+0000 7fa4c2011880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:51:23.413 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:51:23.413+0000 7fa4c2011880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:51:23.434 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T15:51:23.434 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T15:51:23.434 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T15:51:23.434 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T15:51:23.434 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T15:51:23.434 INFO:teuthology.orchestra.run.smithi134.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T15:51:23.434 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T15:51:23.435 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:51:23.435 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T15:51:23.435 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:51:23.435 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T15:51:23.435 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:51:23.435 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:51:23.435 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:51:23.435 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:51:23.435 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 1, 2024-08-18T15:51:23.435 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:51:23.435 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:51:23.435 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:51:23.436 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:51:23.436 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:51:23.436 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:51:23.436 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T15:51:23.436 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:51:23.436 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:51:23.436 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T15:51:23.436 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:51:23.436 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:51:23.436 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:51:23.436 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T15:51:23.436 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:51:23.436 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:51:23.436 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:51:23.437 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:51:23.437 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T15:51:23.437 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:51:23.437 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:51:23.437 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:51:23.437 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:51:23.437 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:51:23.437 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:51:23.437 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:51:23.437 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:51:23.437 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:51:23.437 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:51:23.438 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:51:23.438 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T15:51:23.438 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:51:23.438 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:51:23.438 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723996040.655168_301.1", 2024-08-18T15:51:23.438 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:51:23.439 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:51:23.439 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:51:23.439 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:47:20.655168Z", 2024-08-18T15:51:23.439 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T15:51:23.440 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:51:23.440 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:51:23.440 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T15:51:23.440 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:51:23.440 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T15:51:23.440 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 1, 2024-08-18T15:51:23.440 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 1 2024-08-18T15:51:23.441 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:51:23.441 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T15:51:23Z" 2024-08-18T15:51:23.441 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T15:51:23.442 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-08-18T15:51:23.443 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'MRX9HX7RYBDXO1UM', '--secret', 'hGUsvhUdpGZAYWjQwu3StBaYXJPhuCEj', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:51:23.443 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'MRX9HX7RYBDXO1UM', '--secret', 'hGUsvhUdpGZAYWjQwu3StBaYXJPhuCEj', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:51:23.443 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key MRX9HX7RYBDXO1UM --secret hGUsvhUdpGZAYWjQwu3StBaYXJPhuCEj --cluster c2 --debug-rgw 0 --debug-ms 0 2024-08-18T15:51:23.493 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:51:23.494 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:51:23.508 INFO:teuthology.orchestra.run.smithi193.stderr:NOTICE: set zone's realm_id=2827c6f0-bf0d-4836-944c-9e7fcf22457b 2024-08-18T15:51:23.522 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:51:23.522 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "d995592c-fdf1-445b-8bef-4b5c0926553f", 2024-08-18T15:51:23.522 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "remove", 2024-08-18T15:51:23.522 INFO:teuthology.orchestra.run.smithi193.stdout: "domain_root": "remove.rgw.meta:root", 2024-08-18T15:51:23.522 INFO:teuthology.orchestra.run.smithi193.stdout: "control_pool": "remove.rgw.control", 2024-08-18T15:51:23.522 INFO:teuthology.orchestra.run.smithi193.stdout: "gc_pool": "remove.rgw.log:gc", 2024-08-18T15:51:23.522 INFO:teuthology.orchestra.run.smithi193.stdout: "lc_pool": "remove.rgw.log:lc", 2024-08-18T15:51:23.522 INFO:teuthology.orchestra.run.smithi193.stdout: "log_pool": "remove.rgw.log", 2024-08-18T15:51:23.523 INFO:teuthology.orchestra.run.smithi193.stdout: "intent_log_pool": "remove.rgw.log:intent", 2024-08-18T15:51:23.523 INFO:teuthology.orchestra.run.smithi193.stdout: "usage_log_pool": "remove.rgw.log:usage", 2024-08-18T15:51:23.523 INFO:teuthology.orchestra.run.smithi193.stdout: "roles_pool": "remove.rgw.meta:roles", 2024-08-18T15:51:23.523 INFO:teuthology.orchestra.run.smithi193.stdout: "reshard_pool": "remove.rgw.log:reshard", 2024-08-18T15:51:23.523 INFO:teuthology.orchestra.run.smithi193.stdout: "user_keys_pool": "remove.rgw.meta:users.keys", 2024-08-18T15:51:23.523 INFO:teuthology.orchestra.run.smithi193.stdout: "user_email_pool": "remove.rgw.meta:users.email", 2024-08-18T15:51:23.523 INFO:teuthology.orchestra.run.smithi193.stdout: "user_swift_pool": "remove.rgw.meta:users.swift", 2024-08-18T15:51:23.523 INFO:teuthology.orchestra.run.smithi193.stdout: "user_uid_pool": "remove.rgw.meta:users.uid", 2024-08-18T15:51:23.523 INFO:teuthology.orchestra.run.smithi193.stdout: "otp_pool": "remove.rgw.otp", 2024-08-18T15:51:23.523 INFO:teuthology.orchestra.run.smithi193.stdout: "system_key": { 2024-08-18T15:51:23.523 INFO:teuthology.orchestra.run.smithi193.stdout: "access_key": "MRX9HX7RYBDXO1UM", 2024-08-18T15:51:23.523 INFO:teuthology.orchestra.run.smithi193.stdout: "secret_key": "hGUsvhUdpGZAYWjQwu3StBaYXJPhuCEj" 2024-08-18T15:51:23.524 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:51:23.524 INFO:teuthology.orchestra.run.smithi193.stdout: "placement_pools": [ 2024-08-18T15:51:23.524 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:51:23.524 INFO:teuthology.orchestra.run.smithi193.stdout: "key": "default-placement", 2024-08-18T15:51:23.524 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:51:23.524 INFO:teuthology.orchestra.run.smithi193.stdout: "index_pool": "remove.rgw.buckets.index", 2024-08-18T15:51:23.524 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_classes": { 2024-08-18T15:51:23.524 INFO:teuthology.orchestra.run.smithi193.stdout: "STANDARD": { 2024-08-18T15:51:23.524 INFO:teuthology.orchestra.run.smithi193.stdout: "data_pool": "remove.rgw.buckets.data" 2024-08-18T15:51:23.524 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:51:23.524 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:51:23.524 INFO:teuthology.orchestra.run.smithi193.stdout: "data_extra_pool": "remove.rgw.buckets.non-ec", 2024-08-18T15:51:23.525 INFO:teuthology.orchestra.run.smithi193.stdout: "index_type": 0, 2024-08-18T15:51:23.525 INFO:teuthology.orchestra.run.smithi193.stdout: "inline_data": true 2024-08-18T15:51:23.525 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:51:23.525 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:51:23.525 INFO:teuthology.orchestra.run.smithi193.stdout: ], 2024-08-18T15:51:23.525 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_id": "2827c6f0-bf0d-4836-944c-9e7fcf22457b", 2024-08-18T15:51:23.525 INFO:teuthology.orchestra.run.smithi193.stdout: "notif_pool": "remove.rgw.log:notif" 2024-08-18T15:51:23.525 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:51:23.527 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:51:23.527 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:51:23.527 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2024-08-18T15:51:23.573 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:51:23.573 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:51:23.585 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:51:23.585+0000 7f2032746880 0 period (01050e30-179b-43be-bdbb-3bb23cf37ae8 does not have zone d995592c-fdf1-445b-8bef-4b5c0926553f configured 2024-08-18T15:51:30.177 INFO:teuthology.orchestra.run.smithi193.stderr:Sending period to new master zone 96bf8502-2f39-4d53-8196-3378c28da3b4 2024-08-18T15:51:30.201 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:51:30.201 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T15:51:30.201 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 6, 2024-08-18T15:51:30.201 INFO:teuthology.orchestra.run.smithi193.stdout: "predecessor_uuid": "64309209-7567-46e5-8881-65783d84cf3c", 2024-08-18T15:51:30.202 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": [ 2024-08-18T15:51:30.202 INFO:teuthology.orchestra.run.smithi193.stdout: "", 2024-08-18T15:51:30.202 INFO:teuthology.orchestra.run.smithi193.stdout: "", 2024-08-18T15:51:30.202 INFO:teuthology.orchestra.run.smithi193.stdout: "", 2024-08-18T15:51:30.202 INFO:teuthology.orchestra.run.smithi193.stdout: "1_1723995848.930494_187.1" 2024-08-18T15:51:30.202 INFO:teuthology.orchestra.run.smithi193.stdout: ], 2024-08-18T15:51:30.202 INFO:teuthology.orchestra.run.smithi193.stdout: "period_map": { 2024-08-18T15:51:30.202 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T15:51:30.202 INFO:teuthology.orchestra.run.smithi193.stdout: "zonegroups": [ 2024-08-18T15:51:30.202 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:51:30.202 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "test-zonegroup", 2024-08-18T15:51:30.202 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "test-zonegroup", 2024-08-18T15:51:30.203 INFO:teuthology.orchestra.run.smithi193.stdout: "api_name": "", 2024-08-18T15:51:30.203 INFO:teuthology.orchestra.run.smithi193.stdout: "is_master": true, 2024-08-18T15:51:30.203 INFO:teuthology.orchestra.run.smithi193.stdout: "endpoints": [ 2024-08-18T15:51:30.203 INFO:teuthology.orchestra.run.smithi193.stdout: "http://smithi134.front.sepia.ceph.com:8000" 2024-08-18T15:51:30.203 INFO:teuthology.orchestra.run.smithi193.stdout: ], 2024-08-18T15:51:30.203 INFO:teuthology.orchestra.run.smithi193.stdout: "hostnames": [], 2024-08-18T15:51:30.203 INFO:teuthology.orchestra.run.smithi193.stdout: "hostnames_s3website": [], 2024-08-18T15:51:30.203 INFO:teuthology.orchestra.run.smithi193.stdout: "master_zone": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:51:30.203 INFO:teuthology.orchestra.run.smithi193.stdout: "zones": [ 2024-08-18T15:51:30.203 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:51:30.203 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T15:51:30.203 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "test-zone3", 2024-08-18T15:51:30.203 INFO:teuthology.orchestra.run.smithi193.stdout: "endpoints": [ 2024-08-18T15:51:30.204 INFO:teuthology.orchestra.run.smithi193.stdout: "http://smithi134.front.sepia.ceph.com:8001" 2024-08-18T15:51:30.204 INFO:teuthology.orchestra.run.smithi193.stdout: ], 2024-08-18T15:51:30.204 INFO:teuthology.orchestra.run.smithi193.stdout: "log_meta": false, 2024-08-18T15:51:30.204 INFO:teuthology.orchestra.run.smithi193.stdout: "log_data": true, 2024-08-18T15:51:30.204 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket_index_max_shards": 11, 2024-08-18T15:51:30.204 INFO:teuthology.orchestra.run.smithi193.stdout: "read_only": false, 2024-08-18T15:51:30.204 INFO:teuthology.orchestra.run.smithi193.stdout: "tier_type": "", 2024-08-18T15:51:30.204 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_from_all": true, 2024-08-18T15:51:30.204 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_from": [], 2024-08-18T15:51:30.204 INFO:teuthology.orchestra.run.smithi193.stdout: "redirect_zone": "", 2024-08-18T15:51:30.204 INFO:teuthology.orchestra.run.smithi193.stdout: "supported_features": [ 2024-08-18T15:51:30.204 INFO:teuthology.orchestra.run.smithi193.stdout: "compress-encrypted", 2024-08-18T15:51:30.204 INFO:teuthology.orchestra.run.smithi193.stdout: "resharding" 2024-08-18T15:51:30.204 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:51:30.205 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:51:30.205 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:51:30.205 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "10090ccf-aad6-416d-9775-df8b7b27345a", 2024-08-18T15:51:30.205 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "test-zone2", 2024-08-18T15:51:30.205 INFO:teuthology.orchestra.run.smithi193.stdout: "endpoints": [ 2024-08-18T15:51:30.205 INFO:teuthology.orchestra.run.smithi193.stdout: "http://smithi193.front.sepia.ceph.com:8000" 2024-08-18T15:51:30.205 INFO:teuthology.orchestra.run.smithi193.stdout: ], 2024-08-18T15:51:30.205 INFO:teuthology.orchestra.run.smithi193.stdout: "log_meta": false, 2024-08-18T15:51:30.205 INFO:teuthology.orchestra.run.smithi193.stdout: "log_data": true, 2024-08-18T15:51:30.205 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket_index_max_shards": 11, 2024-08-18T15:51:30.205 INFO:teuthology.orchestra.run.smithi193.stdout: "read_only": false, 2024-08-18T15:51:30.205 INFO:teuthology.orchestra.run.smithi193.stdout: "tier_type": "", 2024-08-18T15:51:30.205 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_from_all": true, 2024-08-18T15:51:30.206 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_from": [], 2024-08-18T15:51:30.206 INFO:teuthology.orchestra.run.smithi193.stdout: "redirect_zone": "", 2024-08-18T15:51:30.206 INFO:teuthology.orchestra.run.smithi193.stdout: "supported_features": [ 2024-08-18T15:51:30.206 INFO:teuthology.orchestra.run.smithi193.stdout: "compress-encrypted", 2024-08-18T15:51:30.206 INFO:teuthology.orchestra.run.smithi193.stdout: "resharding" 2024-08-18T15:51:30.206 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:51:30.206 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:51:30.206 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:51:30.206 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:51:30.206 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "test-zone1", 2024-08-18T15:51:30.206 INFO:teuthology.orchestra.run.smithi193.stdout: "endpoints": [ 2024-08-18T15:51:30.206 INFO:teuthology.orchestra.run.smithi193.stdout: "http://smithi134.front.sepia.ceph.com:8000" 2024-08-18T15:51:30.206 INFO:teuthology.orchestra.run.smithi193.stdout: ], 2024-08-18T15:51:30.207 INFO:teuthology.orchestra.run.smithi193.stdout: "log_meta": false, 2024-08-18T15:51:30.207 INFO:teuthology.orchestra.run.smithi193.stdout: "log_data": true, 2024-08-18T15:51:30.207 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket_index_max_shards": 11, 2024-08-18T15:51:30.207 INFO:teuthology.orchestra.run.smithi193.stdout: "read_only": false, 2024-08-18T15:51:30.207 INFO:teuthology.orchestra.run.smithi193.stdout: "tier_type": "", 2024-08-18T15:51:30.207 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_from_all": true, 2024-08-18T15:51:30.207 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_from": [], 2024-08-18T15:51:30.207 INFO:teuthology.orchestra.run.smithi193.stdout: "redirect_zone": "", 2024-08-18T15:51:30.207 INFO:teuthology.orchestra.run.smithi193.stdout: "supported_features": [ 2024-08-18T15:51:30.207 INFO:teuthology.orchestra.run.smithi193.stdout: "compress-encrypted", 2024-08-18T15:51:30.207 INFO:teuthology.orchestra.run.smithi193.stdout: "resharding" 2024-08-18T15:51:30.207 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:51:30.207 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:51:30.208 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:51:30.208 INFO:teuthology.orchestra.run.smithi193.stdout: "id": "d995592c-fdf1-445b-8bef-4b5c0926553f", 2024-08-18T15:51:30.208 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "remove", 2024-08-18T15:51:30.208 INFO:teuthology.orchestra.run.smithi193.stdout: "endpoints": [], 2024-08-18T15:51:30.208 INFO:teuthology.orchestra.run.smithi193.stdout: "log_meta": false, 2024-08-18T15:51:30.208 INFO:teuthology.orchestra.run.smithi193.stdout: "log_data": true, 2024-08-18T15:51:30.208 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket_index_max_shards": 11, 2024-08-18T15:51:30.208 INFO:teuthology.orchestra.run.smithi193.stdout: "read_only": false, 2024-08-18T15:51:30.208 INFO:teuthology.orchestra.run.smithi193.stdout: "tier_type": "", 2024-08-18T15:51:30.208 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_from_all": true, 2024-08-18T15:51:30.208 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_from": [], 2024-08-18T15:51:30.208 INFO:teuthology.orchestra.run.smithi193.stdout: "redirect_zone": "", 2024-08-18T15:51:30.208 INFO:teuthology.orchestra.run.smithi193.stdout: "supported_features": [ 2024-08-18T15:51:30.208 INFO:teuthology.orchestra.run.smithi193.stdout: "compress-encrypted", 2024-08-18T15:51:30.209 INFO:teuthology.orchestra.run.smithi193.stdout: "resharding" 2024-08-18T15:51:30.209 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:51:30.209 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:51:30.209 INFO:teuthology.orchestra.run.smithi193.stdout: ], 2024-08-18T15:51:30.209 INFO:teuthology.orchestra.run.smithi193.stdout: "placement_targets": [ 2024-08-18T15:51:30.209 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:51:30.213 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "default-placement", 2024-08-18T15:51:30.213 INFO:teuthology.orchestra.run.smithi193.stdout: "tags": [], 2024-08-18T15:51:30.213 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_classes": [ 2024-08-18T15:51:30.214 INFO:teuthology.orchestra.run.smithi193.stdout: "STANDARD" 2024-08-18T15:51:30.214 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:51:30.214 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:51:30.214 INFO:teuthology.orchestra.run.smithi193.stdout: ], 2024-08-18T15:51:30.214 INFO:teuthology.orchestra.run.smithi193.stdout: "default_placement": "default-placement", 2024-08-18T15:51:30.214 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_id": "2827c6f0-bf0d-4836-944c-9e7fcf22457b", 2024-08-18T15:51:30.214 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_policy": { 2024-08-18T15:51:30.214 INFO:teuthology.orchestra.run.smithi193.stdout: "groups": [] 2024-08-18T15:51:30.214 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:51:30.214 INFO:teuthology.orchestra.run.smithi193.stdout: "enabled_features": [ 2024-08-18T15:51:30.214 INFO:teuthology.orchestra.run.smithi193.stdout: "resharding" 2024-08-18T15:51:30.214 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:51:30.214 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:51:30.215 INFO:teuthology.orchestra.run.smithi193.stdout: ], 2024-08-18T15:51:30.215 INFO:teuthology.orchestra.run.smithi193.stdout: "short_zone_ids": [ 2024-08-18T15:51:30.215 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:51:30.215 INFO:teuthology.orchestra.run.smithi193.stdout: "key": "018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T15:51:30.215 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 462036791 2024-08-18T15:51:30.215 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:51:30.215 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:51:30.215 INFO:teuthology.orchestra.run.smithi193.stdout: "key": "10090ccf-aad6-416d-9775-df8b7b27345a", 2024-08-18T15:51:30.215 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 874289624 2024-08-18T15:51:30.215 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:51:30.215 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:51:30.215 INFO:teuthology.orchestra.run.smithi193.stdout: "key": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:51:30.215 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 1721399886 2024-08-18T15:51:30.216 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:51:30.216 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:51:30.216 INFO:teuthology.orchestra.run.smithi193.stdout: "key": "d995592c-fdf1-445b-8bef-4b5c0926553f", 2024-08-18T15:51:30.216 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 563809476 2024-08-18T15:51:30.216 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:51:30.216 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:51:30.216 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:51:30.216 INFO:teuthology.orchestra.run.smithi193.stdout: "master_zonegroup": "test-zonegroup", 2024-08-18T15:51:30.216 INFO:teuthology.orchestra.run.smithi193.stdout: "master_zone": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:51:30.216 INFO:teuthology.orchestra.run.smithi193.stdout: "period_config": { 2024-08-18T15:51:30.216 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket_quota": { 2024-08-18T15:51:30.216 INFO:teuthology.orchestra.run.smithi193.stdout: "enabled": false, 2024-08-18T15:51:30.216 INFO:teuthology.orchestra.run.smithi193.stdout: "check_on_raw": false, 2024-08-18T15:51:30.217 INFO:teuthology.orchestra.run.smithi193.stdout: "max_size": -1, 2024-08-18T15:51:30.217 INFO:teuthology.orchestra.run.smithi193.stdout: "max_size_kb": 0, 2024-08-18T15:51:30.217 INFO:teuthology.orchestra.run.smithi193.stdout: "max_objects": -1 2024-08-18T15:51:30.217 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:51:30.217 INFO:teuthology.orchestra.run.smithi193.stdout: "user_quota": { 2024-08-18T15:51:30.217 INFO:teuthology.orchestra.run.smithi193.stdout: "enabled": false, 2024-08-18T15:51:30.217 INFO:teuthology.orchestra.run.smithi193.stdout: "check_on_raw": false, 2024-08-18T15:51:30.217 INFO:teuthology.orchestra.run.smithi193.stdout: "max_size": -1, 2024-08-18T15:51:30.217 INFO:teuthology.orchestra.run.smithi193.stdout: "max_size_kb": 0, 2024-08-18T15:51:30.217 INFO:teuthology.orchestra.run.smithi193.stdout: "max_objects": -1 2024-08-18T15:51:30.217 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:51:30.217 INFO:teuthology.orchestra.run.smithi193.stdout: "user_ratelimit": { 2024-08-18T15:51:30.217 INFO:teuthology.orchestra.run.smithi193.stdout: "max_read_ops": 0, 2024-08-18T15:51:30.218 INFO:teuthology.orchestra.run.smithi193.stdout: "max_write_ops": 0, 2024-08-18T15:51:30.218 INFO:teuthology.orchestra.run.smithi193.stdout: "max_read_bytes": 0, 2024-08-18T15:51:30.218 INFO:teuthology.orchestra.run.smithi193.stdout: "max_write_bytes": 0, 2024-08-18T15:51:30.218 INFO:teuthology.orchestra.run.smithi193.stdout: "enabled": false 2024-08-18T15:51:30.218 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:51:30.218 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket_ratelimit": { 2024-08-18T15:51:30.218 INFO:teuthology.orchestra.run.smithi193.stdout: "max_read_ops": 0, 2024-08-18T15:51:30.218 INFO:teuthology.orchestra.run.smithi193.stdout: "max_write_ops": 0, 2024-08-18T15:51:30.218 INFO:teuthology.orchestra.run.smithi193.stdout: "max_read_bytes": 0, 2024-08-18T15:51:30.218 INFO:teuthology.orchestra.run.smithi193.stdout: "max_write_bytes": 0, 2024-08-18T15:51:30.218 INFO:teuthology.orchestra.run.smithi193.stdout: "enabled": false 2024-08-18T15:51:30.218 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:51:30.218 INFO:teuthology.orchestra.run.smithi193.stdout: "anonymous_ratelimit": { 2024-08-18T15:51:30.219 INFO:teuthology.orchestra.run.smithi193.stdout: "max_read_ops": 0, 2024-08-18T15:51:30.219 INFO:teuthology.orchestra.run.smithi193.stdout: "max_write_ops": 0, 2024-08-18T15:51:30.219 INFO:teuthology.orchestra.run.smithi193.stdout: "max_read_bytes": 0, 2024-08-18T15:51:30.219 INFO:teuthology.orchestra.run.smithi193.stdout: "max_write_bytes": 0, 2024-08-18T15:51:30.219 INFO:teuthology.orchestra.run.smithi193.stdout: "enabled": false 2024-08-18T15:51:30.219 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:51:30.219 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:51:30.219 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_id": "2827c6f0-bf0d-4836-944c-9e7fcf22457b", 2024-08-18T15:51:30.219 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_name": "test-realm", 2024-08-18T15:51:30.219 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:51:30.219 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:51:30.220 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:51:30.220 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:51:30.221 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T15:51:30.274 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:51:30.274 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:51:30.300 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T15:51:30.300 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "test-zonegroup", 2024-08-18T15:51:30.301 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "test-zonegroup", 2024-08-18T15:51:30.301 INFO:teuthology.orchestra.run.smithi134.stdout: "api_name": "", 2024-08-18T15:51:30.301 INFO:teuthology.orchestra.run.smithi134.stdout: "is_master": true, 2024-08-18T15:51:30.301 INFO:teuthology.orchestra.run.smithi134.stdout: "endpoints": [ 2024-08-18T15:51:30.301 INFO:teuthology.orchestra.run.smithi134.stdout: "http://smithi134.front.sepia.ceph.com:8000" 2024-08-18T15:51:30.301 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T15:51:30.301 INFO:teuthology.orchestra.run.smithi134.stdout: "hostnames": [], 2024-08-18T15:51:30.301 INFO:teuthology.orchestra.run.smithi134.stdout: "hostnames_s3website": [], 2024-08-18T15:51:30.301 INFO:teuthology.orchestra.run.smithi134.stdout: "master_zone": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:51:30.301 INFO:teuthology.orchestra.run.smithi134.stdout: "zones": [ 2024-08-18T15:51:30.301 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:51:30.301 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T15:51:30.301 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "test-zone3", 2024-08-18T15:51:30.302 INFO:teuthology.orchestra.run.smithi134.stdout: "endpoints": [ 2024-08-18T15:51:30.302 INFO:teuthology.orchestra.run.smithi134.stdout: "http://smithi134.front.sepia.ceph.com:8001" 2024-08-18T15:51:30.302 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T15:51:30.302 INFO:teuthology.orchestra.run.smithi134.stdout: "log_meta": false, 2024-08-18T15:51:30.302 INFO:teuthology.orchestra.run.smithi134.stdout: "log_data": true, 2024-08-18T15:51:30.302 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_index_max_shards": 11, 2024-08-18T15:51:30.302 INFO:teuthology.orchestra.run.smithi134.stdout: "read_only": false, 2024-08-18T15:51:30.302 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_type": "", 2024-08-18T15:51:30.302 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from_all": true, 2024-08-18T15:51:30.302 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from": [], 2024-08-18T15:51:30.302 INFO:teuthology.orchestra.run.smithi134.stdout: "redirect_zone": "", 2024-08-18T15:51:30.302 INFO:teuthology.orchestra.run.smithi134.stdout: "supported_features": [ 2024-08-18T15:51:30.302 INFO:teuthology.orchestra.run.smithi134.stdout: "compress-encrypted", 2024-08-18T15:51:30.302 INFO:teuthology.orchestra.run.smithi134.stdout: "resharding" 2024-08-18T15:51:30.303 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T15:51:30.303 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:51:30.303 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:51:30.303 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "10090ccf-aad6-416d-9775-df8b7b27345a", 2024-08-18T15:51:30.303 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "test-zone2", 2024-08-18T15:51:30.303 INFO:teuthology.orchestra.run.smithi134.stdout: "endpoints": [ 2024-08-18T15:51:30.303 INFO:teuthology.orchestra.run.smithi134.stdout: "http://smithi193.front.sepia.ceph.com:8000" 2024-08-18T15:51:30.303 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T15:51:30.303 INFO:teuthology.orchestra.run.smithi134.stdout: "log_meta": false, 2024-08-18T15:51:30.303 INFO:teuthology.orchestra.run.smithi134.stdout: "log_data": true, 2024-08-18T15:51:30.303 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_index_max_shards": 11, 2024-08-18T15:51:30.303 INFO:teuthology.orchestra.run.smithi134.stdout: "read_only": false, 2024-08-18T15:51:30.303 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_type": "", 2024-08-18T15:51:30.304 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from_all": true, 2024-08-18T15:51:30.304 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from": [], 2024-08-18T15:51:30.304 INFO:teuthology.orchestra.run.smithi134.stdout: "redirect_zone": "", 2024-08-18T15:51:30.304 INFO:teuthology.orchestra.run.smithi134.stdout: "supported_features": [ 2024-08-18T15:51:30.304 INFO:teuthology.orchestra.run.smithi134.stdout: "compress-encrypted", 2024-08-18T15:51:30.304 INFO:teuthology.orchestra.run.smithi134.stdout: "resharding" 2024-08-18T15:51:30.304 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T15:51:30.304 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:51:30.304 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:51:30.304 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:51:30.304 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "test-zone1", 2024-08-18T15:51:30.304 INFO:teuthology.orchestra.run.smithi134.stdout: "endpoints": [ 2024-08-18T15:51:30.304 INFO:teuthology.orchestra.run.smithi134.stdout: "http://smithi134.front.sepia.ceph.com:8000" 2024-08-18T15:51:30.304 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T15:51:30.305 INFO:teuthology.orchestra.run.smithi134.stdout: "log_meta": false, 2024-08-18T15:51:30.305 INFO:teuthology.orchestra.run.smithi134.stdout: "log_data": true, 2024-08-18T15:51:30.305 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_index_max_shards": 11, 2024-08-18T15:51:30.305 INFO:teuthology.orchestra.run.smithi134.stdout: "read_only": false, 2024-08-18T15:51:30.305 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_type": "", 2024-08-18T15:51:30.305 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from_all": true, 2024-08-18T15:51:30.305 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from": [], 2024-08-18T15:51:30.305 INFO:teuthology.orchestra.run.smithi134.stdout: "redirect_zone": "", 2024-08-18T15:51:30.305 INFO:teuthology.orchestra.run.smithi134.stdout: "supported_features": [ 2024-08-18T15:51:30.305 INFO:teuthology.orchestra.run.smithi134.stdout: "compress-encrypted", 2024-08-18T15:51:30.305 INFO:teuthology.orchestra.run.smithi134.stdout: "resharding" 2024-08-18T15:51:30.305 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T15:51:30.305 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:51:30.306 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T15:51:30.306 INFO:teuthology.orchestra.run.smithi134.stdout: "placement_targets": [ 2024-08-18T15:51:30.306 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:51:30.306 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "default-placement", 2024-08-18T15:51:30.306 INFO:teuthology.orchestra.run.smithi134.stdout: "tags": [], 2024-08-18T15:51:30.306 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_classes": [ 2024-08-18T15:51:30.306 INFO:teuthology.orchestra.run.smithi134.stdout: "STANDARD" 2024-08-18T15:51:30.306 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T15:51:30.306 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:51:30.306 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T15:51:30.306 INFO:teuthology.orchestra.run.smithi134.stdout: "default_placement": "default-placement", 2024-08-18T15:51:30.306 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_id": "2827c6f0-bf0d-4836-944c-9e7fcf22457b", 2024-08-18T15:51:30.306 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_policy": { 2024-08-18T15:51:30.306 INFO:teuthology.orchestra.run.smithi134.stdout: "groups": [] 2024-08-18T15:51:30.307 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:51:30.307 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled_features": [ 2024-08-18T15:51:30.307 INFO:teuthology.orchestra.run.smithi134.stdout: "resharding" 2024-08-18T15:51:30.307 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T15:51:30.307 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T15:51:30.308 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:51:30.308 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:51:30.308 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T15:51:30.366 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:51:30.367 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:51:30.395 INFO:teuthology.orchestra.run.smithi134.stderr:zone name remove not found in zonegroup test-zonegroup 2024-08-18T15:51:30.404 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-08-18T15:51:30.405 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:51:30.405 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'delete', '--rgw-zone', 'remove', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:51:30.405 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone delete --rgw-zone remove --cluster c2 --debug-rgw 0 --debug-ms 0 2024-08-18T15:51:30.455 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:51:30.455 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:51:30.485 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:51:30.485+0000 7fc4de927880 0 Removed zone from zonegroup test-zonegroup 2024-08-18T15:51:30.493 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:51:30.493 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:51:30.493 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T15:51:30.555 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:51:30.555 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:51:30.582 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:51:30.581+0000 7f5adc53e880 0 WARNING: can't generate connection for zone d995592c-fdf1-445b-8bef-4b5c0926553f id remove: no endpoints defined 2024-08-18T15:51:30.633 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T15:51:30.633 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T15:51:30.633 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 7, 2024-08-18T15:51:30.633 INFO:teuthology.orchestra.run.smithi134.stdout: "predecessor_uuid": "64309209-7567-46e5-8881-65783d84cf3c", 2024-08-18T15:51:30.633 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": [ 2024-08-18T15:51:30.633 INFO:teuthology.orchestra.run.smithi134.stdout: "", 2024-08-18T15:51:30.633 INFO:teuthology.orchestra.run.smithi134.stdout: "", 2024-08-18T15:51:30.633 INFO:teuthology.orchestra.run.smithi134.stdout: "", 2024-08-18T15:51:30.633 INFO:teuthology.orchestra.run.smithi134.stdout: "1_1723995848.930494_187.1" 2024-08-18T15:51:30.633 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T15:51:30.633 INFO:teuthology.orchestra.run.smithi134.stdout: "period_map": { 2024-08-18T15:51:30.634 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T15:51:30.634 INFO:teuthology.orchestra.run.smithi134.stdout: "zonegroups": [ 2024-08-18T15:51:30.634 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:51:30.634 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "test-zonegroup", 2024-08-18T15:51:30.634 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "test-zonegroup", 2024-08-18T15:51:30.634 INFO:teuthology.orchestra.run.smithi134.stdout: "api_name": "", 2024-08-18T15:51:30.634 INFO:teuthology.orchestra.run.smithi134.stdout: "is_master": true, 2024-08-18T15:51:30.634 INFO:teuthology.orchestra.run.smithi134.stdout: "endpoints": [ 2024-08-18T15:51:30.634 INFO:teuthology.orchestra.run.smithi134.stdout: "http://smithi134.front.sepia.ceph.com:8000" 2024-08-18T15:51:30.634 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T15:51:30.634 INFO:teuthology.orchestra.run.smithi134.stdout: "hostnames": [], 2024-08-18T15:51:30.634 INFO:teuthology.orchestra.run.smithi134.stdout: "hostnames_s3website": [], 2024-08-18T15:51:30.634 INFO:teuthology.orchestra.run.smithi134.stdout: "master_zone": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:51:30.635 INFO:teuthology.orchestra.run.smithi134.stdout: "zones": [ 2024-08-18T15:51:30.635 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:51:30.635 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T15:51:30.635 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "test-zone3", 2024-08-18T15:51:30.635 INFO:teuthology.orchestra.run.smithi134.stdout: "endpoints": [ 2024-08-18T15:51:30.635 INFO:teuthology.orchestra.run.smithi134.stdout: "http://smithi134.front.sepia.ceph.com:8001" 2024-08-18T15:51:30.635 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T15:51:30.635 INFO:teuthology.orchestra.run.smithi134.stdout: "log_meta": false, 2024-08-18T15:51:30.635 INFO:teuthology.orchestra.run.smithi134.stdout: "log_data": true, 2024-08-18T15:51:30.635 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_index_max_shards": 11, 2024-08-18T15:51:30.635 INFO:teuthology.orchestra.run.smithi134.stdout: "read_only": false, 2024-08-18T15:51:30.635 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_type": "", 2024-08-18T15:51:30.635 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from_all": true, 2024-08-18T15:51:30.636 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from": [], 2024-08-18T15:51:30.636 INFO:teuthology.orchestra.run.smithi134.stdout: "redirect_zone": "", 2024-08-18T15:51:30.636 INFO:teuthology.orchestra.run.smithi134.stdout: "supported_features": [ 2024-08-18T15:51:30.636 INFO:teuthology.orchestra.run.smithi134.stdout: "compress-encrypted", 2024-08-18T15:51:30.636 INFO:teuthology.orchestra.run.smithi134.stdout: "resharding" 2024-08-18T15:51:30.636 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T15:51:30.636 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:51:30.636 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:51:30.636 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "10090ccf-aad6-416d-9775-df8b7b27345a", 2024-08-18T15:51:30.636 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "test-zone2", 2024-08-18T15:51:30.636 INFO:teuthology.orchestra.run.smithi134.stdout: "endpoints": [ 2024-08-18T15:51:30.636 INFO:teuthology.orchestra.run.smithi134.stdout: "http://smithi193.front.sepia.ceph.com:8000" 2024-08-18T15:51:30.636 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T15:51:30.636 INFO:teuthology.orchestra.run.smithi134.stdout: "log_meta": false, 2024-08-18T15:51:30.637 INFO:teuthology.orchestra.run.smithi134.stdout: "log_data": true, 2024-08-18T15:51:30.637 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_index_max_shards": 11, 2024-08-18T15:51:30.637 INFO:teuthology.orchestra.run.smithi134.stdout: "read_only": false, 2024-08-18T15:51:30.637 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_type": "", 2024-08-18T15:51:30.637 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from_all": true, 2024-08-18T15:51:30.637 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from": [], 2024-08-18T15:51:30.637 INFO:teuthology.orchestra.run.smithi134.stdout: "redirect_zone": "", 2024-08-18T15:51:30.637 INFO:teuthology.orchestra.run.smithi134.stdout: "supported_features": [ 2024-08-18T15:51:30.637 INFO:teuthology.orchestra.run.smithi134.stdout: "compress-encrypted", 2024-08-18T15:51:30.637 INFO:teuthology.orchestra.run.smithi134.stdout: "resharding" 2024-08-18T15:51:30.637 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T15:51:30.637 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:51:30.637 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:51:30.637 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:51:30.638 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "test-zone1", 2024-08-18T15:51:30.638 INFO:teuthology.orchestra.run.smithi134.stdout: "endpoints": [ 2024-08-18T15:51:30.638 INFO:teuthology.orchestra.run.smithi134.stdout: "http://smithi134.front.sepia.ceph.com:8000" 2024-08-18T15:51:30.638 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T15:51:30.638 INFO:teuthology.orchestra.run.smithi134.stdout: "log_meta": false, 2024-08-18T15:51:30.638 INFO:teuthology.orchestra.run.smithi134.stdout: "log_data": true, 2024-08-18T15:51:30.638 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_index_max_shards": 11, 2024-08-18T15:51:30.638 INFO:teuthology.orchestra.run.smithi134.stdout: "read_only": false, 2024-08-18T15:51:30.638 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_type": "", 2024-08-18T15:51:30.638 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from_all": true, 2024-08-18T15:51:30.638 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from": [], 2024-08-18T15:51:30.638 INFO:teuthology.orchestra.run.smithi134.stdout: "redirect_zone": "", 2024-08-18T15:51:30.638 INFO:teuthology.orchestra.run.smithi134.stdout: "supported_features": [ 2024-08-18T15:51:30.639 INFO:teuthology.orchestra.run.smithi134.stdout: "compress-encrypted", 2024-08-18T15:51:30.639 INFO:teuthology.orchestra.run.smithi134.stdout: "resharding" 2024-08-18T15:51:30.639 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T15:51:30.639 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:51:30.639 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T15:51:30.639 INFO:teuthology.orchestra.run.smithi134.stdout: "placement_targets": [ 2024-08-18T15:51:30.639 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:51:30.639 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "default-placement", 2024-08-18T15:51:30.639 INFO:teuthology.orchestra.run.smithi134.stdout: "tags": [], 2024-08-18T15:51:30.639 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_classes": [ 2024-08-18T15:51:30.639 INFO:teuthology.orchestra.run.smithi134.stdout: "STANDARD" 2024-08-18T15:51:30.639 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T15:51:30.639 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:51:30.639 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T15:51:30.640 INFO:teuthology.orchestra.run.smithi134.stdout: "default_placement": "default-placement", 2024-08-18T15:51:30.640 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_id": "2827c6f0-bf0d-4836-944c-9e7fcf22457b", 2024-08-18T15:51:30.640 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_policy": { 2024-08-18T15:51:30.640 INFO:teuthology.orchestra.run.smithi134.stdout: "groups": [] 2024-08-18T15:51:30.640 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:51:30.640 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled_features": [ 2024-08-18T15:51:30.640 INFO:teuthology.orchestra.run.smithi134.stdout: "resharding" 2024-08-18T15:51:30.640 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T15:51:30.640 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:51:30.640 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T15:51:30.640 INFO:teuthology.orchestra.run.smithi134.stdout: "short_zone_ids": [ 2024-08-18T15:51:30.640 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:51:30.640 INFO:teuthology.orchestra.run.smithi134.stdout: "key": "018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T15:51:30.641 INFO:teuthology.orchestra.run.smithi134.stdout: "val": 462036791 2024-08-18T15:51:30.641 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:51:30.641 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:51:30.650 INFO:teuthology.orchestra.run.smithi134.stdout: "key": "10090ccf-aad6-416d-9775-df8b7b27345a", 2024-08-18T15:51:30.650 INFO:teuthology.orchestra.run.smithi134.stdout: "val": 874289624 2024-08-18T15:51:30.650 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:51:30.650 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:51:30.650 INFO:teuthology.orchestra.run.smithi134.stdout: "key": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:51:30.651 INFO:teuthology.orchestra.run.smithi134.stdout: "val": 1721399886 2024-08-18T15:51:30.651 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:51:30.651 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T15:51:30.651 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:51:30.651 INFO:teuthology.orchestra.run.smithi134.stdout: "master_zonegroup": "test-zonegroup", 2024-08-18T15:51:30.651 INFO:teuthology.orchestra.run.smithi134.stdout: "master_zone": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:51:30.651 INFO:teuthology.orchestra.run.smithi134.stdout: "period_config": { 2024-08-18T15:51:30.651 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_quota": { 2024-08-18T15:51:30.651 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false, 2024-08-18T15:51:30.651 INFO:teuthology.orchestra.run.smithi134.stdout: "check_on_raw": false, 2024-08-18T15:51:30.651 INFO:teuthology.orchestra.run.smithi134.stdout: "max_size": -1, 2024-08-18T15:51:30.651 INFO:teuthology.orchestra.run.smithi134.stdout: "max_size_kb": 0, 2024-08-18T15:51:30.651 INFO:teuthology.orchestra.run.smithi134.stdout: "max_objects": -1 2024-08-18T15:51:30.651 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:51:30.652 INFO:teuthology.orchestra.run.smithi134.stdout: "user_quota": { 2024-08-18T15:51:30.652 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false, 2024-08-18T15:51:30.652 INFO:teuthology.orchestra.run.smithi134.stdout: "check_on_raw": false, 2024-08-18T15:51:30.652 INFO:teuthology.orchestra.run.smithi134.stdout: "max_size": -1, 2024-08-18T15:51:30.652 INFO:teuthology.orchestra.run.smithi134.stdout: "max_size_kb": 0, 2024-08-18T15:51:30.652 INFO:teuthology.orchestra.run.smithi134.stdout: "max_objects": -1 2024-08-18T15:51:30.652 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:51:30.652 INFO:teuthology.orchestra.run.smithi134.stdout: "user_ratelimit": { 2024-08-18T15:51:30.652 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_ops": 0, 2024-08-18T15:51:30.652 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_ops": 0, 2024-08-18T15:51:30.652 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_bytes": 0, 2024-08-18T15:51:30.652 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_bytes": 0, 2024-08-18T15:51:30.652 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false 2024-08-18T15:51:30.653 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:51:30.653 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_ratelimit": { 2024-08-18T15:51:30.653 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_ops": 0, 2024-08-18T15:51:30.653 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_ops": 0, 2024-08-18T15:51:30.653 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_bytes": 0, 2024-08-18T15:51:30.653 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_bytes": 0, 2024-08-18T15:51:30.653 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false 2024-08-18T15:51:30.653 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:51:30.653 INFO:teuthology.orchestra.run.smithi134.stdout: "anonymous_ratelimit": { 2024-08-18T15:51:30.653 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_ops": 0, 2024-08-18T15:51:30.653 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_ops": 0, 2024-08-18T15:51:30.653 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_bytes": 0, 2024-08-18T15:51:30.653 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_bytes": 0, 2024-08-18T15:51:30.653 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false 2024-08-18T15:51:30.654 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:51:30.654 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:51:30.654 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_id": "2827c6f0-bf0d-4836-944c-9e7fcf22457b", 2024-08-18T15:51:30.654 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_name": "test-realm", 2024-08-18T15:51:30.654 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T15:51:30.654 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T15:51:30.655 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_zonegroup_remove ... ok 2024-08-18T15:51:30.655 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:51:30.655 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:51:30.655 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup create --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T15:51:30.710 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:51:30.710 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:51:30.740 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T15:51:30.741 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "c4b8d362-8b8d-4499-b4ca-6373270bc1d5", 2024-08-18T15:51:30.741 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "remove_zg", 2024-08-18T15:51:30.741 INFO:teuthology.orchestra.run.smithi134.stdout: "api_name": "remove_zg", 2024-08-18T15:51:30.741 INFO:teuthology.orchestra.run.smithi134.stdout: "is_master": false, 2024-08-18T15:51:30.741 INFO:teuthology.orchestra.run.smithi134.stdout: "endpoints": [], 2024-08-18T15:51:30.741 INFO:teuthology.orchestra.run.smithi134.stdout: "hostnames": [], 2024-08-18T15:51:30.741 INFO:teuthology.orchestra.run.smithi134.stdout: "hostnames_s3website": [], 2024-08-18T15:51:30.741 INFO:teuthology.orchestra.run.smithi134.stdout: "master_zone": "", 2024-08-18T15:51:30.741 INFO:teuthology.orchestra.run.smithi134.stdout: "zones": [], 2024-08-18T15:51:30.741 INFO:teuthology.orchestra.run.smithi134.stdout: "placement_targets": [ 2024-08-18T15:51:30.741 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:51:30.741 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "default-placement", 2024-08-18T15:51:30.742 INFO:teuthology.orchestra.run.smithi134.stdout: "tags": [], 2024-08-18T15:51:30.742 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_classes": [] 2024-08-18T15:51:30.742 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:51:30.742 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T15:51:30.742 INFO:teuthology.orchestra.run.smithi134.stdout: "default_placement": "default-placement", 2024-08-18T15:51:30.742 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_id": "2827c6f0-bf0d-4836-944c-9e7fcf22457b", 2024-08-18T15:51:30.742 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_policy": { 2024-08-18T15:51:30.742 INFO:teuthology.orchestra.run.smithi134.stdout: "groups": [] 2024-08-18T15:51:30.742 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:51:30.742 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled_features": [ 2024-08-18T15:51:30.742 INFO:teuthology.orchestra.run.smithi134.stdout: "resharding" 2024-08-18T15:51:30.742 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T15:51:30.743 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T15:51:30.743 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:51:30.744 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:51:30.744 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T15:51:30.799 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:51:30.800 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:51:30.836 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:51:30.835+0000 7f859da86880 0 NOTICE: promoted remove as new master_zone of zonegroup remove_zg 2024-08-18T15:51:30.842 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T15:51:30.842 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "01bd9a04-f4a8-4e7e-aa07-0ee8d14f6129", 2024-08-18T15:51:30.842 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "remove", 2024-08-18T15:51:30.842 INFO:teuthology.orchestra.run.smithi134.stdout: "domain_root": "remove.rgw.meta:root", 2024-08-18T15:51:30.842 INFO:teuthology.orchestra.run.smithi134.stdout: "control_pool": "remove.rgw.control", 2024-08-18T15:51:30.842 INFO:teuthology.orchestra.run.smithi134.stdout: "gc_pool": "remove.rgw.log:gc", 2024-08-18T15:51:30.842 INFO:teuthology.orchestra.run.smithi134.stdout: "lc_pool": "remove.rgw.log:lc", 2024-08-18T15:51:30.843 INFO:teuthology.orchestra.run.smithi134.stdout: "log_pool": "remove.rgw.log", 2024-08-18T15:51:30.843 INFO:teuthology.orchestra.run.smithi134.stdout: "intent_log_pool": "remove.rgw.log:intent", 2024-08-18T15:51:30.843 INFO:teuthology.orchestra.run.smithi134.stdout: "usage_log_pool": "remove.rgw.log:usage", 2024-08-18T15:51:30.843 INFO:teuthology.orchestra.run.smithi134.stdout: "roles_pool": "remove.rgw.meta:roles", 2024-08-18T15:51:30.843 INFO:teuthology.orchestra.run.smithi134.stdout: "reshard_pool": "remove.rgw.log:reshard", 2024-08-18T15:51:30.843 INFO:teuthology.orchestra.run.smithi134.stdout: "user_keys_pool": "remove.rgw.meta:users.keys", 2024-08-18T15:51:30.843 INFO:teuthology.orchestra.run.smithi134.stdout: "user_email_pool": "remove.rgw.meta:users.email", 2024-08-18T15:51:30.843 INFO:teuthology.orchestra.run.smithi134.stdout: "user_swift_pool": "remove.rgw.meta:users.swift", 2024-08-18T15:51:30.843 INFO:teuthology.orchestra.run.smithi134.stdout: "user_uid_pool": "remove.rgw.meta:users.uid", 2024-08-18T15:51:30.843 INFO:teuthology.orchestra.run.smithi134.stdout: "otp_pool": "remove.rgw.otp", 2024-08-18T15:51:30.844 INFO:teuthology.orchestra.run.smithi134.stdout: "system_key": { 2024-08-18T15:51:30.844 INFO:teuthology.orchestra.run.smithi134.stdout: "access_key": "", 2024-08-18T15:51:30.844 INFO:teuthology.orchestra.run.smithi134.stdout: "secret_key": "" 2024-08-18T15:51:30.844 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:51:30.844 INFO:teuthology.orchestra.run.smithi134.stdout: "placement_pools": [ 2024-08-18T15:51:30.844 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:51:30.844 INFO:teuthology.orchestra.run.smithi134.stdout: "key": "default-placement", 2024-08-18T15:51:30.844 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:51:30.844 INFO:teuthology.orchestra.run.smithi134.stdout: "index_pool": "remove.rgw.buckets.index", 2024-08-18T15:51:30.844 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_classes": { 2024-08-18T15:51:30.844 INFO:teuthology.orchestra.run.smithi134.stdout: "STANDARD": { 2024-08-18T15:51:30.844 INFO:teuthology.orchestra.run.smithi134.stdout: "data_pool": "remove.rgw.buckets.data" 2024-08-18T15:51:30.845 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:51:30.845 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:51:30.845 INFO:teuthology.orchestra.run.smithi134.stdout: "data_extra_pool": "remove.rgw.buckets.non-ec", 2024-08-18T15:51:30.845 INFO:teuthology.orchestra.run.smithi134.stdout: "index_type": 0, 2024-08-18T15:51:30.845 INFO:teuthology.orchestra.run.smithi134.stdout: "inline_data": true 2024-08-18T15:51:30.845 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:51:30.845 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:51:30.845 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T15:51:30.845 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_id": "2827c6f0-bf0d-4836-944c-9e7fcf22457b", 2024-08-18T15:51:30.845 INFO:teuthology.orchestra.run.smithi134.stdout: "notif_pool": "remove.rgw.log:notif" 2024-08-18T15:51:30.845 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T15:51:30.846 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:51:30.846 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:51:30.847 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T15:51:30.915 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:51:30.915 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:51:31.014 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T15:51:31.014 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T15:51:31.014 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 8, 2024-08-18T15:51:31.014 INFO:teuthology.orchestra.run.smithi134.stdout: "predecessor_uuid": "64309209-7567-46e5-8881-65783d84cf3c", 2024-08-18T15:51:31.014 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": [ 2024-08-18T15:51:31.014 INFO:teuthology.orchestra.run.smithi134.stdout: "", 2024-08-18T15:51:31.014 INFO:teuthology.orchestra.run.smithi134.stdout: "", 2024-08-18T15:51:31.014 INFO:teuthology.orchestra.run.smithi134.stdout: "", 2024-08-18T15:51:31.015 INFO:teuthology.orchestra.run.smithi134.stdout: "1_1723995848.930494_187.1" 2024-08-18T15:51:31.015 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T15:51:31.015 INFO:teuthology.orchestra.run.smithi134.stdout: "period_map": { 2024-08-18T15:51:31.015 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T15:51:31.015 INFO:teuthology.orchestra.run.smithi134.stdout: "zonegroups": [ 2024-08-18T15:51:31.015 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:51:31.015 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "c4b8d362-8b8d-4499-b4ca-6373270bc1d5", 2024-08-18T15:51:31.015 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "remove_zg", 2024-08-18T15:51:31.015 INFO:teuthology.orchestra.run.smithi134.stdout: "api_name": "remove_zg", 2024-08-18T15:51:31.015 INFO:teuthology.orchestra.run.smithi134.stdout: "is_master": false, 2024-08-18T15:51:31.015 INFO:teuthology.orchestra.run.smithi134.stdout: "endpoints": [], 2024-08-18T15:51:31.015 INFO:teuthology.orchestra.run.smithi134.stdout: "hostnames": [], 2024-08-18T15:51:31.015 INFO:teuthology.orchestra.run.smithi134.stdout: "hostnames_s3website": [], 2024-08-18T15:51:31.015 INFO:teuthology.orchestra.run.smithi134.stdout: "master_zone": "01bd9a04-f4a8-4e7e-aa07-0ee8d14f6129", 2024-08-18T15:51:31.016 INFO:teuthology.orchestra.run.smithi134.stdout: "zones": [ 2024-08-18T15:51:31.016 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:51:31.016 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "01bd9a04-f4a8-4e7e-aa07-0ee8d14f6129", 2024-08-18T15:51:31.016 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "remove", 2024-08-18T15:51:31.016 INFO:teuthology.orchestra.run.smithi134.stdout: "endpoints": [], 2024-08-18T15:51:31.016 INFO:teuthology.orchestra.run.smithi134.stdout: "log_meta": false, 2024-08-18T15:51:31.016 INFO:teuthology.orchestra.run.smithi134.stdout: "log_data": false, 2024-08-18T15:51:31.016 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_index_max_shards": 11, 2024-08-18T15:51:31.016 INFO:teuthology.orchestra.run.smithi134.stdout: "read_only": false, 2024-08-18T15:51:31.016 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_type": "", 2024-08-18T15:51:31.016 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from_all": true, 2024-08-18T15:51:31.016 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from": [], 2024-08-18T15:51:31.016 INFO:teuthology.orchestra.run.smithi134.stdout: "redirect_zone": "", 2024-08-18T15:51:31.017 INFO:teuthology.orchestra.run.smithi134.stdout: "supported_features": [ 2024-08-18T15:51:31.017 INFO:teuthology.orchestra.run.smithi134.stdout: "compress-encrypted", 2024-08-18T15:51:31.017 INFO:teuthology.orchestra.run.smithi134.stdout: "resharding" 2024-08-18T15:51:31.017 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T15:51:31.017 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:51:31.017 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T15:51:31.017 INFO:teuthology.orchestra.run.smithi134.stdout: "placement_targets": [ 2024-08-18T15:51:31.017 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:51:31.017 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "default-placement", 2024-08-18T15:51:31.017 INFO:teuthology.orchestra.run.smithi134.stdout: "tags": [], 2024-08-18T15:51:31.018 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_classes": [ 2024-08-18T15:51:31.018 INFO:teuthology.orchestra.run.smithi134.stdout: "STANDARD" 2024-08-18T15:51:31.018 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T15:51:31.018 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:51:31.018 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T15:51:31.018 INFO:teuthology.orchestra.run.smithi134.stdout: "default_placement": "default-placement", 2024-08-18T15:51:31.018 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_id": "2827c6f0-bf0d-4836-944c-9e7fcf22457b", 2024-08-18T15:51:31.018 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_policy": { 2024-08-18T15:51:31.018 INFO:teuthology.orchestra.run.smithi134.stdout: "groups": [] 2024-08-18T15:51:31.018 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:51:31.018 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled_features": [ 2024-08-18T15:51:31.018 INFO:teuthology.orchestra.run.smithi134.stdout: "resharding" 2024-08-18T15:51:31.018 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T15:51:31.019 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:51:31.019 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:51:31.019 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "test-zonegroup", 2024-08-18T15:51:31.019 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "test-zonegroup", 2024-08-18T15:51:31.019 INFO:teuthology.orchestra.run.smithi134.stdout: "api_name": "", 2024-08-18T15:51:31.019 INFO:teuthology.orchestra.run.smithi134.stdout: "is_master": true, 2024-08-18T15:51:31.019 INFO:teuthology.orchestra.run.smithi134.stdout: "endpoints": [ 2024-08-18T15:51:31.019 INFO:teuthology.orchestra.run.smithi134.stdout: "http://smithi134.front.sepia.ceph.com:8000" 2024-08-18T15:51:31.019 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T15:51:31.019 INFO:teuthology.orchestra.run.smithi134.stdout: "hostnames": [], 2024-08-18T15:51:31.019 INFO:teuthology.orchestra.run.smithi134.stdout: "hostnames_s3website": [], 2024-08-18T15:51:31.019 INFO:teuthology.orchestra.run.smithi134.stdout: "master_zone": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:51:31.019 INFO:teuthology.orchestra.run.smithi134.stdout: "zones": [ 2024-08-18T15:51:31.020 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:51:31.020 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T15:51:31.020 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "test-zone3", 2024-08-18T15:51:31.020 INFO:teuthology.orchestra.run.smithi134.stdout: "endpoints": [ 2024-08-18T15:51:31.020 INFO:teuthology.orchestra.run.smithi134.stdout: "http://smithi134.front.sepia.ceph.com:8001" 2024-08-18T15:51:31.020 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T15:51:31.020 INFO:teuthology.orchestra.run.smithi134.stdout: "log_meta": false, 2024-08-18T15:51:31.020 INFO:teuthology.orchestra.run.smithi134.stdout: "log_data": true, 2024-08-18T15:51:31.020 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_index_max_shards": 11, 2024-08-18T15:51:31.020 INFO:teuthology.orchestra.run.smithi134.stdout: "read_only": false, 2024-08-18T15:51:31.020 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_type": "", 2024-08-18T15:51:31.020 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from_all": true, 2024-08-18T15:51:31.020 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from": [], 2024-08-18T15:51:31.021 INFO:teuthology.orchestra.run.smithi134.stdout: "redirect_zone": "", 2024-08-18T15:51:31.021 INFO:teuthology.orchestra.run.smithi134.stdout: "supported_features": [ 2024-08-18T15:51:31.021 INFO:teuthology.orchestra.run.smithi134.stdout: "compress-encrypted", 2024-08-18T15:51:31.021 INFO:teuthology.orchestra.run.smithi134.stdout: "resharding" 2024-08-18T15:51:31.021 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T15:51:31.021 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:51:31.021 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:51:31.021 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "10090ccf-aad6-416d-9775-df8b7b27345a", 2024-08-18T15:51:31.021 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "test-zone2", 2024-08-18T15:51:31.021 INFO:teuthology.orchestra.run.smithi134.stdout: "endpoints": [ 2024-08-18T15:51:31.021 INFO:teuthology.orchestra.run.smithi134.stdout: "http://smithi193.front.sepia.ceph.com:8000" 2024-08-18T15:51:31.021 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T15:51:31.021 INFO:teuthology.orchestra.run.smithi134.stdout: "log_meta": false, 2024-08-18T15:51:31.021 INFO:teuthology.orchestra.run.smithi134.stdout: "log_data": true, 2024-08-18T15:51:31.022 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_index_max_shards": 11, 2024-08-18T15:51:31.022 INFO:teuthology.orchestra.run.smithi134.stdout: "read_only": false, 2024-08-18T15:51:31.022 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_type": "", 2024-08-18T15:51:31.022 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from_all": true, 2024-08-18T15:51:31.022 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from": [], 2024-08-18T15:51:31.022 INFO:teuthology.orchestra.run.smithi134.stdout: "redirect_zone": "", 2024-08-18T15:51:31.022 INFO:teuthology.orchestra.run.smithi134.stdout: "supported_features": [ 2024-08-18T15:51:31.022 INFO:teuthology.orchestra.run.smithi134.stdout: "compress-encrypted", 2024-08-18T15:51:31.036 INFO:teuthology.orchestra.run.smithi134.stdout: "resharding" 2024-08-18T15:51:31.036 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T15:51:31.036 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:51:31.036 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:51:31.036 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:51:31.036 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "test-zone1", 2024-08-18T15:51:31.036 INFO:teuthology.orchestra.run.smithi134.stdout: "endpoints": [ 2024-08-18T15:51:31.036 INFO:teuthology.orchestra.run.smithi134.stdout: "http://smithi134.front.sepia.ceph.com:8000" 2024-08-18T15:51:31.037 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T15:51:31.037 INFO:teuthology.orchestra.run.smithi134.stdout: "log_meta": false, 2024-08-18T15:51:31.037 INFO:teuthology.orchestra.run.smithi134.stdout: "log_data": true, 2024-08-18T15:51:31.037 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_index_max_shards": 11, 2024-08-18T15:51:31.037 INFO:teuthology.orchestra.run.smithi134.stdout: "read_only": false, 2024-08-18T15:51:31.037 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_type": "", 2024-08-18T15:51:31.037 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from_all": true, 2024-08-18T15:51:31.037 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from": [], 2024-08-18T15:51:31.037 INFO:teuthology.orchestra.run.smithi134.stdout: "redirect_zone": "", 2024-08-18T15:51:31.037 INFO:teuthology.orchestra.run.smithi134.stdout: "supported_features": [ 2024-08-18T15:51:31.038 INFO:teuthology.orchestra.run.smithi134.stdout: "compress-encrypted", 2024-08-18T15:51:31.038 INFO:teuthology.orchestra.run.smithi134.stdout: "resharding" 2024-08-18T15:51:31.038 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T15:51:31.039 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:51:31.039 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T15:51:31.039 INFO:teuthology.orchestra.run.smithi134.stdout: "placement_targets": [ 2024-08-18T15:51:31.039 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:51:31.039 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "default-placement", 2024-08-18T15:51:31.039 INFO:teuthology.orchestra.run.smithi134.stdout: "tags": [], 2024-08-18T15:51:31.039 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_classes": [ 2024-08-18T15:51:31.039 INFO:teuthology.orchestra.run.smithi134.stdout: "STANDARD" 2024-08-18T15:51:31.039 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T15:51:31.039 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:51:31.039 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T15:51:31.039 INFO:teuthology.orchestra.run.smithi134.stdout: "default_placement": "default-placement", 2024-08-18T15:51:31.039 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_id": "2827c6f0-bf0d-4836-944c-9e7fcf22457b", 2024-08-18T15:51:31.040 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_policy": { 2024-08-18T15:51:31.040 INFO:teuthology.orchestra.run.smithi134.stdout: "groups": [] 2024-08-18T15:51:31.040 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:51:31.040 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled_features": [ 2024-08-18T15:51:31.040 INFO:teuthology.orchestra.run.smithi134.stdout: "resharding" 2024-08-18T15:51:31.040 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T15:51:31.040 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:51:31.040 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T15:51:31.040 INFO:teuthology.orchestra.run.smithi134.stdout: "short_zone_ids": [ 2024-08-18T15:51:31.041 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:51:31.041 INFO:teuthology.orchestra.run.smithi134.stdout: "key": "018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T15:51:31.041 INFO:teuthology.orchestra.run.smithi134.stdout: "val": 462036791 2024-08-18T15:51:31.041 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:51:31.041 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:51:31.041 INFO:teuthology.orchestra.run.smithi134.stdout: "key": "01bd9a04-f4a8-4e7e-aa07-0ee8d14f6129", 2024-08-18T15:51:31.041 INFO:teuthology.orchestra.run.smithi134.stdout: "val": 2353822739 2024-08-18T15:51:31.041 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:51:31.041 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:51:31.041 INFO:teuthology.orchestra.run.smithi134.stdout: "key": "10090ccf-aad6-416d-9775-df8b7b27345a", 2024-08-18T15:51:31.041 INFO:teuthology.orchestra.run.smithi134.stdout: "val": 874289624 2024-08-18T15:51:31.041 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:51:31.042 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:51:31.042 INFO:teuthology.orchestra.run.smithi134.stdout: "key": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:51:31.042 INFO:teuthology.orchestra.run.smithi134.stdout: "val": 1721399886 2024-08-18T15:51:31.042 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:51:31.042 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T15:51:31.042 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:51:31.042 INFO:teuthology.orchestra.run.smithi134.stdout: "master_zonegroup": "test-zonegroup", 2024-08-18T15:51:31.042 INFO:teuthology.orchestra.run.smithi134.stdout: "master_zone": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:51:31.042 INFO:teuthology.orchestra.run.smithi134.stdout: "period_config": { 2024-08-18T15:51:31.042 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_quota": { 2024-08-18T15:51:31.043 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false, 2024-08-18T15:51:31.043 INFO:teuthology.orchestra.run.smithi134.stdout: "check_on_raw": false, 2024-08-18T15:51:31.043 INFO:teuthology.orchestra.run.smithi134.stdout: "max_size": -1, 2024-08-18T15:51:31.043 INFO:teuthology.orchestra.run.smithi134.stdout: "max_size_kb": 0, 2024-08-18T15:51:31.043 INFO:teuthology.orchestra.run.smithi134.stdout: "max_objects": -1 2024-08-18T15:51:31.043 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:51:31.043 INFO:teuthology.orchestra.run.smithi134.stdout: "user_quota": { 2024-08-18T15:51:31.043 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false, 2024-08-18T15:51:31.043 INFO:teuthology.orchestra.run.smithi134.stdout: "check_on_raw": false, 2024-08-18T15:51:31.043 INFO:teuthology.orchestra.run.smithi134.stdout: "max_size": -1, 2024-08-18T15:51:31.043 INFO:teuthology.orchestra.run.smithi134.stdout: "max_size_kb": 0, 2024-08-18T15:51:31.043 INFO:teuthology.orchestra.run.smithi134.stdout: "max_objects": -1 2024-08-18T15:51:31.043 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:51:31.044 INFO:teuthology.orchestra.run.smithi134.stdout: "user_ratelimit": { 2024-08-18T15:51:31.044 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_ops": 0, 2024-08-18T15:51:31.044 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_ops": 0, 2024-08-18T15:51:31.044 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_bytes": 0, 2024-08-18T15:51:31.044 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_bytes": 0, 2024-08-18T15:51:31.044 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false 2024-08-18T15:51:31.044 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:51:31.044 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_ratelimit": { 2024-08-18T15:51:31.044 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_ops": 0, 2024-08-18T15:51:31.044 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_ops": 0, 2024-08-18T15:51:31.044 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_bytes": 0, 2024-08-18T15:51:31.044 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_bytes": 0, 2024-08-18T15:51:31.044 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false 2024-08-18T15:51:31.044 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:51:31.045 INFO:teuthology.orchestra.run.smithi134.stdout: "anonymous_ratelimit": { 2024-08-18T15:51:31.045 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_ops": 0, 2024-08-18T15:51:31.045 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_ops": 0, 2024-08-18T15:51:31.045 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_bytes": 0, 2024-08-18T15:51:31.045 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_bytes": 0, 2024-08-18T15:51:31.045 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false 2024-08-18T15:51:31.045 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:51:31.045 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:51:31.045 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_id": "2827c6f0-bf0d-4836-944c-9e7fcf22457b", 2024-08-18T15:51:31.045 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_name": "test-realm", 2024-08-18T15:51:31.045 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T15:51:31.045 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T15:51:31.046 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:51:31.047 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:51:31.047 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone delete --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T15:51:31.109 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:51:31.109 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:51:31.144 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:51:31.144+0000 7f3021726880 0 NOTICE: removed master_zone of zonegroup remove_zg 2024-08-18T15:51:31.145 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:51:31.145+0000 7f3021726880 0 Removed zone from zonegroup remove_zg 2024-08-18T15:51:31.157 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:51:31.158 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:51:31.158 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T15:51:31.218 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:51:31.218 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:51:31.252 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:51:31.251+0000 7f64e3c12880 0 ERROR: zonegroup remove_zg has a non existent master zone 2024-08-18T15:51:31.258 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-18T15:51:31.259 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:51:31.259 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:51:31.259 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup delete --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T15:51:31.315 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:51:31.315 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:51:31.345 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:51:31.345 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:51:31.345 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T15:51:31.414 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:51:31.414 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:51:31.510 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T15:51:31.510 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T15:51:31.510 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 9, 2024-08-18T15:51:31.511 INFO:teuthology.orchestra.run.smithi134.stdout: "predecessor_uuid": "64309209-7567-46e5-8881-65783d84cf3c", 2024-08-18T15:51:31.511 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": [ 2024-08-18T15:51:31.511 INFO:teuthology.orchestra.run.smithi134.stdout: "", 2024-08-18T15:51:31.511 INFO:teuthology.orchestra.run.smithi134.stdout: "", 2024-08-18T15:51:31.511 INFO:teuthology.orchestra.run.smithi134.stdout: "", 2024-08-18T15:51:31.511 INFO:teuthology.orchestra.run.smithi134.stdout: "1_1723995848.930494_187.1" 2024-08-18T15:51:31.511 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T15:51:31.511 INFO:teuthology.orchestra.run.smithi134.stdout: "period_map": { 2024-08-18T15:51:31.511 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T15:51:31.511 INFO:teuthology.orchestra.run.smithi134.stdout: "zonegroups": [ 2024-08-18T15:51:31.511 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:51:31.511 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "test-zonegroup", 2024-08-18T15:51:31.511 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "test-zonegroup", 2024-08-18T15:51:31.512 INFO:teuthology.orchestra.run.smithi134.stdout: "api_name": "", 2024-08-18T15:51:31.512 INFO:teuthology.orchestra.run.smithi134.stdout: "is_master": true, 2024-08-18T15:51:31.512 INFO:teuthology.orchestra.run.smithi134.stdout: "endpoints": [ 2024-08-18T15:51:31.512 INFO:teuthology.orchestra.run.smithi134.stdout: "http://smithi134.front.sepia.ceph.com:8000" 2024-08-18T15:51:31.512 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T15:51:31.512 INFO:teuthology.orchestra.run.smithi134.stdout: "hostnames": [], 2024-08-18T15:51:31.512 INFO:teuthology.orchestra.run.smithi134.stdout: "hostnames_s3website": [], 2024-08-18T15:51:31.512 INFO:teuthology.orchestra.run.smithi134.stdout: "master_zone": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:51:31.512 INFO:teuthology.orchestra.run.smithi134.stdout: "zones": [ 2024-08-18T15:51:31.512 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:51:31.512 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T15:51:31.512 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "test-zone3", 2024-08-18T15:51:31.512 INFO:teuthology.orchestra.run.smithi134.stdout: "endpoints": [ 2024-08-18T15:51:31.513 INFO:teuthology.orchestra.run.smithi134.stdout: "http://smithi134.front.sepia.ceph.com:8001" 2024-08-18T15:51:31.513 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T15:51:31.513 INFO:teuthology.orchestra.run.smithi134.stdout: "log_meta": false, 2024-08-18T15:51:31.513 INFO:teuthology.orchestra.run.smithi134.stdout: "log_data": true, 2024-08-18T15:51:31.513 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_index_max_shards": 11, 2024-08-18T15:51:31.513 INFO:teuthology.orchestra.run.smithi134.stdout: "read_only": false, 2024-08-18T15:51:31.513 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_type": "", 2024-08-18T15:51:31.513 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from_all": true, 2024-08-18T15:51:31.513 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from": [], 2024-08-18T15:51:31.513 INFO:teuthology.orchestra.run.smithi134.stdout: "redirect_zone": "", 2024-08-18T15:51:31.513 INFO:teuthology.orchestra.run.smithi134.stdout: "supported_features": [ 2024-08-18T15:51:31.513 INFO:teuthology.orchestra.run.smithi134.stdout: "compress-encrypted", 2024-08-18T15:51:31.513 INFO:teuthology.orchestra.run.smithi134.stdout: "resharding" 2024-08-18T15:51:31.514 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T15:51:31.514 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:51:31.514 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:51:31.514 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "10090ccf-aad6-416d-9775-df8b7b27345a", 2024-08-18T15:51:31.514 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "test-zone2", 2024-08-18T15:51:31.514 INFO:teuthology.orchestra.run.smithi134.stdout: "endpoints": [ 2024-08-18T15:51:31.514 INFO:teuthology.orchestra.run.smithi134.stdout: "http://smithi193.front.sepia.ceph.com:8000" 2024-08-18T15:51:31.514 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T15:51:31.514 INFO:teuthology.orchestra.run.smithi134.stdout: "log_meta": false, 2024-08-18T15:51:31.514 INFO:teuthology.orchestra.run.smithi134.stdout: "log_data": true, 2024-08-18T15:51:31.514 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_index_max_shards": 11, 2024-08-18T15:51:31.514 INFO:teuthology.orchestra.run.smithi134.stdout: "read_only": false, 2024-08-18T15:51:31.515 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_type": "", 2024-08-18T15:51:31.515 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from_all": true, 2024-08-18T15:51:31.515 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from": [], 2024-08-18T15:51:31.515 INFO:teuthology.orchestra.run.smithi134.stdout: "redirect_zone": "", 2024-08-18T15:51:31.515 INFO:teuthology.orchestra.run.smithi134.stdout: "supported_features": [ 2024-08-18T15:51:31.515 INFO:teuthology.orchestra.run.smithi134.stdout: "compress-encrypted", 2024-08-18T15:51:31.515 INFO:teuthology.orchestra.run.smithi134.stdout: "resharding" 2024-08-18T15:51:31.515 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T15:51:31.515 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:51:31.515 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:51:31.515 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:51:31.515 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "test-zone1", 2024-08-18T15:51:31.515 INFO:teuthology.orchestra.run.smithi134.stdout: "endpoints": [ 2024-08-18T15:51:31.516 INFO:teuthology.orchestra.run.smithi134.stdout: "http://smithi134.front.sepia.ceph.com:8000" 2024-08-18T15:51:31.516 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T15:51:31.516 INFO:teuthology.orchestra.run.smithi134.stdout: "log_meta": false, 2024-08-18T15:51:31.516 INFO:teuthology.orchestra.run.smithi134.stdout: "log_data": true, 2024-08-18T15:51:31.516 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_index_max_shards": 11, 2024-08-18T15:51:31.516 INFO:teuthology.orchestra.run.smithi134.stdout: "read_only": false, 2024-08-18T15:51:31.516 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_type": "", 2024-08-18T15:51:31.516 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from_all": true, 2024-08-18T15:51:31.516 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from": [], 2024-08-18T15:51:31.516 INFO:teuthology.orchestra.run.smithi134.stdout: "redirect_zone": "", 2024-08-18T15:51:31.516 INFO:teuthology.orchestra.run.smithi134.stdout: "supported_features": [ 2024-08-18T15:51:31.516 INFO:teuthology.orchestra.run.smithi134.stdout: "compress-encrypted", 2024-08-18T15:51:31.516 INFO:teuthology.orchestra.run.smithi134.stdout: "resharding" 2024-08-18T15:51:31.516 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T15:51:31.517 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:51:31.517 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T15:51:31.517 INFO:teuthology.orchestra.run.smithi134.stdout: "placement_targets": [ 2024-08-18T15:51:31.517 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:51:31.517 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "default-placement", 2024-08-18T15:51:31.517 INFO:teuthology.orchestra.run.smithi134.stdout: "tags": [], 2024-08-18T15:51:31.517 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_classes": [ 2024-08-18T15:51:31.517 INFO:teuthology.orchestra.run.smithi134.stdout: "STANDARD" 2024-08-18T15:51:31.517 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T15:51:31.517 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:51:31.517 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T15:51:31.517 INFO:teuthology.orchestra.run.smithi134.stdout: "default_placement": "default-placement", 2024-08-18T15:51:31.517 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_id": "2827c6f0-bf0d-4836-944c-9e7fcf22457b", 2024-08-18T15:51:31.517 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_policy": { 2024-08-18T15:51:31.518 INFO:teuthology.orchestra.run.smithi134.stdout: "groups": [] 2024-08-18T15:51:31.518 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:51:31.518 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled_features": [ 2024-08-18T15:51:31.518 INFO:teuthology.orchestra.run.smithi134.stdout: "resharding" 2024-08-18T15:51:31.518 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T15:51:31.518 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:51:31.518 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T15:51:31.518 INFO:teuthology.orchestra.run.smithi134.stdout: "short_zone_ids": [ 2024-08-18T15:51:31.518 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:51:31.518 INFO:teuthology.orchestra.run.smithi134.stdout: "key": "018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T15:51:31.518 INFO:teuthology.orchestra.run.smithi134.stdout: "val": 462036791 2024-08-18T15:51:31.518 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:51:31.518 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:51:31.533 INFO:teuthology.orchestra.run.smithi134.stdout: "key": "10090ccf-aad6-416d-9775-df8b7b27345a", 2024-08-18T15:51:31.533 INFO:teuthology.orchestra.run.smithi134.stdout: "val": 874289624 2024-08-18T15:51:31.533 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:51:31.533 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:51:31.533 INFO:teuthology.orchestra.run.smithi134.stdout: "key": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:51:31.533 INFO:teuthology.orchestra.run.smithi134.stdout: "val": 1721399886 2024-08-18T15:51:31.533 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:51:31.533 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T15:51:31.533 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:51:31.534 INFO:teuthology.orchestra.run.smithi134.stdout: "master_zonegroup": "test-zonegroup", 2024-08-18T15:51:31.534 INFO:teuthology.orchestra.run.smithi134.stdout: "master_zone": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:51:31.534 INFO:teuthology.orchestra.run.smithi134.stdout: "period_config": { 2024-08-18T15:51:31.534 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_quota": { 2024-08-18T15:51:31.534 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false, 2024-08-18T15:51:31.534 INFO:teuthology.orchestra.run.smithi134.stdout: "check_on_raw": false, 2024-08-18T15:51:31.534 INFO:teuthology.orchestra.run.smithi134.stdout: "max_size": -1, 2024-08-18T15:51:31.534 INFO:teuthology.orchestra.run.smithi134.stdout: "max_size_kb": 0, 2024-08-18T15:51:31.534 INFO:teuthology.orchestra.run.smithi134.stdout: "max_objects": -1 2024-08-18T15:51:31.534 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:51:31.534 INFO:teuthology.orchestra.run.smithi134.stdout: "user_quota": { 2024-08-18T15:51:31.534 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false, 2024-08-18T15:51:31.535 INFO:teuthology.orchestra.run.smithi134.stdout: "check_on_raw": false, 2024-08-18T15:51:31.535 INFO:teuthology.orchestra.run.smithi134.stdout: "max_size": -1, 2024-08-18T15:51:31.535 INFO:teuthology.orchestra.run.smithi134.stdout: "max_size_kb": 0, 2024-08-18T15:51:31.535 INFO:teuthology.orchestra.run.smithi134.stdout: "max_objects": -1 2024-08-18T15:51:31.535 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:51:31.535 INFO:teuthology.orchestra.run.smithi134.stdout: "user_ratelimit": { 2024-08-18T15:51:31.535 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_ops": 0, 2024-08-18T15:51:31.535 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_ops": 0, 2024-08-18T15:51:31.535 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_bytes": 0, 2024-08-18T15:51:31.535 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_bytes": 0, 2024-08-18T15:51:31.535 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false 2024-08-18T15:51:31.535 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:51:31.535 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_ratelimit": { 2024-08-18T15:51:31.535 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_ops": 0, 2024-08-18T15:51:31.536 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_ops": 0, 2024-08-18T15:51:31.536 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_bytes": 0, 2024-08-18T15:51:31.536 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_bytes": 0, 2024-08-18T15:51:31.536 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false 2024-08-18T15:51:31.536 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:51:31.536 INFO:teuthology.orchestra.run.smithi134.stdout: "anonymous_ratelimit": { 2024-08-18T15:51:31.536 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_ops": 0, 2024-08-18T15:51:31.536 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_ops": 0, 2024-08-18T15:51:31.536 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_bytes": 0, 2024-08-18T15:51:31.536 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_bytes": 0, 2024-08-18T15:51:31.536 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false 2024-08-18T15:51:31.536 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:51:31.536 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:51:31.537 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_id": "2827c6f0-bf0d-4836-944c-9e7fcf22457b", 2024-08-18T15:51:31.537 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_name": "test-realm", 2024-08-18T15:51:31.537 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T15:51:31.537 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T15:51:31.538 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_zg_master_zone_delete ... ok 2024-08-18T15:51:31.538 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=kibgmb-53 2024-08-18T15:51:31.950 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=kibgmb-54 2024-08-18T15:51:32.030 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=kibgmb-55 2024-08-18T15:51:32.070 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_set_bucket_website ... SKIP: test_set_bucket_website skipped. Requires rgw_enable_static_website = 1. 2024-08-18T15:51:32.070 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=kibgmb-56 2024-08-18T15:51:32.088 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=kibgmb-57 2024-08-18T15:51:32.116 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=kibgmb-58 2024-08-18T15:51:34.174 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_set_bucket_policy ... ok 2024-08-18T15:51:34.175 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=kibgmb-59 2024-08-18T15:51:34.195 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=kibgmb-60 2024-08-18T15:51:34.225 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=kibgmb-61 2024-08-18T15:51:34.251 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:51:34.251 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:51:34.252 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:51:34.301 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:51:34.301 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:51:34.333 INFO:teuthology.orchestra.run.smithi134.stderr:No --period given, using current period=01050e30-179b-43be-bdbb-3bb23cf37ae8 2024-08-18T15:51:34.346 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T15:51:34.346 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:51:34.346 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T15:51:34.346 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T15:51:34.346 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:51:34.346 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:51:34.346 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T15:51:34.346 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T15:51:34.346 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:51:34.346 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:51:34.347 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T15:51:34.347 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T15:51:34.347 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:51:34.347 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:51:34.347 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723996294.242267_467.1", 2024-08-18T15:51:34.347 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:51:34.242267Z" 2024-08-18T15:51:34.347 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:51:34.347 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T15:51:34.348 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-08-18T15:51:34.348 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:51:34.348 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:51:34.348 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:51:34.403 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:51:34.403 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:51:34.435 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:51:34.435+0000 7f0dbcc13880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:51:34.436 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:51:34.436+0000 7f0dbcc13880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:51:34.450 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:51:34.450 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:51:34.450 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:51:34.450 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:51:34.450 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:51:34.451 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T15:51:34.451 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:51:34.451 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:51:34.451 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:51:34.451 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:51:34.451 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:51:34.451 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:51:34.451 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:51:34.451 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:51:34.451 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:51:34.451 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:51:34.451 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:51:34.452 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:51:34.452 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:51:34.452 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:51:34.452 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:51:34.452 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:51:34.452 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:51:34.452 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:51:34.452 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:51:34.452 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:51:34.452 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:51:34.452 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:51:34.452 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:51:34.452 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:51:34.453 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:51:34.453 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:51:34.453 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:51:34.453 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:51:34.453 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:51:34.453 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:51:34.453 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:51:34.453 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:51:34.453 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:51:34.453 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:51:34.453 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:51:34.453 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:51:34.453 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:51:34.454 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:51:34.454 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:51:34.454 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:51:34.454 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:51:34.454 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:51:34.454 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:51:34.454 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723996040.655168_301.1", 2024-08-18T15:51:34.454 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:51:34.454 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:51:34.454 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:51:34.454 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:47:20.655168Z", 2024-08-18T15:51:34.454 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:51:34.455 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:51:34.455 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:51:34.455 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:51:34.455 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:51:34.455 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:51:34.455 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:51:34.455 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:51:34.455 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:51:34.455 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:51:34Z" 2024-08-18T15:51:34.455 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:51:34.456 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1723996294.242267_467.1 target=1_1723996040.655168_301.1 2024-08-18T15:51:39.457 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:51:39.457 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:51:39.458 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:51:39.512 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:51:39.512 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:51:39.545 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:51:39.544+0000 7f1732812880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:51:39.545 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:51:39.545+0000 7f1732812880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:51:39.559 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:51:39.559 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:51:39.559 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:51:39.559 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:51:39.559 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:51:39.559 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T15:51:39.559 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:51:39.559 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:51:39.559 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:51:39.560 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:51:39.560 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:51:39.560 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:51:39.560 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:51:39.560 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:51:39.560 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:51:39.560 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:51:39.560 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:51:39.560 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:51:39.560 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:51:39.560 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:51:39.560 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:51:39.560 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:51:39.561 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:51:39.561 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:51:39.561 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:51:39.561 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:51:39.561 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:51:39.561 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:51:39.561 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:51:39.561 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:51:39.561 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:51:39.561 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:51:39.561 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:51:39.561 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:51:39.561 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:51:39.562 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:51:39.562 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:51:39.562 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:51:39.562 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:51:39.562 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:51:39.562 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:51:39.562 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:51:39.562 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:51:39.562 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:51:39.562 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:51:39.562 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:51:39.562 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:51:39.562 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:51:39.563 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:51:39.563 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723996040.655168_301.1", 2024-08-18T15:51:39.563 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:51:39.563 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:51:39.563 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:51:39.563 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:47:20.655168Z", 2024-08-18T15:51:39.563 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:51:39.563 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:51:39.563 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:51:39.563 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:51:39.563 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:51:39.563 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:51:39.563 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:51:39.564 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:51:39.564 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:51:39.564 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:51:39Z" 2024-08-18T15:51:39.564 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:51:39.565 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1723996294.242267_467.1 target=1_1723996040.655168_301.1 2024-08-18T15:51:44.565 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:51:44.566 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:51:44.566 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:51:44.623 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:51:44.624 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:51:44.656 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:51:44.655+0000 7f84aac0f880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:51:44.656 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:51:44.656+0000 7f84aac0f880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:51:44.670 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:51:44.670 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:51:44.670 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:51:44.670 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:51:44.670 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:51:44.670 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T15:51:44.671 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:51:44.671 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:51:44.671 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:51:44.671 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:51:44.671 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:51:44.671 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:51:44.671 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:51:44.671 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:51:44.671 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:51:44.671 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:51:44.671 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:51:44.671 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:51:44.671 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:51:44.672 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:51:44.672 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:51:44.672 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:51:44.672 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:51:44.672 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:51:44.672 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:51:44.672 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:51:44.672 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:51:44.672 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:51:44.672 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:51:44.672 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:51:44.672 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:51:44.673 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:51:44.673 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:51:44.673 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:51:44.673 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:51:44.673 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:51:44.673 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:51:44.673 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:51:44.673 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:51:44.673 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:51:44.673 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:51:44.673 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:51:44.673 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:51:44.673 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:51:44.674 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:51:44.674 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:51:44.674 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:51:44.674 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:51:44.674 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:51:44.674 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723996040.655168_301.1", 2024-08-18T15:51:44.674 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:51:44.674 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:51:44.674 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:51:44.674 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:47:20.655168Z", 2024-08-18T15:51:44.674 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:51:44.674 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:51:44.675 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:51:44.675 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:51:44.675 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:51:44.675 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:51:44.675 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:51:44.675 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:51:44.675 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:51:44.675 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:51:44Z" 2024-08-18T15:51:44.675 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:51:44.676 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1723996294.242267_467.1 target=1_1723996040.655168_301.1 2024-08-18T15:51:49.677 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:51:49.678 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:51:49.678 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:51:49.734 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:51:49.734 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:51:49.766 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:51:49.764+0000 7f4abdb36880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:51:49.766 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:51:49.765+0000 7f4abdb36880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:51:49.780 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:51:49.780 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:51:49.780 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:51:49.780 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:51:49.780 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:51:49.780 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T15:51:49.781 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:51:49.781 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:51:49.781 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:51:49.781 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:51:49.781 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:51:49.781 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:51:49.781 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:51:49.781 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:51:49.781 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:51:49.781 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:51:49.781 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:51:49.781 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:51:49.781 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:51:49.782 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:51:49.782 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:51:49.782 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:51:49.782 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:51:49.782 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:51:49.782 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:51:49.782 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:51:49.782 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:51:49.782 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:51:49.782 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:51:49.782 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:51:49.782 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:51:49.783 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:51:49.783 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:51:49.783 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:51:49.783 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:51:49.783 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:51:49.783 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:51:49.783 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:51:49.783 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:51:49.783 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:51:49.783 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:51:49.783 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:51:49.783 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:51:49.783 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:51:49.783 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:51:49.784 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:51:49.784 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:51:49.784 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:51:49.784 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:51:49.784 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723996040.655168_301.1", 2024-08-18T15:51:49.784 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:51:49.784 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:51:49.784 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:51:49.784 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:47:20.655168Z", 2024-08-18T15:51:49.784 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:51:49.784 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:51:49.784 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:51:49.784 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:51:49.785 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:51:49.785 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:51:49.785 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:51:49.785 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:51:49.785 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:51:49.785 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:51:49Z" 2024-08-18T15:51:49.785 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:51:49.786 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1723996294.242267_467.1 target=1_1723996040.655168_301.1 2024-08-18T15:51:54.787 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:51:54.788 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:51:54.788 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:51:54.844 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:51:54.844 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:51:54.876 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:51:54.876+0000 7fed0f13c880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:51:54.877 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:51:54.877+0000 7fed0f13c880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:51:54.891 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:51:54.891 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:51:54.891 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:51:54.891 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:51:54.891 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:51:54.891 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T15:51:54.891 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:51:54.891 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:51:54.891 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:51:54.891 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:51:54.892 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:51:54.892 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:51:54.892 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:51:54.892 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:51:54.892 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:51:54.892 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:51:54.892 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:51:54.892 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:51:54.892 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:51:54.892 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:51:54.892 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:51:54.892 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:51:54.892 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:51:54.892 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:51:54.892 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:51:54.893 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:51:54.893 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:51:54.893 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:51:54.893 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:51:54.893 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:51:54.893 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:51:54.893 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:51:54.893 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:51:54.893 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:51:54.893 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:51:54.893 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:51:54.893 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:51:54.893 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:51:54.894 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:51:54.894 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:51:54.894 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:51:54.894 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:51:54.894 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:51:54.894 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:51:54.894 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:51:54.894 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:51:54.894 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:51:54.894 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:51:54.894 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:51:54.894 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723996040.655168_301.1", 2024-08-18T15:51:54.894 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:51:54.895 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:51:54.895 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:51:54.895 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:47:20.655168Z", 2024-08-18T15:51:54.895 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:51:54.895 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:51:54.895 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:51:54.895 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:51:54.895 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:51:54.895 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:51:54.895 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:51:54.895 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:51:54.895 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:51:54.895 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:51:54Z" 2024-08-18T15:51:54.895 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:51:54.896 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1723996294.242267_467.1 target=1_1723996040.655168_301.1 2024-08-18T15:51:59.898 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:51:59.898 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:51:59.898 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:51:59.955 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:51:59.955 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:51:59.989 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:51:59.989+0000 7f54f6f32880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:51:59.990 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:51:59.990+0000 7f54f6f32880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:52:00.004 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:52:00.004 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:52:00.004 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:52:00.004 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:52:00.004 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:52:00.004 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T15:52:00.004 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:52:00.004 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:52:00.004 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:52:00.004 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:52:00.004 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:52:00.005 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:52:00.005 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:52:00.005 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:52:00.005 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:52:00.005 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:52:00.005 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:52:00.005 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:52:00.005 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:52:00.005 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:52:00.005 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:52:00.005 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:52:00.005 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:52:00.006 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:52:00.006 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:52:00.006 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:52:00.006 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:52:00.006 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:52:00.006 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:52:00.006 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:52:00.006 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:52:00.006 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:52:00.006 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:52:00.006 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:52:00.007 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:52:00.007 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:52:00.007 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:52:00.007 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:52:00.007 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:52:00.007 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:52:00.007 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:52:00.007 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:52:00.007 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:52:00.007 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:52:00.007 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:52:00.007 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:52:00.008 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:52:00.008 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:52:00.008 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:52:00.008 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723996040.655168_301.1", 2024-08-18T15:52:00.008 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:52:00.008 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:52:00.008 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:52:00.008 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:47:20.655168Z", 2024-08-18T15:52:00.008 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:52:00.008 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:52:00.008 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:52:00.008 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:52:00.008 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:52:00.009 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:52:00.009 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:52:00.009 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:52:00.009 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:52:00.009 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:51:59Z" 2024-08-18T15:52:00.009 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:52:00.010 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1723996294.242267_467.1 target=1_1723996040.655168_301.1 2024-08-18T15:52:05.012 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:52:05.012 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:52:05.012 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:52:05.068 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:52:05.069 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:52:05.101 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:52:05.101+0000 7f106b931880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:52:05.102 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:52:05.102+0000 7f106b931880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:52:05.116 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:52:05.116 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:52:05.116 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:52:05.116 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:52:05.116 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:52:05.116 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T15:52:05.117 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:52:05.117 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:52:05.117 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:52:05.117 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:52:05.117 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:52:05.117 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:52:05.117 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:52:05.117 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:52:05.117 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:52:05.117 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:52:05.117 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:52:05.117 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:52:05.117 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:52:05.118 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:52:05.118 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:52:05.118 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:52:05.118 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:52:05.118 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:52:05.118 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:52:05.118 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:52:05.118 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:52:05.118 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:52:05.118 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:52:05.118 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:52:05.118 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:52:05.118 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:52:05.119 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:52:05.119 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:52:05.119 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:52:05.119 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:52:05.119 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:52:05.119 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:52:05.119 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:52:05.119 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:52:05.119 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:52:05.119 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:52:05.120 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:52:05.120 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:52:05.120 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:52:05.120 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:52:05.120 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:52:05.120 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:52:05.120 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:52:05.121 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723996040.655168_301.1", 2024-08-18T15:52:05.121 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:52:05.121 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:52:05.121 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:52:05.121 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:47:20.655168Z", 2024-08-18T15:52:05.121 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:52:05.121 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:52:05.121 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:52:05.121 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:52:05.121 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:52:05.121 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:52:05.121 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:52:05.121 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:52:05.122 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:52:05.122 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:52:05Z" 2024-08-18T15:52:05.122 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:52:05.123 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1723996294.242267_467.1 target=1_1723996040.655168_301.1 2024-08-18T15:52:10.124 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:52:10.125 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:52:10.125 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:52:10.180 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:52:10.180 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:52:10.212 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:52:10.212+0000 7fd881213880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:52:10.213 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:52:10.213+0000 7fd881213880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:52:10.226 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:52:10.227 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:52:10.227 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:52:10.227 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:52:10.227 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:52:10.227 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T15:52:10.227 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:52:10.227 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:52:10.227 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:52:10.227 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:52:10.227 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:52:10.227 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:52:10.227 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:52:10.228 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:52:10.228 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:52:10.228 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:52:10.228 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:52:10.228 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:52:10.228 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:52:10.228 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:52:10.228 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:52:10.228 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:52:10.228 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:52:10.228 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:52:10.228 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:52:10.228 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:52:10.229 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:52:10.229 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:52:10.229 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:52:10.229 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:52:10.229 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:52:10.229 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:52:10.229 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:52:10.229 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:52:10.229 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:52:10.229 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:52:10.230 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:52:10.230 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:52:10.230 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:52:10.230 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:52:10.230 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:52:10.230 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:52:10.230 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:52:10.230 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:52:10.230 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:52:10.230 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:52:10.230 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:52:10.230 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:52:10.230 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:52:10.231 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723996040.655168_301.1", 2024-08-18T15:52:10.231 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:52:10.231 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:52:10.231 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:52:10.231 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:47:20.655168Z", 2024-08-18T15:52:10.231 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:52:10.231 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:52:10.231 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:52:10.231 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:52:10.231 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:52:10.231 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:52:10.231 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:52:10.231 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:52:10.231 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:52:10.231 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:52:10Z" 2024-08-18T15:52:10.232 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:52:10.233 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1723996294.242267_467.1 target=1_1723996040.655168_301.1 2024-08-18T15:52:15.234 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:52:15.234 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:52:15.234 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:52:15.288 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:52:15.288 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:52:15.320 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:52:15.319+0000 7f5543720880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:52:15.321 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:52:15.320+0000 7f5543720880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:52:15.334 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:52:15.334 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:52:15.334 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:52:15.334 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:52:15.335 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:52:15.335 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T15:52:15.335 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:52:15.335 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:52:15.335 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:52:15.335 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:52:15.335 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:52:15.335 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:52:15.335 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:52:15.335 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:52:15.335 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:52:15.335 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:52:15.335 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:52:15.336 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:52:15.336 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:52:15.336 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:52:15.336 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:52:15.336 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:52:15.336 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:52:15.336 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:52:15.336 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:52:15.336 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:52:15.336 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:52:15.336 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:52:15.336 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:52:15.336 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:52:15.336 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:52:15.337 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:52:15.337 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:52:15.337 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:52:15.337 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:52:15.337 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:52:15.337 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:52:15.337 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:52:15.337 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:52:15.337 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:52:15.337 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:52:15.337 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:52:15.337 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:52:15.337 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:52:15.337 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:52:15.338 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:52:15.338 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:52:15.338 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:52:15.338 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:52:15.338 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723996040.655168_301.1", 2024-08-18T15:52:15.338 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:52:15.338 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:52:15.338 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:52:15.338 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:47:20.655168Z", 2024-08-18T15:52:15.338 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:52:15.338 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:52:15.338 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:52:15.338 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:52:15.339 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:52:15.339 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:52:15.339 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:52:15.339 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:52:15.339 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:52:15.339 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:52:15Z" 2024-08-18T15:52:15.339 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:52:15.340 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1723996294.242267_467.1 target=1_1723996040.655168_301.1 2024-08-18T15:52:20.341 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:52:20.341 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:52:20.341 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:52:20.395 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:52:20.396 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:52:20.428 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:52:20.428+0000 7f6aa2a94880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:52:20.429 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:52:20.429+0000 7f6aa2a94880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:52:20.443 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:52:20.444 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:52:20.444 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:52:20.444 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:52:20.444 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:52:20.444 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T15:52:20.444 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:52:20.444 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:52:20.444 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:52:20.444 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:52:20.444 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:52:20.444 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:52:20.444 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:52:20.444 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:52:20.445 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:52:20.445 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:52:20.445 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:52:20.445 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:52:20.445 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:52:20.445 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:52:20.445 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:52:20.445 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:52:20.445 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:52:20.446 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:52:20.446 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:52:20.446 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:52:20.446 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:52:20.446 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:52:20.446 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:52:20.446 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:52:20.446 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:52:20.446 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:52:20.446 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:52:20.446 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:52:20.446 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:52:20.446 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:52:20.447 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:52:20.447 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:52:20.447 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:52:20.447 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:52:20.447 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:52:20.447 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:52:20.447 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:52:20.447 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:52:20.447 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:52:20.447 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:52:20.447 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:52:20.447 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:52:20.447 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:52:20.447 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723996040.655168_301.1", 2024-08-18T15:52:20.448 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:52:20.448 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:52:20.448 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:52:20.448 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:47:20.655168Z", 2024-08-18T15:52:20.448 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:52:20.448 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:52:20.448 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:52:20.448 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:52:20.448 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:52:20.448 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:52:20.448 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:52:20.448 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:52:20.448 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:52:20.449 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:52:20Z" 2024-08-18T15:52:20.449 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:52:20.450 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1723996294.242267_467.1 target=1_1723996040.655168_301.1 2024-08-18T15:52:25.451 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:52:25.451 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:52:25.451 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:52:25.507 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:52:25.507 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:52:25.539 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:52:25.539+0000 7f7e39c15880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:52:25.540 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:52:25.540+0000 7f7e39c15880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:52:25.554 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:52:25.554 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:52:25.554 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:52:25.554 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:52:25.554 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:52:25.554 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T15:52:25.555 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:52:25.555 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:52:25.555 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:52:25.555 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:52:25.555 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:52:25.555 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:52:25.555 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:52:25.555 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:52:25.555 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:52:25.555 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:52:25.555 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:52:25.555 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:52:25.555 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:52:25.556 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:52:25.556 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:52:25.556 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:52:25.556 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:52:25.556 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:52:25.556 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:52:25.556 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:52:25.556 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:52:25.556 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:52:25.556 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:52:25.556 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:52:25.556 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:52:25.556 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:52:25.556 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:52:25.557 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:52:25.557 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:52:25.557 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:52:25.557 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:52:25.557 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:52:25.557 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:52:25.557 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:52:25.557 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:52:25.557 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:52:25.557 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:52:25.557 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:52:25.557 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:52:25.557 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:52:25.557 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:52:25.558 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:52:25.558 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:52:25.558 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723996040.655168_301.1", 2024-08-18T15:52:25.558 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:52:25.558 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:52:25.558 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:52:25.558 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:47:20.655168Z", 2024-08-18T15:52:25.558 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:52:25.558 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:52:25.558 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:52:25.558 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:52:25.558 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:52:25.558 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:52:25.559 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:52:25.559 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:52:25.559 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:52:25.559 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:52:25Z" 2024-08-18T15:52:25.559 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:52:25.560 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1723996294.242267_467.1 target=1_1723996040.655168_301.1 2024-08-18T15:52:30.561 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:52:30.561 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:52:30.561 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:52:30.616 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:52:30.617 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:52:30.650 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:52:30.649+0000 7f2020119880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:52:30.651 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:52:30.650+0000 7f2020119880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:52:30.664 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:52:30.665 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:52:30.665 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:52:30.665 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:52:30.665 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:52:30.665 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T15:52:30.665 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:52:30.665 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:52:30.665 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:52:30.665 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:52:30.665 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:52:30.665 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:52:30.665 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:52:30.665 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:52:30.665 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:52:30.666 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:52:30.666 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:52:30.666 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:52:30.666 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:52:30.666 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:52:30.666 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:52:30.666 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:52:30.666 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:52:30.666 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:52:30.666 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:52:30.666 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:52:30.666 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:52:30.667 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:52:30.667 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:52:30.667 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:52:30.667 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:52:30.667 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:52:30.667 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:52:30.667 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:52:30.667 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:52:30.667 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:52:30.667 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:52:30.667 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:52:30.667 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:52:30.667 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:52:30.667 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:52:30.668 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:52:30.668 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:52:30.668 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:52:30.668 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:52:30.668 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:52:30.668 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:52:30.668 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:52:30.668 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:52:30.668 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723996040.655168_301.1", 2024-08-18T15:52:30.668 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:52:30.668 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:52:30.668 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:52:30.668 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:47:20.655168Z", 2024-08-18T15:52:30.668 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:52:30.669 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:52:30.669 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:52:30.669 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:52:30.669 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:52:30.669 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:52:30.669 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:52:30.669 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:52:30.669 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:52:30.669 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:52:30Z" 2024-08-18T15:52:30.669 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:52:30.670 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1723996294.242267_467.1 target=1_1723996040.655168_301.1 2024-08-18T15:52:35.671 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:52:35.672 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:52:35.672 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:52:35.724 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:52:35.725 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:52:35.756 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:52:35.755+0000 7fbdc4898880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:52:35.756 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:52:35.756+0000 7fbdc4898880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:52:35.770 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:52:35.770 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:52:35.770 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:52:35.770 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:52:35.770 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:52:35.770 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T15:52:35.771 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:52:35.771 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:52:35.771 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:52:35.771 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:52:35.771 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:52:35.771 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:52:35.771 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:52:35.771 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:52:35.771 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:52:35.771 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:52:35.771 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:52:35.771 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:52:35.772 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:52:35.772 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:52:35.772 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:52:35.772 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:52:35.772 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:52:35.772 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:52:35.772 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:52:35.772 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:52:35.772 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:52:35.772 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:52:35.772 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:52:35.772 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:52:35.772 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:52:35.773 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:52:35.773 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:52:35.773 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:52:35.773 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:52:35.773 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:52:35.773 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:52:35.773 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:52:35.773 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:52:35.773 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:52:35.773 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:52:35.773 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:52:35.773 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:52:35.773 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:52:35.773 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:52:35.774 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:52:35.774 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:52:35.774 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:52:35.774 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:52:35.774 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723996040.655168_301.1", 2024-08-18T15:52:35.774 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:52:35.774 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:52:35.774 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:52:35.774 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:47:20.655168Z", 2024-08-18T15:52:35.774 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:52:35.774 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:52:35.774 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:52:35.775 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:52:35.775 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:52:35.775 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:52:35.775 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:52:35.775 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:52:35.775 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:52:35.775 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:52:35Z" 2024-08-18T15:52:35.775 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:52:35.776 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1723996294.242267_467.1 target=1_1723996040.655168_301.1 2024-08-18T15:52:40.777 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:52:40.778 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:52:40.778 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:52:40.833 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:52:40.833 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:52:40.866 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:52:40.866+0000 7fa09eb29880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:52:40.867 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:52:40.867+0000 7fa09eb29880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:52:40.880 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:52:40.881 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:52:40.881 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:52:40.881 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:52:40.881 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:52:40.881 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T15:52:40.881 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:52:40.881 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:52:40.881 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:52:40.881 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:52:40.881 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:52:40.881 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:52:40.882 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:52:40.882 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:52:40.882 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:52:40.882 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:52:40.882 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:52:40.882 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:52:40.882 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:52:40.882 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:52:40.882 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:52:40.882 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:52:40.882 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:52:40.882 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:52:40.883 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:52:40.883 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:52:40.883 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:52:40.883 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:52:40.883 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:52:40.883 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:52:40.883 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:52:40.883 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:52:40.883 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:52:40.883 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:52:40.883 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:52:40.883 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:52:40.883 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:52:40.883 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:52:40.884 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:52:40.884 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:52:40.884 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:52:40.884 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:52:40.884 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:52:40.884 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:52:40.884 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:52:40.884 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:52:40.884 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:52:40.884 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:52:40.884 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:52:40.884 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723996040.655168_301.1", 2024-08-18T15:52:40.884 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:52:40.885 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:52:40.885 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:52:40.885 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:47:20.655168Z", 2024-08-18T15:52:40.885 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:52:40.885 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:52:40.885 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:52:40.885 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:52:40.885 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:52:40.885 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:52:40.885 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:52:40.885 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:52:40.885 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:52:40.885 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:52:40Z" 2024-08-18T15:52:40.885 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:52:40.886 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1723996294.242267_467.1 target=1_1723996040.655168_301.1 2024-08-18T15:52:45.888 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:52:45.888 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:52:45.888 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:52:45.944 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:52:45.944 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:52:45.978 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:52:45.978+0000 7fd616f27880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:52:45.979 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:52:45.979+0000 7fd616f27880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:52:45.995 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:52:45.995 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:52:45.995 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:52:45.995 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:52:45.996 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:52:45.996 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T15:52:45.996 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:52:45.996 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:52:45.996 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:52:45.996 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:52:45.996 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:52:45.996 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:52:45.996 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:52:45.996 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:52:45.996 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:52:45.996 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:52:45.996 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:52:45.997 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:52:45.997 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:52:45.997 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:52:45.997 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:52:45.997 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:52:45.997 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:52:45.997 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:52:45.997 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:52:45.997 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:52:45.997 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:52:45.997 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:52:45.997 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:52:45.997 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:52:45.997 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:52:45.998 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:52:45.998 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:52:45.998 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:52:45.998 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:52:45.998 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:52:45.998 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:52:45.998 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:52:45.998 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:52:45.998 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:52:45.998 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:52:45.998 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:52:45.998 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:52:45.998 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:52:45.999 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:52:45.999 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:52:45.999 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:52:45.999 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:52:45.999 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:52:45.999 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723996040.655168_301.1", 2024-08-18T15:52:45.999 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:52:45.999 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:52:45.999 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:52:45.999 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:47:20.655168Z", 2024-08-18T15:52:46.000 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:52:46.000 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:52:46.000 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:52:46.000 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:52:46.000 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:52:46.000 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:52:46.000 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:52:46.000 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:52:46.000 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:52:46.000 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:52:45Z" 2024-08-18T15:52:46.000 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:52:46.001 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1723996294.242267_467.1 target=1_1723996040.655168_301.1 2024-08-18T15:52:51.002 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:52:51.003 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:52:51.003 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:52:51.058 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:52:51.058 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:52:51.091 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:52:51.089+0000 7f4b32496880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:52:51.091 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:52:51.090+0000 7f4b32496880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:52:51.105 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:52:51.105 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:52:51.105 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:52:51.106 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:52:51.106 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:52:51.106 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T15:52:51.106 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:52:51.106 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:52:51.106 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:52:51.106 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:52:51.106 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:52:51.106 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:52:51.106 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:52:51.106 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:52:51.106 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:52:51.106 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:52:51.107 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:52:51.107 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:52:51.107 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:52:51.107 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:52:51.107 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:52:51.107 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:52:51.107 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:52:51.107 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:52:51.107 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:52:51.107 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:52:51.107 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:52:51.107 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:52:51.107 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:52:51.108 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:52:51.108 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:52:51.108 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:52:51.108 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:52:51.108 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:52:51.108 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:52:51.108 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:52:51.108 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:52:51.108 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:52:51.108 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:52:51.108 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:52:51.108 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:52:51.109 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:52:51.109 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:52:51.109 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:52:51.109 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:52:51.109 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:52:51.109 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:52:51.109 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:52:51.109 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:52:51.109 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723996040.655168_301.1", 2024-08-18T15:52:51.109 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:52:51.109 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:52:51.109 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:52:51.109 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:47:20.655168Z", 2024-08-18T15:52:51.110 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:52:51.110 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:52:51.110 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:52:51.110 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:52:51.110 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:52:51.110 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:52:51.110 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:52:51.110 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:52:51.110 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:52:51.110 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:52:51Z" 2024-08-18T15:52:51.110 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:52:51.111 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1723996294.242267_467.1 target=1_1723996040.655168_301.1 2024-08-18T15:52:56.112 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:52:56.112 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:52:56.113 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:52:56.168 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:52:56.169 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:52:56.202 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:52:56.201+0000 7f2670333880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:52:56.202 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:52:56.202+0000 7f2670333880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:52:56.216 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:52:56.216 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:52:56.217 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:52:56.217 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:52:56.217 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:52:56.217 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T15:52:56.217 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:52:56.217 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:52:56.218 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:52:56.218 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:52:56.218 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:52:56.218 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:52:56.218 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:52:56.219 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:52:56.219 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:52:56.219 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:52:56.219 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:52:56.219 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:52:56.220 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:52:56.220 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:52:56.220 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:52:56.220 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:52:56.220 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:52:56.220 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:52:56.221 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:52:56.221 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:52:56.221 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:52:56.221 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:52:56.221 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:52:56.222 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:52:56.222 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:52:56.222 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:52:56.222 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:52:56.222 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:52:56.222 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:52:56.223 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:52:56.223 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:52:56.223 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:52:56.223 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:52:56.223 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:52:56.224 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:52:56.224 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:52:56.224 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:52:56.224 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:52:56.224 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:52:56.224 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:52:56.224 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:52:56.225 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:52:56.225 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:52:56.225 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723996040.655168_301.1", 2024-08-18T15:52:56.225 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:52:56.225 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:52:56.225 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:52:56.226 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:47:20.655168Z", 2024-08-18T15:52:56.226 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:52:56.226 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:52:56.226 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:52:56.226 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:52:56.226 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:52:56.226 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:52:56.227 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:52:56.227 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:52:56.227 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:52:56.227 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:52:56Z" 2024-08-18T15:52:56.227 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:52:56.228 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1723996294.242267_467.1 target=1_1723996040.655168_301.1 2024-08-18T15:53:01.229 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:53:01.230 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:53:01.230 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:53:01.285 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:53:01.285 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:53:01.318 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:53:01.317+0000 7ff9cb235880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:53:01.318 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:53:01.318+0000 7ff9cb235880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:53:01.332 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:53:01.333 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:53:01.333 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:53:01.333 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:53:01.333 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:53:01.333 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T15:53:01.334 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:53:01.334 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:53:01.334 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:53:01.334 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:53:01.334 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:53:01.334 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:53:01.335 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:53:01.335 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:53:01.335 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:53:01.335 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:53:01.335 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:53:01.335 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:53:01.336 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:53:01.336 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:53:01.336 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:53:01.336 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:53:01.336 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:53:01.336 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:53:01.336 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:53:01.337 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:53:01.337 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:53:01.337 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:53:01.337 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:53:01.337 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:53:01.337 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:53:01.338 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:53:01.338 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:53:01.338 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:53:01.338 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:53:01.338 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:53:01.338 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:53:01.339 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:53:01.339 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:53:01.339 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:53:01.339 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:53:01.339 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:53:01.340 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:53:01.340 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:53:01.340 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:53:01.340 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:53:01.340 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:53:01.340 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:53:01.340 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:53:01.340 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723996040.655168_301.1", 2024-08-18T15:53:01.340 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:53:01.340 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:53:01.340 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:53:01.341 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:47:20.655168Z", 2024-08-18T15:53:01.341 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:53:01.341 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:53:01.341 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:53:01.341 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:53:01.341 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:53:01.341 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:53:01.341 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:53:01.341 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:53:01.341 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:53:01.341 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:53:01Z" 2024-08-18T15:53:01.341 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:53:01.342 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1723996294.242267_467.1 target=1_1723996040.655168_301.1 2024-08-18T15:53:06.343 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:53:06.343 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:53:06.344 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:53:06.399 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:53:06.399 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:53:06.432 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:53:06.431+0000 7f194cf42880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:53:06.432 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:53:06.432+0000 7f194cf42880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:53:06.446 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:53:06.446 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:53:06.446 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:53:06.447 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:53:06.447 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:53:06.447 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T15:53:06.447 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:53:06.447 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:53:06.447 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:53:06.447 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:53:06.447 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:53:06.447 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:53:06.447 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:53:06.447 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:53:06.447 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:53:06.448 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:53:06.448 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:53:06.448 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:53:06.448 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:53:06.448 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:53:06.448 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:53:06.448 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:53:06.448 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:53:06.448 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:53:06.448 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:53:06.448 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:53:06.449 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:53:06.449 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:53:06.449 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:53:06.449 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:53:06.449 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:53:06.449 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:53:06.449 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:53:06.449 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:53:06.449 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:53:06.449 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:53:06.449 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:53:06.449 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:53:06.450 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:53:06.450 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:53:06.450 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:53:06.450 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:53:06.450 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:53:06.450 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:53:06.450 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:53:06.450 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:53:06.450 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:53:06.451 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:53:06.451 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:53:06.451 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723996040.655168_301.1", 2024-08-18T15:53:06.451 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:53:06.451 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:53:06.451 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:53:06.451 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:47:20.655168Z", 2024-08-18T15:53:06.452 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:53:06.452 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:53:06.452 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:53:06.452 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:53:06.452 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:53:06.452 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:53:06.452 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:53:06.452 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:53:06.452 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:53:06.452 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:53:06Z" 2024-08-18T15:53:06.452 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:53:06.454 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1723996294.242267_467.1 target=1_1723996040.655168_301.1 2024-08-18T15:53:11.454 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:53:11.454 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:53:11.455 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:53:11.510 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:53:11.510 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:53:11.542 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:53:11.542+0000 7f6904923880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:53:11.543 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:53:11.543+0000 7f6904923880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:53:11.557 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:53:11.557 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:53:11.557 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:53:11.557 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:53:11.557 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:53:11.557 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T15:53:11.557 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:53:11.557 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:53:11.557 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:53:11.558 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:53:11.558 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:53:11.558 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:53:11.558 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:53:11.558 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:53:11.558 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:53:11.558 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:53:11.558 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:53:11.558 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:53:11.558 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:53:11.558 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:53:11.558 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:53:11.559 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:53:11.559 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:53:11.559 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:53:11.559 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:53:11.559 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:53:11.559 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:53:11.559 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:53:11.559 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:53:11.559 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:53:11.559 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:53:11.559 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:53:11.559 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:53:11.560 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:53:11.560 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:53:11.560 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:53:11.560 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:53:11.560 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:53:11.560 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:53:11.560 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:53:11.560 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:53:11.560 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:53:11.560 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:53:11.560 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:53:11.560 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:53:11.560 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:53:11.561 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:53:11.561 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:53:11.561 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:53:11.561 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723996040.655168_301.1", 2024-08-18T15:53:11.561 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:53:11.561 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:53:11.561 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:53:11.561 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:47:20.655168Z", 2024-08-18T15:53:11.561 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:53:11.561 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:53:11.561 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:53:11.561 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:53:11.561 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:53:11.562 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:53:11.562 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:53:11.562 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:53:11.562 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:53:11.562 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:53:11Z" 2024-08-18T15:53:11.562 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:53:11.563 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1723996294.242267_467.1 target=1_1723996040.655168_301.1 2024-08-18T15:53:16.564 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:53:16.564 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:53:16.564 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:53:16.619 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:53:16.619 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:53:16.652 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:53:16.651+0000 7f5b05b36880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:53:16.652 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:53:16.652+0000 7f5b05b36880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:53:16.666 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:53:16.667 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:53:16.667 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:53:16.667 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:53:16.667 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:53:16.667 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T15:53:16.667 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:53:16.667 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:53:16.667 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:53:16.667 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:53:16.667 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:53:16.667 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:53:16.668 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:53:16.668 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:53:16.668 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:53:16.668 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:53:16.668 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:53:16.668 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:53:16.668 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:53:16.668 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:53:16.668 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:53:16.668 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:53:16.669 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:53:16.669 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:53:16.669 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:53:16.669 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:53:16.669 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:53:16.669 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:53:16.669 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:53:16.669 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:53:16.669 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:53:16.669 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:53:16.669 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:53:16.669 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:53:16.669 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:53:16.669 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:53:16.670 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:53:16.670 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:53:16.670 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:53:16.670 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:53:16.670 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:53:16.670 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:53:16.670 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:53:16.670 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:53:16.670 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:53:16.670 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:53:16.670 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:53:16.671 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:53:16.671 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:53:16.671 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723996040.655168_301.1", 2024-08-18T15:53:16.671 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:53:16.671 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:53:16.671 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:53:16.671 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:47:20.655168Z", 2024-08-18T15:53:16.671 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:53:16.671 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:53:16.671 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:53:16.671 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:53:16.671 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:53:16.671 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:53:16.672 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:53:16.672 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:53:16.672 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:53:16.672 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:53:16Z" 2024-08-18T15:53:16.672 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:53:16.673 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1723996294.242267_467.1 target=1_1723996040.655168_301.1 2024-08-18T15:53:21.674 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:53:21.674 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:53:21.674 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:53:21.729 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:53:21.729 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:53:21.762 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:53:21.761+0000 7fd651013880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:53:21.762 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:53:21.762+0000 7fd651013880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:53:21.776 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:53:21.776 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:53:21.777 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:53:21.777 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:53:21.777 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:53:21.777 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T15:53:21.777 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:53:21.777 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:53:21.777 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:53:21.777 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:53:21.777 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:53:21.777 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:53:21.777 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:53:21.777 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:53:21.777 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:53:21.778 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:53:21.778 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:53:21.778 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:53:21.778 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:53:21.778 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:53:21.778 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:53:21.778 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:53:21.778 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:53:21.778 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:53:21.778 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:53:21.778 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:53:21.778 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:53:21.779 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:53:21.779 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:53:21.779 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:53:21.779 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:53:21.779 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:53:21.779 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:53:21.779 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:53:21.779 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:53:21.779 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:53:21.779 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:53:21.779 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:53:21.779 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:53:21.779 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:53:21.779 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:53:21.779 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:53:21.780 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:53:21.780 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:53:21.780 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:53:21.780 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:53:21.780 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:53:21.780 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:53:21.780 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:53:21.780 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723996040.655168_301.1", 2024-08-18T15:53:21.780 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:53:21.780 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:53:21.780 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:53:21.780 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:47:20.655168Z", 2024-08-18T15:53:21.780 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:53:21.780 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:53:21.781 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:53:21.781 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:53:21.781 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:53:21.781 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:53:21.781 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:53:21.781 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:53:21.781 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:53:21.781 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:53:21Z" 2024-08-18T15:53:21.781 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:53:21.782 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1723996294.242267_467.1 target=1_1723996040.655168_301.1 2024-08-18T15:53:26.783 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:53:26.783 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:53:26.784 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:53:26.838 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:53:26.839 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:53:26.871 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:53:26.870+0000 7fbfd3f22880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:53:26.872 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:53:26.871+0000 7fbfd3f22880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:53:26.886 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:53:26.886 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:53:26.886 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:53:26.886 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:53:26.886 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:53:26.886 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T15:53:26.886 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:53:26.886 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:53:26.886 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:53:26.886 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:53:26.887 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:53:26.887 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:53:26.887 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:53:26.887 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:53:26.887 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:53:26.887 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:53:26.887 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:53:26.887 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:53:26.887 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:53:26.888 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:53:26.888 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:53:26.888 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:53:26.888 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:53:26.888 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:53:26.888 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:53:26.888 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:53:26.888 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:53:26.888 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:53:26.888 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:53:26.888 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:53:26.888 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:53:26.888 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:53:26.888 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:53:26.889 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:53:26.889 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:53:26.889 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:53:26.889 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:53:26.889 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:53:26.889 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:53:26.889 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:53:26.889 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:53:26.889 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:53:26.889 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:53:26.889 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:53:26.889 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:53:26.890 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:53:26.890 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:53:26.890 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:53:26.890 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:53:26.890 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723996040.655168_301.1", 2024-08-18T15:53:26.890 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:53:26.890 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:53:26.890 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:53:26.890 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:47:20.655168Z", 2024-08-18T15:53:26.890 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:53:26.891 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:53:26.891 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:53:26.891 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:53:26.891 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:53:26.891 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:53:26.891 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:53:26.891 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:53:26.892 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:53:26.892 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:53:26Z" 2024-08-18T15:53:26.892 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:53:26.893 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1723996294.242267_467.1 target=1_1723996040.655168_301.1 2024-08-18T15:53:31.894 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:53:31.894 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:53:31.895 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:53:31.950 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:53:31.950 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:53:31.982 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:53:31.981+0000 7f10c2d23880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:53:31.982 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:53:31.982+0000 7f10c2d23880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:53:31.996 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:53:31.996 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:53:31.996 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:53:31.996 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:53:31.997 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:53:31.997 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T15:53:31.997 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:53:31.997 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:53:31.997 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:53:31.997 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:53:31.997 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:53:31.997 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:53:31.997 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:53:31.997 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:53:31.997 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:53:31.997 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:53:31.997 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:53:31.998 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:53:31.998 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:53:31.998 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:53:31.998 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:53:31.998 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:53:31.998 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:53:31.998 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:53:31.998 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:53:31.998 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:53:31.998 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:53:31.998 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:53:31.998 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:53:31.998 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:53:31.999 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:53:31.999 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:53:31.999 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:53:31.999 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:53:31.999 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:53:31.999 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:53:31.999 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:53:31.999 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:53:31.999 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:53:31.999 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:53:31.999 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:53:31.999 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:53:31.999 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:53:32.000 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:53:32.000 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:53:32.000 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:53:32.000 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:53:32.000 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:53:32.000 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:53:32.000 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723996040.655168_301.1", 2024-08-18T15:53:32.000 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:53:32.001 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:53:32.001 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:53:32.001 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:47:20.655168Z", 2024-08-18T15:53:32.001 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:53:32.001 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:53:32.001 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:53:32.001 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:53:32.001 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:53:32.001 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:53:32.001 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:53:32.001 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:53:32.001 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:53:32.001 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:53:31Z" 2024-08-18T15:53:32.002 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:53:32.003 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1723996294.242267_467.1 target=1_1723996040.655168_301.1 2024-08-18T15:53:37.003 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:53:37.003 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:53:37.004 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:53:37.059 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:53:37.059 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:53:37.091 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:53:37.091+0000 7fb53c23f880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:53:37.092 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:53:37.092+0000 7fb53c23f880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:53:37.106 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:53:37.106 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:53:37.106 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:53:37.106 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:53:37.106 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:53:37.107 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T15:53:37.107 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:53:37.107 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:53:37.107 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:53:37.107 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:53:37.107 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:53:37.107 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:53:37.107 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:53:37.107 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:53:37.107 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:53:37.107 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:53:37.107 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:53:37.107 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:53:37.108 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:53:37.108 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:53:37.108 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:53:37.108 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:53:37.108 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:53:37.108 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:53:37.108 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:53:37.108 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:53:37.108 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:53:37.108 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:53:37.108 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:53:37.108 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:53:37.108 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:53:37.108 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:53:37.108 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:53:37.109 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:53:37.109 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:53:37.109 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:53:37.109 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:53:37.109 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:53:37.109 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:53:37.109 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:53:37.109 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:53:37.109 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:53:37.109 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:53:37.109 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:53:37.109 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:53:37.109 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:53:37.110 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:53:37.110 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:53:37.110 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:53:37.110 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723996291.941226_414.1", 2024-08-18T15:53:37.110 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:53:37.110 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:53:37.110 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:53:37.110 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:51:31.941226Z", 2024-08-18T15:53:37.110 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:53:37.110 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:53:37.110 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:53:37.110 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:53:37.111 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:53:37.111 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:53:37.111 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:53:37.111 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:53:37.111 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:53:37.111 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:53:37Z" 2024-08-18T15:53:37.111 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:53:37.112 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1723996294.242267_467.1 target=1_1723996291.941226_414.1 2024-08-18T15:53:42.112 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:53:42.113 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:53:42.113 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:53:42.175 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:53:42.175 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:53:42.207 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:53:42.207+0000 7efdfdd22880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:53:42.208 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:53:42.207+0000 7efdfdd22880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:53:42.221 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:53:42.221 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:53:42.221 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:53:42.222 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:53:42.222 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:53:42.222 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T15:53:42.222 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:53:42.222 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:53:42.222 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:53:42.222 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:53:42.222 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:53:42.222 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:53:42.222 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:53:42.222 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:53:42.222 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:53:42.222 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:53:42.223 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:53:42.223 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:53:42.223 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:53:42.223 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:53:42.223 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:53:42.223 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:53:42.223 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:53:42.223 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:53:42.223 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:53:42.223 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:53:42.223 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:53:42.223 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:53:42.223 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:53:42.224 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:53:42.224 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:53:42.224 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:53:42.224 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:53:42.224 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:53:42.224 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:53:42.224 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:53:42.224 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:53:42.224 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:53:42.224 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:53:42.224 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:53:42.224 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:53:42.224 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:53:42.224 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:53:42.225 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:53:42.225 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:53:42.225 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:53:42.225 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:53:42.225 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:53:42.225 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:53:42.225 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723996292.127213_442.1", 2024-08-18T15:53:42.225 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:53:42.225 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:53:42.225 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:53:42.225 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:51:32.127213Z", 2024-08-18T15:53:42.225 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:53:42.225 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:53:42.226 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:53:42.226 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:53:42.226 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:53:42.226 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:53:42.226 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:53:42.226 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:53:42.226 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:53:42.226 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:53:42Z" 2024-08-18T15:53:42.226 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:53:42.227 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1723996294.242267_467.1 target=1_1723996292.127213_442.1 2024-08-18T15:53:47.228 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:53:47.228 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:53:47.229 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:53:47.283 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:53:47.283 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:53:47.315 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:53:47.314+0000 7fc487d20880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:53:47.316 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:53:47.315+0000 7fc487d20880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:53:47.329 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:53:47.329 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:53:47.329 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:53:47.330 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:53:47.330 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:53:47.330 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T15:53:47.330 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:53:47.330 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:53:47.330 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:53:47.330 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:53:47.330 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:53:47.330 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:53:47.330 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:53:47.330 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:53:47.330 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:53:47.330 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:53:47.331 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:53:47.331 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:53:47.331 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:53:47.331 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:53:47.331 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:53:47.331 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:53:47.331 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:53:47.331 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:53:47.331 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:53:47.331 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:53:47.331 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:53:47.331 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:53:47.331 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:53:47.331 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:53:47.332 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:53:47.332 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:53:47.332 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:53:47.332 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:53:47.332 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:53:47.332 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:53:47.332 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:53:47.332 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:53:47.332 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:53:47.332 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:53:47.332 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:53:47.332 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:53:47.332 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:53:47.333 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:53:47.333 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:53:47.333 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:53:47.333 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:53:47.333 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:53:47.333 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:53:47.333 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723996292.055702_431.1", 2024-08-18T15:53:47.333 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:53:47.333 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:53:47.333 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:53:47.333 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:51:32.055702Z", 2024-08-18T15:53:47.333 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:53:47.333 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:53:47.333 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:53:47.334 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:53:47.334 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:53:47.334 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:53:47.334 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:53:47.334 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:53:47.334 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:53:47.334 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:53:47Z" 2024-08-18T15:53:47.334 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:53:47.335 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1723996294.242267_467.1 target=1_1723996292.055702_431.1 2024-08-18T15:53:52.336 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:53:52.336 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:53:52.336 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:53:52.391 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:53:52.392 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:53:52.423 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:53:52.422+0000 7fbdf9624880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:53:52.424 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:53:52.423+0000 7fbdf9624880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:53:52.438 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:53:52.438 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:53:52.439 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:53:52.439 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:53:52.439 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:53:52.439 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T15:53:52.439 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:53:52.439 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:53:52.439 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:53:52.439 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:53:52.439 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:53:52.439 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:53:52.439 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:53:52.439 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:53:52.439 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:53:52.440 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:53:52.440 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:53:52.440 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:53:52.440 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:53:52.440 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:53:52.440 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:53:52.440 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:53:52.440 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:53:52.440 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:53:52.440 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:53:52.440 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:53:52.440 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:53:52.440 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:53:52.440 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:53:52.441 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:53:52.441 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:53:52.441 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:53:52.441 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:53:52.441 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:53:52.441 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:53:52.441 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:53:52.441 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:53:52.441 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:53:52.441 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:53:52.441 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:53:52.441 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:53:52.441 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:53:52.442 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:53:52.442 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:53:52.442 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:53:52.442 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:53:52.442 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:53:52.442 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:53:52.442 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:53:52.442 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723996294.242267_467.1", 2024-08-18T15:53:52.442 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:53:52.442 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:53:52.442 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:53:52.442 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:51:34.242267Z", 2024-08-18T15:53:52.442 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:53:52.443 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:53:52.443 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:53:52.443 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:53:52.443 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:53:52.443 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:53:52.443 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:53:52.443 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:53:52.443 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:53:52.443 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:53:52Z" 2024-08-18T15:53:52.443 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:53:52.444 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-08-18T15:53:52.444 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-08-18T15:53:52.445 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:53:52.445 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:53:52.445 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:53:52.501 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:53:52.501 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:53:52.533 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:53:52.532+0000 7f9c44538880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:53:52.533 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:53:52.533+0000 7f9c44538880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:53:52.546 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T15:53:52.547 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T15:53:52.547 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T15:53:52.547 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T15:53:52.547 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T15:53:52.547 INFO:teuthology.orchestra.run.smithi134.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T15:53:52.547 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T15:53:52.547 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:53:52.547 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T15:53:52.547 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:53:52.547 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T15:53:52.547 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:53:52.547 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:53:52.548 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:53:52.548 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:53:52.548 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 1, 2024-08-18T15:53:52.548 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:53:52.548 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:53:52.548 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:53:52.548 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:53:52.548 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:53:52.548 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:53:52.548 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T15:53:52.548 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:53:52.548 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:53:52.548 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T15:53:52.548 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:53:52.549 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:53:52.549 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:53:52.549 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T15:53:52.549 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:53:52.549 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:53:52.549 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:53:52.549 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:53:52.549 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T15:53:52.549 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:53:52.549 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:53:52.549 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:53:52.549 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:53:52.549 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:53:52.550 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:53:52.550 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:53:52.550 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:53:52.550 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:53:52.550 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:53:52.550 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:53:52.550 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T15:53:52.550 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:53:52.550 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:53:52.550 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723996294.242267_467.1", 2024-08-18T15:53:52.550 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:53:52.550 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:53:52.550 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:53:52.550 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:51:34.242267Z", 2024-08-18T15:53:52.551 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T15:53:52.551 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:53:52.551 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:53:52.551 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T15:53:52.551 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:53:52.551 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T15:53:52.551 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 1, 2024-08-18T15:53:52.551 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 1 2024-08-18T15:53:52.551 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:53:52.551 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T15:53:52Z" 2024-08-18T15:53:52.551 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T15:53:52.552 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-08-18T15:53:52.553 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'kibgmb-59', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:53:52.553 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'kibgmb-59', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:53:52.553 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket kibgmb-59 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T15:53:52.603 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:53:52.604 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:53:52.670 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'kibgmb-60', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:53:52.670 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'kibgmb-60', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:53:52.671 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket kibgmb-60 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T15:53:52.718 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:53:52.718 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:53:52.786 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'kibgmb-61', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:53:52.786 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'kibgmb-61', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T15:53:52.786 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket kibgmb-61 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T15:53:52.834 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:53:52.834 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:53:52.902 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['log', 'list', '--rgw-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:53:52.902 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'log', 'list', '--rgw-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:53:52.902 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 log list --rgw-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:53:52.946 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:53:52.946 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:53:53.035 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T15:53:53.036 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:53:53.036 INFO:teuthology.orchestra.run.smithi134.stdout: "sync.error-log.29", 2024-08-18T15:53:53.036 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000078", 2024-08-18T15:53:53.036 INFO:teuthology.orchestra.run.smithi134.stdout: "meta.history", 2024-08-18T15:53:53.036 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:53:53.036 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:53:53.036 INFO:teuthology.orchestra.run.smithi134.stdout: "sync.error-log.23", 2024-08-18T15:53:53.036 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000070", 2024-08-18T15:53:53.036 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000104", 2024-08-18T15:53:53.036 INFO:teuthology.orchestra.run.smithi134.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2.retry", 2024-08-18T15:53:53.036 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:53:53.037 INFO:teuthology.orchestra.run.smithi134.stdout: "datalog.sync-status.shard.10090ccf-aad6-416d-9775-df8b7b27345a.0.retry", 2024-08-18T15:53:53.037 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:53:53.037 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:53:53.037 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000026", 2024-08-18T15:53:53.037 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:53:53.037 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:53:53.037 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:53:53.037 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:53:53.037 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:53:53.037 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:53:53.037 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:53:53.037 INFO:teuthology.orchestra.run.smithi134.stdout: "sync.error-log.8", 2024-08-18T15:53:53.037 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:53:53.038 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:53:53.038 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000028", 2024-08-18T15:53:53.038 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:53:53.038 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.full-sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:53:53.038 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:53:53.038 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:53:53.038 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:53:53.038 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:53:53.038 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:53:53.038 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.full-sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:53:53.038 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:53:53.039 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:53:53.039 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:53:53.039 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:53:53.039 INFO:teuthology.orchestra.run.smithi134.stdout: "sync.error-log.30", 2024-08-18T15:53:53.039 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:53:53.039 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:53:53.039 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:53:53.039 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:53:53.039 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:53:53.039 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000040", 2024-08-18T15:53:53.039 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:53:53.039 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:53:53.039 INFO:teuthology.orchestra.run.smithi134.stdout: "sync.error-log.6", 2024-08-18T15:53:53.039 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:53:53.040 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:53:53.040 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:53:53.041 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:53:53.041 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000015", 2024-08-18T15:53:53.041 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:53:53.041 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:53:53.041 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000069", 2024-08-18T15:53:53.042 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:53:53.042 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000095", 2024-08-18T15:53:53.042 INFO:teuthology.orchestra.run.smithi134.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0", 2024-08-18T15:53:53.042 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:53:53.042 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:53:53.042 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000003", 2024-08-18T15:53:53.042 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000047", 2024-08-18T15:53:53.042 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:53:53.042 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:53:53.043 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:53:53.043 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:53:53.043 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:53:53.043 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:53:53.043 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:53:53.043 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000035", 2024-08-18T15:53:53.043 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000037", 2024-08-18T15:53:53.043 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:53:53.043 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.full-sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:53:53.043 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:53:53.043 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:53:53.043 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:53:53.044 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:53:53.044 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:53:53.044 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:53:53.044 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:53:53.044 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:53:53.044 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:53:53.044 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000024", 2024-08-18T15:53:53.044 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:53:53.044 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:53:53.044 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000000", 2024-08-18T15:53:53.044 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:53:53.044 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:53:53.044 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000031", 2024-08-18T15:53:53.045 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:53:53.045 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:53:53.045 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:53:53.045 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:53:53.045 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:53:53.045 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:53:53.045 INFO:teuthology.orchestra.run.smithi134.stdout: "data_log.3", 2024-08-18T15:53:53.045 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:53:53.045 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:53:53.045 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:53:53.045 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:53:53.046 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:53:53.046 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:53:53.046 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:53:53.046 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:53:53.047 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:53:53.047 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:53:53.047 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:53:53.047 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:53:53.047 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:53:53.047 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:53:53.047 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:53:53.047 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:53:53.047 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:53:53.047 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:53:53.048 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:53:53.048 INFO:teuthology.orchestra.run.smithi134.stdout: "datalog.sync-status.shard.10090ccf-aad6-416d-9775-df8b7b27345a.1", 2024-08-18T15:53:53.048 INFO:teuthology.orchestra.run.smithi134.stdout: "data_loggenerations_metadata", 2024-08-18T15:53:53.048 INFO:teuthology.orchestra.run.smithi134.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2", 2024-08-18T15:53:53.048 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:53:53.048 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:53:53.048 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:53:53.048 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:53:53.048 INFO:teuthology.orchestra.run.smithi134.stdout: "datalog.sync-status.018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T15:53:53.048 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:53:53.048 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:53:53.048 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:53:53.048 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:53:53.049 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:53:53.049 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000076", 2024-08-18T15:53:53.049 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:53:53.049 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:53:53.049 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000113", 2024-08-18T15:53:53.049 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000005", 2024-08-18T15:53:53.049 INFO:teuthology.orchestra.run.smithi134.stdout: "datalog.sync-status.shard.10090ccf-aad6-416d-9775-df8b7b27345a.3", 2024-08-18T15:53:53.049 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:53:53.049 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000011", 2024-08-18T15:53:53.049 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:53:53.049 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000058", 2024-08-18T15:53:53.049 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:53:53.050 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:53:53.050 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000052", 2024-08-18T15:53:53.050 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:53:53.050 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:53:53.050 INFO:teuthology.orchestra.run.smithi134.stdout: "sync.error-log.21", 2024-08-18T15:53:53.050 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:53:53.050 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:53:53.050 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:53:53.050 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:53:53.050 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:53:53.050 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:53:53.050 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:53:53.050 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:53:53.051 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:53:53.051 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:53:53.051 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:53:53.051 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:53:53.051 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:53:53.051 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:53:53.051 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000088", 2024-08-18T15:53:53.051 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:53:53.051 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000080", 2024-08-18T15:53:53.051 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:53:53.052 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:53:53.052 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:53:53.052 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000090", 2024-08-18T15:53:53.052 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:53:53.052 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:53:53.052 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:53:53.052 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000110", 2024-08-18T15:53:53.052 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:53:53.052 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:53:53.053 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:53:53.053 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:53:53.053 INFO:teuthology.orchestra.run.smithi134.stdout: "sync.error-log.0", 2024-08-18T15:53:53.053 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:53:53.053 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:53:53.053 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:53:53.053 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000096", 2024-08-18T15:53:53.053 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:53:53.053 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:53:53.053 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:53:53.053 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:53:53.053 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:53:53.053 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000087", 2024-08-18T15:53:53.054 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:53:53.054 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:53:53.054 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:53:53.054 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000008", 2024-08-18T15:53:53.054 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:53:53.054 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:53:53.054 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:53:53.054 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:53:53.054 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:53:53.054 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:53:53.054 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:53:53.054 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:53:53.054 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:53:53.055 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000006", 2024-08-18T15:53:53.055 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000029", 2024-08-18T15:53:53.055 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:53:53.055 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:53:53.055 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:53:53.055 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:53:53.055 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000089", 2024-08-18T15:53:53.055 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.full-sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:53:53.055 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:53:53.055 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:53:53.055 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000022", 2024-08-18T15:53:53.055 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:53:53.055 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:53:53.056 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:53:53.056 INFO:teuthology.orchestra.run.smithi134.stdout: "data_log.1.ECB4DUXwata.0", 2024-08-18T15:53:53.056 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:53:53.056 INFO:teuthology.orchestra.run.smithi134.stdout: "bilog.trim", 2024-08-18T15:53:53.056 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000016", 2024-08-18T15:53:53.056 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:53:53.056 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:53:53.056 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:53:53.056 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:53:53.056 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:53:53.056 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:53:53.056 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:53:53.056 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000041", 2024-08-18T15:53:53.057 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:53:53.057 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:53:53.057 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:53:53.057 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:53:53.057 INFO:teuthology.orchestra.run.smithi134.stdout: "meta.log.01050e30-179b-43be-bdbb-3bb23cf37ae8.3", 2024-08-18T15:53:53.057 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:53:53.057 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000018", 2024-08-18T15:53:53.057 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000092", 2024-08-18T15:53:53.057 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.full-sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:53:53.057 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.full-sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:53:53.057 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:53:53.057 INFO:teuthology.orchestra.run.smithi134.stdout: "data_log.3.bb3HL9RjdJb.0", 2024-08-18T15:53:53.057 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:53:53.058 INFO:teuthology.orchestra.run.smithi134.stdout: "sync.error-log.27", 2024-08-18T15:53:53.058 INFO:teuthology.orchestra.run.smithi134.stdout: "sync.error-log.4", 2024-08-18T15:53:53.058 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:53:53.058 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:53:53.058 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:53:53.058 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:53:53.058 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:53:53.058 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000014", 2024-08-18T15:53:53.058 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:53:53.058 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:53:53.058 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:53:53.058 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:53:53.059 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000112", 2024-08-18T15:53:53.059 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:53:53.059 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:53:53.059 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000007", 2024-08-18T15:53:53.059 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000021", 2024-08-18T15:53:53.059 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:53:53.059 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:53:53.059 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:53:53.059 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:53:53.059 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:53:53.059 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:53:53.059 INFO:teuthology.orchestra.run.smithi134.stdout: "sync.error-log.16", 2024-08-18T15:53:53.060 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:53:53.060 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000064", 2024-08-18T15:53:53.060 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:53:53.060 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:53:53.060 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:53:53.060 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:53:53.060 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000071", 2024-08-18T15:53:53.060 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:53:53.060 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:53:53.060 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:53:53.060 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000074", 2024-08-18T15:53:53.061 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:53:53.061 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:53:53.061 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:53:53.061 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:53:53.061 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:53:53.061 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:53:53.061 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:53:53.061 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:53:53.061 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:53:53.061 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:53:53.061 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:53:53.061 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:53:53.061 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:53:53.062 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:53:53.062 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:53:53.062 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000081", 2024-08-18T15:53:53.062 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:53:53.062 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000009", 2024-08-18T15:53:53.062 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000121", 2024-08-18T15:53:53.062 INFO:teuthology.orchestra.run.smithi134.stdout: "data_log.1", 2024-08-18T15:53:53.062 INFO:teuthology.orchestra.run.smithi134.stdout: "data_log.0.rIgTkbumpbF.0", 2024-08-18T15:53:53.062 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.full-sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:53:53.062 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:53:53.062 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:53:53.062 INFO:teuthology.orchestra.run.smithi134.stdout: "sync.error-log.17", 2024-08-18T15:53:53.062 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000125", 2024-08-18T15:53:53.063 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000082", 2024-08-18T15:53:53.063 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:53:53.063 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.full-sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:53:53.063 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:53:53.063 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:53:53.063 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:53:53.063 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:53:53.063 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:53:53.063 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000105", 2024-08-18T15:53:53.063 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000059", 2024-08-18T15:53:53.063 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:53:53.063 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:53:53.063 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:53:53.064 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:53:53.064 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:53:53.064 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:53:53.064 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:53:53.064 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000077", 2024-08-18T15:53:53.064 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000032", 2024-08-18T15:53:53.064 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:53:53.064 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:53:53.064 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:53:53.064 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:53:53.065 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:53:53.065 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:53:53.065 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:53:53.065 INFO:teuthology.orchestra.run.smithi134.stdout: "sync.error-log.24", 2024-08-18T15:53:53.065 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:53:53.065 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:53:53.065 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:53:53.065 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:53:53.065 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:53:53.065 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:53:53.065 INFO:teuthology.orchestra.run.smithi134.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3.retry", 2024-08-18T15:53:53.065 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:53:53.066 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:53:53.066 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:53:53.066 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:53:53.066 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000053", 2024-08-18T15:53:53.066 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:53:53.066 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000091", 2024-08-18T15:53:53.066 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:53:53.066 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:53:53.066 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:53:53.066 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:53:53.066 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000065", 2024-08-18T15:53:53.066 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:53:53.067 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:53:53.067 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000083", 2024-08-18T15:53:53.067 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000010", 2024-08-18T15:53:53.067 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:53:53.067 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:53:53.067 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:53:53.067 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:53:53.067 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:53:53.067 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:53:53.067 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:53:53.067 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:53:53.067 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:53:53.067 INFO:teuthology.orchestra.run.smithi134.stdout: "sync.error-log.7", 2024-08-18T15:53:53.068 INFO:teuthology.orchestra.run.smithi134.stdout: "sync.error-log.3", 2024-08-18T15:53:53.068 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:53:53.068 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:53:53.068 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:53:53.068 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:53:53.068 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:53:53.068 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:53:53.068 INFO:teuthology.orchestra.run.smithi134.stdout: "mdlog.sync-status.shard.2", 2024-08-18T15:53:53.068 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:53:53.068 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:53:53.068 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:53:53.068 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:53:53.068 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:53:53.069 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:53:53.069 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000045", 2024-08-18T15:53:53.069 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:53:53.069 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:53:53.069 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:53:53.069 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000002", 2024-08-18T15:53:53.069 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:53:53.069 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:53:53.069 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:53:53.070 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:53:53.070 INFO:teuthology.orchestra.run.smithi134.stdout: "sync.error-log.12", 2024-08-18T15:53:53.070 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:53:53.070 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:53:53.070 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:53:53.070 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:53:53.070 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:53:53.070 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:53:53.070 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000116", 2024-08-18T15:53:53.071 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:53:53.071 INFO:teuthology.orchestra.run.smithi134.stdout: "sync.error-log.25", 2024-08-18T15:53:53.071 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.full-sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:53:53.071 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:53:53.071 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:53:53.071 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:53:53.071 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:53:53.071 INFO:teuthology.orchestra.run.smithi134.stdout: "datalog.sync-status.shard.10090ccf-aad6-416d-9775-df8b7b27345a.0", 2024-08-18T15:53:53.071 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:53:53.071 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.full-sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:53:53.071 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000034", 2024-08-18T15:53:53.072 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:53:53.072 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:53:53.072 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:53:53.072 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:53:53.072 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:53:53.072 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000101", 2024-08-18T15:53:53.072 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:53:53.072 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:53:53.072 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:53:53.072 INFO:teuthology.orchestra.run.smithi134.stdout: "sync.error-log.13", 2024-08-18T15:53:53.072 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.full-sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:53:53.072 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:53:53.073 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:53:53.073 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:53:53.073 INFO:teuthology.orchestra.run.smithi134.stdout: "mdlog.sync-status.shard.1", 2024-08-18T15:53:53.073 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:53:53.073 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000079", 2024-08-18T15:53:53.073 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000049", 2024-08-18T15:53:53.073 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:53:53.073 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:53:53.073 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000117", 2024-08-18T15:53:53.073 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000044", 2024-08-18T15:53:53.073 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:53:53.074 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:53:53.074 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.full-sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:53:53.074 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:53:53.074 INFO:teuthology.orchestra.run.smithi134.stdout: "sync.error-log.14", 2024-08-18T15:53:53.074 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:53:53.074 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000066", 2024-08-18T15:53:53.074 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:53:53.074 INFO:teuthology.orchestra.run.smithi134.stdout: "data_log.2.ykAHQYS7gOC.0", 2024-08-18T15:53:53.074 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:53:53.074 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:53:53.074 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:53:53.074 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000068", 2024-08-18T15:53:53.075 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000085", 2024-08-18T15:53:53.075 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:53:53.075 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:53:53.075 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:53:53.075 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:53:53.075 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:53:53.075 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:53:53.075 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:53:53.075 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:53:53.075 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:53:53.075 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:53:53.075 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:53:53.075 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:53:53.076 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000073", 2024-08-18T15:53:53.076 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:53:53.076 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:53:53.076 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000038", 2024-08-18T15:53:53.076 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:53:53.076 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:53:53.076 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:53:53.076 INFO:teuthology.orchestra.run.smithi134.stdout: "sync.error-log.10", 2024-08-18T15:53:53.076 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:53:53.076 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:53:53.076 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000118", 2024-08-18T15:53:53.076 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:53:53.076 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:53:53.076 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:53:53.077 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:53:53.077 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:53:53.077 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000036", 2024-08-18T15:53:53.077 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:53:53.077 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:53:53.077 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000103", 2024-08-18T15:53:53.077 INFO:teuthology.orchestra.run.smithi134.stdout: "mdlog.sync-status.shard.3", 2024-08-18T15:53:53.077 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:53:53.077 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:53:53.077 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000119", 2024-08-18T15:53:53.077 INFO:teuthology.orchestra.run.smithi134.stdout: "datalog.sync-status.shard.10090ccf-aad6-416d-9775-df8b7b27345a.3.retry", 2024-08-18T15:53:53.077 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:53:53.077 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000098", 2024-08-18T15:53:53.077 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:53:53.078 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:53:53.078 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:53:53.078 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:53:53.078 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:53:53.078 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:53:53.078 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:53:53.078 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:53:53.078 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:53:53.078 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.full-sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:53:53.078 INFO:teuthology.orchestra.run.smithi134.stdout: "mdlog.sync-status.shard.0", 2024-08-18T15:53:53.078 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:53:53.078 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:53:53.079 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:53:53.079 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:53:53.079 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000027", 2024-08-18T15:53:53.079 INFO:teuthology.orchestra.run.smithi134.stdout: "mdlog.sync-status", 2024-08-18T15:53:53.079 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:53:53.079 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000019", 2024-08-18T15:53:53.079 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:53:53.079 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:53:53.079 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:53:53.079 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:53:53.079 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:53:53.079 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:53:53.079 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:53:53.079 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:53:53.079 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000039", 2024-08-18T15:53:53.080 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:53:53.080 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:53:53.080 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:53:53.080 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:53:53.080 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:53:53.080 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.full-sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:53:53.080 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:53:53.080 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:53:53.080 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:53:53.080 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:53:53.080 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:53:53.080 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:53:53.080 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:53:53.080 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:53:53.081 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:53:53.081 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:53:53.081 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.full-sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:53:53.081 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000100", 2024-08-18T15:53:53.081 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.full-sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:53:53.081 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:53:53.081 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:53:53.081 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:53:53.081 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:53:53.081 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000093", 2024-08-18T15:53:53.081 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:53:53.081 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:53:53.081 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:53:53.082 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000004", 2024-08-18T15:53:53.082 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000063", 2024-08-18T15:53:53.082 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000122", 2024-08-18T15:53:53.082 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:53:53.082 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000057", 2024-08-18T15:53:53.082 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:53:53.082 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:53:53.082 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000054", 2024-08-18T15:53:53.082 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000114", 2024-08-18T15:53:53.082 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:53:53.082 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:53:53.082 INFO:teuthology.orchestra.run.smithi134.stdout: "sync.error-log.2", 2024-08-18T15:53:53.082 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:53:53.082 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:53:53.083 INFO:teuthology.orchestra.run.smithi134.stdout: "data_log.0", 2024-08-18T15:53:53.083 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:53:53.083 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:53:53.083 INFO:teuthology.orchestra.run.smithi134.stdout: "datalog.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a", 2024-08-18T15:53:53.083 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:53:53.083 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:53:53.083 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:53:53.083 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000012", 2024-08-18T15:53:53.083 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:53:53.083 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:53:53.083 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:53:53.083 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:53:53.083 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:53:53.084 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:53:53.084 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:53:53.084 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:53:53.084 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:53:53.084 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000084", 2024-08-18T15:53:53.084 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000043", 2024-08-18T15:53:53.084 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:53:53.084 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:53:53.084 INFO:teuthology.orchestra.run.smithi134.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3", 2024-08-18T15:53:53.084 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:53:53.084 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:53:53.084 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:53:53.084 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:53:53.085 INFO:teuthology.orchestra.run.smithi134.stdout: "sync.error-log.11", 2024-08-18T15:53:53.085 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:53:53.085 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:53:53.085 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:53:53.085 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:53:53.085 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:53:53.085 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:53:53.085 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:53:53.085 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.full-sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:53:53.085 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:53:53.085 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000111", 2024-08-18T15:53:53.085 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:53:53.085 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000048", 2024-08-18T15:53:53.085 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000020", 2024-08-18T15:53:53.086 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:53:53.086 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:53:53.086 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:53:53.086 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:53:53.086 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:53:53.086 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000099", 2024-08-18T15:53:53.086 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000056", 2024-08-18T15:53:53.086 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:53:53.086 INFO:teuthology.orchestra.run.smithi134.stdout: "sync.error-log.19", 2024-08-18T15:53:53.087 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:53:53.087 INFO:teuthology.orchestra.run.smithi134.stdout: "sync.error-log.28", 2024-08-18T15:53:53.087 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:53:53.087 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:53:53.087 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:53:53.087 INFO:teuthology.orchestra.run.smithi134.stdout: "datalog.sync-status.shard.10090ccf-aad6-416d-9775-df8b7b27345a.1.retry", 2024-08-18T15:53:53.087 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:53:53.087 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:53:53.087 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000072", 2024-08-18T15:53:53.087 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:53:53.087 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:53:53.087 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000062", 2024-08-18T15:53:53.088 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:53:53.088 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:53:53.088 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:53:53.088 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000115", 2024-08-18T15:53:53.088 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:53:53.088 INFO:teuthology.orchestra.run.smithi134.stdout: "sync.error-log.1", 2024-08-18T15:53:53.088 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000033", 2024-08-18T15:53:53.088 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:53:53.088 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:53:53.088 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:53:53.088 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:53:53.088 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:53:53.089 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:53:53.089 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:53:53.089 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:53:53.089 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:53:53.089 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:53:53.089 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:53:53.089 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:53:53.089 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000025", 2024-08-18T15:53:53.089 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.full-sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:53:53.089 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:53:53.089 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000123", 2024-08-18T15:53:53.089 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:53:53.089 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:53:53.090 INFO:teuthology.orchestra.run.smithi134.stdout: "sync.error-log.31", 2024-08-18T15:53:53.090 INFO:teuthology.orchestra.run.smithi134.stdout: "sync.error-log.18", 2024-08-18T15:53:53.090 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:53:53.090 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:53:53.090 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.full-sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:53:53.090 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:53:53.090 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:53:53.090 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.full-sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:53:53.090 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:53:53.090 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000108", 2024-08-18T15:53:53.090 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:53:53.090 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000094", 2024-08-18T15:53:53.091 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:53:53.091 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:53:53.091 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:53:53.091 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:53:53.091 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:53:53.091 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000060", 2024-08-18T15:53:53.091 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:53:53.091 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:53:53.091 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:53:53.091 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:53:53.091 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000109", 2024-08-18T15:53:53.091 INFO:teuthology.orchestra.run.smithi134.stdout: "sync.error-log.15", 2024-08-18T15:53:53.091 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.full-sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:53:53.092 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:53:53.092 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:53:53.092 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:53:53.092 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:53:53.092 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000013", 2024-08-18T15:53:53.092 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:53:53.092 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:53:53.092 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:53:53.092 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:53:53.092 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:53:53.092 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:53:53.092 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:53:53.093 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:53:53.093 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:53:53.093 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:53:53.093 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:53:53.093 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:53:53.093 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:53:53.093 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:53:53.093 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:53:53.093 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:53:53.093 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.full-sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:53:53.093 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000051", 2024-08-18T15:53:53.093 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:53:53.093 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:53:53.094 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:53:53.094 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:53:53.094 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000106", 2024-08-18T15:53:53.094 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:53:53.094 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:53:53.094 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:53:53.094 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:53:53.094 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:53:53.094 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000017", 2024-08-18T15:53:53.094 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:53:53.094 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:53:53.094 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:53:53.095 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:53:53.095 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000023", 2024-08-18T15:53:53.095 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:53:53.095 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:53:53.095 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:53:53.095 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:53:53.095 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000107", 2024-08-18T15:53:53.095 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:53:53.095 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:53:53.095 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:53:53.095 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:53:53.095 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:53:53.095 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000046", 2024-08-18T15:53:53.096 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:53:53.096 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:53:53.096 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.full-sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:53:53.096 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:53:53.096 INFO:teuthology.orchestra.run.smithi134.stdout: "sync.error-log.26", 2024-08-18T15:53:53.096 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9", 2024-08-18T15:53:53.096 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:53:53.097 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:53:53.097 INFO:teuthology.orchestra.run.smithi134.stdout: "datalog.sync-status.shard.10090ccf-aad6-416d-9775-df8b7b27345a.2.retry", 2024-08-18T15:53:53.097 INFO:teuthology.orchestra.run.smithi134.stdout: "sync.error-log.22", 2024-08-18T15:53:53.097 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:53:53.097 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:53:53.097 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:53:53.097 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:53:53.097 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:53:53.097 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:53:53.097 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:53:53.097 INFO:teuthology.orchestra.run.smithi134.stdout: "data_log.2", 2024-08-18T15:53:53.097 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.full-sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:53:53.097 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:53:53.098 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:53:53.098 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:53:53.098 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.full-sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:53:53.098 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000067", 2024-08-18T15:53:53.098 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:53:53.098 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:53:53.098 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:53:53.098 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:53:53.098 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:53:53.098 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:53:53.098 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:53:53.099 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:53:53.099 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:53:53.099 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:53:53.099 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:53:53.099 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:53:53.099 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:53:53.099 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:53:53.099 INFO:teuthology.orchestra.run.smithi134.stdout: "sync.error-log.9", 2024-08-18T15:53:53.099 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000086", 2024-08-18T15:53:53.099 INFO:teuthology.orchestra.run.smithi134.stdout: "sync.error-log.5", 2024-08-18T15:53:53.099 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:53:53.099 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:53:53.100 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:53:53.100 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:53:53.100 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:53:53.100 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:53:53.100 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:53:53.100 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:53:53.100 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:53:53.100 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:53:53.100 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:53:53.100 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:53:53.100 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:53:53.100 INFO:teuthology.orchestra.run.smithi134.stdout: "sync.error-log.20", 2024-08-18T15:53:53.100 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:53:53.101 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:53:53.101 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:53:53.101 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:53:53.101 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:53:53.101 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:53:53.101 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:53:53.101 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:53:53.101 INFO:teuthology.orchestra.run.smithi134.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1.retry", 2024-08-18T15:53:53.101 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:53:53.101 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:53:53.101 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:53:53.101 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:53:53.101 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:53:53.102 INFO:teuthology.orchestra.run.smithi134.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0.retry", 2024-08-18T15:53:53.102 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000050", 2024-08-18T15:53:53.102 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000102", 2024-08-18T15:53:53.102 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:53:53.103 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:53:53.103 INFO:teuthology.orchestra.run.smithi134.stdout: "datalog.sync-status.shard.10090ccf-aad6-416d-9775-df8b7b27345a.2", 2024-08-18T15:53:53.103 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:53:53.103 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:53:53.103 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:53:53.103 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:53:53.103 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000124", 2024-08-18T15:53:53.103 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:53:53.103 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:53:53.103 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:53:53.103 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:53:53.103 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000001", 2024-08-18T15:53:53.104 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:53:53.104 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:53:53.104 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:53:53.104 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:53:53.104 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:53:53.104 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:53:53.104 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000126", 2024-08-18T15:53:53.104 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:53:53.104 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:53:53.104 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000055", 2024-08-18T15:53:53.104 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:53:53.104 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.full-sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:53:53.104 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000061", 2024-08-18T15:53:53.105 INFO:teuthology.orchestra.run.smithi134.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1", 2024-08-18T15:53:53.105 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:53:53.105 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:53:53.105 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:53:53.105 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:53:53.105 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000097", 2024-08-18T15:53:53.105 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:53:53.105 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:53:53.105 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:53:53.105 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:53:53.105 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:53:53.105 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:53:53.105 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:53:53.106 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:53:53.106 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:53:53.106 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:53:53.106 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000042", 2024-08-18T15:53:53.106 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.full-sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:53:53.106 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:53:53.106 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:53:53.106 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:53:53.106 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:53:53.106 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:53:53.106 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:53:53.106 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:53:53.106 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:53:53.107 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000075", 2024-08-18T15:53:53.107 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:53:53.107 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000120", 2024-08-18T15:53:53.107 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:53:53.107 INFO:teuthology.orchestra.run.smithi134.stdout: "obj_delete_at_hint.0000000030", 2024-08-18T15:53:53.107 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:53:53.107 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:53:53.107 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:53:53.107 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:53:53.107 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:53:53.107 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9", 2024-08-18T15:53:53.107 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:53:53.107 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket.sync-status.10090ccf-aad6-416d-9775-df8b7b27345a:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5" 2024-08-18T15:53:53.108 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T15:53:53.108 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-08-18T15:53:53.109 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:53:53.109 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:53:53.109 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:53:53.158 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:53:53.158 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:53:53.220 INFO:teuthology.orchestra.run.smithi193.stdout:[ 2024-08-18T15:53:53.220 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:53:53.220 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:53:53.220 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.29", 2024-08-18T15:53:53.220 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000078", 2024-08-18T15:53:53.220 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.history", 2024-08-18T15:53:53.220 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:53:53.220 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:53:53.220 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:53:53.221 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:53:53.221 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.23", 2024-08-18T15:53:53.221 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000070", 2024-08-18T15:53:53.221 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000104", 2024-08-18T15:53:53.221 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:53:53.221 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2.retry", 2024-08-18T15:53:53.221 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:53:53.221 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:53:53.221 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:53:53.221 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:53:53.221 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000026", 2024-08-18T15:53:53.221 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:53:53.221 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:53:53.222 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:53:53.222 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:53:53.222 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:53:53.222 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:53:53.222 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.8", 2024-08-18T15:53:53.222 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:53:53.222 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:53:53.222 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:53:53.222 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:53:53.222 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000028", 2024-08-18T15:53:53.222 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:53:53.222 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:53:53.223 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:53:53.223 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:53:53.223 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:53:53.223 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:53:53.223 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:53:53.223 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2.retry", 2024-08-18T15:53:53.223 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:53:53.223 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.30", 2024-08-18T15:53:53.223 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:53:53.223 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:53:53.223 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:53:53.223 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:53:53.224 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000040", 2024-08-18T15:53:53.224 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:53:53.224 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.6", 2024-08-18T15:53:53.224 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:53:53.224 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:53:53.224 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:53:53.224 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:53:53.224 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:53:53.224 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:53:53.224 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000015", 2024-08-18T15:53:53.224 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:53:53.224 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:53:53.225 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000069", 2024-08-18T15:53:53.225 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:53:53.225 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:53:53.225 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:53:53.225 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000095", 2024-08-18T15:53:53.225 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0", 2024-08-18T15:53:53.225 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000003", 2024-08-18T15:53:53.225 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000047", 2024-08-18T15:53:53.225 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:53:53.225 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:53:53.225 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:53:53.225 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:53:53.225 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:53:53.226 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:53:53.226 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:53:53.226 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:53:53.226 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:53:53.226 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000035", 2024-08-18T15:53:53.226 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:53:53.226 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:53:53.226 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000037", 2024-08-18T15:53:53.226 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:53:53.226 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:53:53.226 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:53:53.226 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:53:53.227 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:53:53.227 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:53:53.227 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:53:53.227 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:53:53.227 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:53:53.227 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:53:53.227 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:53:53.227 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:53:53.227 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:53:53.227 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:53:53.227 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000024", 2024-08-18T15:53:53.227 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:53:53.227 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:53:53.228 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:53:53.228 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:53:53.228 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000000", 2024-08-18T15:53:53.228 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:53:53.228 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000031", 2024-08-18T15:53:53.228 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:53:53.228 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:53:53.228 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:53:53.228 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:53:53.228 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:53:53.228 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:53:53.228 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3", 2024-08-18T15:53:53.228 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:53:53.229 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:53:53.229 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:53:53.229 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:53:53.229 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:53:53.229 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:53:53.229 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:53:53.229 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:53:53.229 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:53:53.229 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:53:53.229 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:53:53.229 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:53:53.229 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:53:53.229 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:53:53.229 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:53:53.230 INFO:teuthology.orchestra.run.smithi193.stdout: "data_loggenerations_metadata", 2024-08-18T15:53:53.230 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2", 2024-08-18T15:53:53.230 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:53:53.230 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:53:53.230 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:53:53.230 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:53:53.230 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T15:53:53.230 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:53:53.230 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:53:53.230 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:53:53.230 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:53:53.230 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:53:53.230 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000076", 2024-08-18T15:53:53.231 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:53:53.231 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:53:53.231 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000113", 2024-08-18T15:53:53.231 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000005", 2024-08-18T15:53:53.231 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:53:53.231 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000011", 2024-08-18T15:53:53.231 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:53:53.231 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000058", 2024-08-18T15:53:53.231 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:53:53.231 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:53:53.231 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:53:53.231 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000052", 2024-08-18T15:53:53.231 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:53:53.231 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:53:53.232 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:53:53.232 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:53:53.232 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.21", 2024-08-18T15:53:53.232 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:53:53.232 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:53:53.232 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:53:53.232 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:53:53.232 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:53:53.232 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:53:53.232 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:53:53.232 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:53:53.232 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:53:53.233 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:53:53.233 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:53:53.233 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:53:53.233 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:53:53.233 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:53:53.233 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:53:53.233 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:53:53.233 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:53:53.233 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:53:53.233 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:53:53.233 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000088", 2024-08-18T15:53:53.233 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000080", 2024-08-18T15:53:53.233 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:53:53.233 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:53:53.234 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000090", 2024-08-18T15:53:53.234 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:53:53.234 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:53:53.234 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000110", 2024-08-18T15:53:53.234 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:53:53.234 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:53:53.234 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:53:53.234 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.0", 2024-08-18T15:53:53.234 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:53:53.234 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000096", 2024-08-18T15:53:53.234 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:53:53.234 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:53:53.234 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:53:53.235 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:53:53.235 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:53:53.235 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000087", 2024-08-18T15:53:53.235 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:53:53.235 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:53:53.235 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:53:53.235 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:53:53.235 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000008", 2024-08-18T15:53:53.235 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:53:53.235 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:53:53.235 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:53:53.235 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:53:53.235 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:53:53.236 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000006", 2024-08-18T15:53:53.236 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000029", 2024-08-18T15:53:53.236 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:53:53.236 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000089", 2024-08-18T15:53:53.236 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:53:53.236 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:53:53.236 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000022", 2024-08-18T15:53:53.236 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:53:53.236 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:53:53.236 INFO:teuthology.orchestra.run.smithi193.stdout: "bilog.trim", 2024-08-18T15:53:53.236 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000016", 2024-08-18T15:53:53.236 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:53:53.236 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:53:53.236 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:53:53.237 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:53:53.237 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:53:53.237 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000041", 2024-08-18T15:53:53.237 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:53:53.237 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:53:53.237 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.log.01050e30-179b-43be-bdbb-3bb23cf37ae8.3", 2024-08-18T15:53:53.237 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:53:53.237 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000018", 2024-08-18T15:53:53.237 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000092", 2024-08-18T15:53:53.237 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:53:53.237 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:53:53.237 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:53:53.237 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:53:53.238 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.27", 2024-08-18T15:53:53.238 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.4", 2024-08-18T15:53:53.238 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:53:53.238 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:53:53.238 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:53:53.238 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:53:53.238 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:53:53.238 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:53:53.238 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:53:53.238 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000014", 2024-08-18T15:53:53.238 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:53:53.238 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3.retry", 2024-08-18T15:53:53.238 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:53:53.238 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:53:53.239 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:53:53.239 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:53:53.239 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:53:53.239 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000112", 2024-08-18T15:53:53.239 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:53:53.239 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000007", 2024-08-18T15:53:53.239 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000021", 2024-08-18T15:53:53.239 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:53:53.239 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:53:53.239 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:53:53.239 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:53:53.239 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:53:53.240 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.16", 2024-08-18T15:53:53.240 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:53:53.240 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000064", 2024-08-18T15:53:53.240 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:53:53.240 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:53:53.240 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:53:53.240 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:53:53.240 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:53:53.240 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:53:53.240 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:53:53.240 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000071", 2024-08-18T15:53:53.240 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:53:53.240 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:53:53.240 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:53:53.241 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000074", 2024-08-18T15:53:53.241 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:53:53.241 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:53:53.241 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:53:53.241 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:53:53.241 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:53:53.241 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:53:53.241 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:53:53.241 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:53:53.241 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:53:53.241 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:53:53.241 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:53:53.241 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:53:53.241 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:53:53.242 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:53:53.242 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:53:53.242 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000081", 2024-08-18T15:53:53.242 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:53:53.242 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000009", 2024-08-18T15:53:53.242 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000121", 2024-08-18T15:53:53.242 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1", 2024-08-18T15:53:53.242 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:53:53.242 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:53:53.242 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:53:53.242 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:53:53.242 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:53:53.243 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:53:53.243 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.17", 2024-08-18T15:53:53.243 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000125", 2024-08-18T15:53:53.243 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:53:53.243 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000082", 2024-08-18T15:53:53.243 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:53:53.243 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:53:53.243 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:53:53.243 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:53:53.243 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:53:53.243 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:53:53.243 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:53:53.243 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:53:53.244 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:53:53.244 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:53:53.244 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:53:53.244 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000105", 2024-08-18T15:53:53.244 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000059", 2024-08-18T15:53:53.244 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:53:53.244 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:53:53.244 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:53:53.244 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:53:53.244 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:53:53.244 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:53:53.244 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:53:53.244 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:53:53.245 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:53:53.245 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:53:53.245 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:53:53.245 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000077", 2024-08-18T15:53:53.245 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000032", 2024-08-18T15:53:53.245 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:53:53.245 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:53:53.245 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:53:53.245 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:53:53.245 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:53:53.245 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:53:53.245 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.24", 2024-08-18T15:53:53.245 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:53:53.246 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:53:53.246 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:53:53.246 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:53:53.246 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3.retry", 2024-08-18T15:53:53.246 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:53:53.246 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:53:53.246 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:53:53.246 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:53:53.246 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:53:53.246 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:53:53.246 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000053", 2024-08-18T15:53:53.246 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000091", 2024-08-18T15:53:53.247 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:53:53.247 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:53:53.247 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:53:53.247 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000065", 2024-08-18T15:53:53.247 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:53:53.247 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:53:53.247 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:53:53.247 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:53:53.247 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000083", 2024-08-18T15:53:53.247 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000010", 2024-08-18T15:53:53.247 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:53:53.247 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:53:53.247 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:53:53.247 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:53:53.248 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:53:53.248 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:53:53.248 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:53:53.248 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:53:53.248 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:53:53.248 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:53:53.248 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:53:53.248 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.7", 2024-08-18T15:53:53.248 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.3", 2024-08-18T15:53:53.248 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:53:53.248 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:53:53.248 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:53:53.248 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:53:53.249 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:53:53.249 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:53:53.249 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:53:53.249 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:53:53.249 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.2", 2024-08-18T15:53:53.249 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:53:53.249 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:53:53.249 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:53:53.249 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:53:53.249 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:53:53.249 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:53:53.249 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000045", 2024-08-18T15:53:53.249 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000002", 2024-08-18T15:53:53.249 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:53:53.250 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3.2UbiLidwSTy.0", 2024-08-18T15:53:53.250 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:53:53.250 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.12", 2024-08-18T15:53:53.250 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:53:53.250 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:53:53.250 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:53:53.250 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:53:53.250 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:53:53.250 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:53:53.250 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000116", 2024-08-18T15:53:53.250 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.25", 2024-08-18T15:53:53.250 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:53:53.251 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:53:53.251 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:53:53.251 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:53:53.251 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000034", 2024-08-18T15:53:53.251 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:53:53.251 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:53:53.251 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000101", 2024-08-18T15:53:53.251 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:53:53.251 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:53:53.251 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.13", 2024-08-18T15:53:53.251 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:53:53.251 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:53:53.251 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.1", 2024-08-18T15:53:53.252 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0.retry", 2024-08-18T15:53:53.252 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:53:53.252 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:53:53.252 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000079", 2024-08-18T15:53:53.252 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000049", 2024-08-18T15:53:53.252 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:53:53.252 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0", 2024-08-18T15:53:53.252 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:53:53.252 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:53:53.252 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000117", 2024-08-18T15:53:53.252 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:53:53.252 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000044", 2024-08-18T15:53:53.252 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:53:53.253 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:53:53.253 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:53:53.253 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:53:53.253 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:53:53.253 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:53:53.253 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.14", 2024-08-18T15:53:53.253 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:53:53.253 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000066", 2024-08-18T15:53:53.253 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:53:53.253 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:53:53.253 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:53:53.253 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:53:53.254 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:53:53.254 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000068", 2024-08-18T15:53:53.254 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:53:53.254 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000085", 2024-08-18T15:53:53.254 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:53:53.254 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:53:53.254 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:53:53.254 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:53:53.254 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:53:53.254 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:53:53.254 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:53:53.254 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:53:53.254 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:53:53.255 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:53:53.255 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:53:53.255 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:53:53.255 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000073", 2024-08-18T15:53:53.255 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:53:53.255 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:53:53.255 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:53:53.255 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000038", 2024-08-18T15:53:53.255 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:53:53.255 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:53:53.255 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:53:53.255 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:53:53.255 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.10", 2024-08-18T15:53:53.256 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:53:53.256 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:53:53.256 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:53:53.256 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000118", 2024-08-18T15:53:53.256 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:53:53.256 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3", 2024-08-18T15:53:53.256 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:53:53.256 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000036", 2024-08-18T15:53:53.256 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:53:53.256 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:53:53.256 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:53:53.256 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:53:53.256 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:53:53.257 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000103", 2024-08-18T15:53:53.257 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.3", 2024-08-18T15:53:53.257 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:53:53.257 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000119", 2024-08-18T15:53:53.257 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000098", 2024-08-18T15:53:53.257 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:53:53.257 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:53:53.257 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:53:53.257 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:53:53.257 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:53:53.257 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:53:53.257 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:53:53.257 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:53:53.257 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.0", 2024-08-18T15:53:53.258 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:53:53.258 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:53:53.258 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:53:53.258 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:53:53.258 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:53:53.258 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:53:53.258 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:53:53.258 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:53:53.258 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000027", 2024-08-18T15:53:53.258 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:53:53.258 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status", 2024-08-18T15:53:53.258 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:53:53.258 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000019", 2024-08-18T15:53:53.259 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:53:53.259 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:53:53.259 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:53:53.259 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:53:53.259 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1", 2024-08-18T15:53:53.259 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:53:53.259 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:53:53.259 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:53:53.259 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:53:53.259 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000039", 2024-08-18T15:53:53.259 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:53:53.259 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:53:53.259 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:53:53.260 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:53:53.260 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:53:53.260 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:53:53.260 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:53:53.260 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:53:53.260 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:53:53.260 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:53:53.260 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:53:53.260 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000100", 2024-08-18T15:53:53.260 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:53:53.260 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:53:53.260 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:53:53.260 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:53:53.260 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:53:53.261 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000093", 2024-08-18T15:53:53.261 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:53:53.261 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000004", 2024-08-18T15:53:53.261 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000063", 2024-08-18T15:53:53.261 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000122", 2024-08-18T15:53:53.261 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:53:53.261 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:53:53.261 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:53:53.261 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000057", 2024-08-18T15:53:53.261 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:53:53.261 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:53:53.261 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:53:53.261 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:53:53.261 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000054", 2024-08-18T15:53:53.262 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:53:53.262 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000114", 2024-08-18T15:53:53.262 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:53:53.262 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:53:53.262 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:53:53.262 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.2", 2024-08-18T15:53:53.262 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:53:53.262 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:53:53.262 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0", 2024-08-18T15:53:53.262 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:53:53.262 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:53:53.262 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:53:53.262 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:53:53.263 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:53:53.263 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:53:53.263 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000012", 2024-08-18T15:53:53.263 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:53:53.263 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:53:53.263 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:53:53.263 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000084", 2024-08-18T15:53:53.263 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000043", 2024-08-18T15:53:53.263 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3", 2024-08-18T15:53:53.263 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:53:53.263 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:53:53.263 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.11", 2024-08-18T15:53:53.263 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:53:53.264 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:53:53.264 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:53:53.264 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:53:53.264 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:53:53.264 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:53:53.264 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:53:53.264 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:53:53.264 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000111", 2024-08-18T15:53:53.264 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:53:53.264 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000048", 2024-08-18T15:53:53.264 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:53:53.264 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000020", 2024-08-18T15:53:53.264 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:53:53.265 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:53:53.265 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:53:53.265 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:53:53.265 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:53:53.265 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000099", 2024-08-18T15:53:53.265 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000056", 2024-08-18T15:53:53.265 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:53:53.265 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:53:53.265 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.19", 2024-08-18T15:53:53.265 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:53:53.265 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.28", 2024-08-18T15:53:53.265 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:53:53.265 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:53:53.265 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:53:53.266 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:53:53.266 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000072", 2024-08-18T15:53:53.266 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:53:53.266 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:53:53.266 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000062", 2024-08-18T15:53:53.266 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:53:53.266 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:53:53.266 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:53:53.266 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000115", 2024-08-18T15:53:53.266 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:53:53.266 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.1", 2024-08-18T15:53:53.266 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2", 2024-08-18T15:53:53.266 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000033", 2024-08-18T15:53:53.267 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:53:53.267 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:53:53.267 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:53:53.267 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:53:53.267 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:53:53.267 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:53:53.267 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:53:53.267 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:53:53.267 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:53:53.267 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:53:53.267 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:53:53.267 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:53:53.267 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:53:53.268 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:53:53.268 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:53:53.268 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:53:53.268 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:53:53.268 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:53:53.268 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000025", 2024-08-18T15:53:53.268 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:53:53.268 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000123", 2024-08-18T15:53:53.268 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.31", 2024-08-18T15:53:53.268 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:53:53.268 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:53:53.268 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:53:53.268 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:53:53.268 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.18", 2024-08-18T15:53:53.269 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:53:53.269 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:53:53.269 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:53:53.269 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:53:53.269 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:53:53.269 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000108", 2024-08-18T15:53:53.269 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:53:53.269 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:53:53.269 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000094", 2024-08-18T15:53:53.269 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:53:53.269 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:53:53.269 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:53:53.269 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:53:53.269 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:53:53.269 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:53:53.270 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000060", 2024-08-18T15:53:53.270 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:53:53.270 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:53:53.270 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:53:53.270 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000109", 2024-08-18T15:53:53.270 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.15", 2024-08-18T15:53:53.270 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:53:53.270 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:53:53.270 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:53:53.270 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:53:53.270 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:53:53.270 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:53:53.271 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:53:53.271 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:53:53.271 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:53:53.271 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000013", 2024-08-18T15:53:53.271 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:53:53.271 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:53:53.271 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:53:53.271 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:53:53.271 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:53:53.271 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:53:53.271 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:53:53.271 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:53:53.271 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:53:53.271 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:53:53.272 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:53:53.272 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:53:53.272 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:53:53.272 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:53:53.272 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000051", 2024-08-18T15:53:53.272 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:53:53.272 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:53:53.272 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:53:53.272 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:53:53.272 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000106", 2024-08-18T15:53:53.272 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:53:53.272 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:53:53.272 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:53:53.273 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:53:53.273 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:53:53.273 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000017", 2024-08-18T15:53:53.273 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:53:53.273 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:53:53.273 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:53:53.273 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000023", 2024-08-18T15:53:53.273 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:53:53.273 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:53:53.273 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:53:53.273 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:53:53.273 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:53:53.273 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:53:53.274 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:53:53.274 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000107", 2024-08-18T15:53:53.274 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:53:53.274 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:53:53.274 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000046", 2024-08-18T15:53:53.274 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:53:53.274 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:53:53.274 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:53:53.274 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.26", 2024-08-18T15:53:53.274 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:53:53.274 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:53:53.274 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9", 2024-08-18T15:53:53.274 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.22", 2024-08-18T15:53:53.275 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:53:53.275 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:53:53.275 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:53:53.275 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:53:53.275 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:53:53.275 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:53:53.275 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:53:53.275 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2", 2024-08-18T15:53:53.275 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:53:53.275 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:53:53.275 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000067", 2024-08-18T15:53:53.275 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:53:53.275 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:53:53.276 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:53:53.276 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:53:53.276 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:53:53.276 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:53:53.276 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:53:53.276 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:53:53.276 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:53:53.276 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:53:53.276 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:53:53.276 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:53:53.276 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:53:53.276 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:53:53.276 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:53:53.277 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:53:53.277 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:53:53.277 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:53:53.277 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:53:53.277 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.9", 2024-08-18T15:53:53.277 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000086", 2024-08-18T15:53:53.277 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.5", 2024-08-18T15:53:53.277 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:53:53.277 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:53:53.278 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:53:53.278 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:53:53.278 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:53:53.278 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:53:53.278 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2.MFXJ583gCr1.0", 2024-08-18T15:53:53.278 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:53:53.278 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1.GPnO16liyZ+.0", 2024-08-18T15:53:53.278 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:53:53.278 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:53:53.278 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:53:53.278 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:53:53.278 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:53:53.279 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:53:53.279 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:53:53.279 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:53:53.279 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:53:53.279 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.20", 2024-08-18T15:53:53.279 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:53:53.279 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:53:53.279 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:53:53.279 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:53:53.279 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:53:53.280 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:53:53.280 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1.retry", 2024-08-18T15:53:53.280 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:53:53.280 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:53:53.280 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:53:53.280 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0.retry", 2024-08-18T15:53:53.280 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000050", 2024-08-18T15:53:53.280 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:53:53.280 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000102", 2024-08-18T15:53:53.280 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:53:53.280 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:53:53.280 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:53:53.280 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:53:53.281 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:53:53.281 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:53:53.281 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:53:53.281 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:53:53.281 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:53:53.281 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:53:53.281 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:53:53.281 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000124", 2024-08-18T15:53:53.281 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:53:53.281 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:53:53.281 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:53:53.281 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000001", 2024-08-18T15:53:53.281 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:53:53.282 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:53:53.282 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:53:53.282 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:53:53.282 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000126", 2024-08-18T15:53:53.282 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:53:53.282 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0.fZhctfLXnSt.0", 2024-08-18T15:53:53.282 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:53:53.282 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000055", 2024-08-18T15:53:53.282 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:53:53.282 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000061", 2024-08-18T15:53:53.282 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1", 2024-08-18T15:53:53.282 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:53:53.282 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:53:53.283 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:53:53.283 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000097", 2024-08-18T15:53:53.283 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:53:53.283 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:53:53.283 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:53:53.283 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:53:53.283 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:53:53.283 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:53:53.283 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:53:53.283 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:53:53.283 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000042", 2024-08-18T15:53:53.283 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:53:53.283 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:53:53.283 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:53:53.284 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1.retry", 2024-08-18T15:53:53.284 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:53:53.284 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:53:53.284 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:53:53.284 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:53:53.284 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:53:53.284 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:53:53.284 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:53:53.284 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000075", 2024-08-18T15:53:53.284 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:53:53.284 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000120", 2024-08-18T15:53:53.284 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000030", 2024-08-18T15:53:53.284 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:53:53.285 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:53:53.285 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:53:53.285 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:53:53.285 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:53:53.285 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:53:53.285 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9" 2024-08-18T15:53:53.285 INFO:teuthology.orchestra.run.smithi193.stdout:] 2024-08-18T15:53:53.285 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:53:58.287 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:53:58.288 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:53:58.288 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:53:58.340 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:53:58.340 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:53:58.404 INFO:teuthology.orchestra.run.smithi193.stdout:[ 2024-08-18T15:53:58.404 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:53:58.404 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:53:58.404 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.29", 2024-08-18T15:53:58.404 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000078", 2024-08-18T15:53:58.404 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.history", 2024-08-18T15:53:58.404 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:53:58.404 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:53:58.405 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:53:58.405 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:53:58.405 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.23", 2024-08-18T15:53:58.405 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000070", 2024-08-18T15:53:58.405 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000104", 2024-08-18T15:53:58.405 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:53:58.405 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2.retry", 2024-08-18T15:53:58.405 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:53:58.405 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:53:58.405 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:53:58.405 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:53:58.405 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000026", 2024-08-18T15:53:58.405 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:53:58.406 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:53:58.406 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:53:58.406 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:53:58.406 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:53:58.406 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:53:58.406 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.8", 2024-08-18T15:53:58.406 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:53:58.406 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:53:58.406 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:53:58.406 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:53:58.406 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000028", 2024-08-18T15:53:58.407 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:53:58.407 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:53:58.407 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:53:58.407 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:53:58.407 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:53:58.407 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:53:58.407 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:53:58.407 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2.retry", 2024-08-18T15:53:58.407 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:53:58.407 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.30", 2024-08-18T15:53:58.407 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:53:58.407 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:53:58.408 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:53:58.408 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:53:58.408 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000040", 2024-08-18T15:53:58.408 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:53:58.408 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.6", 2024-08-18T15:53:58.408 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:53:58.408 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:53:58.408 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:53:58.409 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:53:58.409 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:53:58.409 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:53:58.409 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000015", 2024-08-18T15:53:58.409 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:53:58.409 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:53:58.409 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000069", 2024-08-18T15:53:58.409 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:53:58.409 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:53:58.409 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:53:58.409 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000095", 2024-08-18T15:53:58.409 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0", 2024-08-18T15:53:58.409 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000003", 2024-08-18T15:53:58.409 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000047", 2024-08-18T15:53:58.410 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:53:58.410 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:53:58.410 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:53:58.410 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:53:58.410 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:53:58.410 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:53:58.410 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:53:58.410 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:53:58.410 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:53:58.410 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000035", 2024-08-18T15:53:58.410 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:53:58.410 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:53:58.411 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000037", 2024-08-18T15:53:58.411 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:53:58.411 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:53:58.411 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:53:58.411 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:53:58.411 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:53:58.411 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:53:58.411 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:53:58.411 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:53:58.411 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:53:58.411 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:53:58.411 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:53:58.412 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:53:58.412 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:53:58.412 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:53:58.412 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000024", 2024-08-18T15:53:58.412 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:53:58.412 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:53:58.412 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:53:58.412 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:53:58.412 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000000", 2024-08-18T15:53:58.412 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:53:58.412 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000031", 2024-08-18T15:53:58.412 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:53:58.412 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:53:58.413 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:53:58.413 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:53:58.413 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:53:58.413 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:53:58.413 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3", 2024-08-18T15:53:58.413 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:53:58.413 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:53:58.413 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:53:58.413 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:53:58.413 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:53:58.413 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:53:58.413 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:53:58.413 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:53:58.413 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:53:58.414 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:53:58.414 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:53:58.414 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:53:58.414 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:53:58.414 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:53:58.414 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:53:58.414 INFO:teuthology.orchestra.run.smithi193.stdout: "data_loggenerations_metadata", 2024-08-18T15:53:58.414 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2", 2024-08-18T15:53:58.414 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:53:58.414 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:53:58.414 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:53:58.414 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:53:58.414 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T15:53:58.415 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:53:58.415 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:53:58.415 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:53:58.415 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:53:58.415 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:53:58.415 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000076", 2024-08-18T15:53:58.415 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:53:58.415 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:53:58.415 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000113", 2024-08-18T15:53:58.415 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000005", 2024-08-18T15:53:58.415 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:53:58.415 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000011", 2024-08-18T15:53:58.416 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:53:58.416 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000058", 2024-08-18T15:53:58.416 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:53:58.416 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:53:58.416 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:53:58.416 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000052", 2024-08-18T15:53:58.417 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:53:58.417 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:53:58.417 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:53:58.417 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:53:58.417 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.21", 2024-08-18T15:53:58.417 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:53:58.418 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:53:58.418 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:53:58.418 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:53:58.418 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:53:58.418 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:53:58.418 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:53:58.419 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:53:58.419 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:53:58.419 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:53:58.419 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:53:58.419 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:53:58.420 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:53:58.420 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:53:58.420 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:53:58.420 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:53:58.420 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:53:58.420 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:53:58.421 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:53:58.421 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000088", 2024-08-18T15:53:58.421 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000080", 2024-08-18T15:53:58.421 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:53:58.421 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:53:58.422 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000090", 2024-08-18T15:53:58.422 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:53:58.422 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:53:58.422 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000110", 2024-08-18T15:53:58.422 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:53:58.422 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:53:58.423 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:53:58.423 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.0", 2024-08-18T15:53:58.423 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:53:58.423 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000096", 2024-08-18T15:53:58.423 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:53:58.424 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:53:58.424 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:53:58.424 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:53:58.424 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:53:58.424 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000087", 2024-08-18T15:53:58.425 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:53:58.425 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:53:58.425 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:53:58.425 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:53:58.425 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000008", 2024-08-18T15:53:58.425 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:53:58.426 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:53:58.426 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:53:58.426 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:53:58.426 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:53:58.426 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000006", 2024-08-18T15:53:58.427 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000029", 2024-08-18T15:53:58.427 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:53:58.427 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000089", 2024-08-18T15:53:58.427 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:53:58.427 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:53:58.427 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000022", 2024-08-18T15:53:58.428 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:53:58.428 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:53:58.428 INFO:teuthology.orchestra.run.smithi193.stdout: "bilog.trim", 2024-08-18T15:53:58.428 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000016", 2024-08-18T15:53:58.428 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:53:58.429 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:53:58.429 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:53:58.429 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:53:58.429 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:53:58.429 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000041", 2024-08-18T15:53:58.430 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:53:58.430 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:53:58.430 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.log.01050e30-179b-43be-bdbb-3bb23cf37ae8.3", 2024-08-18T15:53:58.430 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:53:58.430 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000018", 2024-08-18T15:53:58.431 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000092", 2024-08-18T15:53:58.431 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:53:58.431 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:53:58.431 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:53:58.431 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:53:58.431 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.27", 2024-08-18T15:53:58.432 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.4", 2024-08-18T15:53:58.432 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:53:58.432 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:53:58.432 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:53:58.432 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:53:58.433 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:53:58.433 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:53:58.433 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:53:58.433 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000014", 2024-08-18T15:53:58.433 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:53:58.434 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3.retry", 2024-08-18T15:53:58.434 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:53:58.434 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:53:58.434 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:53:58.435 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:53:58.435 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:53:58.435 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000112", 2024-08-18T15:53:58.435 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:53:58.435 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000007", 2024-08-18T15:53:58.436 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000021", 2024-08-18T15:53:58.436 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:53:58.436 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:53:58.436 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:53:58.436 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:53:58.436 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:53:58.437 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.16", 2024-08-18T15:53:58.437 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:53:58.437 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000064", 2024-08-18T15:53:58.437 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:53:58.437 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:53:58.438 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:53:58.438 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:53:58.438 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:53:58.438 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:53:58.438 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:53:58.439 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000071", 2024-08-18T15:53:58.439 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:53:58.439 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:53:58.439 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:53:58.439 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000074", 2024-08-18T15:53:58.440 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:53:58.440 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:53:58.440 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:53:58.440 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:53:58.440 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:53:58.441 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:53:58.441 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:53:58.441 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:53:58.441 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:53:58.441 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:53:58.442 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:53:58.442 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:53:58.442 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:53:58.442 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:53:58.442 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:53:58.442 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000081", 2024-08-18T15:53:58.443 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:53:58.443 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000009", 2024-08-18T15:53:58.443 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000121", 2024-08-18T15:53:58.443 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1", 2024-08-18T15:53:58.443 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:53:58.443 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:53:58.443 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:53:58.443 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:53:58.444 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:53:58.444 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:53:58.444 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.17", 2024-08-18T15:53:58.444 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000125", 2024-08-18T15:53:58.444 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:53:58.445 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000082", 2024-08-18T15:53:58.445 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:53:58.445 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:53:58.445 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:53:58.445 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:53:58.445 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:53:58.445 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:53:58.445 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:53:58.445 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:53:58.446 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:53:58.446 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:53:58.446 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:53:58.446 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000105", 2024-08-18T15:53:58.446 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000059", 2024-08-18T15:53:58.446 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:53:58.446 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:53:58.446 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:53:58.446 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:53:58.446 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:53:58.446 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:53:58.446 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:53:58.447 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:53:58.447 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:53:58.447 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:53:58.447 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:53:58.447 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000077", 2024-08-18T15:53:58.447 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000032", 2024-08-18T15:53:58.447 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:53:58.447 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:53:58.447 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:53:58.447 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:53:58.447 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:53:58.448 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:53:58.448 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.24", 2024-08-18T15:53:58.448 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:53:58.448 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:53:58.448 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:53:58.448 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:53:58.448 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3.retry", 2024-08-18T15:53:58.448 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:53:58.448 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:53:58.448 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:53:58.448 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:53:58.448 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:53:58.448 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:53:58.449 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000053", 2024-08-18T15:53:58.449 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000091", 2024-08-18T15:53:58.449 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:53:58.449 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:53:58.449 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:53:58.449 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000065", 2024-08-18T15:53:58.449 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:53:58.449 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:53:58.449 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:53:58.449 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:53:58.449 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000083", 2024-08-18T15:53:58.449 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000010", 2024-08-18T15:53:58.449 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:53:58.450 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:53:58.450 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:53:58.450 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:53:58.450 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:53:58.450 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:53:58.450 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:53:58.450 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:53:58.450 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:53:58.450 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:53:58.450 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:53:58.454 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.7", 2024-08-18T15:53:58.454 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.3", 2024-08-18T15:53:58.454 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:53:58.455 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:53:58.455 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:53:58.455 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:53:58.455 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:53:58.455 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:53:58.455 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:53:58.455 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:53:58.455 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.2", 2024-08-18T15:53:58.455 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:53:58.455 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:53:58.455 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:53:58.455 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:53:58.456 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:53:58.456 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:53:58.456 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000045", 2024-08-18T15:53:58.456 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000002", 2024-08-18T15:53:58.456 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:53:58.456 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3.2UbiLidwSTy.0", 2024-08-18T15:53:58.456 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:53:58.456 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.12", 2024-08-18T15:53:58.456 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:53:58.456 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:53:58.456 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:53:58.456 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:53:58.456 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:53:58.457 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:53:58.457 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000116", 2024-08-18T15:53:58.457 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.25", 2024-08-18T15:53:58.457 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:53:58.457 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:53:58.457 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:53:58.457 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:53:58.457 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000034", 2024-08-18T15:53:58.457 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:53:58.457 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:53:58.457 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000101", 2024-08-18T15:53:58.457 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:53:58.458 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:53:58.458 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.13", 2024-08-18T15:53:58.458 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:53:58.458 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:53:58.458 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.1", 2024-08-18T15:53:58.458 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0.retry", 2024-08-18T15:53:58.458 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:53:58.458 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:53:58.458 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000079", 2024-08-18T15:53:58.458 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000049", 2024-08-18T15:53:58.458 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:53:58.459 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0", 2024-08-18T15:53:58.459 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:53:58.459 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:53:58.459 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000117", 2024-08-18T15:53:58.459 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:53:58.459 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000044", 2024-08-18T15:53:58.459 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:53:58.459 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:53:58.459 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:53:58.459 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:53:58.459 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:53:58.459 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:53:58.459 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.14", 2024-08-18T15:53:58.460 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:53:58.460 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000066", 2024-08-18T15:53:58.460 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:53:58.460 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:53:58.460 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:53:58.460 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:53:58.460 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:53:58.460 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000068", 2024-08-18T15:53:58.460 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:53:58.460 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000085", 2024-08-18T15:53:58.460 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:53:58.460 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:53:58.460 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:53:58.461 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:53:58.461 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:53:58.461 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:53:58.461 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:53:58.461 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:53:58.461 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:53:58.461 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:53:58.461 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:53:58.461 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:53:58.461 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000073", 2024-08-18T15:53:58.461 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:53:58.461 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:53:58.462 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:53:58.462 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000038", 2024-08-18T15:53:58.462 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:53:58.462 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:53:58.462 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:53:58.462 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:53:58.462 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.10", 2024-08-18T15:53:58.462 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:53:58.462 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:53:58.462 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:53:58.463 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000118", 2024-08-18T15:53:58.463 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:53:58.463 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3", 2024-08-18T15:53:58.463 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:53:58.463 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000036", 2024-08-18T15:53:58.463 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:53:58.463 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:53:58.463 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:53:58.463 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:53:58.463 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:53:58.463 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000103", 2024-08-18T15:53:58.463 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.3", 2024-08-18T15:53:58.464 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:53:58.464 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000119", 2024-08-18T15:53:58.464 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000098", 2024-08-18T15:53:58.464 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:53:58.464 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:53:58.464 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:53:58.464 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:53:58.464 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:53:58.464 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:53:58.464 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:53:58.464 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:53:58.464 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.0", 2024-08-18T15:53:58.464 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:53:58.465 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:53:58.465 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:53:58.465 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:53:58.465 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:53:58.465 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:53:58.465 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:53:58.465 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:53:58.465 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000027", 2024-08-18T15:53:58.465 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:53:58.465 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status", 2024-08-18T15:53:58.465 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:53:58.465 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000019", 2024-08-18T15:53:58.466 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:53:58.466 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:53:58.466 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:53:58.466 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:53:58.466 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1", 2024-08-18T15:53:58.466 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:53:58.466 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:53:58.466 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:53:58.466 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:53:58.466 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000039", 2024-08-18T15:53:58.466 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:53:58.466 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:53:58.467 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:53:58.467 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:53:58.467 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:53:58.467 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:53:58.467 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:53:58.467 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:53:58.467 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:53:58.467 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:53:58.467 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:53:58.467 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000100", 2024-08-18T15:53:58.467 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:53:58.467 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:53:58.468 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:53:58.468 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:53:58.468 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:53:58.468 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000093", 2024-08-18T15:53:58.468 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:53:58.468 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000004", 2024-08-18T15:53:58.468 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000063", 2024-08-18T15:53:58.468 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000122", 2024-08-18T15:53:58.468 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:53:58.468 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:53:58.468 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:53:58.468 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000057", 2024-08-18T15:53:58.468 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:53:58.469 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:53:58.469 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:53:58.469 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:53:58.469 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000054", 2024-08-18T15:53:58.469 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:53:58.469 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000114", 2024-08-18T15:53:58.469 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:53:58.469 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:53:58.469 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:53:58.469 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.2", 2024-08-18T15:53:58.469 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:53:58.470 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:53:58.470 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0", 2024-08-18T15:53:58.470 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:53:58.470 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:53:58.470 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:53:58.470 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:53:58.470 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:53:58.470 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:53:58.471 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000012", 2024-08-18T15:53:58.471 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:53:58.471 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:53:58.471 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:53:58.471 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000084", 2024-08-18T15:53:58.471 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000043", 2024-08-18T15:53:58.471 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3", 2024-08-18T15:53:58.471 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:53:58.471 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:53:58.471 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.11", 2024-08-18T15:53:58.471 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:53:58.471 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:53:58.471 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:53:58.472 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:53:58.472 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:53:58.472 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:53:58.472 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:53:58.472 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:53:58.472 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000111", 2024-08-18T15:53:58.472 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:53:58.472 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000048", 2024-08-18T15:53:58.472 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:53:58.472 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000020", 2024-08-18T15:53:58.473 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:53:58.473 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:53:58.473 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:53:58.473 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:53:58.473 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:53:58.473 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000099", 2024-08-18T15:53:58.473 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000056", 2024-08-18T15:53:58.473 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:53:58.473 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:53:58.473 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.19", 2024-08-18T15:53:58.473 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:53:58.473 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.28", 2024-08-18T15:53:58.473 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:53:58.474 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:53:58.474 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:53:58.474 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:53:58.474 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000072", 2024-08-18T15:53:58.474 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:53:58.474 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:53:58.474 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000062", 2024-08-18T15:53:58.474 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:53:58.474 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:53:58.474 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:53:58.474 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000115", 2024-08-18T15:53:58.474 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:53:58.474 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.1", 2024-08-18T15:53:58.475 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2", 2024-08-18T15:53:58.475 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000033", 2024-08-18T15:53:58.475 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:53:58.475 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:53:58.475 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:53:58.475 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:53:58.475 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:53:58.475 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:53:58.475 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:53:58.475 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:53:58.475 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:53:58.475 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:53:58.476 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:53:58.476 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:53:58.476 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:53:58.476 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:53:58.476 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:53:58.476 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:53:58.476 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:53:58.476 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:53:58.476 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000025", 2024-08-18T15:53:58.476 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:53:58.476 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000123", 2024-08-18T15:53:58.476 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.31", 2024-08-18T15:53:58.476 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:53:58.476 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:53:58.477 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:53:58.477 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:53:58.477 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.18", 2024-08-18T15:53:58.477 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:53:58.477 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:53:58.477 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:53:58.477 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:53:58.477 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:53:58.477 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000108", 2024-08-18T15:53:58.478 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:53:58.478 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:53:58.478 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000094", 2024-08-18T15:53:58.478 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:53:58.478 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:53:58.478 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:53:58.478 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:53:58.478 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:53:58.478 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:53:58.478 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000060", 2024-08-18T15:53:58.478 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:53:58.479 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:53:58.479 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:53:58.479 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000109", 2024-08-18T15:53:58.479 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.15", 2024-08-18T15:53:58.479 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:53:58.479 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:53:58.479 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:53:58.479 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:53:58.479 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:53:58.479 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:53:58.479 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:53:58.479 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:53:58.480 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:53:58.480 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000013", 2024-08-18T15:53:58.480 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:53:58.480 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:53:58.480 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:53:58.480 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:53:58.480 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:53:58.480 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:53:58.480 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:53:58.481 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:53:58.481 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:53:58.481 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:53:58.481 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:53:58.481 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:53:58.481 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:53:58.481 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:53:58.481 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000051", 2024-08-18T15:53:58.481 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:53:58.481 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:53:58.481 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:53:58.482 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:53:58.482 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000106", 2024-08-18T15:53:58.482 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:53:58.482 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:53:58.482 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:53:58.482 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:53:58.482 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:53:58.482 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000017", 2024-08-18T15:53:58.482 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:53:58.482 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:53:58.482 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:53:58.482 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000023", 2024-08-18T15:53:58.483 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:53:58.483 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:53:58.483 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:53:58.483 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:53:58.483 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:53:58.483 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:53:58.483 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:53:58.483 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000107", 2024-08-18T15:53:58.483 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:53:58.483 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:53:58.483 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000046", 2024-08-18T15:53:58.483 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:53:58.483 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:53:58.484 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:53:58.484 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.26", 2024-08-18T15:53:58.484 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:53:58.484 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:53:58.484 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9", 2024-08-18T15:53:58.484 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.22", 2024-08-18T15:53:58.484 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:53:58.484 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:53:58.484 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:53:58.484 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:53:58.484 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:53:58.484 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:53:58.484 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:53:58.485 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2", 2024-08-18T15:53:58.485 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:53:58.485 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:53:58.485 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000067", 2024-08-18T15:53:58.485 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:53:58.485 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:53:58.485 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:53:58.485 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:53:58.485 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:53:58.485 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:53:58.485 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:53:58.485 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:53:58.485 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:53:58.486 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:53:58.486 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:53:58.486 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:53:58.486 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:53:58.486 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:53:58.486 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:53:58.486 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:53:58.486 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:53:58.486 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:53:58.486 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:53:58.486 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.9", 2024-08-18T15:53:58.486 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000086", 2024-08-18T15:53:58.486 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.5", 2024-08-18T15:53:58.487 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:53:58.487 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:53:58.487 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:53:58.487 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:53:58.487 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:53:58.487 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:53:58.487 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2.MFXJ583gCr1.0", 2024-08-18T15:53:58.487 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:53:58.487 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1.GPnO16liyZ+.0", 2024-08-18T15:53:58.487 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:53:58.487 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:53:58.488 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:53:58.488 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:53:58.488 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:53:58.488 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:53:58.488 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:53:58.488 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:53:58.488 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:53:58.488 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.20", 2024-08-18T15:53:58.488 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:53:58.488 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:53:58.488 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:53:58.488 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:53:58.488 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:53:58.489 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:53:58.489 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1.retry", 2024-08-18T15:53:58.489 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:53:58.489 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:53:58.489 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:53:58.489 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0.retry", 2024-08-18T15:53:58.489 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000050", 2024-08-18T15:53:58.489 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:53:58.489 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000102", 2024-08-18T15:53:58.489 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:53:58.489 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:53:58.489 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:53:58.489 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:53:58.489 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:53:58.490 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:53:58.490 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:53:58.490 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:53:58.490 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:53:58.490 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:53:58.490 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:53:58.490 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000124", 2024-08-18T15:53:58.490 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:53:58.490 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:53:58.490 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:53:58.490 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000001", 2024-08-18T15:53:58.490 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:53:58.490 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:53:58.491 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:53:58.491 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:53:58.491 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000126", 2024-08-18T15:53:58.491 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:53:58.491 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0.fZhctfLXnSt.0", 2024-08-18T15:53:58.491 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:53:58.492 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000055", 2024-08-18T15:53:58.492 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:53:58.492 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000061", 2024-08-18T15:53:58.492 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1", 2024-08-18T15:53:58.492 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:53:58.492 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:53:58.492 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:53:58.492 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000097", 2024-08-18T15:53:58.492 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:53:58.492 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:53:58.492 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:53:58.493 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:53:58.493 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:53:58.493 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:53:58.493 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:53:58.493 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:53:58.493 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000042", 2024-08-18T15:53:58.493 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:53:58.493 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:53:58.493 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:53:58.493 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1.retry", 2024-08-18T15:53:58.493 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:53:58.494 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:53:58.494 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:53:58.494 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:53:58.494 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:53:58.494 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:53:58.494 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:53:58.494 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000075", 2024-08-18T15:53:58.494 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:53:58.495 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000120", 2024-08-18T15:53:58.495 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000030", 2024-08-18T15:53:58.495 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:53:58.495 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:53:58.495 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:53:58.495 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:53:58.495 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:53:58.495 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:53:58.495 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9" 2024-08-18T15:53:58.495 INFO:teuthology.orchestra.run.smithi193.stdout:] 2024-08-18T15:53:58.495 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:54:03.498 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:54:03.498 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:54:03.498 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:54:03.551 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:54:03.551 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:54:03.613 INFO:teuthology.orchestra.run.smithi193.stdout:[ 2024-08-18T15:54:03.614 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:54:03.614 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:54:03.614 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.29", 2024-08-18T15:54:03.614 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000078", 2024-08-18T15:54:03.614 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.history", 2024-08-18T15:54:03.614 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:54:03.614 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:54:03.614 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:54:03.614 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:54:03.614 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.23", 2024-08-18T15:54:03.614 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000070", 2024-08-18T15:54:03.614 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000104", 2024-08-18T15:54:03.615 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:54:03.615 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2.retry", 2024-08-18T15:54:03.615 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:54:03.615 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:54:03.615 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:54:03.615 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:54:03.615 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000026", 2024-08-18T15:54:03.615 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:54:03.615 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:54:03.615 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:54:03.615 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:54:03.615 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:54:03.616 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:54:03.616 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.8", 2024-08-18T15:54:03.616 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:54:03.616 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:54:03.616 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:54:03.616 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:54:03.616 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000028", 2024-08-18T15:54:03.616 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:54:03.616 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:54:03.616 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:54:03.616 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:54:03.616 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:54:03.616 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:54:03.617 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:54:03.617 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2.retry", 2024-08-18T15:54:03.617 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:54:03.617 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.30", 2024-08-18T15:54:03.617 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:54:03.617 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:54:03.617 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:54:03.617 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:54:03.617 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000040", 2024-08-18T15:54:03.617 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:54:03.617 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.6", 2024-08-18T15:54:03.617 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:54:03.618 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:54:03.618 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:54:03.618 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:54:03.618 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:54:03.618 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:54:03.618 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000015", 2024-08-18T15:54:03.618 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:54:03.618 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:54:03.618 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000069", 2024-08-18T15:54:03.618 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:54:03.618 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:54:03.618 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:54:03.618 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000095", 2024-08-18T15:54:03.619 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0", 2024-08-18T15:54:03.619 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000003", 2024-08-18T15:54:03.619 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000047", 2024-08-18T15:54:03.619 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:54:03.619 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:54:03.619 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:54:03.619 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:54:03.619 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:54:03.619 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:54:03.619 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:54:03.619 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:54:03.619 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:54:03.620 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000035", 2024-08-18T15:54:03.620 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:54:03.620 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:54:03.620 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000037", 2024-08-18T15:54:03.620 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:54:03.620 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:54:03.620 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:54:03.620 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:54:03.620 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:54:03.620 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:54:03.620 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:54:03.620 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:54:03.621 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:54:03.621 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:54:03.621 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:54:03.621 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:54:03.621 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:54:03.621 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:54:03.621 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000024", 2024-08-18T15:54:03.621 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:54:03.621 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:54:03.621 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:54:03.622 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:54:03.622 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000000", 2024-08-18T15:54:03.622 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:54:03.622 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000031", 2024-08-18T15:54:03.622 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:54:03.622 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:54:03.622 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:54:03.622 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:54:03.622 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:54:03.622 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:54:03.622 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3", 2024-08-18T15:54:03.622 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:54:03.623 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:54:03.623 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:54:03.623 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:54:03.623 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:54:03.623 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:54:03.623 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:54:03.623 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:54:03.623 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:54:03.623 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:54:03.623 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:54:03.623 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:54:03.623 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:54:03.624 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:54:03.624 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:54:03.624 INFO:teuthology.orchestra.run.smithi193.stdout: "data_loggenerations_metadata", 2024-08-18T15:54:03.624 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2", 2024-08-18T15:54:03.624 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:54:03.624 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:54:03.624 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:54:03.624 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:54:03.624 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T15:54:03.624 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:54:03.624 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:54:03.624 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:54:03.624 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:54:03.625 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:54:03.625 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000076", 2024-08-18T15:54:03.625 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:54:03.625 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:54:03.625 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000113", 2024-08-18T15:54:03.625 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000005", 2024-08-18T15:54:03.625 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:54:03.625 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000011", 2024-08-18T15:54:03.625 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:54:03.625 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000058", 2024-08-18T15:54:03.625 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:54:03.625 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:54:03.626 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:54:03.626 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000052", 2024-08-18T15:54:03.626 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:54:03.626 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:54:03.626 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:54:03.626 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:54:03.626 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.21", 2024-08-18T15:54:03.626 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:54:03.626 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:54:03.626 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:54:03.626 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:54:03.626 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:54:03.627 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:54:03.627 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:54:03.627 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:54:03.627 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:54:03.627 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:54:03.627 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:54:03.627 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:54:03.627 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:54:03.627 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:54:03.627 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:54:03.627 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:54:03.627 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:54:03.628 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:54:03.628 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:54:03.629 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000088", 2024-08-18T15:54:03.629 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000080", 2024-08-18T15:54:03.629 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:54:03.629 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:54:03.629 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000090", 2024-08-18T15:54:03.629 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:54:03.629 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:54:03.629 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000110", 2024-08-18T15:54:03.629 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:54:03.629 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:54:03.629 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:54:03.629 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.0", 2024-08-18T15:54:03.630 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:54:03.630 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000096", 2024-08-18T15:54:03.630 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:54:03.630 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:54:03.630 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:54:03.630 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:54:03.630 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:54:03.630 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000087", 2024-08-18T15:54:03.630 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:54:03.630 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:54:03.630 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:54:03.630 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:54:03.631 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000008", 2024-08-18T15:54:03.631 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:54:03.631 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:54:03.631 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:54:03.631 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:54:03.631 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:54:03.631 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000006", 2024-08-18T15:54:03.631 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000029", 2024-08-18T15:54:03.631 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:54:03.631 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000089", 2024-08-18T15:54:03.631 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:54:03.631 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:54:03.631 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000022", 2024-08-18T15:54:03.632 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:54:03.632 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:54:03.632 INFO:teuthology.orchestra.run.smithi193.stdout: "bilog.trim", 2024-08-18T15:54:03.632 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000016", 2024-08-18T15:54:03.632 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:54:03.632 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:54:03.632 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:54:03.632 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:54:03.632 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:54:03.632 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000041", 2024-08-18T15:54:03.632 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:54:03.632 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:54:03.632 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.log.01050e30-179b-43be-bdbb-3bb23cf37ae8.3", 2024-08-18T15:54:03.633 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:54:03.633 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000018", 2024-08-18T15:54:03.633 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000092", 2024-08-18T15:54:03.633 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:54:03.633 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:54:03.633 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:54:03.633 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:54:03.633 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.27", 2024-08-18T15:54:03.633 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.4", 2024-08-18T15:54:03.633 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:54:03.633 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:54:03.633 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:54:03.633 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:54:03.634 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:54:03.634 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:54:03.634 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:54:03.634 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000014", 2024-08-18T15:54:03.634 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:54:03.634 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3.retry", 2024-08-18T15:54:03.634 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:54:03.634 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:54:03.634 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:54:03.634 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:54:03.634 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:54:03.634 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000112", 2024-08-18T15:54:03.635 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:54:03.635 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000007", 2024-08-18T15:54:03.635 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000021", 2024-08-18T15:54:03.635 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:54:03.635 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:54:03.635 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:54:03.635 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:54:03.635 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:54:03.635 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.16", 2024-08-18T15:54:03.635 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:54:03.635 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000064", 2024-08-18T15:54:03.635 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:54:03.635 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:54:03.636 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:54:03.636 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:54:03.636 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:54:03.636 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:54:03.636 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:54:03.636 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000071", 2024-08-18T15:54:03.636 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:54:03.636 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:54:03.636 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:54:03.636 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000074", 2024-08-18T15:54:03.636 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:54:03.636 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:54:03.636 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:54:03.637 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:54:03.637 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:54:03.637 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:54:03.637 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:54:03.637 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:54:03.637 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:54:03.637 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:54:03.637 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:54:03.637 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:54:03.637 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:54:03.637 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:54:03.637 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:54:03.637 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000081", 2024-08-18T15:54:03.638 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:54:03.638 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000009", 2024-08-18T15:54:03.638 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000121", 2024-08-18T15:54:03.638 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1", 2024-08-18T15:54:03.638 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:54:03.638 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:54:03.638 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:54:03.638 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:54:03.638 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:54:03.638 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:54:03.638 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.17", 2024-08-18T15:54:03.638 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000125", 2024-08-18T15:54:03.639 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:54:03.639 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000082", 2024-08-18T15:54:03.639 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:54:03.639 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:54:03.639 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:54:03.639 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:54:03.639 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:54:03.639 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:54:03.639 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:54:03.639 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:54:03.639 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:54:03.639 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:54:03.639 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:54:03.640 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000105", 2024-08-18T15:54:03.640 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000059", 2024-08-18T15:54:03.640 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:54:03.640 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:54:03.640 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:54:03.640 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:54:03.640 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:54:03.640 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:54:03.640 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:54:03.640 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:54:03.640 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:54:03.640 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:54:03.640 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:54:03.641 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000077", 2024-08-18T15:54:03.641 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000032", 2024-08-18T15:54:03.641 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:54:03.641 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:54:03.641 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:54:03.641 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:54:03.641 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:54:03.641 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:54:03.641 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.24", 2024-08-18T15:54:03.641 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:54:03.641 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:54:03.641 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:54:03.641 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:54:03.642 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3.retry", 2024-08-18T15:54:03.642 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:54:03.642 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:54:03.642 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:54:03.642 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:54:03.642 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:54:03.642 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:54:03.642 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000053", 2024-08-18T15:54:03.642 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000091", 2024-08-18T15:54:03.642 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:54:03.642 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:54:03.642 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:54:03.643 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000065", 2024-08-18T15:54:03.643 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:54:03.643 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:54:03.643 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:54:03.643 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:54:03.643 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000083", 2024-08-18T15:54:03.643 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000010", 2024-08-18T15:54:03.643 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:54:03.643 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:54:03.643 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:54:03.643 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:54:03.643 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:54:03.643 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:54:03.644 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:54:03.644 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:54:03.644 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:54:03.644 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:54:03.644 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:54:03.644 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.7", 2024-08-18T15:54:03.644 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.3", 2024-08-18T15:54:03.644 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:54:03.644 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:54:03.644 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:54:03.644 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:54:03.644 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:54:03.645 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:54:03.645 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:54:03.645 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:54:03.645 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.2", 2024-08-18T15:54:03.645 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:54:03.645 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:54:03.645 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:54:03.645 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:54:03.645 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:54:03.645 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:54:03.645 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000045", 2024-08-18T15:54:03.645 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000002", 2024-08-18T15:54:03.645 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:54:03.646 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3.2UbiLidwSTy.0", 2024-08-18T15:54:03.646 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:54:03.646 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.12", 2024-08-18T15:54:03.646 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:54:03.646 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:54:03.646 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:54:03.646 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:54:03.646 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:54:03.646 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:54:03.646 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000116", 2024-08-18T15:54:03.646 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.25", 2024-08-18T15:54:03.646 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:54:03.647 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:54:03.647 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:54:03.647 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:54:03.647 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000034", 2024-08-18T15:54:03.647 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:54:03.647 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:54:03.647 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000101", 2024-08-18T15:54:03.647 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:54:03.647 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:54:03.647 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.13", 2024-08-18T15:54:03.647 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:54:03.647 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:54:03.648 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.1", 2024-08-18T15:54:03.648 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0.retry", 2024-08-18T15:54:03.648 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:54:03.648 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:54:03.648 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000079", 2024-08-18T15:54:03.648 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000049", 2024-08-18T15:54:03.648 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:54:03.648 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0", 2024-08-18T15:54:03.648 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:54:03.648 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:54:03.648 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000117", 2024-08-18T15:54:03.648 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:54:03.648 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000044", 2024-08-18T15:54:03.649 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:54:03.649 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:54:03.649 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:54:03.649 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:54:03.649 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:54:03.649 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:54:03.649 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.14", 2024-08-18T15:54:03.649 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:54:03.649 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000066", 2024-08-18T15:54:03.649 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:54:03.649 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:54:03.649 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:54:03.649 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:54:03.650 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:54:03.650 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000068", 2024-08-18T15:54:03.650 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:54:03.650 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000085", 2024-08-18T15:54:03.650 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:54:03.650 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:54:03.650 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:54:03.650 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:54:03.650 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:54:03.650 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:54:03.650 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:54:03.650 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:54:03.650 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:54:03.651 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:54:03.651 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:54:03.651 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:54:03.651 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000073", 2024-08-18T15:54:03.651 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:54:03.651 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:54:03.651 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:54:03.651 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000038", 2024-08-18T15:54:03.651 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:54:03.651 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:54:03.651 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:54:03.651 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:54:03.651 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.10", 2024-08-18T15:54:03.652 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:54:03.652 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:54:03.652 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:54:03.652 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000118", 2024-08-18T15:54:03.652 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:54:03.652 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3", 2024-08-18T15:54:03.652 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:54:03.652 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000036", 2024-08-18T15:54:03.652 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:54:03.652 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:54:03.652 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:54:03.652 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:54:03.652 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:54:03.653 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000103", 2024-08-18T15:54:03.653 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.3", 2024-08-18T15:54:03.653 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:54:03.653 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000119", 2024-08-18T15:54:03.653 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000098", 2024-08-18T15:54:03.653 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:54:03.653 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:54:03.653 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:54:03.653 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:54:03.653 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:54:03.653 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:54:03.653 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:54:03.653 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:54:03.654 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.0", 2024-08-18T15:54:03.654 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:54:03.654 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:54:03.654 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:54:03.654 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:54:03.654 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:54:03.654 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:54:03.654 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:54:03.654 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:54:03.654 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000027", 2024-08-18T15:54:03.654 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:54:03.654 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status", 2024-08-18T15:54:03.655 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:54:03.655 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000019", 2024-08-18T15:54:03.655 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:54:03.655 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:54:03.655 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:54:03.655 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:54:03.655 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1", 2024-08-18T15:54:03.655 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:54:03.655 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:54:03.655 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:54:03.655 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:54:03.655 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000039", 2024-08-18T15:54:03.655 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:54:03.656 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:54:03.656 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:54:03.656 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:54:03.656 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:54:03.656 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:54:03.656 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:54:03.656 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:54:03.656 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:54:03.656 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:54:03.656 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:54:03.656 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000100", 2024-08-18T15:54:03.656 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:54:03.656 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:54:03.657 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:54:03.657 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:54:03.657 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:54:03.657 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000093", 2024-08-18T15:54:03.657 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:54:03.657 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000004", 2024-08-18T15:54:03.657 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000063", 2024-08-18T15:54:03.657 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000122", 2024-08-18T15:54:03.657 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:54:03.657 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:54:03.657 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:54:03.657 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000057", 2024-08-18T15:54:03.657 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:54:03.657 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:54:03.658 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:54:03.658 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:54:03.658 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000054", 2024-08-18T15:54:03.658 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:54:03.658 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000114", 2024-08-18T15:54:03.658 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:54:03.658 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:54:03.658 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:54:03.658 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.2", 2024-08-18T15:54:03.658 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:54:03.658 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:54:03.659 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0", 2024-08-18T15:54:03.659 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:54:03.659 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:54:03.659 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:54:03.659 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:54:03.659 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:54:03.659 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:54:03.659 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000012", 2024-08-18T15:54:03.659 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:54:03.659 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:54:03.659 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:54:03.659 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000084", 2024-08-18T15:54:03.659 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000043", 2024-08-18T15:54:03.660 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3", 2024-08-18T15:54:03.660 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:54:03.660 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:54:03.660 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.11", 2024-08-18T15:54:03.660 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:54:03.660 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:54:03.660 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:54:03.660 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:54:03.660 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:54:03.660 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:54:03.660 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:54:03.661 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:54:03.661 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000111", 2024-08-18T15:54:03.661 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:54:03.661 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000048", 2024-08-18T15:54:03.661 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:54:03.661 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000020", 2024-08-18T15:54:03.661 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:54:03.661 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:54:03.661 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:54:03.661 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:54:03.661 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:54:03.661 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000099", 2024-08-18T15:54:03.661 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000056", 2024-08-18T15:54:03.662 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:54:03.662 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:54:03.662 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.19", 2024-08-18T15:54:03.662 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:54:03.662 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.28", 2024-08-18T15:54:03.662 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:54:03.662 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:54:03.662 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:54:03.662 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:54:03.662 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000072", 2024-08-18T15:54:03.662 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:54:03.662 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:54:03.662 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000062", 2024-08-18T15:54:03.663 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:54:03.663 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:54:03.663 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:54:03.663 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000115", 2024-08-18T15:54:03.663 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:54:03.663 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.1", 2024-08-18T15:54:03.663 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2", 2024-08-18T15:54:03.663 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000033", 2024-08-18T15:54:03.663 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:54:03.663 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:54:03.663 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:54:03.663 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:54:03.663 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:54:03.664 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:54:03.664 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:54:03.664 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:54:03.664 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:54:03.664 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:54:03.664 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:54:03.664 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:54:03.664 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:54:03.664 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:54:03.664 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:54:03.664 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:54:03.664 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:54:03.664 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:54:03.665 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000025", 2024-08-18T15:54:03.665 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:54:03.665 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000123", 2024-08-18T15:54:03.665 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.31", 2024-08-18T15:54:03.665 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:54:03.665 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:54:03.665 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:54:03.665 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:54:03.665 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.18", 2024-08-18T15:54:03.665 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:54:03.665 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:54:03.665 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:54:03.665 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:54:03.666 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:54:03.666 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000108", 2024-08-18T15:54:03.666 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:54:03.666 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:54:03.666 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000094", 2024-08-18T15:54:03.666 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:54:03.666 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:54:03.666 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:54:03.666 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:54:03.666 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:54:03.666 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:54:03.666 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000060", 2024-08-18T15:54:03.666 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:54:03.667 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:54:03.667 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:54:03.667 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000109", 2024-08-18T15:54:03.667 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.15", 2024-08-18T15:54:03.667 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:54:03.667 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:54:03.667 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:54:03.667 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:54:03.667 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:54:03.667 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:54:03.667 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:54:03.667 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:54:03.667 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:54:03.668 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000013", 2024-08-18T15:54:03.668 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:54:03.668 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:54:03.668 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:54:03.668 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:54:03.668 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:54:03.668 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:54:03.668 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:54:03.668 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:54:03.668 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:54:03.668 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:54:03.668 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:54:03.668 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:54:03.669 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:54:03.669 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:54:03.669 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000051", 2024-08-18T15:54:03.669 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:54:03.669 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:54:03.669 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:54:03.669 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:54:03.669 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000106", 2024-08-18T15:54:03.669 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:54:03.669 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:54:03.669 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:54:03.669 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:54:03.669 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:54:03.670 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000017", 2024-08-18T15:54:03.670 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:54:03.670 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:54:03.670 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:54:03.670 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000023", 2024-08-18T15:54:03.670 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:54:03.670 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:54:03.670 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:54:03.670 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:54:03.670 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:54:03.670 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:54:03.670 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:54:03.670 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000107", 2024-08-18T15:54:03.671 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:54:03.671 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:54:03.671 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000046", 2024-08-18T15:54:03.671 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:54:03.671 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:54:03.671 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:54:03.671 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.26", 2024-08-18T15:54:03.671 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:54:03.671 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:54:03.671 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9", 2024-08-18T15:54:03.671 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.22", 2024-08-18T15:54:03.671 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:54:03.671 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:54:03.672 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:54:03.672 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:54:03.672 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:54:03.672 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:54:03.672 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:54:03.672 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2", 2024-08-18T15:54:03.672 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:54:03.672 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:54:03.672 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000067", 2024-08-18T15:54:03.672 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:54:03.672 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:54:03.672 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:54:03.672 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:54:03.673 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:54:03.673 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:54:03.673 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:54:03.673 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:54:03.673 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:54:03.673 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:54:03.673 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:54:03.673 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:54:03.673 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:54:03.673 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:54:03.673 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:54:03.673 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:54:03.673 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:54:03.674 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:54:03.674 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:54:03.674 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.9", 2024-08-18T15:54:03.674 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000086", 2024-08-18T15:54:03.674 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.5", 2024-08-18T15:54:03.674 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:54:03.674 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:54:03.674 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:54:03.674 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:54:03.674 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:54:03.674 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:54:03.674 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2.MFXJ583gCr1.0", 2024-08-18T15:54:03.675 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:54:03.675 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1.GPnO16liyZ+.0", 2024-08-18T15:54:03.675 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:54:03.675 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:54:03.675 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:54:03.675 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:54:03.675 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:54:03.675 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:54:03.675 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:54:03.675 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:54:03.675 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:54:03.675 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.20", 2024-08-18T15:54:03.675 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:54:03.675 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:54:03.676 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:54:03.676 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:54:03.676 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:54:03.676 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:54:03.676 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1.retry", 2024-08-18T15:54:03.676 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:54:03.676 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:54:03.676 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:54:03.676 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0.retry", 2024-08-18T15:54:03.676 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000050", 2024-08-18T15:54:03.676 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:54:03.676 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000102", 2024-08-18T15:54:03.676 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:54:03.677 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:54:03.677 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:54:03.677 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:54:03.677 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:54:03.677 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:54:03.677 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:54:03.677 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:54:03.677 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:54:03.677 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:54:03.677 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:54:03.677 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000124", 2024-08-18T15:54:03.677 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:54:03.677 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:54:03.678 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:54:03.678 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000001", 2024-08-18T15:54:03.678 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:54:03.678 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:54:03.678 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:54:03.678 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:54:03.678 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000126", 2024-08-18T15:54:03.678 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:54:03.678 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0.fZhctfLXnSt.0", 2024-08-18T15:54:03.678 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:54:03.678 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000055", 2024-08-18T15:54:03.678 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:54:03.679 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000061", 2024-08-18T15:54:03.679 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1", 2024-08-18T15:54:03.679 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:54:03.679 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:54:03.679 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:54:03.679 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000097", 2024-08-18T15:54:03.679 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:54:03.679 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:54:03.679 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:54:03.679 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:54:03.679 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:54:03.679 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:54:03.679 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:54:03.680 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:54:03.680 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000042", 2024-08-18T15:54:03.680 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:54:03.680 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:54:03.680 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:54:03.680 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1.retry", 2024-08-18T15:54:03.680 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:54:03.680 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:54:03.680 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:54:03.680 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:54:03.680 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:54:03.680 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:54:03.681 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:54:03.681 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000075", 2024-08-18T15:54:03.681 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:54:03.681 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000120", 2024-08-18T15:54:03.681 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000030", 2024-08-18T15:54:03.681 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:54:03.681 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:54:03.681 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:54:03.681 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:54:03.681 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:54:03.681 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:54:03.681 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9" 2024-08-18T15:54:03.681 INFO:teuthology.orchestra.run.smithi193.stdout:] 2024-08-18T15:54:03.682 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:54:08.683 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:54:08.684 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:54:08.684 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:54:08.736 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:54:08.737 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:54:08.801 INFO:teuthology.orchestra.run.smithi193.stdout:[ 2024-08-18T15:54:08.801 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:54:08.801 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:54:08.801 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.29", 2024-08-18T15:54:08.801 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000078", 2024-08-18T15:54:08.801 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.history", 2024-08-18T15:54:08.801 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:54:08.801 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:54:08.801 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:54:08.802 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:54:08.802 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.23", 2024-08-18T15:54:08.802 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000070", 2024-08-18T15:54:08.802 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000104", 2024-08-18T15:54:08.802 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:54:08.802 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2.retry", 2024-08-18T15:54:08.802 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:54:08.802 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:54:08.802 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:54:08.802 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:54:08.802 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000026", 2024-08-18T15:54:08.802 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:54:08.803 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:54:08.803 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:54:08.803 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:54:08.803 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:54:08.803 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:54:08.803 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.8", 2024-08-18T15:54:08.803 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:54:08.803 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:54:08.803 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:54:08.803 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:54:08.803 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000028", 2024-08-18T15:54:08.803 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:54:08.804 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:54:08.804 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:54:08.804 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:54:08.804 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:54:08.804 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:54:08.804 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:54:08.804 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2.retry", 2024-08-18T15:54:08.804 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:54:08.804 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.30", 2024-08-18T15:54:08.804 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:54:08.804 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:54:08.804 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:54:08.805 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:54:08.805 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000040", 2024-08-18T15:54:08.805 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:54:08.805 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.6", 2024-08-18T15:54:08.805 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:54:08.805 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:54:08.805 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:54:08.805 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:54:08.805 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:54:08.805 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:54:08.805 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000015", 2024-08-18T15:54:08.805 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:54:08.805 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:54:08.806 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000069", 2024-08-18T15:54:08.806 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:54:08.806 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:54:08.806 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:54:08.806 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000095", 2024-08-18T15:54:08.806 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0", 2024-08-18T15:54:08.806 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000003", 2024-08-18T15:54:08.806 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000047", 2024-08-18T15:54:08.806 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:54:08.806 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:54:08.806 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:54:08.806 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:54:08.806 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:54:08.807 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:54:08.807 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:54:08.807 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:54:08.807 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:54:08.807 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000035", 2024-08-18T15:54:08.807 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:54:08.807 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:54:08.807 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000037", 2024-08-18T15:54:08.807 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:54:08.807 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:54:08.807 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:54:08.807 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:54:08.807 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:54:08.808 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:54:08.808 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:54:08.808 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:54:08.808 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:54:08.808 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:54:08.808 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:54:08.808 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:54:08.808 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:54:08.808 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:54:08.808 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000024", 2024-08-18T15:54:08.808 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:54:08.808 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:54:08.809 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:54:08.809 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:54:08.809 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000000", 2024-08-18T15:54:08.809 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:54:08.809 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000031", 2024-08-18T15:54:08.809 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:54:08.809 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:54:08.809 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:54:08.809 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:54:08.809 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:54:08.809 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:54:08.809 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3", 2024-08-18T15:54:08.810 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:54:08.810 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:54:08.810 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:54:08.810 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:54:08.810 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:54:08.810 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:54:08.810 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:54:08.810 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:54:08.810 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:54:08.810 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:54:08.810 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:54:08.810 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:54:08.811 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:54:08.811 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:54:08.811 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:54:08.811 INFO:teuthology.orchestra.run.smithi193.stdout: "data_loggenerations_metadata", 2024-08-18T15:54:08.811 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2", 2024-08-18T15:54:08.811 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:54:08.811 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:54:08.811 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:54:08.811 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:54:08.811 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T15:54:08.811 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:54:08.811 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:54:08.811 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:54:08.812 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:54:08.812 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:54:08.812 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000076", 2024-08-18T15:54:08.812 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:54:08.812 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:54:08.812 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000113", 2024-08-18T15:54:08.812 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000005", 2024-08-18T15:54:08.812 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:54:08.812 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000011", 2024-08-18T15:54:08.812 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:54:08.812 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000058", 2024-08-18T15:54:08.812 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:54:08.812 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:54:08.813 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:54:08.813 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000052", 2024-08-18T15:54:08.813 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:54:08.813 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:54:08.813 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:54:08.813 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:54:08.813 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.21", 2024-08-18T15:54:08.813 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:54:08.813 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:54:08.813 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:54:08.813 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:54:08.813 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:54:08.813 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:54:08.814 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:54:08.814 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:54:08.814 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:54:08.814 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:54:08.814 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:54:08.814 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:54:08.814 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:54:08.814 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:54:08.814 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:54:08.814 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:54:08.814 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:54:08.814 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:54:08.815 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:54:08.815 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000088", 2024-08-18T15:54:08.815 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000080", 2024-08-18T15:54:08.815 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:54:08.815 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:54:08.815 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000090", 2024-08-18T15:54:08.815 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:54:08.815 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:54:08.815 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000110", 2024-08-18T15:54:08.815 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:54:08.815 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:54:08.815 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:54:08.815 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.0", 2024-08-18T15:54:08.816 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:54:08.816 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000096", 2024-08-18T15:54:08.816 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:54:08.816 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:54:08.816 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:54:08.816 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:54:08.816 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:54:08.816 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000087", 2024-08-18T15:54:08.816 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:54:08.816 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:54:08.816 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:54:08.816 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:54:08.816 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000008", 2024-08-18T15:54:08.817 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:54:08.817 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:54:08.817 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:54:08.817 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:54:08.817 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:54:08.817 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000006", 2024-08-18T15:54:08.817 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000029", 2024-08-18T15:54:08.817 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:54:08.817 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000089", 2024-08-18T15:54:08.817 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:54:08.817 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:54:08.817 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000022", 2024-08-18T15:54:08.817 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:54:08.818 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:54:08.818 INFO:teuthology.orchestra.run.smithi193.stdout: "bilog.trim", 2024-08-18T15:54:08.818 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000016", 2024-08-18T15:54:08.818 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:54:08.818 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:54:08.818 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:54:08.818 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:54:08.818 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:54:08.818 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000041", 2024-08-18T15:54:08.818 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:54:08.818 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:54:08.818 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.log.01050e30-179b-43be-bdbb-3bb23cf37ae8.3", 2024-08-18T15:54:08.819 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:54:08.819 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000018", 2024-08-18T15:54:08.819 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000092", 2024-08-18T15:54:08.819 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:54:08.819 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:54:08.819 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:54:08.819 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:54:08.819 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.27", 2024-08-18T15:54:08.819 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.4", 2024-08-18T15:54:08.819 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:54:08.819 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:54:08.819 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:54:08.819 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:54:08.820 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:54:08.820 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:54:08.820 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:54:08.820 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000014", 2024-08-18T15:54:08.820 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:54:08.820 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3.retry", 2024-08-18T15:54:08.820 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:54:08.820 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:54:08.820 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:54:08.820 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:54:08.820 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:54:08.820 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000112", 2024-08-18T15:54:08.821 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:54:08.821 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000007", 2024-08-18T15:54:08.821 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000021", 2024-08-18T15:54:08.821 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:54:08.821 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:54:08.821 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:54:08.821 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:54:08.821 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:54:08.821 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.16", 2024-08-18T15:54:08.821 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:54:08.821 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000064", 2024-08-18T15:54:08.821 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:54:08.821 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:54:08.822 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:54:08.822 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:54:08.822 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:54:08.822 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:54:08.822 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:54:08.822 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000071", 2024-08-18T15:54:08.822 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:54:08.822 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:54:08.822 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:54:08.822 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000074", 2024-08-18T15:54:08.822 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:54:08.822 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:54:08.822 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:54:08.823 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:54:08.823 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:54:08.823 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:54:08.823 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:54:08.823 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:54:08.823 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:54:08.823 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:54:08.823 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:54:08.823 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:54:08.823 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:54:08.823 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:54:08.823 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:54:08.823 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000081", 2024-08-18T15:54:08.824 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:54:08.824 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000009", 2024-08-18T15:54:08.824 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000121", 2024-08-18T15:54:08.824 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1", 2024-08-18T15:54:08.824 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:54:08.824 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:54:08.824 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:54:08.824 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:54:08.824 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:54:08.824 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:54:08.824 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.17", 2024-08-18T15:54:08.824 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000125", 2024-08-18T15:54:08.824 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:54:08.825 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000082", 2024-08-18T15:54:08.825 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:54:08.825 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:54:08.825 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:54:08.825 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:54:08.825 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:54:08.825 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:54:08.825 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:54:08.825 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:54:08.825 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:54:08.825 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:54:08.825 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:54:08.825 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000105", 2024-08-18T15:54:08.826 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000059", 2024-08-18T15:54:08.826 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:54:08.826 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:54:08.826 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:54:08.826 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:54:08.826 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:54:08.826 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:54:08.826 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:54:08.826 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:54:08.826 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:54:08.826 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:54:08.826 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:54:08.826 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000077", 2024-08-18T15:54:08.827 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000032", 2024-08-18T15:54:08.827 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:54:08.827 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:54:08.827 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:54:08.827 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:54:08.827 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:54:08.827 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:54:08.827 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.24", 2024-08-18T15:54:08.827 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:54:08.827 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:54:08.827 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:54:08.827 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:54:08.828 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3.retry", 2024-08-18T15:54:08.828 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:54:08.828 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:54:08.828 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:54:08.828 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:54:08.828 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:54:08.828 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:54:08.828 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000053", 2024-08-18T15:54:08.828 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000091", 2024-08-18T15:54:08.828 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:54:08.828 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:54:08.828 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:54:08.828 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000065", 2024-08-18T15:54:08.829 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:54:08.829 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:54:08.829 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:54:08.829 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:54:08.829 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000083", 2024-08-18T15:54:08.829 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000010", 2024-08-18T15:54:08.829 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:54:08.829 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:54:08.829 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:54:08.829 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:54:08.829 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:54:08.829 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:54:08.829 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:54:08.830 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:54:08.830 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:54:08.830 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:54:08.830 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:54:08.830 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.7", 2024-08-18T15:54:08.830 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.3", 2024-08-18T15:54:08.830 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:54:08.830 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:54:08.830 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:54:08.831 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:54:08.831 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:54:08.831 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:54:08.831 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:54:08.831 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:54:08.831 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.2", 2024-08-18T15:54:08.831 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:54:08.831 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:54:08.831 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:54:08.831 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:54:08.831 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:54:08.831 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:54:08.831 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000045", 2024-08-18T15:54:08.832 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000002", 2024-08-18T15:54:08.832 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:54:08.832 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3.2UbiLidwSTy.0", 2024-08-18T15:54:08.832 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:54:08.832 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.12", 2024-08-18T15:54:08.832 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:54:08.832 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:54:08.832 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:54:08.832 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:54:08.832 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:54:08.832 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:54:08.832 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000116", 2024-08-18T15:54:08.833 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.25", 2024-08-18T15:54:08.833 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:54:08.833 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:54:08.833 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:54:08.833 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:54:08.833 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000034", 2024-08-18T15:54:08.833 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:54:08.833 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:54:08.833 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000101", 2024-08-18T15:54:08.833 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:54:08.833 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:54:08.834 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.13", 2024-08-18T15:54:08.834 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:54:08.834 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:54:08.834 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.1", 2024-08-18T15:54:08.834 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0.retry", 2024-08-18T15:54:08.834 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:54:08.834 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:54:08.834 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000079", 2024-08-18T15:54:08.834 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000049", 2024-08-18T15:54:08.834 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:54:08.834 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0", 2024-08-18T15:54:08.835 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:54:08.835 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:54:08.835 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000117", 2024-08-18T15:54:08.835 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:54:08.835 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000044", 2024-08-18T15:54:08.835 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:54:08.835 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:54:08.835 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:54:08.835 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:54:08.835 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:54:08.835 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:54:08.835 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.14", 2024-08-18T15:54:08.836 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:54:08.836 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000066", 2024-08-18T15:54:08.836 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:54:08.836 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:54:08.836 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:54:08.836 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:54:08.836 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:54:08.836 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000068", 2024-08-18T15:54:08.836 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:54:08.836 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000085", 2024-08-18T15:54:08.836 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:54:08.836 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:54:08.836 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:54:08.836 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:54:08.837 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:54:08.837 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:54:08.837 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:54:08.837 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:54:08.837 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:54:08.837 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:54:08.837 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:54:08.837 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:54:08.837 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000073", 2024-08-18T15:54:08.837 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:54:08.837 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:54:08.837 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:54:08.837 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000038", 2024-08-18T15:54:08.838 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:54:08.838 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:54:08.838 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:54:08.838 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:54:08.838 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.10", 2024-08-18T15:54:08.838 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:54:08.838 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:54:08.838 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:54:08.838 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000118", 2024-08-18T15:54:08.838 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:54:08.838 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3", 2024-08-18T15:54:08.838 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:54:08.839 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000036", 2024-08-18T15:54:08.839 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:54:08.839 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:54:08.839 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:54:08.839 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:54:08.839 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:54:08.839 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000103", 2024-08-18T15:54:08.839 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.3", 2024-08-18T15:54:08.839 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:54:08.839 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000119", 2024-08-18T15:54:08.839 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000098", 2024-08-18T15:54:08.839 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:54:08.839 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:54:08.839 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:54:08.840 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:54:08.840 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:54:08.840 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:54:08.840 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:54:08.840 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:54:08.840 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.0", 2024-08-18T15:54:08.840 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:54:08.840 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:54:08.840 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:54:08.840 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:54:08.840 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:54:08.840 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:54:08.841 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:54:08.841 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:54:08.841 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000027", 2024-08-18T15:54:08.841 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:54:08.841 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status", 2024-08-18T15:54:08.841 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:54:08.841 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000019", 2024-08-18T15:54:08.841 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:54:08.841 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:54:08.841 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:54:08.841 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:54:08.841 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1", 2024-08-18T15:54:08.841 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:54:08.841 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:54:08.842 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:54:08.842 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:54:08.842 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000039", 2024-08-18T15:54:08.842 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:54:08.842 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:54:08.842 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:54:08.842 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:54:08.842 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:54:08.842 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:54:08.842 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:54:08.842 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:54:08.842 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:54:08.842 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:54:08.843 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:54:08.843 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000100", 2024-08-18T15:54:08.843 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:54:08.843 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:54:08.843 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:54:08.843 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:54:08.843 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:54:08.843 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000093", 2024-08-18T15:54:08.843 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:54:08.843 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000004", 2024-08-18T15:54:08.843 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000063", 2024-08-18T15:54:08.843 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000122", 2024-08-18T15:54:08.843 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:54:08.844 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:54:08.844 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:54:08.844 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000057", 2024-08-18T15:54:08.844 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:54:08.844 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:54:08.844 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:54:08.844 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:54:08.844 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000054", 2024-08-18T15:54:08.844 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:54:08.844 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000114", 2024-08-18T15:54:08.844 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:54:08.844 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:54:08.844 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:54:08.844 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.2", 2024-08-18T15:54:08.845 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:54:08.845 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:54:08.845 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0", 2024-08-18T15:54:08.845 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:54:08.845 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:54:08.845 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:54:08.845 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:54:08.845 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:54:08.845 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:54:08.845 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000012", 2024-08-18T15:54:08.845 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:54:08.845 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:54:08.845 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:54:08.845 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000084", 2024-08-18T15:54:08.846 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000043", 2024-08-18T15:54:08.846 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3", 2024-08-18T15:54:08.846 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:54:08.846 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:54:08.846 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.11", 2024-08-18T15:54:08.846 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:54:08.846 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:54:08.846 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:54:08.846 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:54:08.846 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:54:08.846 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:54:08.846 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:54:08.847 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:54:08.847 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000111", 2024-08-18T15:54:08.847 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:54:08.847 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000048", 2024-08-18T15:54:08.847 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:54:08.847 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000020", 2024-08-18T15:54:08.847 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:54:08.847 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:54:08.847 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:54:08.847 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:54:08.847 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:54:08.847 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000099", 2024-08-18T15:54:08.847 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000056", 2024-08-18T15:54:08.847 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:54:08.848 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:54:08.848 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.19", 2024-08-18T15:54:08.848 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:54:08.848 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.28", 2024-08-18T15:54:08.848 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:54:08.848 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:54:08.848 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:54:08.848 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:54:08.848 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000072", 2024-08-18T15:54:08.848 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:54:08.848 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:54:08.849 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000062", 2024-08-18T15:54:08.849 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:54:08.849 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:54:08.849 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:54:08.849 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000115", 2024-08-18T15:54:08.849 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:54:08.849 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.1", 2024-08-18T15:54:08.849 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2", 2024-08-18T15:54:08.849 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000033", 2024-08-18T15:54:08.849 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:54:08.849 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:54:08.849 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:54:08.849 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:54:08.849 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:54:08.850 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:54:08.850 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:54:08.850 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:54:08.850 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:54:08.850 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:54:08.850 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:54:08.850 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:54:08.850 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:54:08.850 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:54:08.850 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:54:08.850 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:54:08.850 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:54:08.851 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:54:08.851 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000025", 2024-08-18T15:54:08.851 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:54:08.851 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000123", 2024-08-18T15:54:08.851 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.31", 2024-08-18T15:54:08.851 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:54:08.851 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:54:08.851 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:54:08.851 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:54:08.851 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.18", 2024-08-18T15:54:08.851 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:54:08.851 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:54:08.851 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:54:08.852 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:54:08.852 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:54:08.852 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000108", 2024-08-18T15:54:08.852 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:54:08.852 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:54:08.852 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000094", 2024-08-18T15:54:08.852 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:54:08.852 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:54:08.852 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:54:08.852 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:54:08.852 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:54:08.852 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:54:08.852 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000060", 2024-08-18T15:54:08.853 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:54:08.853 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:54:08.853 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:54:08.853 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000109", 2024-08-18T15:54:08.853 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.15", 2024-08-18T15:54:08.853 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:54:08.853 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:54:08.853 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:54:08.853 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:54:08.853 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:54:08.853 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:54:08.853 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:54:08.853 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:54:08.854 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:54:08.854 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000013", 2024-08-18T15:54:08.854 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:54:08.854 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:54:08.854 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:54:08.854 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:54:08.854 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:54:08.854 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:54:08.854 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:54:08.854 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:54:08.854 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:54:08.854 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:54:08.855 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:54:08.855 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:54:08.855 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:54:08.855 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:54:08.855 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000051", 2024-08-18T15:54:08.855 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:54:08.855 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:54:08.855 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:54:08.855 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:54:08.855 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000106", 2024-08-18T15:54:08.855 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:54:08.855 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:54:08.855 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:54:08.856 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:54:08.856 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:54:08.856 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000017", 2024-08-18T15:54:08.856 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:54:08.856 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:54:08.856 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:54:08.856 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000023", 2024-08-18T15:54:08.856 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:54:08.856 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:54:08.856 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:54:08.856 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:54:08.856 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:54:08.856 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:54:08.856 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:54:08.857 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000107", 2024-08-18T15:54:08.857 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:54:08.857 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:54:08.857 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000046", 2024-08-18T15:54:08.857 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:54:08.857 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:54:08.857 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:54:08.857 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.26", 2024-08-18T15:54:08.857 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:54:08.857 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:54:08.857 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9", 2024-08-18T15:54:08.857 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.22", 2024-08-18T15:54:08.857 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:54:08.857 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:54:08.858 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:54:08.858 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:54:08.858 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:54:08.858 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:54:08.858 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:54:08.858 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2", 2024-08-18T15:54:08.858 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:54:08.858 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:54:08.858 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000067", 2024-08-18T15:54:08.858 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:54:08.858 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:54:08.858 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:54:08.858 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:54:08.859 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:54:08.859 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:54:08.859 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:54:08.859 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:54:08.859 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:54:08.859 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:54:08.859 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:54:08.859 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:54:08.859 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:54:08.859 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:54:08.859 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:54:08.859 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:54:08.859 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:54:08.860 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:54:08.860 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:54:08.860 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.9", 2024-08-18T15:54:08.860 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000086", 2024-08-18T15:54:08.860 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.5", 2024-08-18T15:54:08.860 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:54:08.860 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:54:08.860 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:54:08.860 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:54:08.860 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:54:08.860 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:54:08.860 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2.MFXJ583gCr1.0", 2024-08-18T15:54:08.860 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:54:08.860 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1.GPnO16liyZ+.0", 2024-08-18T15:54:08.861 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:54:08.861 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:54:08.861 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:54:08.861 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:54:08.861 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:54:08.861 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:54:08.861 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:54:08.861 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:54:08.861 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:54:08.861 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.20", 2024-08-18T15:54:08.861 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:54:08.861 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:54:08.861 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:54:08.862 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:54:08.862 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:54:08.862 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:54:08.862 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1.retry", 2024-08-18T15:54:08.862 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:54:08.862 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:54:08.862 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:54:08.862 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0.retry", 2024-08-18T15:54:08.862 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000050", 2024-08-18T15:54:08.862 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:54:08.862 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000102", 2024-08-18T15:54:08.862 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:54:08.862 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:54:08.863 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:54:08.863 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:54:08.863 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:54:08.863 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:54:08.863 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:54:08.863 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:54:08.863 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:54:08.863 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:54:08.863 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:54:08.863 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000124", 2024-08-18T15:54:08.863 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:54:08.863 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:54:08.863 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:54:08.863 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000001", 2024-08-18T15:54:08.864 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:54:08.864 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:54:08.864 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:54:08.864 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:54:08.864 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000126", 2024-08-18T15:54:08.864 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:54:08.864 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0.fZhctfLXnSt.0", 2024-08-18T15:54:08.864 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:54:08.864 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000055", 2024-08-18T15:54:08.864 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:54:08.864 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000061", 2024-08-18T15:54:08.864 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1", 2024-08-18T15:54:08.864 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:54:08.865 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:54:08.865 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:54:08.865 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000097", 2024-08-18T15:54:08.865 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:54:08.865 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:54:08.865 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:54:08.865 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:54:08.865 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:54:08.865 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:54:08.865 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:54:08.865 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:54:08.865 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000042", 2024-08-18T15:54:08.865 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:54:08.865 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:54:08.866 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:54:08.866 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1.retry", 2024-08-18T15:54:08.866 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:54:08.866 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:54:08.866 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:54:08.866 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:54:08.866 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:54:08.866 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:54:08.866 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:54:08.866 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000075", 2024-08-18T15:54:08.866 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:54:08.866 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000120", 2024-08-18T15:54:08.866 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000030", 2024-08-18T15:54:08.867 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:54:08.867 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:54:08.867 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:54:08.867 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:54:08.867 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:54:08.867 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:54:08.867 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9" 2024-08-18T15:54:08.867 INFO:teuthology.orchestra.run.smithi193.stdout:] 2024-08-18T15:54:08.867 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:54:13.869 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:54:13.869 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:54:13.869 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:54:13.921 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:54:13.922 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:54:13.987 INFO:teuthology.orchestra.run.smithi193.stdout:[ 2024-08-18T15:54:13.987 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:54:13.987 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:54:13.987 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.29", 2024-08-18T15:54:13.987 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000078", 2024-08-18T15:54:13.987 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.history", 2024-08-18T15:54:13.988 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:54:13.988 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:54:13.988 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:54:13.988 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:54:13.988 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.23", 2024-08-18T15:54:13.988 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000070", 2024-08-18T15:54:13.988 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000104", 2024-08-18T15:54:13.988 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:54:13.988 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2.retry", 2024-08-18T15:54:13.988 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:54:13.988 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:54:13.988 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:54:13.988 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:54:13.989 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000026", 2024-08-18T15:54:13.989 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:54:13.989 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:54:13.989 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:54:13.989 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:54:13.989 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:54:13.989 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:54:13.989 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.8", 2024-08-18T15:54:13.989 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:54:13.989 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:54:13.989 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:54:13.989 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:54:13.989 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000028", 2024-08-18T15:54:13.990 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:54:13.990 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:54:13.990 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:54:13.990 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:54:13.990 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:54:13.990 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:54:13.990 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:54:13.990 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2.retry", 2024-08-18T15:54:13.990 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:54:13.990 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.30", 2024-08-18T15:54:13.990 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:54:13.990 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:54:13.990 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:54:13.991 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:54:13.991 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000040", 2024-08-18T15:54:13.991 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:54:13.991 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.6", 2024-08-18T15:54:13.991 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:54:13.991 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:54:13.991 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:54:13.991 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:54:13.991 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:54:13.991 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:54:13.991 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000015", 2024-08-18T15:54:13.991 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:54:13.991 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:54:13.991 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000069", 2024-08-18T15:54:13.992 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:54:13.992 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:54:13.992 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:54:13.992 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000095", 2024-08-18T15:54:13.992 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0", 2024-08-18T15:54:13.992 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000003", 2024-08-18T15:54:13.992 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000047", 2024-08-18T15:54:13.992 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:54:13.992 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:54:13.992 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:54:13.992 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:54:13.992 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:54:13.992 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:54:13.993 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:54:13.993 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:54:13.993 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:54:13.993 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000035", 2024-08-18T15:54:13.993 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:54:13.993 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:54:13.993 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000037", 2024-08-18T15:54:13.993 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:54:13.993 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:54:13.993 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:54:13.993 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:54:13.993 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:54:13.994 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:54:13.994 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:54:13.994 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:54:13.994 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:54:13.994 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:54:13.994 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:54:13.994 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:54:13.994 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:54:13.994 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:54:13.994 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000024", 2024-08-18T15:54:13.994 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:54:13.994 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:54:13.995 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:54:13.995 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:54:13.995 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000000", 2024-08-18T15:54:13.995 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:54:13.995 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000031", 2024-08-18T15:54:13.995 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:54:13.995 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:54:13.995 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:54:13.995 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:54:13.996 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:54:13.996 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:54:13.996 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3", 2024-08-18T15:54:13.996 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:54:13.996 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:54:13.996 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:54:13.996 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:54:13.996 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:54:13.996 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:54:13.996 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:54:13.996 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:54:13.996 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:54:13.996 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:54:13.997 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:54:13.997 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:54:13.997 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:54:13.997 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:54:13.997 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:54:13.997 INFO:teuthology.orchestra.run.smithi193.stdout: "data_loggenerations_metadata", 2024-08-18T15:54:13.997 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2", 2024-08-18T15:54:13.997 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:54:13.997 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:54:13.997 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:54:13.998 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:54:13.998 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T15:54:13.998 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:54:13.998 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:54:13.998 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:54:13.998 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:54:13.998 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:54:13.998 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000076", 2024-08-18T15:54:13.999 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:54:13.999 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:54:13.999 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000113", 2024-08-18T15:54:13.999 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000005", 2024-08-18T15:54:13.999 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:54:13.999 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000011", 2024-08-18T15:54:13.999 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:54:13.999 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000058", 2024-08-18T15:54:13.999 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:54:13.999 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:54:13.999 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:54:13.999 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000052", 2024-08-18T15:54:14.000 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:54:14.000 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:54:14.000 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:54:14.000 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:54:14.000 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.21", 2024-08-18T15:54:14.000 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:54:14.000 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:54:14.000 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:54:14.000 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:54:14.000 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:54:14.000 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:54:14.000 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:54:14.000 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:54:14.001 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:54:14.001 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:54:14.001 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:54:14.001 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:54:14.001 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:54:14.001 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:54:14.001 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:54:14.001 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:54:14.001 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:54:14.002 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:54:14.002 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:54:14.002 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000088", 2024-08-18T15:54:14.002 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000080", 2024-08-18T15:54:14.002 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:54:14.002 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:54:14.002 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000090", 2024-08-18T15:54:14.002 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:54:14.002 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:54:14.002 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000110", 2024-08-18T15:54:14.003 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:54:14.003 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:54:14.003 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:54:14.003 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.0", 2024-08-18T15:54:14.003 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:54:14.003 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000096", 2024-08-18T15:54:14.003 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:54:14.003 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:54:14.003 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:54:14.003 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:54:14.004 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:54:14.004 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000087", 2024-08-18T15:54:14.004 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:54:14.004 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:54:14.004 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:54:14.004 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:54:14.004 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000008", 2024-08-18T15:54:14.004 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:54:14.004 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:54:14.004 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:54:14.005 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:54:14.005 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:54:14.005 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000006", 2024-08-18T15:54:14.005 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000029", 2024-08-18T15:54:14.005 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:54:14.005 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000089", 2024-08-18T15:54:14.005 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:54:14.005 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:54:14.005 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000022", 2024-08-18T15:54:14.005 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:54:14.006 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:54:14.006 INFO:teuthology.orchestra.run.smithi193.stdout: "bilog.trim", 2024-08-18T15:54:14.006 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000016", 2024-08-18T15:54:14.006 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:54:14.006 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:54:14.006 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:54:14.006 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:54:14.006 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:54:14.006 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000041", 2024-08-18T15:54:14.006 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:54:14.007 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:54:14.007 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.log.01050e30-179b-43be-bdbb-3bb23cf37ae8.3", 2024-08-18T15:54:14.007 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:54:14.007 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000018", 2024-08-18T15:54:14.007 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000092", 2024-08-18T15:54:14.007 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:54:14.007 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:54:14.007 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:54:14.007 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:54:14.007 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.27", 2024-08-18T15:54:14.007 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.4", 2024-08-18T15:54:14.007 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:54:14.007 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:54:14.008 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:54:14.008 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:54:14.008 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:54:14.008 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:54:14.008 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:54:14.008 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000014", 2024-08-18T15:54:14.008 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:54:14.008 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3.retry", 2024-08-18T15:54:14.008 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:54:14.008 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:54:14.008 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:54:14.009 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:54:14.009 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:54:14.009 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000112", 2024-08-18T15:54:14.009 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:54:14.009 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000007", 2024-08-18T15:54:14.009 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000021", 2024-08-18T15:54:14.009 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:54:14.009 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:54:14.009 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:54:14.009 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:54:14.009 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:54:14.009 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.16", 2024-08-18T15:54:14.009 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:54:14.010 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000064", 2024-08-18T15:54:14.010 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:54:14.010 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:54:14.010 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:54:14.010 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:54:14.010 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:54:14.010 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:54:14.010 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:54:14.010 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000071", 2024-08-18T15:54:14.010 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:54:14.010 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:54:14.011 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:54:14.011 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000074", 2024-08-18T15:54:14.011 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:54:14.011 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:54:14.011 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:54:14.011 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:54:14.011 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:54:14.011 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:54:14.011 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:54:14.011 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:54:14.011 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:54:14.012 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:54:14.012 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:54:14.012 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:54:14.012 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:54:14.012 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:54:14.012 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:54:14.012 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000081", 2024-08-18T15:54:14.012 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:54:14.012 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000009", 2024-08-18T15:54:14.012 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000121", 2024-08-18T15:54:14.012 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1", 2024-08-18T15:54:14.012 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:54:14.012 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:54:14.013 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:54:14.013 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:54:14.013 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:54:14.013 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:54:14.013 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.17", 2024-08-18T15:54:14.013 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000125", 2024-08-18T15:54:14.013 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:54:14.013 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000082", 2024-08-18T15:54:14.013 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:54:14.013 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:54:14.013 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:54:14.013 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:54:14.014 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:54:14.014 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:54:14.014 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:54:14.014 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:54:14.014 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:54:14.014 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:54:14.014 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:54:14.014 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000105", 2024-08-18T15:54:14.014 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000059", 2024-08-18T15:54:14.014 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:54:14.014 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:54:14.015 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:54:14.015 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:54:14.015 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:54:14.015 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:54:14.015 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:54:14.015 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:54:14.015 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:54:14.015 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:54:14.015 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:54:14.015 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000077", 2024-08-18T15:54:14.015 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000032", 2024-08-18T15:54:14.016 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:54:14.016 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:54:14.016 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:54:14.016 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:54:14.016 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:54:14.016 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:54:14.016 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.24", 2024-08-18T15:54:14.016 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:54:14.016 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:54:14.016 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:54:14.016 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:54:14.016 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3.retry", 2024-08-18T15:54:14.017 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:54:14.017 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:54:14.017 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:54:14.017 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:54:14.017 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:54:14.017 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:54:14.017 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000053", 2024-08-18T15:54:14.017 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000091", 2024-08-18T15:54:14.017 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:54:14.017 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:54:14.017 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:54:14.017 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000065", 2024-08-18T15:54:14.018 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:54:14.018 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:54:14.018 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:54:14.018 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:54:14.018 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000083", 2024-08-18T15:54:14.018 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000010", 2024-08-18T15:54:14.018 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:54:14.018 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:54:14.018 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:54:14.018 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:54:14.018 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:54:14.019 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:54:14.019 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:54:14.019 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:54:14.019 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:54:14.019 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:54:14.019 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:54:14.019 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.7", 2024-08-18T15:54:14.019 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.3", 2024-08-18T15:54:14.019 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:54:14.019 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:54:14.019 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:54:14.019 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:54:14.020 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:54:14.020 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:54:14.020 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:54:14.020 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:54:14.020 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.2", 2024-08-18T15:54:14.020 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:54:14.020 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:54:14.020 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:54:14.020 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:54:14.020 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:54:14.020 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:54:14.020 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:54:14.021 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000045", 2024-08-18T15:54:14.021 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000002", 2024-08-18T15:54:14.021 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:54:14.021 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3.2UbiLidwSTy.0", 2024-08-18T15:54:14.021 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:54:14.021 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.12", 2024-08-18T15:54:14.021 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:54:14.021 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:54:14.021 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:54:14.021 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:54:14.021 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:54:14.021 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:54:14.022 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000116", 2024-08-18T15:54:14.022 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.25", 2024-08-18T15:54:14.022 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:54:14.022 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:54:14.022 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:54:14.022 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:54:14.022 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000034", 2024-08-18T15:54:14.022 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:54:14.022 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:54:14.022 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000101", 2024-08-18T15:54:14.022 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:54:14.023 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:54:14.023 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.13", 2024-08-18T15:54:14.023 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:54:14.023 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:54:14.023 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.1", 2024-08-18T15:54:14.023 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0.retry", 2024-08-18T15:54:14.023 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:54:14.023 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:54:14.023 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000079", 2024-08-18T15:54:14.023 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000049", 2024-08-18T15:54:14.023 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:54:14.023 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0", 2024-08-18T15:54:14.024 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:54:14.024 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:54:14.024 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000117", 2024-08-18T15:54:14.024 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:54:14.024 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000044", 2024-08-18T15:54:14.024 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:54:14.024 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:54:14.024 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:54:14.024 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:54:14.024 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:54:14.024 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:54:14.025 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.14", 2024-08-18T15:54:14.025 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:54:14.025 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000066", 2024-08-18T15:54:14.025 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:54:14.025 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:54:14.025 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:54:14.025 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:54:14.025 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:54:14.025 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000068", 2024-08-18T15:54:14.025 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:54:14.025 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000085", 2024-08-18T15:54:14.025 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:54:14.025 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:54:14.026 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:54:14.026 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:54:14.026 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:54:14.026 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:54:14.026 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:54:14.026 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:54:14.026 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:54:14.026 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:54:14.026 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:54:14.026 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:54:14.027 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000073", 2024-08-18T15:54:14.027 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:54:14.027 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:54:14.027 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:54:14.027 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000038", 2024-08-18T15:54:14.027 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:54:14.027 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:54:14.027 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:54:14.027 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:54:14.027 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.10", 2024-08-18T15:54:14.027 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:54:14.027 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:54:14.028 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:54:14.028 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000118", 2024-08-18T15:54:14.028 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:54:14.028 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3", 2024-08-18T15:54:14.028 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:54:14.028 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000036", 2024-08-18T15:54:14.028 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:54:14.028 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:54:14.028 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:54:14.028 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:54:14.028 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:54:14.028 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000103", 2024-08-18T15:54:14.028 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.3", 2024-08-18T15:54:14.029 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:54:14.029 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000119", 2024-08-18T15:54:14.029 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000098", 2024-08-18T15:54:14.029 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:54:14.029 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:54:14.029 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:54:14.029 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:54:14.029 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:54:14.029 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:54:14.029 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:54:14.029 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:54:14.029 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.0", 2024-08-18T15:54:14.030 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:54:14.030 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:54:14.030 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:54:14.030 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:54:14.030 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:54:14.030 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:54:14.030 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:54:14.030 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:54:14.030 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000027", 2024-08-18T15:54:14.030 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:54:14.030 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status", 2024-08-18T15:54:14.031 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:54:14.031 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000019", 2024-08-18T15:54:14.031 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:54:14.031 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:54:14.031 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:54:14.031 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:54:14.031 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1", 2024-08-18T15:54:14.031 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:54:14.031 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:54:14.032 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:54:14.032 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:54:14.032 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000039", 2024-08-18T15:54:14.032 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:54:14.032 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:54:14.032 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:54:14.032 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:54:14.032 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:54:14.032 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:54:14.032 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:54:14.032 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:54:14.032 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:54:14.033 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:54:14.033 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:54:14.033 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000100", 2024-08-18T15:54:14.033 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:54:14.033 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:54:14.033 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:54:14.033 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:54:14.033 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:54:14.033 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000093", 2024-08-18T15:54:14.033 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:54:14.033 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000004", 2024-08-18T15:54:14.034 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000063", 2024-08-18T15:54:14.034 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000122", 2024-08-18T15:54:14.034 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:54:14.034 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:54:14.034 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:54:14.034 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000057", 2024-08-18T15:54:14.034 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:54:14.034 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:54:14.034 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:54:14.034 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:54:14.034 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000054", 2024-08-18T15:54:14.035 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:54:14.035 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000114", 2024-08-18T15:54:14.035 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:54:14.035 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:54:14.035 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:54:14.035 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.2", 2024-08-18T15:54:14.035 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:54:14.035 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:54:14.035 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0", 2024-08-18T15:54:14.035 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:54:14.036 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:54:14.036 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:54:14.036 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:54:14.036 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:54:14.036 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:54:14.036 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000012", 2024-08-18T15:54:14.036 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:54:14.036 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:54:14.036 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:54:14.036 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000084", 2024-08-18T15:54:14.036 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000043", 2024-08-18T15:54:14.037 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3", 2024-08-18T15:54:14.037 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:54:14.037 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:54:14.037 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.11", 2024-08-18T15:54:14.037 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:54:14.037 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:54:14.037 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:54:14.037 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:54:14.037 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:54:14.037 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:54:14.037 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:54:14.037 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:54:14.038 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000111", 2024-08-18T15:54:14.038 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:54:14.038 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000048", 2024-08-18T15:54:14.038 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:54:14.038 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000020", 2024-08-18T15:54:14.038 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:54:14.038 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:54:14.038 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:54:14.038 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:54:14.038 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:54:14.039 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000099", 2024-08-18T15:54:14.039 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000056", 2024-08-18T15:54:14.039 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:54:14.039 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:54:14.039 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.19", 2024-08-18T15:54:14.039 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:54:14.039 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.28", 2024-08-18T15:54:14.039 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:54:14.039 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:54:14.039 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:54:14.039 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:54:14.039 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:54:14.040 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000072", 2024-08-18T15:54:14.040 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:54:14.040 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:54:14.040 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000062", 2024-08-18T15:54:14.040 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:54:14.040 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:54:14.040 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:54:14.040 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000115", 2024-08-18T15:54:14.040 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:54:14.040 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.1", 2024-08-18T15:54:14.040 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2", 2024-08-18T15:54:14.040 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000033", 2024-08-18T15:54:14.041 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:54:14.041 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:54:14.041 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:54:14.041 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:54:14.041 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:54:14.041 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:54:14.041 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:54:14.041 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:54:14.041 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:54:14.041 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:54:14.041 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:54:14.042 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:54:14.042 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:54:14.042 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:54:14.042 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:54:14.042 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:54:14.042 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:54:14.042 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:54:14.042 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:54:14.042 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000025", 2024-08-18T15:54:14.042 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:54:14.042 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000123", 2024-08-18T15:54:14.043 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.31", 2024-08-18T15:54:14.043 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:54:14.043 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:54:14.043 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:54:14.043 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:54:14.043 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.18", 2024-08-18T15:54:14.043 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:54:14.043 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:54:14.043 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:54:14.043 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:54:14.043 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:54:14.044 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000108", 2024-08-18T15:54:14.044 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:54:14.044 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:54:14.044 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000094", 2024-08-18T15:54:14.044 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:54:14.044 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:54:14.044 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:54:14.044 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:54:14.044 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:54:14.044 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:54:14.045 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000060", 2024-08-18T15:54:14.045 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:54:14.045 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:54:14.045 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:54:14.045 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000109", 2024-08-18T15:54:14.045 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.15", 2024-08-18T15:54:14.045 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:54:14.045 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:54:14.045 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:54:14.045 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:54:14.045 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:54:14.046 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:54:14.046 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:54:14.046 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:54:14.046 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:54:14.046 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000013", 2024-08-18T15:54:14.046 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:54:14.046 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:54:14.046 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:54:14.046 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:54:14.046 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:54:14.047 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:54:14.047 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:54:14.047 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:54:14.047 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:54:14.047 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:54:14.047 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:54:14.047 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:54:14.047 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:54:14.047 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:54:14.047 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000051", 2024-08-18T15:54:14.047 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:54:14.048 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:54:14.048 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:54:14.048 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:54:14.048 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000106", 2024-08-18T15:54:14.048 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:54:14.048 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:54:14.048 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:54:14.048 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:54:14.048 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:54:14.048 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000017", 2024-08-18T15:54:14.048 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:54:14.049 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:54:14.049 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:54:14.049 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000023", 2024-08-18T15:54:14.049 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:54:14.049 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:54:14.049 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:54:14.049 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:54:14.049 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:54:14.049 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:54:14.049 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:54:14.049 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000107", 2024-08-18T15:54:14.050 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:54:14.050 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:54:14.050 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000046", 2024-08-18T15:54:14.050 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:54:14.050 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:54:14.050 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:54:14.050 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.26", 2024-08-18T15:54:14.050 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:54:14.050 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:54:14.051 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9", 2024-08-18T15:54:14.051 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.22", 2024-08-18T15:54:14.051 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:54:14.051 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:54:14.051 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:54:14.051 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:54:14.051 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:54:14.051 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:54:14.051 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:54:14.051 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2", 2024-08-18T15:54:14.051 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:54:14.052 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:54:14.052 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000067", 2024-08-18T15:54:14.052 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:54:14.052 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:54:14.052 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:54:14.052 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:54:14.052 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:54:14.052 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:54:14.052 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:54:14.052 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:54:14.053 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:54:14.053 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:54:14.053 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:54:14.053 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:54:14.053 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:54:14.053 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:54:14.053 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:54:14.053 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:54:14.053 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:54:14.053 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:54:14.053 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:54:14.054 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.9", 2024-08-18T15:54:14.054 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000086", 2024-08-18T15:54:14.054 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.5", 2024-08-18T15:54:14.054 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:54:14.054 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:54:14.054 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:54:14.054 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:54:14.054 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:54:14.054 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:54:14.054 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2.MFXJ583gCr1.0", 2024-08-18T15:54:14.054 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:54:14.055 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1.GPnO16liyZ+.0", 2024-08-18T15:54:14.055 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:54:14.055 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:54:14.055 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:54:14.055 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:54:14.055 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:54:14.055 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:54:14.055 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:54:14.055 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:54:14.055 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:54:14.056 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.20", 2024-08-18T15:54:14.056 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:54:14.056 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:54:14.056 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:54:14.056 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:54:14.056 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:54:14.056 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:54:14.056 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1.retry", 2024-08-18T15:54:14.056 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:54:14.056 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:54:14.056 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:54:14.057 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0.retry", 2024-08-18T15:54:14.057 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000050", 2024-08-18T15:54:14.057 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:54:14.057 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000102", 2024-08-18T15:54:14.057 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:54:14.057 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:54:14.057 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:54:14.057 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:54:14.057 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:54:14.057 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:54:14.057 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:54:14.058 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:54:14.058 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:54:14.058 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:54:14.058 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:54:14.058 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000124", 2024-08-18T15:54:14.058 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:54:14.058 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:54:14.058 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:54:14.058 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000001", 2024-08-18T15:54:14.058 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:54:14.058 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:54:14.058 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:54:14.059 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:54:14.059 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000126", 2024-08-18T15:54:14.059 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:54:14.059 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0.fZhctfLXnSt.0", 2024-08-18T15:54:14.059 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:54:14.059 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000055", 2024-08-18T15:54:14.059 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:54:14.059 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000061", 2024-08-18T15:54:14.059 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1", 2024-08-18T15:54:14.059 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:54:14.060 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:54:14.060 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:54:14.060 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000097", 2024-08-18T15:54:14.060 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:54:14.060 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:54:14.060 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:54:14.060 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:54:14.060 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:54:14.060 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:54:14.060 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:54:14.060 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:54:14.061 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000042", 2024-08-18T15:54:14.061 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:54:14.061 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:54:14.061 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:54:14.061 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1.retry", 2024-08-18T15:54:14.061 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:54:14.061 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:54:14.061 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:54:14.061 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:54:14.061 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:54:14.061 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:54:14.062 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:54:14.062 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000075", 2024-08-18T15:54:14.062 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:54:14.062 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000120", 2024-08-18T15:54:14.062 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000030", 2024-08-18T15:54:14.062 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:54:14.062 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:54:14.062 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:54:14.062 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:54:14.062 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:54:14.062 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:54:14.063 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9" 2024-08-18T15:54:14.063 INFO:teuthology.orchestra.run.smithi193.stdout:] 2024-08-18T15:54:14.063 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:54:19.065 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:54:19.065 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:54:19.065 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:54:19.117 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:54:19.118 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:54:19.181 INFO:teuthology.orchestra.run.smithi193.stdout:[ 2024-08-18T15:54:19.181 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:54:19.181 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:54:19.181 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.29", 2024-08-18T15:54:19.181 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000078", 2024-08-18T15:54:19.181 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.history", 2024-08-18T15:54:19.182 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:54:19.182 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:54:19.182 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:54:19.182 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:54:19.182 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.23", 2024-08-18T15:54:19.182 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000070", 2024-08-18T15:54:19.182 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000104", 2024-08-18T15:54:19.182 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:54:19.182 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2.retry", 2024-08-18T15:54:19.182 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:54:19.182 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:54:19.182 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:54:19.182 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:54:19.183 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000026", 2024-08-18T15:54:19.183 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:54:19.183 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:54:19.183 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:54:19.183 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:54:19.183 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:54:19.183 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:54:19.183 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.8", 2024-08-18T15:54:19.183 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:54:19.183 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:54:19.183 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:54:19.183 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:54:19.184 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000028", 2024-08-18T15:54:19.184 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:54:19.184 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:54:19.184 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:54:19.184 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:54:19.184 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:54:19.184 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:54:19.184 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:54:19.184 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2.retry", 2024-08-18T15:54:19.184 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:54:19.184 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.30", 2024-08-18T15:54:19.184 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:54:19.184 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:54:19.185 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:54:19.185 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:54:19.185 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000040", 2024-08-18T15:54:19.185 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:54:19.185 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.6", 2024-08-18T15:54:19.185 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:54:19.185 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:54:19.185 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:54:19.185 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:54:19.185 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:54:19.185 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:54:19.185 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000015", 2024-08-18T15:54:19.186 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:54:19.186 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:54:19.186 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000069", 2024-08-18T15:54:19.186 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:54:19.186 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:54:19.186 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:54:19.186 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000095", 2024-08-18T15:54:19.186 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0", 2024-08-18T15:54:19.186 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000003", 2024-08-18T15:54:19.186 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000047", 2024-08-18T15:54:19.186 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:54:19.186 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:54:19.187 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:54:19.187 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:54:19.187 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:54:19.187 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:54:19.187 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:54:19.187 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:54:19.187 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:54:19.187 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000035", 2024-08-18T15:54:19.187 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:54:19.187 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:54:19.187 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000037", 2024-08-18T15:54:19.187 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:54:19.187 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:54:19.188 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:54:19.188 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:54:19.188 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:54:19.188 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:54:19.188 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:54:19.188 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:54:19.188 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:54:19.188 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:54:19.188 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:54:19.188 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:54:19.188 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:54:19.189 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:54:19.189 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000024", 2024-08-18T15:54:19.189 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:54:19.189 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:54:19.189 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:54:19.189 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:54:19.189 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000000", 2024-08-18T15:54:19.189 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:54:19.189 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000031", 2024-08-18T15:54:19.189 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:54:19.189 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:54:19.189 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:54:19.189 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:54:19.190 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:54:19.190 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:54:19.190 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3", 2024-08-18T15:54:19.190 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:54:19.190 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:54:19.190 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:54:19.190 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:54:19.190 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:54:19.190 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:54:19.190 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:54:19.190 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:54:19.190 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:54:19.191 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:54:19.191 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:54:19.191 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:54:19.191 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:54:19.191 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:54:19.191 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:54:19.191 INFO:teuthology.orchestra.run.smithi193.stdout: "data_loggenerations_metadata", 2024-08-18T15:54:19.191 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2", 2024-08-18T15:54:19.191 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:54:19.191 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:54:19.191 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:54:19.191 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:54:19.191 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T15:54:19.192 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:54:19.192 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:54:19.192 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:54:19.192 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:54:19.192 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:54:19.192 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000076", 2024-08-18T15:54:19.192 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:54:19.192 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:54:19.192 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000113", 2024-08-18T15:54:19.192 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000005", 2024-08-18T15:54:19.192 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:54:19.192 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000011", 2024-08-18T15:54:19.193 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:54:19.193 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000058", 2024-08-18T15:54:19.193 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:54:19.193 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:54:19.193 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:54:19.193 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000052", 2024-08-18T15:54:19.193 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:54:19.193 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:54:19.193 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:54:19.193 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:54:19.193 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.21", 2024-08-18T15:54:19.193 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:54:19.194 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:54:19.194 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:54:19.194 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:54:19.194 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:54:19.194 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:54:19.194 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:54:19.194 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:54:19.194 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:54:19.194 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:54:19.194 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:54:19.194 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:54:19.194 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:54:19.195 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:54:19.195 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:54:19.195 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:54:19.195 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:54:19.195 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:54:19.195 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:54:19.195 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000088", 2024-08-18T15:54:19.195 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000080", 2024-08-18T15:54:19.195 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:54:19.195 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:54:19.195 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000090", 2024-08-18T15:54:19.195 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:54:19.195 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:54:19.195 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000110", 2024-08-18T15:54:19.195 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:54:19.196 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:54:19.196 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:54:19.196 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.0", 2024-08-18T15:54:19.196 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:54:19.196 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000096", 2024-08-18T15:54:19.196 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:54:19.196 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:54:19.196 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:54:19.196 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:54:19.196 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:54:19.196 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000087", 2024-08-18T15:54:19.196 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:54:19.197 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:54:19.197 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:54:19.197 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:54:19.197 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000008", 2024-08-18T15:54:19.197 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:54:19.197 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:54:19.197 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:54:19.197 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:54:19.197 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:54:19.197 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000006", 2024-08-18T15:54:19.197 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000029", 2024-08-18T15:54:19.197 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:54:19.197 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000089", 2024-08-18T15:54:19.197 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:54:19.198 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:54:19.198 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000022", 2024-08-18T15:54:19.198 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:54:19.198 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:54:19.198 INFO:teuthology.orchestra.run.smithi193.stdout: "bilog.trim", 2024-08-18T15:54:19.198 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000016", 2024-08-18T15:54:19.198 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:54:19.198 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:54:19.198 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:54:19.198 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:54:19.198 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:54:19.198 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000041", 2024-08-18T15:54:19.198 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:54:19.199 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:54:19.199 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.log.01050e30-179b-43be-bdbb-3bb23cf37ae8.3", 2024-08-18T15:54:19.199 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:54:19.199 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000018", 2024-08-18T15:54:19.199 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000092", 2024-08-18T15:54:19.199 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:54:19.199 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:54:19.199 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:54:19.199 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:54:19.199 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.27", 2024-08-18T15:54:19.199 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.4", 2024-08-18T15:54:19.199 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:54:19.199 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:54:19.199 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:54:19.200 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:54:19.200 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:54:19.200 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:54:19.200 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:54:19.200 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000014", 2024-08-18T15:54:19.200 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:54:19.200 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3.retry", 2024-08-18T15:54:19.200 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:54:19.200 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:54:19.200 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:54:19.200 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:54:19.200 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:54:19.200 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000112", 2024-08-18T15:54:19.201 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:54:19.201 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000007", 2024-08-18T15:54:19.201 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000021", 2024-08-18T15:54:19.201 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:54:19.201 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:54:19.201 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:54:19.201 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:54:19.201 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:54:19.201 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.16", 2024-08-18T15:54:19.201 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:54:19.201 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000064", 2024-08-18T15:54:19.201 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:54:19.201 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:54:19.202 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:54:19.202 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:54:19.202 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:54:19.202 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:54:19.202 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:54:19.202 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000071", 2024-08-18T15:54:19.202 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:54:19.202 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:54:19.202 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:54:19.202 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000074", 2024-08-18T15:54:19.202 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:54:19.202 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:54:19.203 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:54:19.203 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:54:19.203 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:54:19.203 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:54:19.203 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:54:19.203 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:54:19.203 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:54:19.203 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:54:19.203 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:54:19.203 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:54:19.203 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:54:19.203 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:54:19.203 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:54:19.204 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000081", 2024-08-18T15:54:19.204 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:54:19.204 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000009", 2024-08-18T15:54:19.204 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000121", 2024-08-18T15:54:19.204 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1", 2024-08-18T15:54:19.204 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:54:19.204 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:54:19.204 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:54:19.204 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:54:19.204 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:54:19.204 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:54:19.204 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.17", 2024-08-18T15:54:19.204 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000125", 2024-08-18T15:54:19.205 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:54:19.205 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000082", 2024-08-18T15:54:19.205 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:54:19.205 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:54:19.205 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:54:19.205 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:54:19.205 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:54:19.205 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:54:19.205 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:54:19.205 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:54:19.205 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:54:19.206 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:54:19.206 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:54:19.206 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000105", 2024-08-18T15:54:19.206 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000059", 2024-08-18T15:54:19.206 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:54:19.206 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:54:19.206 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:54:19.206 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:54:19.206 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:54:19.206 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:54:19.206 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:54:19.207 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:54:19.207 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:54:19.207 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:54:19.207 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:54:19.207 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000077", 2024-08-18T15:54:19.207 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000032", 2024-08-18T15:54:19.207 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:54:19.207 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:54:19.207 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:54:19.207 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:54:19.207 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:54:19.207 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:54:19.207 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.24", 2024-08-18T15:54:19.208 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:54:19.208 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:54:19.208 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:54:19.208 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:54:19.208 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3.retry", 2024-08-18T15:54:19.208 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:54:19.208 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:54:19.208 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:54:19.208 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:54:19.208 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:54:19.208 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:54:19.208 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000053", 2024-08-18T15:54:19.208 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000091", 2024-08-18T15:54:19.209 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:54:19.209 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:54:19.209 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:54:19.209 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000065", 2024-08-18T15:54:19.209 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:54:19.209 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:54:19.209 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:54:19.209 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:54:19.209 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000083", 2024-08-18T15:54:19.209 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000010", 2024-08-18T15:54:19.209 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:54:19.209 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:54:19.209 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:54:19.209 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:54:19.210 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:54:19.210 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:54:19.210 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:54:19.210 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:54:19.210 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:54:19.210 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:54:19.210 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:54:19.210 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.7", 2024-08-18T15:54:19.210 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.3", 2024-08-18T15:54:19.210 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:54:19.210 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:54:19.210 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:54:19.210 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:54:19.211 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:54:19.211 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:54:19.211 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:54:19.211 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:54:19.211 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.2", 2024-08-18T15:54:19.211 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:54:19.211 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:54:19.211 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:54:19.211 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:54:19.211 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:54:19.211 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:54:19.211 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:54:19.212 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000045", 2024-08-18T15:54:19.212 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000002", 2024-08-18T15:54:19.212 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:54:19.212 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3.2UbiLidwSTy.0", 2024-08-18T15:54:19.212 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:54:19.212 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.12", 2024-08-18T15:54:19.212 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:54:19.212 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:54:19.212 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:54:19.212 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:54:19.212 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:54:19.212 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:54:19.212 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000116", 2024-08-18T15:54:19.212 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.25", 2024-08-18T15:54:19.213 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:54:19.213 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:54:19.213 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:54:19.213 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:54:19.213 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000034", 2024-08-18T15:54:19.213 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:54:19.213 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:54:19.213 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000101", 2024-08-18T15:54:19.213 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:54:19.213 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:54:19.213 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.13", 2024-08-18T15:54:19.213 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:54:19.213 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:54:19.214 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.1", 2024-08-18T15:54:19.214 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0.retry", 2024-08-18T15:54:19.214 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:54:19.214 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:54:19.214 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000079", 2024-08-18T15:54:19.214 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000049", 2024-08-18T15:54:19.214 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:54:19.214 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0", 2024-08-18T15:54:19.214 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:54:19.214 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:54:19.214 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000117", 2024-08-18T15:54:19.214 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:54:19.214 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000044", 2024-08-18T15:54:19.215 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:54:19.215 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:54:19.215 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:54:19.215 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:54:19.215 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:54:19.215 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:54:19.215 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.14", 2024-08-18T15:54:19.215 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:54:19.215 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000066", 2024-08-18T15:54:19.215 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:54:19.215 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:54:19.215 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:54:19.215 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:54:19.216 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:54:19.216 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000068", 2024-08-18T15:54:19.216 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:54:19.216 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000085", 2024-08-18T15:54:19.216 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:54:19.216 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:54:19.216 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:54:19.216 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:54:19.216 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:54:19.216 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:54:19.216 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:54:19.216 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:54:19.216 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:54:19.217 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:54:19.217 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:54:19.217 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:54:19.217 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000073", 2024-08-18T15:54:19.217 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:54:19.217 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:54:19.217 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:54:19.217 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000038", 2024-08-18T15:54:19.217 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:54:19.217 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:54:19.217 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:54:19.217 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:54:19.218 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.10", 2024-08-18T15:54:19.218 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:54:19.218 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:54:19.218 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:54:19.218 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000118", 2024-08-18T15:54:19.218 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:54:19.218 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3", 2024-08-18T15:54:19.218 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:54:19.218 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000036", 2024-08-18T15:54:19.219 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:54:19.219 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:54:19.219 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:54:19.219 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:54:19.219 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:54:19.219 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000103", 2024-08-18T15:54:19.219 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.3", 2024-08-18T15:54:19.219 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:54:19.219 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000119", 2024-08-18T15:54:19.219 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000098", 2024-08-18T15:54:19.219 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:54:19.219 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:54:19.220 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:54:19.220 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:54:19.220 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:54:19.220 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:54:19.220 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:54:19.220 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:54:19.220 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.0", 2024-08-18T15:54:19.220 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:54:19.220 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:54:19.220 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:54:19.220 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:54:19.220 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:54:19.220 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:54:19.221 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:54:19.221 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:54:19.221 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000027", 2024-08-18T15:54:19.221 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:54:19.221 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status", 2024-08-18T15:54:19.221 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:54:19.221 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000019", 2024-08-18T15:54:19.221 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:54:19.221 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:54:19.221 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:54:19.221 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:54:19.221 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1", 2024-08-18T15:54:19.221 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:54:19.222 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:54:19.222 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:54:19.222 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:54:19.222 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000039", 2024-08-18T15:54:19.222 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:54:19.222 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:54:19.222 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:54:19.222 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:54:19.222 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:54:19.222 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:54:19.222 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:54:19.222 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:54:19.223 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:54:19.223 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:54:19.223 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:54:19.223 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000100", 2024-08-18T15:54:19.223 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:54:19.223 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:54:19.223 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:54:19.223 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:54:19.223 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:54:19.223 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000093", 2024-08-18T15:54:19.223 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:54:19.223 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000004", 2024-08-18T15:54:19.223 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000063", 2024-08-18T15:54:19.224 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000122", 2024-08-18T15:54:19.224 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:54:19.224 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:54:19.224 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:54:19.224 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000057", 2024-08-18T15:54:19.224 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:54:19.224 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:54:19.224 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:54:19.224 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:54:19.224 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000054", 2024-08-18T15:54:19.224 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:54:19.225 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000114", 2024-08-18T15:54:19.225 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:54:19.225 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:54:19.225 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:54:19.225 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.2", 2024-08-18T15:54:19.225 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:54:19.225 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:54:19.225 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0", 2024-08-18T15:54:19.225 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:54:19.225 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:54:19.225 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:54:19.225 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:54:19.225 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:54:19.226 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:54:19.226 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000012", 2024-08-18T15:54:19.226 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:54:19.226 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:54:19.226 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:54:19.226 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000084", 2024-08-18T15:54:19.226 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000043", 2024-08-18T15:54:19.226 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3", 2024-08-18T15:54:19.226 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:54:19.226 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:54:19.226 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.11", 2024-08-18T15:54:19.226 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:54:19.226 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:54:19.227 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:54:19.227 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:54:19.227 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:54:19.227 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:54:19.227 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:54:19.227 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:54:19.227 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000111", 2024-08-18T15:54:19.227 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:54:19.227 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000048", 2024-08-18T15:54:19.228 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:54:19.228 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000020", 2024-08-18T15:54:19.228 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:54:19.228 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:54:19.228 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:54:19.228 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:54:19.228 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:54:19.228 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000099", 2024-08-18T15:54:19.228 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000056", 2024-08-18T15:54:19.228 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:54:19.228 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:54:19.228 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.19", 2024-08-18T15:54:19.228 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:54:19.229 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.28", 2024-08-18T15:54:19.229 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:54:19.229 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:54:19.229 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:54:19.229 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:54:19.229 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:54:19.229 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000072", 2024-08-18T15:54:19.229 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:54:19.230 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:54:19.230 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000062", 2024-08-18T15:54:19.230 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:54:19.230 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:54:19.230 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:54:19.230 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000115", 2024-08-18T15:54:19.230 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:54:19.230 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.1", 2024-08-18T15:54:19.230 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2", 2024-08-18T15:54:19.230 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000033", 2024-08-18T15:54:19.230 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:54:19.231 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:54:19.231 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:54:19.231 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:54:19.231 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:54:19.231 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:54:19.231 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:54:19.231 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:54:19.231 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:54:19.231 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:54:19.231 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:54:19.231 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:54:19.231 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:54:19.231 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:54:19.231 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:54:19.232 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:54:19.232 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:54:19.232 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:54:19.232 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:54:19.232 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000025", 2024-08-18T15:54:19.232 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:54:19.232 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000123", 2024-08-18T15:54:19.232 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.31", 2024-08-18T15:54:19.232 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:54:19.232 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:54:19.232 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:54:19.232 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:54:19.233 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.18", 2024-08-18T15:54:19.233 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:54:19.233 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:54:19.233 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:54:19.233 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:54:19.233 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:54:19.233 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000108", 2024-08-18T15:54:19.233 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:54:19.233 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:54:19.233 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000094", 2024-08-18T15:54:19.234 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:54:19.234 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:54:19.234 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:54:19.234 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:54:19.234 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:54:19.234 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:54:19.234 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000060", 2024-08-18T15:54:19.234 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:54:19.234 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:54:19.234 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:54:19.234 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000109", 2024-08-18T15:54:19.234 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.15", 2024-08-18T15:54:19.235 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:54:19.235 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:54:19.235 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:54:19.235 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:54:19.235 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:54:19.235 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:54:19.235 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:54:19.235 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:54:19.235 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:54:19.235 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000013", 2024-08-18T15:54:19.235 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:54:19.235 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:54:19.236 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:54:19.236 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:54:19.236 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:54:19.236 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:54:19.236 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:54:19.236 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:54:19.237 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:54:19.237 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:54:19.237 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:54:19.237 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:54:19.237 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:54:19.237 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:54:19.237 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000051", 2024-08-18T15:54:19.237 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:54:19.237 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:54:19.237 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:54:19.237 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:54:19.237 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000106", 2024-08-18T15:54:19.237 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:54:19.238 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:54:19.238 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:54:19.238 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:54:19.238 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:54:19.238 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000017", 2024-08-18T15:54:19.238 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:54:19.238 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:54:19.238 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:54:19.238 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000023", 2024-08-18T15:54:19.238 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:54:19.238 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:54:19.238 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:54:19.239 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:54:19.239 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:54:19.239 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:54:19.239 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:54:19.239 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000107", 2024-08-18T15:54:19.239 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:54:19.239 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:54:19.239 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000046", 2024-08-18T15:54:19.239 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:54:19.239 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:54:19.239 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:54:19.239 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.26", 2024-08-18T15:54:19.239 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:54:19.240 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:54:19.240 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9", 2024-08-18T15:54:19.240 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.22", 2024-08-18T15:54:19.240 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:54:19.240 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:54:19.240 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:54:19.240 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:54:19.240 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:54:19.240 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:54:19.240 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:54:19.240 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2", 2024-08-18T15:54:19.240 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:54:19.240 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:54:19.241 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000067", 2024-08-18T15:54:19.241 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:54:19.241 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:54:19.241 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:54:19.241 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:54:19.241 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:54:19.241 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:54:19.241 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:54:19.241 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:54:19.241 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:54:19.241 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:54:19.241 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:54:19.242 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:54:19.242 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:54:19.242 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:54:19.242 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:54:19.242 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:54:19.242 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:54:19.242 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:54:19.242 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:54:19.242 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.9", 2024-08-18T15:54:19.242 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000086", 2024-08-18T15:54:19.242 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.5", 2024-08-18T15:54:19.242 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:54:19.243 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:54:19.243 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:54:19.243 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:54:19.243 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:54:19.243 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:54:19.243 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2.MFXJ583gCr1.0", 2024-08-18T15:54:19.243 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:54:19.243 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1.GPnO16liyZ+.0", 2024-08-18T15:54:19.243 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:54:19.243 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:54:19.243 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:54:19.243 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:54:19.244 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:54:19.244 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:54:19.244 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:54:19.244 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:54:19.244 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:54:19.244 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.20", 2024-08-18T15:54:19.244 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:54:19.244 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:54:19.244 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:54:19.244 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:54:19.244 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:54:19.244 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:54:19.244 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1.retry", 2024-08-18T15:54:19.245 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:54:19.245 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:54:19.245 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:54:19.245 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0.retry", 2024-08-18T15:54:19.245 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000050", 2024-08-18T15:54:19.245 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:54:19.245 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000102", 2024-08-18T15:54:19.245 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:54:19.245 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:54:19.245 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:54:19.245 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:54:19.245 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:54:19.246 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:54:19.246 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:54:19.246 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:54:19.246 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:54:19.246 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:54:19.246 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:54:19.246 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000124", 2024-08-18T15:54:19.246 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:54:19.246 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:54:19.246 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:54:19.246 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000001", 2024-08-18T15:54:19.246 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:54:19.247 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:54:19.247 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:54:19.247 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:54:19.247 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000126", 2024-08-18T15:54:19.247 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:54:19.247 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0.fZhctfLXnSt.0", 2024-08-18T15:54:19.247 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:54:19.247 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000055", 2024-08-18T15:54:19.247 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:54:19.247 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000061", 2024-08-18T15:54:19.247 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1", 2024-08-18T15:54:19.247 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:54:19.247 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:54:19.248 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:54:19.248 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000097", 2024-08-18T15:54:19.248 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:54:19.248 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:54:19.248 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:54:19.248 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:54:19.248 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:54:19.248 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:54:19.248 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:54:19.248 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:54:19.248 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000042", 2024-08-18T15:54:19.248 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:54:19.248 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:54:19.248 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:54:19.249 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1.retry", 2024-08-18T15:54:19.249 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:54:19.249 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:54:19.249 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:54:19.249 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:54:19.249 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:54:19.249 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:54:19.249 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:54:19.249 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000075", 2024-08-18T15:54:19.249 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:54:19.249 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000120", 2024-08-18T15:54:19.249 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000030", 2024-08-18T15:54:19.250 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:54:19.250 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:54:19.250 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:54:19.250 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:54:19.250 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:54:19.250 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:54:19.250 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9" 2024-08-18T15:54:19.250 INFO:teuthology.orchestra.run.smithi193.stdout:] 2024-08-18T15:54:19.250 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:54:24.252 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:54:24.252 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:54:24.252 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:54:24.301 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:54:24.301 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:54:24.362 INFO:teuthology.orchestra.run.smithi193.stdout:[ 2024-08-18T15:54:24.362 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:54:24.362 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:54:24.362 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.29", 2024-08-18T15:54:24.362 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000078", 2024-08-18T15:54:24.363 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.history", 2024-08-18T15:54:24.363 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:54:24.363 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:54:24.363 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:54:24.363 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:54:24.363 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.23", 2024-08-18T15:54:24.363 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000070", 2024-08-18T15:54:24.363 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000104", 2024-08-18T15:54:24.363 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:54:24.363 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2.retry", 2024-08-18T15:54:24.363 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:54:24.363 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:54:24.363 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:54:24.364 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:54:24.364 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000026", 2024-08-18T15:54:24.364 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:54:24.364 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:54:24.364 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:54:24.364 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:54:24.364 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:54:24.364 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:54:24.364 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.8", 2024-08-18T15:54:24.364 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:54:24.364 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:54:24.364 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:54:24.364 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:54:24.365 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000028", 2024-08-18T15:54:24.365 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:54:24.365 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:54:24.365 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:54:24.365 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:54:24.365 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:54:24.365 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:54:24.365 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:54:24.365 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2.retry", 2024-08-18T15:54:24.365 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:54:24.365 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.30", 2024-08-18T15:54:24.365 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:54:24.366 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:54:24.366 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:54:24.366 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:54:24.366 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000040", 2024-08-18T15:54:24.366 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:54:24.366 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.6", 2024-08-18T15:54:24.366 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:54:24.366 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:54:24.366 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:54:24.366 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:54:24.366 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:54:24.366 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:54:24.367 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000015", 2024-08-18T15:54:24.367 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:54:24.367 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:54:24.367 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000069", 2024-08-18T15:54:24.367 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:54:24.367 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:54:24.367 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:54:24.367 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000095", 2024-08-18T15:54:24.367 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0", 2024-08-18T15:54:24.367 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000003", 2024-08-18T15:54:24.367 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000047", 2024-08-18T15:54:24.367 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:54:24.367 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:54:24.368 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:54:24.368 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:54:24.368 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:54:24.368 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:54:24.368 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:54:24.368 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:54:24.368 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:54:24.368 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000035", 2024-08-18T15:54:24.368 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:54:24.368 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:54:24.368 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000037", 2024-08-18T15:54:24.368 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:54:24.369 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:54:24.369 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:54:24.369 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:54:24.369 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:54:24.369 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:54:24.369 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:54:24.369 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:54:24.369 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:54:24.369 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:54:24.369 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:54:24.369 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:54:24.369 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:54:24.370 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:54:24.370 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000024", 2024-08-18T15:54:24.370 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:54:24.370 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:54:24.370 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:54:24.370 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:54:24.370 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000000", 2024-08-18T15:54:24.370 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:54:24.370 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000031", 2024-08-18T15:54:24.370 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:54:24.371 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:54:24.371 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:54:24.371 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:54:24.371 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:54:24.371 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:54:24.371 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3", 2024-08-18T15:54:24.371 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:54:24.371 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:54:24.371 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:54:24.371 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:54:24.371 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:54:24.371 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:54:24.372 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:54:24.372 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:54:24.372 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:54:24.372 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:54:24.372 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:54:24.372 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:54:24.372 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:54:24.372 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:54:24.372 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:54:24.372 INFO:teuthology.orchestra.run.smithi193.stdout: "data_loggenerations_metadata", 2024-08-18T15:54:24.372 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2", 2024-08-18T15:54:24.372 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:54:24.372 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:54:24.373 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:54:24.373 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:54:24.373 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T15:54:24.373 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:54:24.373 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:54:24.373 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:54:24.373 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:54:24.373 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:54:24.373 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000076", 2024-08-18T15:54:24.373 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:54:24.373 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:54:24.373 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000113", 2024-08-18T15:54:24.373 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000005", 2024-08-18T15:54:24.374 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:54:24.374 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000011", 2024-08-18T15:54:24.374 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:54:24.374 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000058", 2024-08-18T15:54:24.374 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:54:24.374 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:54:24.374 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:54:24.374 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000052", 2024-08-18T15:54:24.374 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:54:24.374 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:54:24.374 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:54:24.374 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:54:24.375 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.21", 2024-08-18T15:54:24.375 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:54:24.375 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:54:24.375 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:54:24.375 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:54:24.375 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:54:24.375 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:54:24.375 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:54:24.375 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:54:24.375 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:54:24.375 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:54:24.375 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:54:24.376 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:54:24.376 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:54:24.376 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:54:24.376 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:54:24.376 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:54:24.376 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:54:24.376 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:54:24.376 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:54:24.376 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000088", 2024-08-18T15:54:24.376 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000080", 2024-08-18T15:54:24.376 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:54:24.376 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:54:24.376 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000090", 2024-08-18T15:54:24.377 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:54:24.377 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:54:24.377 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000110", 2024-08-18T15:54:24.377 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:54:24.377 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:54:24.377 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:54:24.377 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.0", 2024-08-18T15:54:24.377 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:54:24.377 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000096", 2024-08-18T15:54:24.377 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:54:24.377 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:54:24.377 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:54:24.377 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:54:24.377 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:54:24.378 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000087", 2024-08-18T15:54:24.378 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:54:24.378 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:54:24.378 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:54:24.378 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:54:24.378 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000008", 2024-08-18T15:54:24.378 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:54:24.378 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:54:24.378 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:54:24.378 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:54:24.378 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:54:24.378 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000006", 2024-08-18T15:54:24.379 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000029", 2024-08-18T15:54:24.379 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:54:24.379 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000089", 2024-08-18T15:54:24.379 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:54:24.379 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:54:24.379 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000022", 2024-08-18T15:54:24.379 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:54:24.379 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:54:24.379 INFO:teuthology.orchestra.run.smithi193.stdout: "bilog.trim", 2024-08-18T15:54:24.379 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000016", 2024-08-18T15:54:24.379 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:54:24.379 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:54:24.379 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:54:24.379 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:54:24.380 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:54:24.380 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000041", 2024-08-18T15:54:24.380 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:54:24.380 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:54:24.380 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.log.01050e30-179b-43be-bdbb-3bb23cf37ae8.3", 2024-08-18T15:54:24.380 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:54:24.380 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000018", 2024-08-18T15:54:24.380 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000092", 2024-08-18T15:54:24.380 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:54:24.380 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:54:24.380 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:54:24.380 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:54:24.380 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.27", 2024-08-18T15:54:24.380 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.4", 2024-08-18T15:54:24.381 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:54:24.381 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:54:24.381 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:54:24.381 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:54:24.381 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:54:24.381 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:54:24.381 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:54:24.381 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000014", 2024-08-18T15:54:24.381 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:54:24.381 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3.retry", 2024-08-18T15:54:24.381 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:54:24.381 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:54:24.381 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:54:24.382 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:54:24.382 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:54:24.382 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000112", 2024-08-18T15:54:24.382 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:54:24.382 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000007", 2024-08-18T15:54:24.382 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000021", 2024-08-18T15:54:24.382 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:54:24.382 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:54:24.382 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:54:24.382 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:54:24.382 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:54:24.382 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.16", 2024-08-18T15:54:24.383 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:54:24.383 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000064", 2024-08-18T15:54:24.383 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:54:24.383 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:54:24.383 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:54:24.383 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:54:24.383 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:54:24.383 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:54:24.383 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:54:24.383 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000071", 2024-08-18T15:54:24.383 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:54:24.383 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:54:24.383 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:54:24.383 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000074", 2024-08-18T15:54:24.383 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:54:24.384 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:54:24.384 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:54:24.384 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:54:24.384 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:54:24.384 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:54:24.384 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:54:24.384 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:54:24.384 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:54:24.384 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:54:24.384 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:54:24.384 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:54:24.385 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:54:24.385 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:54:24.385 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:54:24.385 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000081", 2024-08-18T15:54:24.385 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:54:24.385 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000009", 2024-08-18T15:54:24.385 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000121", 2024-08-18T15:54:24.385 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1", 2024-08-18T15:54:24.385 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:54:24.385 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:54:24.385 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:54:24.385 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:54:24.386 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:54:24.386 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:54:24.386 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.17", 2024-08-18T15:54:24.386 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000125", 2024-08-18T15:54:24.386 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:54:24.386 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000082", 2024-08-18T15:54:24.386 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:54:24.386 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:54:24.386 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:54:24.386 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:54:24.386 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:54:24.386 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:54:24.386 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:54:24.387 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:54:24.387 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:54:24.387 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:54:24.387 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:54:24.387 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000105", 2024-08-18T15:54:24.387 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000059", 2024-08-18T15:54:24.387 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:54:24.387 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:54:24.387 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:54:24.387 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:54:24.387 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:54:24.387 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:54:24.387 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:54:24.388 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:54:24.388 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:54:24.388 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:54:24.388 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:54:24.388 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000077", 2024-08-18T15:54:24.388 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000032", 2024-08-18T15:54:24.388 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:54:24.388 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:54:24.388 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:54:24.388 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:54:24.388 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:54:24.388 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:54:24.388 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.24", 2024-08-18T15:54:24.388 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:54:24.389 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:54:24.389 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:54:24.389 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:54:24.389 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3.retry", 2024-08-18T15:54:24.389 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:54:24.389 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:54:24.389 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:54:24.389 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:54:24.389 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:54:24.389 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:54:24.389 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000053", 2024-08-18T15:54:24.389 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000091", 2024-08-18T15:54:24.389 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:54:24.389 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:54:24.389 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:54:24.390 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000065", 2024-08-18T15:54:24.390 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:54:24.390 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:54:24.390 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:54:24.390 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:54:24.390 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000083", 2024-08-18T15:54:24.390 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000010", 2024-08-18T15:54:24.390 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:54:24.390 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:54:24.390 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:54:24.390 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:54:24.390 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:54:24.391 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:54:24.391 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:54:24.391 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:54:24.391 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:54:24.391 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:54:24.391 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:54:24.391 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.7", 2024-08-18T15:54:24.391 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.3", 2024-08-18T15:54:24.391 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:54:24.391 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:54:24.391 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:54:24.391 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:54:24.391 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:54:24.392 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:54:24.392 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:54:24.392 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:54:24.392 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.2", 2024-08-18T15:54:24.392 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:54:24.392 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:54:24.392 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:54:24.392 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:54:24.392 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:54:24.392 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:54:24.392 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:54:24.392 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000045", 2024-08-18T15:54:24.392 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000002", 2024-08-18T15:54:24.393 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:54:24.393 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3.2UbiLidwSTy.0", 2024-08-18T15:54:24.393 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:54:24.393 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.12", 2024-08-18T15:54:24.393 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:54:24.393 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:54:24.393 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:54:24.393 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:54:24.393 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:54:24.393 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:54:24.393 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000116", 2024-08-18T15:54:24.393 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.25", 2024-08-18T15:54:24.393 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:54:24.394 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:54:24.394 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:54:24.394 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:54:24.394 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000034", 2024-08-18T15:54:24.394 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:54:24.394 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:54:24.394 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000101", 2024-08-18T15:54:24.394 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:54:24.394 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:54:24.394 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.13", 2024-08-18T15:54:24.394 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:54:24.394 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:54:24.395 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.1", 2024-08-18T15:54:24.395 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0.retry", 2024-08-18T15:54:24.395 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:54:24.395 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:54:24.395 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000079", 2024-08-18T15:54:24.395 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000049", 2024-08-18T15:54:24.395 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:54:24.395 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0", 2024-08-18T15:54:24.395 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:54:24.395 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:54:24.395 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000117", 2024-08-18T15:54:24.395 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:54:24.395 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000044", 2024-08-18T15:54:24.396 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:54:24.396 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:54:24.396 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:54:24.396 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:54:24.396 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:54:24.396 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:54:24.396 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.14", 2024-08-18T15:54:24.396 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:54:24.396 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000066", 2024-08-18T15:54:24.396 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:54:24.396 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:54:24.396 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:54:24.396 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:54:24.397 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:54:24.397 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000068", 2024-08-18T15:54:24.397 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:54:24.397 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000085", 2024-08-18T15:54:24.397 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:54:24.397 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:54:24.397 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:54:24.397 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:54:24.397 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:54:24.397 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:54:24.397 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:54:24.397 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:54:24.397 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:54:24.397 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:54:24.398 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:54:24.398 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:54:24.398 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000073", 2024-08-18T15:54:24.398 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:54:24.398 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:54:24.398 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:54:24.398 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000038", 2024-08-18T15:54:24.398 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:54:24.398 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:54:24.398 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:54:24.398 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:54:24.398 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.10", 2024-08-18T15:54:24.398 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:54:24.399 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:54:24.399 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:54:24.399 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000118", 2024-08-18T15:54:24.399 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:54:24.399 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3", 2024-08-18T15:54:24.399 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:54:24.399 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000036", 2024-08-18T15:54:24.399 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:54:24.399 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:54:24.399 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:54:24.399 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:54:24.399 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:54:24.399 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000103", 2024-08-18T15:54:24.400 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.3", 2024-08-18T15:54:24.400 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:54:24.400 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000119", 2024-08-18T15:54:24.400 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000098", 2024-08-18T15:54:24.400 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:54:24.400 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:54:24.400 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:54:24.400 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:54:24.400 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:54:24.400 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:54:24.400 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:54:24.400 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:54:24.401 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.0", 2024-08-18T15:54:24.401 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:54:24.401 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:54:24.401 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:54:24.401 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:54:24.401 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:54:24.401 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:54:24.401 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:54:24.401 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:54:24.401 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000027", 2024-08-18T15:54:24.401 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:54:24.401 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status", 2024-08-18T15:54:24.401 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:54:24.401 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000019", 2024-08-18T15:54:24.402 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:54:24.402 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:54:24.402 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:54:24.402 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:54:24.402 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1", 2024-08-18T15:54:24.402 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:54:24.402 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:54:24.402 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:54:24.402 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:54:24.402 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000039", 2024-08-18T15:54:24.402 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:54:24.402 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:54:24.403 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:54:24.403 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:54:24.403 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:54:24.403 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:54:24.403 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:54:24.403 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:54:24.403 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:54:24.403 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:54:24.403 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:54:24.403 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000100", 2024-08-18T15:54:24.403 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:54:24.403 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:54:24.403 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:54:24.403 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:54:24.404 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:54:24.404 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000093", 2024-08-18T15:54:24.404 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:54:24.404 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000004", 2024-08-18T15:54:24.404 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000063", 2024-08-18T15:54:24.404 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000122", 2024-08-18T15:54:24.404 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:54:24.404 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:54:24.404 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:54:24.404 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000057", 2024-08-18T15:54:24.404 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:54:24.404 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:54:24.404 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:54:24.405 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:54:24.405 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000054", 2024-08-18T15:54:24.405 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:54:24.405 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000114", 2024-08-18T15:54:24.405 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:54:24.405 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:54:24.405 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:54:24.405 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.2", 2024-08-18T15:54:24.405 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:54:24.405 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:54:24.405 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0", 2024-08-18T15:54:24.405 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:54:24.405 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:54:24.406 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:54:24.406 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:54:24.406 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:54:24.406 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:54:24.406 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000012", 2024-08-18T15:54:24.406 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:54:24.406 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:54:24.406 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:54:24.406 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000084", 2024-08-18T15:54:24.406 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000043", 2024-08-18T15:54:24.406 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3", 2024-08-18T15:54:24.406 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:54:24.406 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:54:24.407 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.11", 2024-08-18T15:54:24.407 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:54:24.407 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:54:24.407 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:54:24.407 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:54:24.407 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:54:24.407 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:54:24.407 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:54:24.407 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:54:24.407 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000111", 2024-08-18T15:54:24.407 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:54:24.407 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000048", 2024-08-18T15:54:24.407 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:54:24.407 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000020", 2024-08-18T15:54:24.408 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:54:24.408 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:54:24.408 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:54:24.408 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:54:24.408 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:54:24.408 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000099", 2024-08-18T15:54:24.408 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000056", 2024-08-18T15:54:24.408 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:54:24.408 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:54:24.408 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.19", 2024-08-18T15:54:24.408 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:54:24.408 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.28", 2024-08-18T15:54:24.408 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:54:24.408 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:54:24.409 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:54:24.409 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:54:24.409 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:54:24.409 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000072", 2024-08-18T15:54:24.409 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:54:24.409 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:54:24.409 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000062", 2024-08-18T15:54:24.409 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:54:24.409 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:54:24.409 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:54:24.409 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000115", 2024-08-18T15:54:24.409 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:54:24.409 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.1", 2024-08-18T15:54:24.410 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2", 2024-08-18T15:54:24.410 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000033", 2024-08-18T15:54:24.410 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:54:24.410 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:54:24.410 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:54:24.410 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:54:24.410 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:54:24.410 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:54:24.410 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:54:24.410 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:54:24.410 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:54:24.410 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:54:24.410 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:54:24.411 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:54:24.411 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:54:24.411 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:54:24.411 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:54:24.411 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:54:24.411 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:54:24.411 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:54:24.411 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:54:24.411 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000025", 2024-08-18T15:54:24.411 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:54:24.411 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000123", 2024-08-18T15:54:24.411 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.31", 2024-08-18T15:54:24.411 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:54:24.411 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:54:24.412 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:54:24.412 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:54:24.412 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.18", 2024-08-18T15:54:24.412 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:54:24.412 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:54:24.412 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:54:24.412 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:54:24.412 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:54:24.412 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000108", 2024-08-18T15:54:24.412 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:54:24.412 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:54:24.412 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000094", 2024-08-18T15:54:24.412 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:54:24.413 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:54:24.413 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:54:24.413 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:54:24.413 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:54:24.413 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:54:24.413 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000060", 2024-08-18T15:54:24.413 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:54:24.413 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:54:24.413 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:54:24.413 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000109", 2024-08-18T15:54:24.413 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.15", 2024-08-18T15:54:24.413 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:54:24.413 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:54:24.414 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:54:24.414 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:54:24.414 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:54:24.414 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:54:24.414 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:54:24.414 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:54:24.414 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:54:24.414 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000013", 2024-08-18T15:54:24.414 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:54:24.414 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:54:24.414 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:54:24.414 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:54:24.415 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:54:24.415 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:54:24.415 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:54:24.415 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:54:24.415 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:54:24.415 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:54:24.415 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:54:24.415 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:54:24.415 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:54:24.415 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:54:24.415 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000051", 2024-08-18T15:54:24.415 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:54:24.415 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:54:24.416 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:54:24.416 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:54:24.416 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000106", 2024-08-18T15:54:24.416 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:54:24.416 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:54:24.416 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:54:24.416 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:54:24.416 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:54:24.416 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000017", 2024-08-18T15:54:24.416 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:54:24.416 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:54:24.416 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:54:24.416 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000023", 2024-08-18T15:54:24.416 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:54:24.417 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:54:24.417 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:54:24.417 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:54:24.417 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:54:24.417 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:54:24.417 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:54:24.417 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000107", 2024-08-18T15:54:24.417 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:54:24.417 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:54:24.417 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000046", 2024-08-18T15:54:24.417 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:54:24.417 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:54:24.417 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:54:24.418 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.26", 2024-08-18T15:54:24.418 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:54:24.418 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:54:24.418 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9", 2024-08-18T15:54:24.418 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.22", 2024-08-18T15:54:24.418 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:54:24.418 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:54:24.418 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:54:24.418 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:54:24.418 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:54:24.418 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:54:24.418 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:54:24.418 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2", 2024-08-18T15:54:24.419 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:54:24.419 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:54:24.419 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000067", 2024-08-18T15:54:24.419 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:54:24.419 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:54:24.419 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:54:24.419 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:54:24.419 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:54:24.419 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:54:24.419 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:54:24.419 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:54:24.419 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:54:24.419 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:54:24.420 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:54:24.420 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:54:24.420 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:54:24.420 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:54:24.420 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:54:24.420 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:54:24.420 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:54:24.420 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:54:24.420 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:54:24.420 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.9", 2024-08-18T15:54:24.420 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000086", 2024-08-18T15:54:24.420 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.5", 2024-08-18T15:54:24.420 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:54:24.421 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:54:24.421 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:54:24.421 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:54:24.421 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:54:24.421 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:54:24.421 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2.MFXJ583gCr1.0", 2024-08-18T15:54:24.421 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:54:24.421 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1.GPnO16liyZ+.0", 2024-08-18T15:54:24.421 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:54:24.421 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:54:24.421 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:54:24.421 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:54:24.421 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:54:24.422 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:54:24.422 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:54:24.422 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:54:24.422 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:54:24.422 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.20", 2024-08-18T15:54:24.422 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:54:24.422 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:54:24.422 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:54:24.422 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:54:24.422 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:54:24.422 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:54:24.422 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1.retry", 2024-08-18T15:54:24.422 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:54:24.422 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:54:24.423 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:54:24.423 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0.retry", 2024-08-18T15:54:24.423 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000050", 2024-08-18T15:54:24.423 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:54:24.423 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000102", 2024-08-18T15:54:24.423 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:54:24.423 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:54:24.423 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:54:24.423 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:54:24.423 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:54:24.423 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:54:24.423 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:54:24.423 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:54:24.424 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:54:24.424 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:54:24.424 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:54:24.424 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000124", 2024-08-18T15:54:24.424 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:54:24.424 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:54:24.424 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:54:24.424 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000001", 2024-08-18T15:54:24.424 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:54:24.424 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:54:24.424 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:54:24.424 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:54:24.424 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000126", 2024-08-18T15:54:24.425 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:54:24.425 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0.fZhctfLXnSt.0", 2024-08-18T15:54:24.425 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:54:24.425 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000055", 2024-08-18T15:54:24.425 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:54:24.425 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000061", 2024-08-18T15:54:24.425 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1", 2024-08-18T15:54:24.425 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:54:24.425 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:54:24.425 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:54:24.425 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000097", 2024-08-18T15:54:24.425 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:54:24.425 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:54:24.426 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:54:24.426 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:54:24.426 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:54:24.426 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:54:24.426 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:54:24.426 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:54:24.426 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000042", 2024-08-18T15:54:24.426 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:54:24.426 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:54:24.426 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:54:24.426 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1.retry", 2024-08-18T15:54:24.426 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:54:24.426 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:54:24.427 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:54:24.427 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:54:24.427 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:54:24.427 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:54:24.427 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:54:24.427 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000075", 2024-08-18T15:54:24.427 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:54:24.427 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000120", 2024-08-18T15:54:24.427 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000030", 2024-08-18T15:54:24.427 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:54:24.427 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:54:24.427 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:54:24.427 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:54:24.428 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:54:24.428 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:54:24.428 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9" 2024-08-18T15:54:24.428 INFO:teuthology.orchestra.run.smithi193.stdout:] 2024-08-18T15:54:24.428 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:54:29.430 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:54:29.431 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:54:29.431 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:54:29.483 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:54:29.484 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:54:29.548 INFO:teuthology.orchestra.run.smithi193.stdout:[ 2024-08-18T15:54:29.548 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:54:29.548 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:54:29.548 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.29", 2024-08-18T15:54:29.549 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000078", 2024-08-18T15:54:29.549 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.history", 2024-08-18T15:54:29.549 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:54:29.549 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:54:29.549 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:54:29.550 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:54:29.550 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.23", 2024-08-18T15:54:29.550 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000070", 2024-08-18T15:54:29.550 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000104", 2024-08-18T15:54:29.550 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:54:29.551 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2.retry", 2024-08-18T15:54:29.551 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:54:29.551 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:54:29.551 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:54:29.551 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:54:29.551 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000026", 2024-08-18T15:54:29.552 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:54:29.552 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:54:29.552 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:54:29.552 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:54:29.552 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:54:29.552 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:54:29.553 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.8", 2024-08-18T15:54:29.553 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:54:29.553 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:54:29.553 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:54:29.553 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:54:29.554 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000028", 2024-08-18T15:54:29.554 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:54:29.554 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:54:29.554 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:54:29.554 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:54:29.554 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:54:29.555 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:54:29.555 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:54:29.555 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2.retry", 2024-08-18T15:54:29.555 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:54:29.555 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.30", 2024-08-18T15:54:29.556 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:54:29.556 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:54:29.556 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:54:29.556 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:54:29.556 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000040", 2024-08-18T15:54:29.557 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:54:29.557 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.6", 2024-08-18T15:54:29.557 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:54:29.557 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:54:29.557 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:54:29.557 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:54:29.557 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:54:29.558 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:54:29.558 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000015", 2024-08-18T15:54:29.558 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:54:29.558 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:54:29.558 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000069", 2024-08-18T15:54:29.558 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:54:29.559 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:54:29.559 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:54:29.559 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000095", 2024-08-18T15:54:29.559 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0", 2024-08-18T15:54:29.559 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000003", 2024-08-18T15:54:29.559 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000047", 2024-08-18T15:54:29.559 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:54:29.559 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:54:29.559 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:54:29.559 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:54:29.559 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:54:29.559 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:54:29.559 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:54:29.560 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:54:29.560 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:54:29.560 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000035", 2024-08-18T15:54:29.560 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:54:29.560 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:54:29.560 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000037", 2024-08-18T15:54:29.560 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:54:29.560 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:54:29.561 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:54:29.561 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:54:29.561 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:54:29.561 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:54:29.561 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:54:29.561 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:54:29.562 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:54:29.562 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:54:29.562 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:54:29.562 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:54:29.562 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:54:29.562 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:54:29.562 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000024", 2024-08-18T15:54:29.562 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:54:29.562 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:54:29.562 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:54:29.563 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:54:29.563 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000000", 2024-08-18T15:54:29.563 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:54:29.563 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000031", 2024-08-18T15:54:29.563 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:54:29.563 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:54:29.563 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:54:29.563 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:54:29.563 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:54:29.563 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:54:29.563 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3", 2024-08-18T15:54:29.563 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:54:29.563 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:54:29.564 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:54:29.564 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:54:29.564 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:54:29.564 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:54:29.564 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:54:29.564 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:54:29.564 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:54:29.564 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:54:29.564 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:54:29.564 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:54:29.564 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:54:29.564 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:54:29.564 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:54:29.564 INFO:teuthology.orchestra.run.smithi193.stdout: "data_loggenerations_metadata", 2024-08-18T15:54:29.565 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2", 2024-08-18T15:54:29.565 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:54:29.565 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:54:29.565 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:54:29.565 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:54:29.565 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T15:54:29.565 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:54:29.565 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:54:29.565 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:54:29.565 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:54:29.565 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:54:29.565 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000076", 2024-08-18T15:54:29.565 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:54:29.566 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:54:29.566 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000113", 2024-08-18T15:54:29.566 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000005", 2024-08-18T15:54:29.566 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:54:29.566 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000011", 2024-08-18T15:54:29.566 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:54:29.566 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000058", 2024-08-18T15:54:29.566 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:54:29.566 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:54:29.566 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:54:29.566 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000052", 2024-08-18T15:54:29.566 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:54:29.566 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:54:29.567 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:54:29.567 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:54:29.567 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.21", 2024-08-18T15:54:29.567 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:54:29.567 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:54:29.567 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:54:29.567 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:54:29.567 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:54:29.567 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:54:29.567 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:54:29.567 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:54:29.568 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:54:29.568 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:54:29.568 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:54:29.568 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:54:29.568 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:54:29.568 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:54:29.568 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:54:29.568 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:54:29.568 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:54:29.568 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:54:29.568 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:54:29.568 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000088", 2024-08-18T15:54:29.568 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000080", 2024-08-18T15:54:29.569 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:54:29.569 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:54:29.569 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000090", 2024-08-18T15:54:29.569 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:54:29.569 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:54:29.569 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000110", 2024-08-18T15:54:29.569 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:54:29.569 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:54:29.569 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:54:29.569 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.0", 2024-08-18T15:54:29.569 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:54:29.569 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000096", 2024-08-18T15:54:29.569 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:54:29.570 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:54:29.570 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:54:29.570 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:54:29.570 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:54:29.570 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000087", 2024-08-18T15:54:29.570 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:54:29.570 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:54:29.570 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:54:29.570 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:54:29.570 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000008", 2024-08-18T15:54:29.570 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:54:29.570 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:54:29.570 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:54:29.571 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:54:29.571 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:54:29.571 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000006", 2024-08-18T15:54:29.571 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000029", 2024-08-18T15:54:29.571 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:54:29.571 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000089", 2024-08-18T15:54:29.571 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:54:29.571 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:54:29.571 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000022", 2024-08-18T15:54:29.571 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:54:29.571 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:54:29.571 INFO:teuthology.orchestra.run.smithi193.stdout: "bilog.trim", 2024-08-18T15:54:29.571 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000016", 2024-08-18T15:54:29.572 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:54:29.572 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:54:29.572 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:54:29.572 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:54:29.572 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:54:29.572 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000041", 2024-08-18T15:54:29.572 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:54:29.572 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:54:29.572 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.log.01050e30-179b-43be-bdbb-3bb23cf37ae8.3", 2024-08-18T15:54:29.572 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:54:29.572 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000018", 2024-08-18T15:54:29.572 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000092", 2024-08-18T15:54:29.572 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:54:29.572 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:54:29.573 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:54:29.573 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:54:29.573 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.27", 2024-08-18T15:54:29.573 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.4", 2024-08-18T15:54:29.573 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:54:29.573 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:54:29.573 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:54:29.573 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:54:29.573 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:54:29.573 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:54:29.573 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:54:29.573 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000014", 2024-08-18T15:54:29.573 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:54:29.574 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3.retry", 2024-08-18T15:54:29.574 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:54:29.574 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:54:29.574 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:54:29.574 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:54:29.574 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:54:29.574 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000112", 2024-08-18T15:54:29.574 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:54:29.574 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000007", 2024-08-18T15:54:29.574 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000021", 2024-08-18T15:54:29.575 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:54:29.575 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:54:29.575 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:54:29.575 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:54:29.575 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:54:29.575 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.16", 2024-08-18T15:54:29.575 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:54:29.575 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000064", 2024-08-18T15:54:29.575 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:54:29.575 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:54:29.575 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:54:29.575 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:54:29.575 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:54:29.576 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:54:29.576 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:54:29.576 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000071", 2024-08-18T15:54:29.576 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:54:29.576 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:54:29.576 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:54:29.576 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000074", 2024-08-18T15:54:29.576 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:54:29.576 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:54:29.576 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:54:29.576 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:54:29.576 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:54:29.576 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:54:29.577 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:54:29.577 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:54:29.577 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:54:29.577 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:54:29.577 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:54:29.577 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:54:29.577 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:54:29.577 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:54:29.577 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:54:29.577 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000081", 2024-08-18T15:54:29.577 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:54:29.577 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000009", 2024-08-18T15:54:29.577 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000121", 2024-08-18T15:54:29.577 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1", 2024-08-18T15:54:29.578 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:54:29.578 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:54:29.578 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:54:29.578 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:54:29.578 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:54:29.578 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:54:29.578 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.17", 2024-08-18T15:54:29.578 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000125", 2024-08-18T15:54:29.578 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:54:29.578 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000082", 2024-08-18T15:54:29.578 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:54:29.578 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:54:29.578 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:54:29.579 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:54:29.579 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:54:29.579 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:54:29.579 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:54:29.579 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:54:29.579 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:54:29.579 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:54:29.579 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:54:29.579 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000105", 2024-08-18T15:54:29.579 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000059", 2024-08-18T15:54:29.579 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:54:29.579 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:54:29.579 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:54:29.580 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:54:29.580 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:54:29.580 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:54:29.580 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:54:29.580 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:54:29.580 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:54:29.580 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:54:29.580 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:54:29.580 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000077", 2024-08-18T15:54:29.580 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000032", 2024-08-18T15:54:29.580 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:54:29.580 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:54:29.580 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:54:29.580 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:54:29.581 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:54:29.581 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:54:29.581 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.24", 2024-08-18T15:54:29.581 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:54:29.581 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:54:29.581 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:54:29.581 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:54:29.581 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3.retry", 2024-08-18T15:54:29.581 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:54:29.581 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:54:29.581 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:54:29.582 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:54:29.582 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:54:29.582 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:54:29.582 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000053", 2024-08-18T15:54:29.582 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000091", 2024-08-18T15:54:29.582 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:54:29.582 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:54:29.582 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:54:29.582 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000065", 2024-08-18T15:54:29.582 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:54:29.583 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:54:29.583 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:54:29.583 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:54:29.583 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000083", 2024-08-18T15:54:29.583 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000010", 2024-08-18T15:54:29.583 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:54:29.583 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:54:29.583 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:54:29.583 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:54:29.583 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:54:29.583 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:54:29.583 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:54:29.583 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:54:29.583 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:54:29.584 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:54:29.584 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:54:29.584 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.7", 2024-08-18T15:54:29.584 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.3", 2024-08-18T15:54:29.584 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:54:29.584 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:54:29.584 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:54:29.584 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:54:29.584 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:54:29.584 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:54:29.584 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:54:29.584 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:54:29.584 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.2", 2024-08-18T15:54:29.585 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:54:29.585 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:54:29.585 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:54:29.585 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:54:29.585 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:54:29.585 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:54:29.585 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:54:29.585 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000045", 2024-08-18T15:54:29.585 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000002", 2024-08-18T15:54:29.585 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:54:29.585 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3.2UbiLidwSTy.0", 2024-08-18T15:54:29.585 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:54:29.585 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.12", 2024-08-18T15:54:29.585 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:54:29.586 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:54:29.586 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:54:29.586 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:54:29.586 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:54:29.586 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:54:29.586 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000116", 2024-08-18T15:54:29.586 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.25", 2024-08-18T15:54:29.586 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:54:29.586 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:54:29.586 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:54:29.586 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:54:29.586 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000034", 2024-08-18T15:54:29.587 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:54:29.587 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:54:29.587 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000101", 2024-08-18T15:54:29.587 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:54:29.587 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:54:29.587 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.13", 2024-08-18T15:54:29.587 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:54:29.587 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:54:29.587 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.1", 2024-08-18T15:54:29.587 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0.retry", 2024-08-18T15:54:29.587 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:54:29.587 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:54:29.587 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000079", 2024-08-18T15:54:29.588 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000049", 2024-08-18T15:54:29.588 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:54:29.588 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0", 2024-08-18T15:54:29.588 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:54:29.588 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:54:29.588 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000117", 2024-08-18T15:54:29.588 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:54:29.588 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000044", 2024-08-18T15:54:29.588 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:54:29.588 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:54:29.588 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:54:29.588 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:54:29.588 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:54:29.588 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:54:29.589 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.14", 2024-08-18T15:54:29.589 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:54:29.589 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000066", 2024-08-18T15:54:29.589 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:54:29.589 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:54:29.589 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:54:29.589 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:54:29.589 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:54:29.589 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000068", 2024-08-18T15:54:29.589 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:54:29.589 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000085", 2024-08-18T15:54:29.589 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:54:29.589 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:54:29.590 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:54:29.590 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:54:29.590 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:54:29.590 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:54:29.590 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:54:29.590 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:54:29.590 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:54:29.590 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:54:29.590 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:54:29.590 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:54:29.590 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000073", 2024-08-18T15:54:29.590 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:54:29.591 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:54:29.591 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:54:29.591 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000038", 2024-08-18T15:54:29.591 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:54:29.591 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:54:29.591 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:54:29.591 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:54:29.591 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.10", 2024-08-18T15:54:29.591 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:54:29.591 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:54:29.592 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:54:29.592 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000118", 2024-08-18T15:54:29.592 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:54:29.592 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3", 2024-08-18T15:54:29.592 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:54:29.592 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000036", 2024-08-18T15:54:29.592 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:54:29.592 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:54:29.592 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:54:29.592 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:54:29.592 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:54:29.592 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000103", 2024-08-18T15:54:29.592 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.3", 2024-08-18T15:54:29.592 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:54:29.593 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000119", 2024-08-18T15:54:29.593 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000098", 2024-08-18T15:54:29.593 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:54:29.593 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:54:29.593 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:54:29.593 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:54:29.593 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:54:29.593 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:54:29.593 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:54:29.593 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:54:29.593 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.0", 2024-08-18T15:54:29.593 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:54:29.594 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:54:29.594 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:54:29.594 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:54:29.594 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:54:29.594 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:54:29.594 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:54:29.594 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:54:29.594 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000027", 2024-08-18T15:54:29.594 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:54:29.594 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status", 2024-08-18T15:54:29.594 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:54:29.594 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000019", 2024-08-18T15:54:29.595 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:54:29.595 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:54:29.595 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:54:29.595 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:54:29.595 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1", 2024-08-18T15:54:29.595 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:54:29.595 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:54:29.595 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:54:29.595 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:54:29.595 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000039", 2024-08-18T15:54:29.595 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:54:29.595 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:54:29.595 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:54:29.596 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:54:29.596 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:54:29.596 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:54:29.596 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:54:29.596 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:54:29.596 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:54:29.596 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:54:29.596 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:54:29.596 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000100", 2024-08-18T15:54:29.596 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:54:29.596 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:54:29.596 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:54:29.596 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:54:29.596 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:54:29.597 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000093", 2024-08-18T15:54:29.597 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:54:29.598 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000004", 2024-08-18T15:54:29.598 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000063", 2024-08-18T15:54:29.598 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000122", 2024-08-18T15:54:29.598 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:54:29.598 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:54:29.598 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:54:29.598 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000057", 2024-08-18T15:54:29.598 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:54:29.598 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:54:29.598 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:54:29.599 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:54:29.599 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000054", 2024-08-18T15:54:29.599 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:54:29.599 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000114", 2024-08-18T15:54:29.599 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:54:29.599 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:54:29.599 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:54:29.599 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.2", 2024-08-18T15:54:29.600 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:54:29.600 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:54:29.600 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0", 2024-08-18T15:54:29.600 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:54:29.600 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:54:29.600 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:54:29.600 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:54:29.600 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:54:29.600 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:54:29.601 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000012", 2024-08-18T15:54:29.601 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:54:29.601 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:54:29.601 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:54:29.601 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000084", 2024-08-18T15:54:29.601 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000043", 2024-08-18T15:54:29.601 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3", 2024-08-18T15:54:29.601 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:54:29.601 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:54:29.601 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.11", 2024-08-18T15:54:29.601 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:54:29.601 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:54:29.601 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:54:29.602 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:54:29.602 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:54:29.602 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:54:29.602 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:54:29.602 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:54:29.602 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000111", 2024-08-18T15:54:29.602 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:54:29.602 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000048", 2024-08-18T15:54:29.602 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:54:29.602 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000020", 2024-08-18T15:54:29.602 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:54:29.602 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:54:29.603 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:54:29.603 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:54:29.603 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:54:29.603 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000099", 2024-08-18T15:54:29.603 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000056", 2024-08-18T15:54:29.603 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:54:29.603 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:54:29.603 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.19", 2024-08-18T15:54:29.603 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:54:29.603 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.28", 2024-08-18T15:54:29.604 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:54:29.604 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:54:29.604 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:54:29.604 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:54:29.604 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:54:29.604 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000072", 2024-08-18T15:54:29.605 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:54:29.605 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:54:29.605 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000062", 2024-08-18T15:54:29.605 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:54:29.605 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:54:29.605 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:54:29.605 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000115", 2024-08-18T15:54:29.605 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:54:29.606 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.1", 2024-08-18T15:54:29.606 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2", 2024-08-18T15:54:29.606 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000033", 2024-08-18T15:54:29.606 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:54:29.606 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:54:29.606 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:54:29.606 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:54:29.606 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:54:29.606 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:54:29.606 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:54:29.606 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:54:29.607 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:54:29.607 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:54:29.607 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:54:29.607 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:54:29.607 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:54:29.607 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:54:29.607 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:54:29.607 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:54:29.607 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:54:29.607 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:54:29.607 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:54:29.607 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000025", 2024-08-18T15:54:29.608 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:54:29.608 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000123", 2024-08-18T15:54:29.608 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.31", 2024-08-18T15:54:29.608 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:54:29.608 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:54:29.608 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:54:29.608 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:54:29.608 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.18", 2024-08-18T15:54:29.608 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:54:29.608 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:54:29.609 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:54:29.609 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:54:29.609 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:54:29.609 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000108", 2024-08-18T15:54:29.609 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:54:29.609 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:54:29.609 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000094", 2024-08-18T15:54:29.609 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:54:29.609 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:54:29.609 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:54:29.609 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:54:29.609 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:54:29.609 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:54:29.610 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000060", 2024-08-18T15:54:29.610 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:54:29.610 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:54:29.610 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:54:29.610 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000109", 2024-08-18T15:54:29.610 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.15", 2024-08-18T15:54:29.610 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:54:29.610 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:54:29.610 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:54:29.610 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:54:29.610 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:54:29.610 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:54:29.611 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:54:29.611 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:54:29.611 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:54:29.611 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000013", 2024-08-18T15:54:29.611 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:54:29.611 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:54:29.611 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:54:29.611 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:54:29.611 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:54:29.611 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:54:29.611 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:54:29.611 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:54:29.611 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:54:29.612 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:54:29.612 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:54:29.612 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:54:29.612 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:54:29.612 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:54:29.612 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000051", 2024-08-18T15:54:29.612 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:54:29.612 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:54:29.612 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:54:29.612 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:54:29.612 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000106", 2024-08-18T15:54:29.612 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:54:29.612 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:54:29.613 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:54:29.613 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:54:29.613 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:54:29.613 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000017", 2024-08-18T15:54:29.613 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:54:29.613 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:54:29.613 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:54:29.613 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000023", 2024-08-18T15:54:29.613 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:54:29.613 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:54:29.613 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:54:29.613 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:54:29.613 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:54:29.614 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:54:29.614 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:54:29.614 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000107", 2024-08-18T15:54:29.614 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:54:29.614 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:54:29.614 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000046", 2024-08-18T15:54:29.614 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:54:29.614 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:54:29.614 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:54:29.614 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.26", 2024-08-18T15:54:29.614 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:54:29.614 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:54:29.615 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9", 2024-08-18T15:54:29.615 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.22", 2024-08-18T15:54:29.615 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:54:29.615 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:54:29.615 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:54:29.615 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:54:29.615 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:54:29.615 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:54:29.615 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:54:29.615 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2", 2024-08-18T15:54:29.615 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:54:29.616 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:54:29.616 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000067", 2024-08-18T15:54:29.616 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:54:29.616 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:54:29.616 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:54:29.616 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:54:29.617 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:54:29.617 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:54:29.617 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:54:29.617 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:54:29.617 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:54:29.617 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:54:29.618 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:54:29.618 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:54:29.618 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:54:29.618 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:54:29.618 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:54:29.618 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:54:29.619 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:54:29.619 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:54:29.619 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:54:29.619 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.9", 2024-08-18T15:54:29.619 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000086", 2024-08-18T15:54:29.619 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.5", 2024-08-18T15:54:29.620 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:54:29.620 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:54:29.620 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:54:29.620 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:54:29.620 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:54:29.620 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:54:29.620 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2.MFXJ583gCr1.0", 2024-08-18T15:54:29.621 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:54:29.621 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1.GPnO16liyZ+.0", 2024-08-18T15:54:29.621 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:54:29.621 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:54:29.621 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:54:29.621 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:54:29.621 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:54:29.622 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:54:29.622 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:54:29.622 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:54:29.622 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:54:29.622 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.20", 2024-08-18T15:54:29.622 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:54:29.623 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:54:29.623 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:54:29.623 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:54:29.623 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:54:29.623 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:54:29.623 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1.retry", 2024-08-18T15:54:29.624 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:54:29.624 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:54:29.624 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:54:29.624 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0.retry", 2024-08-18T15:54:29.624 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000050", 2024-08-18T15:54:29.624 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:54:29.624 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000102", 2024-08-18T15:54:29.625 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:54:29.625 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:54:29.625 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:54:29.625 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:54:29.625 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:54:29.625 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:54:29.625 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:54:29.626 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:54:29.626 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:54:29.626 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:54:29.626 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:54:29.626 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000124", 2024-08-18T15:54:29.626 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:54:29.627 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:54:29.627 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:54:29.627 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000001", 2024-08-18T15:54:29.627 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:54:29.627 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:54:29.628 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:54:29.628 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:54:29.628 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000126", 2024-08-18T15:54:29.628 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:54:29.628 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0.fZhctfLXnSt.0", 2024-08-18T15:54:29.628 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:54:29.629 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000055", 2024-08-18T15:54:29.629 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:54:29.629 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000061", 2024-08-18T15:54:29.629 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1", 2024-08-18T15:54:29.629 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:54:29.630 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:54:29.630 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:54:29.630 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000097", 2024-08-18T15:54:29.630 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:54:29.630 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:54:29.631 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:54:29.631 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:54:29.631 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:54:29.631 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:54:29.631 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:54:29.631 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:54:29.631 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000042", 2024-08-18T15:54:29.631 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:54:29.632 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:54:29.632 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:54:29.632 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1.retry", 2024-08-18T15:54:29.632 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:54:29.632 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:54:29.632 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:54:29.632 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:54:29.632 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:54:29.633 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:54:29.633 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:54:29.633 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000075", 2024-08-18T15:54:29.633 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:54:29.633 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000120", 2024-08-18T15:54:29.633 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000030", 2024-08-18T15:54:29.633 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:54:29.633 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:54:29.634 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:54:29.634 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:54:29.634 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:54:29.634 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:54:29.634 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9" 2024-08-18T15:54:29.634 INFO:teuthology.orchestra.run.smithi193.stdout:] 2024-08-18T15:54:29.634 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:54:34.637 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:54:34.637 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:54:34.638 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:54:34.691 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:54:34.691 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:54:34.753 INFO:teuthology.orchestra.run.smithi193.stdout:[ 2024-08-18T15:54:34.753 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:54:34.753 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:54:34.753 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.29", 2024-08-18T15:54:34.753 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000078", 2024-08-18T15:54:34.753 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.history", 2024-08-18T15:54:34.754 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:54:34.754 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:54:34.754 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:54:34.754 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:54:34.754 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.23", 2024-08-18T15:54:34.754 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000070", 2024-08-18T15:54:34.754 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000104", 2024-08-18T15:54:34.754 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:54:34.754 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2.retry", 2024-08-18T15:54:34.754 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:54:34.754 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:54:34.754 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:54:34.755 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:54:34.755 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000026", 2024-08-18T15:54:34.755 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:54:34.755 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:54:34.755 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:54:34.755 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:54:34.755 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:54:34.755 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:54:34.755 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.8", 2024-08-18T15:54:34.755 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:54:34.755 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:54:34.755 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:54:34.756 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:54:34.756 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000028", 2024-08-18T15:54:34.756 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:54:34.756 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:54:34.756 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:54:34.756 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:54:34.756 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:54:34.756 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:54:34.756 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:54:34.756 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2.retry", 2024-08-18T15:54:34.757 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:54:34.757 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.30", 2024-08-18T15:54:34.757 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:54:34.757 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:54:34.757 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:54:34.757 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:54:34.757 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000040", 2024-08-18T15:54:34.757 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:54:34.757 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.6", 2024-08-18T15:54:34.757 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:54:34.758 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:54:34.758 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:54:34.758 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:54:34.758 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:54:34.758 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:54:34.758 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000015", 2024-08-18T15:54:34.758 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:54:34.758 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:54:34.758 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000069", 2024-08-18T15:54:34.758 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:54:34.758 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:54:34.758 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:54:34.759 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000095", 2024-08-18T15:54:34.759 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0", 2024-08-18T15:54:34.759 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000003", 2024-08-18T15:54:34.759 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000047", 2024-08-18T15:54:34.759 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:54:34.759 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:54:34.759 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:54:34.759 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:54:34.759 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:54:34.759 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:54:34.760 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:54:34.760 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:54:34.760 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:54:34.760 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000035", 2024-08-18T15:54:34.760 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:54:34.760 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:54:34.760 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000037", 2024-08-18T15:54:34.760 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:54:34.760 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:54:34.760 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:54:34.760 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:54:34.760 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:54:34.761 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:54:34.761 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:54:34.761 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:54:34.761 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:54:34.761 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:54:34.761 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:54:34.761 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:54:34.761 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:54:34.761 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:54:34.761 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000024", 2024-08-18T15:54:34.761 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:54:34.761 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:54:34.762 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:54:34.762 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:54:34.762 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000000", 2024-08-18T15:54:34.762 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:54:34.762 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000031", 2024-08-18T15:54:34.762 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:54:34.762 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:54:34.762 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:54:34.762 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:54:34.762 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:54:34.762 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:54:34.763 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3", 2024-08-18T15:54:34.763 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:54:34.763 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:54:34.763 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:54:34.763 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:54:34.763 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:54:34.763 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:54:34.763 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:54:34.763 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:54:34.763 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:54:34.764 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:54:34.764 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:54:34.764 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:54:34.764 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:54:34.764 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:54:34.764 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:54:34.764 INFO:teuthology.orchestra.run.smithi193.stdout: "data_loggenerations_metadata", 2024-08-18T15:54:34.764 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2", 2024-08-18T15:54:34.764 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:54:34.764 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:54:34.765 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:54:34.765 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:54:34.765 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T15:54:34.765 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:54:34.765 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:54:34.765 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:54:34.765 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:54:34.765 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:54:34.765 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000076", 2024-08-18T15:54:34.766 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:54:34.766 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:54:34.766 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000113", 2024-08-18T15:54:34.766 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000005", 2024-08-18T15:54:34.766 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:54:34.766 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000011", 2024-08-18T15:54:34.766 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:54:34.766 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000058", 2024-08-18T15:54:34.766 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:54:34.766 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:54:34.766 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:54:34.766 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000052", 2024-08-18T15:54:34.767 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:54:34.767 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:54:34.767 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:54:34.767 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:54:34.767 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.21", 2024-08-18T15:54:34.767 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:54:34.767 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:54:34.767 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:54:34.767 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:54:34.767 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:54:34.767 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:54:34.767 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:54:34.768 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:54:34.768 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:54:34.768 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:54:34.768 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:54:34.768 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:54:34.768 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:54:34.768 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:54:34.768 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:54:34.768 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:54:34.768 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:54:34.769 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:54:34.769 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:54:34.769 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000088", 2024-08-18T15:54:34.769 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000080", 2024-08-18T15:54:34.769 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:54:34.769 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:54:34.769 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000090", 2024-08-18T15:54:34.769 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:54:34.769 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:54:34.769 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000110", 2024-08-18T15:54:34.769 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:54:34.769 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:54:34.770 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:54:34.770 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.0", 2024-08-18T15:54:34.770 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:54:34.770 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000096", 2024-08-18T15:54:34.770 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:54:34.770 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:54:34.770 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:54:34.770 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:54:34.770 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:54:34.770 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000087", 2024-08-18T15:54:34.770 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:54:34.771 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:54:34.771 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:54:34.771 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:54:34.771 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000008", 2024-08-18T15:54:34.771 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:54:34.771 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:54:34.771 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:54:34.771 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:54:34.771 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:54:34.771 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000006", 2024-08-18T15:54:34.771 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000029", 2024-08-18T15:54:34.771 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:54:34.771 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000089", 2024-08-18T15:54:34.771 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:54:34.771 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:54:34.772 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000022", 2024-08-18T15:54:34.772 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:54:34.772 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:54:34.772 INFO:teuthology.orchestra.run.smithi193.stdout: "bilog.trim", 2024-08-18T15:54:34.772 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000016", 2024-08-18T15:54:34.772 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:54:34.772 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:54:34.772 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:54:34.772 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:54:34.772 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:54:34.772 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000041", 2024-08-18T15:54:34.772 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:54:34.772 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:54:34.772 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.log.01050e30-179b-43be-bdbb-3bb23cf37ae8.3", 2024-08-18T15:54:34.773 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:54:34.773 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000018", 2024-08-18T15:54:34.773 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000092", 2024-08-18T15:54:34.773 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:54:34.773 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:54:34.773 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:54:34.773 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:54:34.773 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.27", 2024-08-18T15:54:34.773 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.4", 2024-08-18T15:54:34.773 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:54:34.773 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:54:34.773 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:54:34.773 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:54:34.773 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:54:34.774 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:54:34.774 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:54:34.774 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000014", 2024-08-18T15:54:34.774 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:54:34.774 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3.retry", 2024-08-18T15:54:34.774 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:54:34.774 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:54:34.774 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:54:34.774 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:54:34.774 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:54:34.775 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000112", 2024-08-18T15:54:34.775 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:54:34.775 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000007", 2024-08-18T15:54:34.775 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000021", 2024-08-18T15:54:34.775 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:54:34.775 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:54:34.775 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:54:34.775 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:54:34.775 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:54:34.775 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.16", 2024-08-18T15:54:34.775 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:54:34.775 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000064", 2024-08-18T15:54:34.776 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:54:34.776 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:54:34.776 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:54:34.776 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:54:34.776 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:54:34.776 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:54:34.776 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:54:34.776 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000071", 2024-08-18T15:54:34.776 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:54:34.776 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:54:34.776 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:54:34.776 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000074", 2024-08-18T15:54:34.776 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:54:34.777 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:54:34.777 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:54:34.777 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:54:34.777 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:54:34.777 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:54:34.777 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:54:34.777 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:54:34.777 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:54:34.777 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:54:34.777 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:54:34.777 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:54:34.777 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:54:34.778 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:54:34.778 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:54:34.778 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000081", 2024-08-18T15:54:34.778 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:54:34.778 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000009", 2024-08-18T15:54:34.778 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000121", 2024-08-18T15:54:34.778 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1", 2024-08-18T15:54:34.778 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:54:34.778 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:54:34.778 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:54:34.778 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:54:34.778 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:54:34.779 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:54:34.779 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.17", 2024-08-18T15:54:34.779 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000125", 2024-08-18T15:54:34.779 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:54:34.779 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000082", 2024-08-18T15:54:34.779 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:54:34.779 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:54:34.779 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:54:34.779 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:54:34.779 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:54:34.779 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:54:34.779 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:54:34.779 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:54:34.780 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:54:34.780 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:54:34.780 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:54:34.780 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000105", 2024-08-18T15:54:34.780 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000059", 2024-08-18T15:54:34.780 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:54:34.780 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:54:34.780 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:54:34.780 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:54:34.780 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:54:34.780 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:54:34.780 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:54:34.780 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:54:34.781 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:54:34.781 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:54:34.781 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:54:34.781 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000077", 2024-08-18T15:54:34.781 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000032", 2024-08-18T15:54:34.781 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:54:34.781 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:54:34.781 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:54:34.781 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:54:34.781 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:54:34.781 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:54:34.781 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.24", 2024-08-18T15:54:34.781 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:54:34.782 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:54:34.782 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:54:34.782 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:54:34.782 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3.retry", 2024-08-18T15:54:34.782 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:54:34.782 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:54:34.782 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:54:34.782 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:54:34.782 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:54:34.782 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:54:34.782 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000053", 2024-08-18T15:54:34.782 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000091", 2024-08-18T15:54:34.782 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:54:34.783 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:54:34.783 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:54:34.783 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000065", 2024-08-18T15:54:34.783 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:54:34.783 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:54:34.783 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:54:34.783 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:54:34.783 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000083", 2024-08-18T15:54:34.783 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000010", 2024-08-18T15:54:34.783 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:54:34.783 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:54:34.783 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:54:34.784 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:54:34.784 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:54:34.784 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:54:34.784 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:54:34.784 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:54:34.784 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:54:34.784 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:54:34.784 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:54:34.784 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.7", 2024-08-18T15:54:34.784 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.3", 2024-08-18T15:54:34.784 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:54:34.785 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:54:34.785 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:54:34.785 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:54:34.785 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:54:34.785 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:54:34.785 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:54:34.785 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:54:34.785 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.2", 2024-08-18T15:54:34.785 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:54:34.785 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:54:34.785 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:54:34.786 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:54:34.786 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:54:34.786 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:54:34.786 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:54:34.786 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000045", 2024-08-18T15:54:34.786 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000002", 2024-08-18T15:54:34.786 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:54:34.786 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3.2UbiLidwSTy.0", 2024-08-18T15:54:34.786 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:54:34.786 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.12", 2024-08-18T15:54:34.786 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:54:34.786 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:54:34.786 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:54:34.787 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:54:34.787 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:54:34.787 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:54:34.787 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000116", 2024-08-18T15:54:34.787 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.25", 2024-08-18T15:54:34.787 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:54:34.787 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:54:34.787 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:54:34.787 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:54:34.787 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000034", 2024-08-18T15:54:34.787 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:54:34.787 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:54:34.788 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000101", 2024-08-18T15:54:34.788 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:54:34.788 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:54:34.788 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.13", 2024-08-18T15:54:34.788 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:54:34.788 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:54:34.788 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.1", 2024-08-18T15:54:34.788 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0.retry", 2024-08-18T15:54:34.788 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:54:34.788 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:54:34.788 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000079", 2024-08-18T15:54:34.788 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000049", 2024-08-18T15:54:34.789 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:54:34.789 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0", 2024-08-18T15:54:34.789 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:54:34.789 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:54:34.789 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000117", 2024-08-18T15:54:34.789 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:54:34.789 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000044", 2024-08-18T15:54:34.789 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:54:34.789 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:54:34.789 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:54:34.789 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:54:34.789 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:54:34.790 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:54:34.790 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.14", 2024-08-18T15:54:34.790 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:54:34.790 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000066", 2024-08-18T15:54:34.790 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:54:34.790 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:54:34.790 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:54:34.790 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:54:34.790 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:54:34.790 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000068", 2024-08-18T15:54:34.790 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:54:34.790 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000085", 2024-08-18T15:54:34.791 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:54:34.791 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:54:34.791 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:54:34.791 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:54:34.791 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:54:34.791 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:54:34.791 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:54:34.791 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:54:34.791 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:54:34.791 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:54:34.791 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:54:34.792 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:54:34.792 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000073", 2024-08-18T15:54:34.792 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:54:34.792 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:54:34.792 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:54:34.792 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000038", 2024-08-18T15:54:34.792 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:54:34.792 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:54:34.792 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:54:34.792 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:54:34.792 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.10", 2024-08-18T15:54:34.792 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:54:34.792 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:54:34.793 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:54:34.793 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000118", 2024-08-18T15:54:34.793 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:54:34.793 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3", 2024-08-18T15:54:34.793 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:54:34.793 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000036", 2024-08-18T15:54:34.793 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:54:34.793 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:54:34.793 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:54:34.793 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:54:34.793 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:54:34.793 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000103", 2024-08-18T15:54:34.793 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.3", 2024-08-18T15:54:34.794 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:54:34.794 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000119", 2024-08-18T15:54:34.794 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000098", 2024-08-18T15:54:34.794 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:54:34.794 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:54:34.794 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:54:34.794 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:54:34.794 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:54:34.794 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:54:34.794 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:54:34.794 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:54:34.794 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.0", 2024-08-18T15:54:34.795 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:54:34.795 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:54:34.795 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:54:34.795 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:54:34.795 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:54:34.795 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:54:34.795 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:54:34.795 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:54:34.795 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000027", 2024-08-18T15:54:34.795 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:54:34.795 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status", 2024-08-18T15:54:34.795 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:54:34.795 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000019", 2024-08-18T15:54:34.796 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:54:34.796 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:54:34.796 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:54:34.796 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:54:34.796 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1", 2024-08-18T15:54:34.796 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:54:34.796 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:54:34.796 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:54:34.796 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:54:34.796 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000039", 2024-08-18T15:54:34.796 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:54:34.796 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:54:34.797 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:54:34.797 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:54:34.797 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:54:34.797 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:54:34.797 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:54:34.797 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:54:34.797 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:54:34.797 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:54:34.797 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:54:34.797 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000100", 2024-08-18T15:54:34.797 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:54:34.797 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:54:34.797 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:54:34.798 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:54:34.798 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:54:34.798 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000093", 2024-08-18T15:54:34.798 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:54:34.798 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000004", 2024-08-18T15:54:34.798 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000063", 2024-08-18T15:54:34.798 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000122", 2024-08-18T15:54:34.798 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:54:34.798 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:54:34.798 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:54:34.798 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000057", 2024-08-18T15:54:34.798 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:54:34.798 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:54:34.799 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:54:34.799 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:54:34.799 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000054", 2024-08-18T15:54:34.799 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:54:34.799 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000114", 2024-08-18T15:54:34.799 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:54:34.799 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:54:34.799 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:54:34.799 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.2", 2024-08-18T15:54:34.799 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:54:34.799 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:54:34.799 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0", 2024-08-18T15:54:34.799 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:54:34.800 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:54:34.800 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:54:34.800 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:54:34.800 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:54:34.800 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:54:34.800 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000012", 2024-08-18T15:54:34.800 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:54:34.800 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:54:34.800 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:54:34.800 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000084", 2024-08-18T15:54:34.800 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000043", 2024-08-18T15:54:34.800 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3", 2024-08-18T15:54:34.800 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:54:34.801 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:54:34.801 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.11", 2024-08-18T15:54:34.801 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:54:34.801 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:54:34.801 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:54:34.801 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:54:34.801 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:54:34.801 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:54:34.801 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:54:34.801 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:54:34.801 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000111", 2024-08-18T15:54:34.802 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:54:34.802 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000048", 2024-08-18T15:54:34.802 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:54:34.802 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000020", 2024-08-18T15:54:34.802 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:54:34.802 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:54:34.802 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:54:34.802 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:54:34.802 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:54:34.802 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000099", 2024-08-18T15:54:34.802 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000056", 2024-08-18T15:54:34.802 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:54:34.803 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:54:34.803 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.19", 2024-08-18T15:54:34.803 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:54:34.803 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.28", 2024-08-18T15:54:34.803 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:54:34.803 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:54:34.803 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:54:34.803 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:54:34.803 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:54:34.803 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000072", 2024-08-18T15:54:34.803 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:54:34.803 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:54:34.803 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000062", 2024-08-18T15:54:34.804 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:54:34.804 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:54:34.804 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:54:34.804 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000115", 2024-08-18T15:54:34.804 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:54:34.804 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.1", 2024-08-18T15:54:34.804 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2", 2024-08-18T15:54:34.804 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000033", 2024-08-18T15:54:34.804 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:54:34.804 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:54:34.804 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:54:34.804 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:54:34.805 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:54:34.805 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:54:34.805 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:54:34.805 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:54:34.805 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:54:34.805 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:54:34.805 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:54:34.805 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:54:34.805 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:54:34.805 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:54:34.805 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:54:34.805 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:54:34.805 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:54:34.806 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:54:34.806 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:54:34.806 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000025", 2024-08-18T15:54:34.806 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:54:34.806 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000123", 2024-08-18T15:54:34.806 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.31", 2024-08-18T15:54:34.806 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:54:34.806 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:54:34.806 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:54:34.806 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:54:34.806 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.18", 2024-08-18T15:54:34.806 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:54:34.807 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:54:34.807 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:54:34.807 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:54:34.807 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:54:34.807 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000108", 2024-08-18T15:54:34.807 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:54:34.807 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:54:34.807 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000094", 2024-08-18T15:54:34.807 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:54:34.807 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:54:34.807 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:54:34.807 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:54:34.807 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:54:34.808 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:54:34.808 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000060", 2024-08-18T15:54:34.808 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:54:34.808 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:54:34.808 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:54:34.808 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000109", 2024-08-18T15:54:34.808 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.15", 2024-08-18T15:54:34.808 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:54:34.808 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:54:34.808 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:54:34.809 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:54:34.809 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:54:34.809 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:54:34.809 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:54:34.809 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:54:34.809 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:54:34.809 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000013", 2024-08-18T15:54:34.809 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:54:34.809 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:54:34.809 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:54:34.809 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:54:34.809 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:54:34.809 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:54:34.810 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:54:34.810 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:54:34.810 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:54:34.810 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:54:34.810 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:54:34.810 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:54:34.810 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:54:34.810 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:54:34.810 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000051", 2024-08-18T15:54:34.810 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:54:34.810 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:54:34.810 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:54:34.810 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:54:34.810 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000106", 2024-08-18T15:54:34.811 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:54:34.811 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:54:34.811 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:54:34.811 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:54:34.811 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:54:34.811 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000017", 2024-08-18T15:54:34.811 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:54:34.811 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:54:34.811 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:54:34.811 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000023", 2024-08-18T15:54:34.811 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:54:34.811 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:54:34.812 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:54:34.812 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:54:34.812 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:54:34.812 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:54:34.812 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:54:34.812 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000107", 2024-08-18T15:54:34.812 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:54:34.812 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:54:34.812 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000046", 2024-08-18T15:54:34.812 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:54:34.812 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:54:34.812 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:54:34.812 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.26", 2024-08-18T15:54:34.813 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:54:34.813 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:54:34.813 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9", 2024-08-18T15:54:34.813 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.22", 2024-08-18T15:54:34.813 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:54:34.813 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:54:34.813 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:54:34.813 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:54:34.813 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:54:34.813 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:54:34.813 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:54:34.813 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2", 2024-08-18T15:54:34.814 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:54:34.814 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:54:34.814 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000067", 2024-08-18T15:54:34.814 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:54:34.814 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:54:34.814 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:54:34.814 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:54:34.814 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:54:34.814 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:54:34.814 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:54:34.814 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:54:34.814 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:54:34.815 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:54:34.815 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:54:34.815 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:54:34.815 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:54:34.815 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:54:34.815 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:54:34.815 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:54:34.815 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:54:34.815 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:54:34.815 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:54:34.815 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.9", 2024-08-18T15:54:34.815 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000086", 2024-08-18T15:54:34.815 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.5", 2024-08-18T15:54:34.816 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:54:34.816 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:54:34.816 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:54:34.816 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:54:34.816 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:54:34.816 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:54:34.816 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2.MFXJ583gCr1.0", 2024-08-18T15:54:34.816 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:54:34.816 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1.GPnO16liyZ+.0", 2024-08-18T15:54:34.816 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:54:34.816 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:54:34.816 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:54:34.816 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:54:34.816 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:54:34.817 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:54:34.817 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:54:34.817 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:54:34.817 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:54:34.817 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.20", 2024-08-18T15:54:34.817 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:54:34.817 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:54:34.817 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:54:34.817 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:54:34.817 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:54:34.817 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:54:34.817 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1.retry", 2024-08-18T15:54:34.818 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:54:34.818 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:54:34.818 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:54:34.818 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0.retry", 2024-08-18T15:54:34.818 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000050", 2024-08-18T15:54:34.818 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:54:34.818 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000102", 2024-08-18T15:54:34.818 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:54:34.818 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:54:34.818 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:54:34.818 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:54:34.818 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:54:34.819 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:54:34.819 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:54:34.819 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:54:34.819 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:54:34.819 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:54:34.819 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:54:34.819 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000124", 2024-08-18T15:54:34.819 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:54:34.819 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:54:34.819 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:54:34.819 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000001", 2024-08-18T15:54:34.820 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:54:34.820 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:54:34.820 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:54:34.820 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:54:34.820 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000126", 2024-08-18T15:54:34.820 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:54:34.820 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0.fZhctfLXnSt.0", 2024-08-18T15:54:34.820 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:54:34.820 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000055", 2024-08-18T15:54:34.820 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:54:34.820 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000061", 2024-08-18T15:54:34.820 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1", 2024-08-18T15:54:34.820 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:54:34.821 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:54:34.821 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:54:34.821 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000097", 2024-08-18T15:54:34.821 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:54:34.821 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:54:34.821 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:54:34.821 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:54:34.821 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:54:34.821 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:54:34.821 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:54:34.821 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:54:34.821 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000042", 2024-08-18T15:54:34.821 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:54:34.822 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:54:34.822 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:54:34.822 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1.retry", 2024-08-18T15:54:34.822 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:54:34.822 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:54:34.822 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:54:34.822 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:54:34.822 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:54:34.822 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:54:34.822 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:54:34.822 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000075", 2024-08-18T15:54:34.822 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:54:34.823 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000120", 2024-08-18T15:54:34.823 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000030", 2024-08-18T15:54:34.823 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:54:34.823 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:54:34.823 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:54:34.823 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:54:34.823 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:54:34.823 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:54:34.823 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9" 2024-08-18T15:54:34.823 INFO:teuthology.orchestra.run.smithi193.stdout:] 2024-08-18T15:54:34.823 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:54:39.825 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:54:39.826 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:54:39.826 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:54:39.878 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:54:39.878 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:54:39.943 INFO:teuthology.orchestra.run.smithi193.stdout:[ 2024-08-18T15:54:39.943 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:54:39.943 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:54:39.943 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.29", 2024-08-18T15:54:39.943 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000078", 2024-08-18T15:54:39.943 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.history", 2024-08-18T15:54:39.943 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:54:39.943 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:54:39.943 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:54:39.944 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:54:39.944 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.23", 2024-08-18T15:54:39.944 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000070", 2024-08-18T15:54:39.944 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000104", 2024-08-18T15:54:39.944 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:54:39.944 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2.retry", 2024-08-18T15:54:39.944 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:54:39.944 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:54:39.944 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:54:39.944 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:54:39.944 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000026", 2024-08-18T15:54:39.944 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:54:39.944 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:54:39.944 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:54:39.945 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:54:39.945 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:54:39.945 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:54:39.945 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.8", 2024-08-18T15:54:39.945 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:54:39.945 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:54:39.945 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:54:39.945 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:54:39.946 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000028", 2024-08-18T15:54:39.946 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:54:39.946 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:54:39.946 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:54:39.946 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:54:39.946 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:54:39.946 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:54:39.946 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:54:39.946 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2.retry", 2024-08-18T15:54:39.946 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:54:39.946 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.30", 2024-08-18T15:54:39.946 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:54:39.946 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:54:39.947 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:54:39.947 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:54:39.947 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000040", 2024-08-18T15:54:39.947 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:54:39.947 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.6", 2024-08-18T15:54:39.947 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:54:39.947 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:54:39.947 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:54:39.947 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:54:39.948 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:54:39.948 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:54:39.948 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000015", 2024-08-18T15:54:39.948 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:54:39.948 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:54:39.948 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000069", 2024-08-18T15:54:39.948 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:54:39.948 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:54:39.948 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:54:39.948 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000095", 2024-08-18T15:54:39.949 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0", 2024-08-18T15:54:39.949 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000003", 2024-08-18T15:54:39.949 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000047", 2024-08-18T15:54:39.949 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:54:39.949 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:54:39.949 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:54:39.949 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:54:39.949 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:54:39.949 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:54:39.949 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:54:39.949 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:54:39.949 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:54:39.950 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000035", 2024-08-18T15:54:39.950 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:54:39.950 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:54:39.950 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000037", 2024-08-18T15:54:39.950 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:54:39.950 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:54:39.950 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:54:39.950 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:54:39.950 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:54:39.950 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:54:39.950 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:54:39.950 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:54:39.950 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:54:39.951 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:54:39.951 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:54:39.951 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:54:39.951 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:54:39.951 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:54:39.951 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000024", 2024-08-18T15:54:39.951 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:54:39.951 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:54:39.951 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:54:39.951 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:54:39.951 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000000", 2024-08-18T15:54:39.951 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:54:39.951 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000031", 2024-08-18T15:54:39.952 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:54:39.952 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:54:39.952 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:54:39.952 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:54:39.952 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:54:39.952 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:54:39.952 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3", 2024-08-18T15:54:39.952 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:54:39.952 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:54:39.952 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:54:39.952 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:54:39.952 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:54:39.952 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:54:39.953 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:54:39.953 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:54:39.953 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:54:39.953 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:54:39.953 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:54:39.953 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:54:39.953 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:54:39.953 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:54:39.953 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:54:39.953 INFO:teuthology.orchestra.run.smithi193.stdout: "data_loggenerations_metadata", 2024-08-18T15:54:39.953 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2", 2024-08-18T15:54:39.953 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:54:39.954 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:54:39.954 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:54:39.954 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:54:39.954 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T15:54:39.954 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:54:39.954 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:54:39.954 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:54:39.954 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:54:39.954 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:54:39.954 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000076", 2024-08-18T15:54:39.954 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:54:39.954 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:54:39.955 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000113", 2024-08-18T15:54:39.955 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000005", 2024-08-18T15:54:39.955 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:54:39.955 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000011", 2024-08-18T15:54:39.955 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:54:39.955 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000058", 2024-08-18T15:54:39.955 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:54:39.955 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:54:39.955 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:54:39.955 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000052", 2024-08-18T15:54:39.955 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:54:39.955 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:54:39.955 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:54:39.956 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:54:39.956 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.21", 2024-08-18T15:54:39.956 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:54:39.956 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:54:39.956 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:54:39.956 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:54:39.956 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:54:39.956 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:54:39.956 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:54:39.957 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:54:39.957 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:54:39.957 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:54:39.957 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:54:39.957 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:54:39.957 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:54:39.957 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:54:39.957 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:54:39.957 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:54:39.957 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:54:39.957 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:54:39.957 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:54:39.957 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000088", 2024-08-18T15:54:39.958 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000080", 2024-08-18T15:54:39.958 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:54:39.958 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:54:39.958 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000090", 2024-08-18T15:54:39.958 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:54:39.958 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:54:39.958 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000110", 2024-08-18T15:54:39.958 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:54:39.958 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:54:39.958 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:54:39.958 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.0", 2024-08-18T15:54:39.958 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:54:39.958 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000096", 2024-08-18T15:54:39.959 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:54:39.959 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:54:39.959 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:54:39.959 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:54:39.959 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:54:39.959 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000087", 2024-08-18T15:54:39.959 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:54:39.959 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:54:39.959 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:54:39.959 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:54:39.959 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000008", 2024-08-18T15:54:39.959 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:54:39.959 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:54:39.959 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:54:39.960 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:54:39.960 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:54:39.960 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000006", 2024-08-18T15:54:39.960 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000029", 2024-08-18T15:54:39.960 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:54:39.960 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000089", 2024-08-18T15:54:39.960 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:54:39.960 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:54:39.960 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000022", 2024-08-18T15:54:39.960 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:54:39.960 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:54:39.960 INFO:teuthology.orchestra.run.smithi193.stdout: "bilog.trim", 2024-08-18T15:54:39.960 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000016", 2024-08-18T15:54:39.961 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:54:39.961 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:54:39.961 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:54:39.961 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:54:39.961 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:54:39.961 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000041", 2024-08-18T15:54:39.961 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:54:39.961 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:54:39.961 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.log.01050e30-179b-43be-bdbb-3bb23cf37ae8.3", 2024-08-18T15:54:39.961 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:54:39.961 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000018", 2024-08-18T15:54:39.961 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000092", 2024-08-18T15:54:39.961 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:54:39.961 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:54:39.962 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:54:39.962 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:54:39.962 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.27", 2024-08-18T15:54:39.962 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.4", 2024-08-18T15:54:39.962 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:54:39.962 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:54:39.962 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:54:39.962 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:54:39.962 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:54:39.962 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:54:39.963 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:54:39.963 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000014", 2024-08-18T15:54:39.963 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:54:39.963 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3.retry", 2024-08-18T15:54:39.963 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:54:39.963 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:54:39.963 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:54:39.963 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:54:39.963 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:54:39.963 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000112", 2024-08-18T15:54:39.963 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:54:39.963 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000007", 2024-08-18T15:54:39.963 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000021", 2024-08-18T15:54:39.964 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:54:39.964 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:54:39.964 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:54:39.964 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:54:39.964 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:54:39.964 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.16", 2024-08-18T15:54:39.964 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:54:39.964 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000064", 2024-08-18T15:54:39.964 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:54:39.964 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:54:39.964 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:54:39.964 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:54:39.964 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:54:39.964 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:54:39.965 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:54:39.965 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000071", 2024-08-18T15:54:39.965 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:54:39.965 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:54:39.965 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:54:39.965 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000074", 2024-08-18T15:54:39.965 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:54:39.965 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:54:39.965 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:54:39.965 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:54:39.965 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:54:39.965 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:54:39.965 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:54:39.965 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:54:39.966 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:54:39.966 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:54:39.966 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:54:39.966 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:54:39.966 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:54:39.966 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:54:39.966 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:54:39.966 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000081", 2024-08-18T15:54:39.966 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:54:39.966 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000009", 2024-08-18T15:54:39.966 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000121", 2024-08-18T15:54:39.966 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1", 2024-08-18T15:54:39.966 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:54:39.967 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:54:39.967 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:54:39.967 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:54:39.967 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:54:39.967 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:54:39.967 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.17", 2024-08-18T15:54:39.967 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000125", 2024-08-18T15:54:39.967 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:54:39.967 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000082", 2024-08-18T15:54:39.967 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:54:39.967 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:54:39.967 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:54:39.967 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:54:39.967 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:54:39.968 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:54:39.968 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:54:39.968 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:54:39.968 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:54:39.968 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:54:39.968 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:54:39.968 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000105", 2024-08-18T15:54:39.968 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000059", 2024-08-18T15:54:39.968 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:54:39.969 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:54:39.969 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:54:39.969 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:54:39.969 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:54:39.969 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:54:39.969 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:54:39.969 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:54:39.969 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:54:39.969 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:54:39.969 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:54:39.970 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000077", 2024-08-18T15:54:39.970 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000032", 2024-08-18T15:54:39.970 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:54:39.970 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:54:39.970 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:54:39.970 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:54:39.970 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:54:39.970 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:54:39.971 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.24", 2024-08-18T15:54:39.971 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:54:39.971 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:54:39.971 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:54:39.971 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:54:39.971 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3.retry", 2024-08-18T15:54:39.971 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:54:39.971 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:54:39.971 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:54:39.971 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:54:39.972 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:54:39.972 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:54:39.972 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000053", 2024-08-18T15:54:39.972 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000091", 2024-08-18T15:54:39.972 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:54:39.972 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:54:39.972 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:54:39.972 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000065", 2024-08-18T15:54:39.972 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:54:39.972 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:54:39.972 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:54:39.973 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:54:39.973 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000083", 2024-08-18T15:54:39.973 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000010", 2024-08-18T15:54:39.973 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:54:39.973 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:54:39.973 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:54:39.973 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:54:39.973 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:54:39.973 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:54:39.973 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:54:39.973 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:54:39.973 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:54:39.973 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:54:39.974 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:54:39.974 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.7", 2024-08-18T15:54:39.974 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.3", 2024-08-18T15:54:39.974 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:54:39.974 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:54:39.974 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:54:39.974 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:54:39.974 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:54:39.974 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:54:39.974 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:54:39.974 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:54:39.974 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.2", 2024-08-18T15:54:39.974 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:54:39.975 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:54:39.975 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:54:39.975 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:54:39.975 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:54:39.975 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:54:39.975 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:54:39.975 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000045", 2024-08-18T15:54:39.975 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000002", 2024-08-18T15:54:39.975 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:54:39.975 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3.2UbiLidwSTy.0", 2024-08-18T15:54:39.975 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:54:39.975 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.12", 2024-08-18T15:54:39.975 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:54:39.976 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:54:39.976 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:54:39.976 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:54:39.976 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:54:39.976 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:54:39.976 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000116", 2024-08-18T15:54:39.976 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.25", 2024-08-18T15:54:39.976 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:54:39.976 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:54:39.976 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:54:39.976 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:54:39.976 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000034", 2024-08-18T15:54:39.976 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:54:39.977 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:54:39.977 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000101", 2024-08-18T15:54:39.977 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:54:39.977 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:54:39.977 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.13", 2024-08-18T15:54:39.977 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:54:39.977 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:54:39.977 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.1", 2024-08-18T15:54:39.977 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0.retry", 2024-08-18T15:54:39.977 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:54:39.977 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:54:39.977 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000079", 2024-08-18T15:54:39.977 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000049", 2024-08-18T15:54:39.978 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:54:39.978 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0", 2024-08-18T15:54:39.978 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:54:39.978 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:54:39.978 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000117", 2024-08-18T15:54:39.978 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:54:39.978 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000044", 2024-08-18T15:54:39.978 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:54:39.978 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:54:39.978 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:54:39.978 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:54:39.978 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:54:39.979 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:54:39.979 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.14", 2024-08-18T15:54:39.979 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:54:39.979 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000066", 2024-08-18T15:54:39.979 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:54:39.979 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:54:39.979 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:54:39.979 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:54:39.979 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:54:39.979 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000068", 2024-08-18T15:54:39.979 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:54:39.979 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000085", 2024-08-18T15:54:39.979 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:54:39.979 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:54:39.980 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:54:39.980 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:54:39.980 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:54:39.980 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:54:39.980 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:54:39.980 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:54:39.980 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:54:39.980 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:54:39.980 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:54:39.980 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:54:39.980 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000073", 2024-08-18T15:54:39.980 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:54:39.980 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:54:39.981 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:54:39.981 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000038", 2024-08-18T15:54:39.981 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:54:39.981 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:54:39.981 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:54:39.981 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:54:39.981 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.10", 2024-08-18T15:54:39.981 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:54:39.981 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:54:39.981 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:54:39.981 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000118", 2024-08-18T15:54:39.981 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:54:39.981 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3", 2024-08-18T15:54:39.982 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:54:39.982 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000036", 2024-08-18T15:54:39.982 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:54:39.982 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:54:39.982 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:54:39.982 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:54:39.982 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:54:39.982 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000103", 2024-08-18T15:54:39.982 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.3", 2024-08-18T15:54:39.982 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:54:39.982 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000119", 2024-08-18T15:54:39.983 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000098", 2024-08-18T15:54:39.983 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:54:39.983 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:54:39.983 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:54:39.983 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:54:39.983 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:54:39.983 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:54:39.983 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:54:39.983 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:54:39.983 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.0", 2024-08-18T15:54:39.983 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:54:39.983 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:54:39.983 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:54:39.984 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:54:39.984 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:54:39.984 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:54:39.984 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:54:39.984 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:54:39.984 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000027", 2024-08-18T15:54:39.984 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:54:39.984 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status", 2024-08-18T15:54:39.984 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:54:39.984 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000019", 2024-08-18T15:54:39.984 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:54:39.984 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:54:39.984 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:54:39.985 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:54:39.985 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1", 2024-08-18T15:54:39.985 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:54:39.985 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:54:39.985 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:54:39.985 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:54:39.985 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000039", 2024-08-18T15:54:39.985 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:54:39.985 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:54:39.985 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:54:39.985 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:54:39.985 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:54:39.985 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:54:39.986 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:54:39.986 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:54:39.986 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:54:39.986 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:54:39.986 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:54:39.986 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000100", 2024-08-18T15:54:39.986 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:54:39.986 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:54:39.986 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:54:39.986 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:54:39.986 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:54:39.986 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000093", 2024-08-18T15:54:39.987 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:54:39.987 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000004", 2024-08-18T15:54:39.987 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000063", 2024-08-18T15:54:39.987 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000122", 2024-08-18T15:54:39.987 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:54:39.987 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:54:39.987 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:54:39.987 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000057", 2024-08-18T15:54:39.987 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:54:39.987 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:54:39.987 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:54:39.987 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:54:39.987 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000054", 2024-08-18T15:54:39.987 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:54:39.988 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000114", 2024-08-18T15:54:39.988 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:54:39.988 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:54:39.988 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:54:39.988 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.2", 2024-08-18T15:54:39.988 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:54:39.988 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:54:39.988 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0", 2024-08-18T15:54:39.988 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:54:39.988 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:54:39.988 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:54:39.988 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:54:39.988 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:54:39.988 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:54:39.989 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000012", 2024-08-18T15:54:39.989 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:54:39.989 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:54:39.989 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:54:39.989 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000084", 2024-08-18T15:54:39.989 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000043", 2024-08-18T15:54:39.989 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3", 2024-08-18T15:54:39.989 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:54:39.989 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:54:39.989 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.11", 2024-08-18T15:54:39.989 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:54:39.989 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:54:39.989 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:54:39.990 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:54:39.990 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:54:39.990 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:54:39.990 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:54:39.990 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:54:39.990 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000111", 2024-08-18T15:54:39.990 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:54:39.990 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000048", 2024-08-18T15:54:39.990 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:54:39.990 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000020", 2024-08-18T15:54:39.990 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:54:39.990 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:54:39.991 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:54:39.991 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:54:39.991 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:54:39.991 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000099", 2024-08-18T15:54:39.991 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000056", 2024-08-18T15:54:39.991 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:54:39.991 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:54:39.991 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.19", 2024-08-18T15:54:39.991 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:54:39.991 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.28", 2024-08-18T15:54:39.991 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:54:39.991 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:54:39.991 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:54:39.991 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:54:39.992 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:54:39.992 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000072", 2024-08-18T15:54:39.992 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:54:39.992 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:54:39.992 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000062", 2024-08-18T15:54:39.992 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:54:39.992 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:54:39.992 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:54:39.992 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000115", 2024-08-18T15:54:39.992 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:54:39.992 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.1", 2024-08-18T15:54:39.992 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2", 2024-08-18T15:54:39.992 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000033", 2024-08-18T15:54:39.993 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:54:39.993 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:54:39.993 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:54:39.993 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:54:39.993 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:54:39.993 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:54:39.993 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:54:39.993 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:54:39.993 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:54:39.993 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:54:39.993 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:54:39.993 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:54:39.993 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:54:39.994 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:54:39.994 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:54:39.994 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:54:39.994 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:54:39.994 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:54:39.994 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:54:39.994 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000025", 2024-08-18T15:54:39.994 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:54:39.994 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000123", 2024-08-18T15:54:39.994 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.31", 2024-08-18T15:54:39.994 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:54:39.994 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:54:39.994 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:54:39.994 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:54:39.995 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.18", 2024-08-18T15:54:39.995 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:54:39.995 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:54:39.995 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:54:39.995 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:54:39.995 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:54:39.995 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000108", 2024-08-18T15:54:39.995 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:54:39.995 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:54:39.995 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000094", 2024-08-18T15:54:39.995 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:54:39.995 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:54:39.995 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:54:39.995 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:54:39.996 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:54:39.996 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:54:39.996 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000060", 2024-08-18T15:54:39.996 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:54:39.996 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:54:39.996 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:54:39.996 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000109", 2024-08-18T15:54:39.996 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.15", 2024-08-18T15:54:39.996 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:54:39.996 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:54:39.996 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:54:39.996 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:54:39.996 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:54:39.997 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:54:39.997 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:54:39.997 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:54:39.997 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:54:39.997 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000013", 2024-08-18T15:54:39.997 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:54:39.997 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:54:39.997 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:54:39.997 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:54:39.997 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:54:39.997 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:54:39.997 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:54:39.997 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:54:39.997 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:54:39.998 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:54:39.998 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:54:39.998 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:54:39.998 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:54:39.998 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:54:39.998 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000051", 2024-08-18T15:54:39.998 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:54:39.998 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:54:39.998 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:54:39.998 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:54:39.998 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000106", 2024-08-18T15:54:39.998 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:54:39.999 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:54:39.999 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:54:39.999 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:54:39.999 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:54:39.999 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000017", 2024-08-18T15:54:39.999 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:54:39.999 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:54:39.999 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:54:39.999 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000023", 2024-08-18T15:54:39.999 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:54:39.999 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:54:39.999 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:54:39.999 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:54:39.999 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:54:40.000 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:54:40.000 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:54:40.000 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000107", 2024-08-18T15:54:40.000 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:54:40.000 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:54:40.000 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000046", 2024-08-18T15:54:40.000 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:54:40.000 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:54:40.000 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:54:40.000 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.26", 2024-08-18T15:54:40.000 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:54:40.000 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:54:40.000 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9", 2024-08-18T15:54:40.000 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.22", 2024-08-18T15:54:40.001 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:54:40.001 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:54:40.001 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:54:40.001 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:54:40.001 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:54:40.001 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:54:40.001 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:54:40.001 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2", 2024-08-18T15:54:40.001 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:54:40.001 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:54:40.001 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000067", 2024-08-18T15:54:40.001 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:54:40.001 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:54:40.001 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:54:40.002 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:54:40.002 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:54:40.002 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:54:40.002 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:54:40.002 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:54:40.002 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:54:40.002 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:54:40.002 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:54:40.002 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:54:40.002 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:54:40.002 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:54:40.002 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:54:40.002 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:54:40.003 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:54:40.003 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:54:40.003 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:54:40.003 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.9", 2024-08-18T15:54:40.003 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000086", 2024-08-18T15:54:40.003 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.5", 2024-08-18T15:54:40.003 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:54:40.003 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:54:40.003 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:54:40.003 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:54:40.003 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:54:40.004 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:54:40.004 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2.MFXJ583gCr1.0", 2024-08-18T15:54:40.004 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:54:40.004 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1.GPnO16liyZ+.0", 2024-08-18T15:54:40.004 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:54:40.004 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:54:40.004 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:54:40.004 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:54:40.004 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:54:40.004 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:54:40.004 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:54:40.004 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:54:40.005 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:54:40.005 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.20", 2024-08-18T15:54:40.005 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:54:40.005 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:54:40.005 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:54:40.005 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:54:40.005 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:54:40.005 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:54:40.005 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1.retry", 2024-08-18T15:54:40.005 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:54:40.005 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:54:40.005 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:54:40.006 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0.retry", 2024-08-18T15:54:40.006 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000050", 2024-08-18T15:54:40.006 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:54:40.006 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000102", 2024-08-18T15:54:40.006 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:54:40.006 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:54:40.006 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:54:40.006 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:54:40.006 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:54:40.006 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:54:40.006 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:54:40.006 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:54:40.006 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:54:40.007 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:54:40.007 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:54:40.007 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000124", 2024-08-18T15:54:40.007 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:54:40.007 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:54:40.007 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:54:40.007 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000001", 2024-08-18T15:54:40.007 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:54:40.007 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:54:40.007 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:54:40.007 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:54:40.007 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000126", 2024-08-18T15:54:40.007 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:54:40.008 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0.fZhctfLXnSt.0", 2024-08-18T15:54:40.008 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:54:40.008 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000055", 2024-08-18T15:54:40.008 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:54:40.008 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000061", 2024-08-18T15:54:40.008 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1", 2024-08-18T15:54:40.008 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:54:40.008 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:54:40.008 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:54:40.008 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000097", 2024-08-18T15:54:40.008 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:54:40.008 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:54:40.009 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:54:40.009 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:54:40.009 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:54:40.009 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:54:40.009 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:54:40.009 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:54:40.009 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000042", 2024-08-18T15:54:40.009 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:54:40.009 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:54:40.009 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:54:40.009 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1.retry", 2024-08-18T15:54:40.009 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:54:40.009 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:54:40.010 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:54:40.010 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:54:40.010 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:54:40.010 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:54:40.010 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:54:40.010 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000075", 2024-08-18T15:54:40.010 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:54:40.010 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000120", 2024-08-18T15:54:40.010 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000030", 2024-08-18T15:54:40.010 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:54:40.010 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:54:40.010 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:54:40.010 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:54:40.011 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:54:40.011 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:54:40.011 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9" 2024-08-18T15:54:40.011 INFO:teuthology.orchestra.run.smithi193.stdout:] 2024-08-18T15:54:40.011 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:54:45.012 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:54:45.013 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:54:45.013 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:54:45.065 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:54:45.066 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:54:45.128 INFO:teuthology.orchestra.run.smithi193.stdout:[ 2024-08-18T15:54:45.129 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:54:45.129 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:54:45.129 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.29", 2024-08-18T15:54:45.129 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000078", 2024-08-18T15:54:45.129 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.history", 2024-08-18T15:54:45.129 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:54:45.129 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:54:45.129 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:54:45.129 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:54:45.129 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.23", 2024-08-18T15:54:45.130 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000070", 2024-08-18T15:54:45.130 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000104", 2024-08-18T15:54:45.130 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:54:45.130 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2.retry", 2024-08-18T15:54:45.130 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:54:45.130 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:54:45.130 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:54:45.130 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:54:45.130 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000026", 2024-08-18T15:54:45.130 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:54:45.130 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:54:45.130 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:54:45.131 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:54:45.131 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:54:45.131 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:54:45.131 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.8", 2024-08-18T15:54:45.131 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:54:45.131 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:54:45.131 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:54:45.131 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:54:45.131 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000028", 2024-08-18T15:54:45.131 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:54:45.131 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:54:45.131 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:54:45.131 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:54:45.131 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:54:45.132 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:54:45.132 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:54:45.132 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2.retry", 2024-08-18T15:54:45.132 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:54:45.132 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.30", 2024-08-18T15:54:45.132 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:54:45.132 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:54:45.132 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:54:45.132 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:54:45.132 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000040", 2024-08-18T15:54:45.132 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:54:45.132 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.6", 2024-08-18T15:54:45.132 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:54:45.133 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:54:45.133 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:54:45.133 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:54:45.133 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:54:45.133 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:54:45.133 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000015", 2024-08-18T15:54:45.133 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:54:45.133 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:54:45.133 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000069", 2024-08-18T15:54:45.133 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:54:45.133 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:54:45.133 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:54:45.133 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000095", 2024-08-18T15:54:45.133 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0", 2024-08-18T15:54:45.134 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000003", 2024-08-18T15:54:45.134 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000047", 2024-08-18T15:54:45.134 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:54:45.134 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:54:45.134 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:54:45.134 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:54:45.134 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:54:45.134 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:54:45.134 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:54:45.134 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:54:45.134 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:54:45.134 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000035", 2024-08-18T15:54:45.134 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:54:45.135 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:54:45.135 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000037", 2024-08-18T15:54:45.135 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:54:45.135 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:54:45.135 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:54:45.135 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:54:45.135 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:54:45.135 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:54:45.135 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:54:45.135 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:54:45.135 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:54:45.135 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:54:45.135 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:54:45.136 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:54:45.136 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:54:45.136 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:54:45.136 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000024", 2024-08-18T15:54:45.136 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:54:45.136 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:54:45.136 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:54:45.136 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:54:45.136 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000000", 2024-08-18T15:54:45.136 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:54:45.136 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000031", 2024-08-18T15:54:45.136 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:54:45.136 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:54:45.136 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:54:45.137 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:54:45.137 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:54:45.137 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:54:45.137 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3", 2024-08-18T15:54:45.137 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:54:45.137 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:54:45.137 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:54:45.137 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:54:45.137 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:54:45.137 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:54:45.137 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:54:45.137 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:54:45.137 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:54:45.138 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:54:45.138 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:54:45.138 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:54:45.138 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:54:45.138 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:54:45.138 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:54:45.138 INFO:teuthology.orchestra.run.smithi193.stdout: "data_loggenerations_metadata", 2024-08-18T15:54:45.138 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2", 2024-08-18T15:54:45.138 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:54:45.138 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:54:45.138 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:54:45.138 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:54:45.138 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T15:54:45.139 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:54:45.139 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:54:45.139 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:54:45.139 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:54:45.139 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:54:45.139 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000076", 2024-08-18T15:54:45.139 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:54:45.139 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:54:45.139 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000113", 2024-08-18T15:54:45.139 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000005", 2024-08-18T15:54:45.139 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:54:45.139 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000011", 2024-08-18T15:54:45.139 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:54:45.139 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000058", 2024-08-18T15:54:45.140 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:54:45.140 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:54:45.140 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:54:45.140 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000052", 2024-08-18T15:54:45.140 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:54:45.140 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:54:45.140 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:54:45.140 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:54:45.140 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.21", 2024-08-18T15:54:45.140 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:54:45.140 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:54:45.140 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:54:45.140 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:54:45.140 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:54:45.141 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:54:45.141 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:54:45.141 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:54:45.141 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:54:45.141 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:54:45.141 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:54:45.141 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:54:45.141 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:54:45.141 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:54:45.141 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:54:45.141 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:54:45.141 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:54:45.142 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:54:45.142 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:54:45.142 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000088", 2024-08-18T15:54:45.142 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000080", 2024-08-18T15:54:45.142 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:54:45.142 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:54:45.142 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000090", 2024-08-18T15:54:45.142 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:54:45.142 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:54:45.142 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000110", 2024-08-18T15:54:45.142 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:54:45.142 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:54:45.142 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:54:45.143 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.0", 2024-08-18T15:54:45.143 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:54:45.143 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000096", 2024-08-18T15:54:45.143 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:54:45.143 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:54:45.143 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:54:45.143 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:54:45.143 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:54:45.143 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000087", 2024-08-18T15:54:45.143 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:54:45.143 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:54:45.143 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:54:45.143 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:54:45.144 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000008", 2024-08-18T15:54:45.144 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:54:45.144 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:54:45.144 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:54:45.144 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:54:45.144 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:54:45.144 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000006", 2024-08-18T15:54:45.144 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000029", 2024-08-18T15:54:45.144 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:54:45.144 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000089", 2024-08-18T15:54:45.144 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:54:45.144 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:54:45.144 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000022", 2024-08-18T15:54:45.145 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:54:45.145 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:54:45.145 INFO:teuthology.orchestra.run.smithi193.stdout: "bilog.trim", 2024-08-18T15:54:45.145 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000016", 2024-08-18T15:54:45.145 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:54:45.145 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:54:45.145 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:54:45.145 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:54:45.145 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:54:45.145 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000041", 2024-08-18T15:54:45.145 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:54:45.145 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:54:45.145 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.log.01050e30-179b-43be-bdbb-3bb23cf37ae8.3", 2024-08-18T15:54:45.146 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:54:45.146 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000018", 2024-08-18T15:54:45.146 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000092", 2024-08-18T15:54:45.146 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:54:45.146 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:54:45.146 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:54:45.146 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:54:45.146 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.27", 2024-08-18T15:54:45.146 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.4", 2024-08-18T15:54:45.146 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:54:45.146 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:54:45.146 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:54:45.146 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:54:45.147 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:54:45.147 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:54:45.147 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:54:45.147 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000014", 2024-08-18T15:54:45.147 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:54:45.147 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3.retry", 2024-08-18T15:54:45.147 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:54:45.147 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:54:45.147 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:54:45.147 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:54:45.147 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:54:45.147 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000112", 2024-08-18T15:54:45.147 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:54:45.148 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000007", 2024-08-18T15:54:45.148 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000021", 2024-08-18T15:54:45.148 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:54:45.148 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:54:45.148 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:54:45.148 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:54:45.148 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:54:45.148 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.16", 2024-08-18T15:54:45.148 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:54:45.148 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000064", 2024-08-18T15:54:45.148 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:54:45.148 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:54:45.148 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:54:45.148 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:54:45.149 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:54:45.149 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:54:45.149 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:54:45.149 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000071", 2024-08-18T15:54:45.149 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:54:45.149 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:54:45.149 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:54:45.149 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000074", 2024-08-18T15:54:45.149 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:54:45.149 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:54:45.149 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:54:45.149 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:54:45.149 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:54:45.150 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:54:45.150 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:54:45.150 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:54:45.150 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:54:45.150 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:54:45.150 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:54:45.150 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:54:45.150 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:54:45.150 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:54:45.150 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:54:45.150 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000081", 2024-08-18T15:54:45.150 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:54:45.151 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000009", 2024-08-18T15:54:45.151 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000121", 2024-08-18T15:54:45.151 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1", 2024-08-18T15:54:45.151 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:54:45.151 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:54:45.151 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:54:45.151 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:54:45.151 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:54:45.151 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:54:45.151 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.17", 2024-08-18T15:54:45.151 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000125", 2024-08-18T15:54:45.151 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:54:45.151 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000082", 2024-08-18T15:54:45.151 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:54:45.152 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:54:45.152 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:54:45.152 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:54:45.152 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:54:45.152 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:54:45.152 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:54:45.152 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:54:45.152 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:54:45.152 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:54:45.152 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:54:45.152 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000105", 2024-08-18T15:54:45.152 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000059", 2024-08-18T15:54:45.152 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:54:45.152 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:54:45.153 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:54:45.153 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:54:45.153 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:54:45.153 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:54:45.153 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:54:45.153 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:54:45.153 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:54:45.153 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:54:45.153 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:54:45.153 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000077", 2024-08-18T15:54:45.153 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000032", 2024-08-18T15:54:45.153 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:54:45.153 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:54:45.154 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:54:45.154 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:54:45.154 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:54:45.154 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:54:45.154 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.24", 2024-08-18T15:54:45.154 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:54:45.154 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:54:45.154 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:54:45.154 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:54:45.154 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3.retry", 2024-08-18T15:54:45.154 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:54:45.154 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:54:45.154 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:54:45.155 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:54:45.155 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:54:45.155 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:54:45.155 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000053", 2024-08-18T15:54:45.155 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000091", 2024-08-18T15:54:45.155 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:54:45.155 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:54:45.155 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:54:45.155 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000065", 2024-08-18T15:54:45.155 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:54:45.155 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:54:45.155 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:54:45.155 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:54:45.155 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000083", 2024-08-18T15:54:45.156 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000010", 2024-08-18T15:54:45.156 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:54:45.156 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:54:45.156 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:54:45.156 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:54:45.156 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:54:45.156 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:54:45.156 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:54:45.156 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:54:45.156 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:54:45.156 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:54:45.156 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:54:45.156 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.7", 2024-08-18T15:54:45.157 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.3", 2024-08-18T15:54:45.157 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:54:45.157 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:54:45.157 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:54:45.157 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:54:45.157 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:54:45.157 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:54:45.157 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:54:45.157 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:54:45.157 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.2", 2024-08-18T15:54:45.157 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:54:45.157 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:54:45.157 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:54:45.157 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:54:45.158 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:54:45.158 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:54:45.158 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:54:45.158 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000045", 2024-08-18T15:54:45.158 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000002", 2024-08-18T15:54:45.158 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:54:45.158 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3.2UbiLidwSTy.0", 2024-08-18T15:54:45.158 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:54:45.158 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.12", 2024-08-18T15:54:45.158 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:54:45.158 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:54:45.158 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:54:45.158 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:54:45.159 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:54:45.159 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:54:45.159 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000116", 2024-08-18T15:54:45.159 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.25", 2024-08-18T15:54:45.159 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:54:45.159 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:54:45.159 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:54:45.159 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:54:45.159 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000034", 2024-08-18T15:54:45.159 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:54:45.159 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:54:45.159 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000101", 2024-08-18T15:54:45.159 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:54:45.160 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:54:45.160 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.13", 2024-08-18T15:54:45.160 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:54:45.160 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:54:45.160 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.1", 2024-08-18T15:54:45.160 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0.retry", 2024-08-18T15:54:45.160 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:54:45.160 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:54:45.160 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000079", 2024-08-18T15:54:45.160 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000049", 2024-08-18T15:54:45.160 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:54:45.160 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0", 2024-08-18T15:54:45.160 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:54:45.161 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:54:45.161 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000117", 2024-08-18T15:54:45.161 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:54:45.161 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000044", 2024-08-18T15:54:45.161 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:54:45.161 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:54:45.161 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:54:45.161 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:54:45.161 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:54:45.161 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:54:45.161 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.14", 2024-08-18T15:54:45.161 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:54:45.161 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000066", 2024-08-18T15:54:45.161 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:54:45.162 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:54:45.162 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:54:45.162 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:54:45.162 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:54:45.162 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000068", 2024-08-18T15:54:45.162 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:54:45.162 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000085", 2024-08-18T15:54:45.162 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:54:45.162 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:54:45.162 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:54:45.162 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:54:45.162 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:54:45.163 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:54:45.163 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:54:45.163 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:54:45.163 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:54:45.163 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:54:45.163 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:54:45.163 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:54:45.163 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000073", 2024-08-18T15:54:45.163 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:54:45.163 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:54:45.163 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:54:45.163 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000038", 2024-08-18T15:54:45.163 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:54:45.164 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:54:45.164 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:54:45.164 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:54:45.164 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.10", 2024-08-18T15:54:45.164 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:54:45.164 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:54:45.164 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:54:45.164 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000118", 2024-08-18T15:54:45.164 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:54:45.164 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3", 2024-08-18T15:54:45.164 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:54:45.164 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000036", 2024-08-18T15:54:45.164 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:54:45.164 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:54:45.165 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:54:45.165 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:54:45.165 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:54:45.165 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000103", 2024-08-18T15:54:45.165 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.3", 2024-08-18T15:54:45.165 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:54:45.165 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000119", 2024-08-18T15:54:45.165 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000098", 2024-08-18T15:54:45.165 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:54:45.165 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:54:45.165 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:54:45.165 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:54:45.165 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:54:45.165 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:54:45.166 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:54:45.166 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:54:45.166 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.0", 2024-08-18T15:54:45.166 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:54:45.166 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:54:45.166 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:54:45.166 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:54:45.166 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:54:45.166 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:54:45.166 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:54:45.166 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:54:45.166 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000027", 2024-08-18T15:54:45.167 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:54:45.167 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status", 2024-08-18T15:54:45.167 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:54:45.167 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000019", 2024-08-18T15:54:45.167 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:54:45.167 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:54:45.167 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:54:45.167 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:54:45.167 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1", 2024-08-18T15:54:45.167 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:54:45.167 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:54:45.167 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:54:45.167 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:54:45.167 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000039", 2024-08-18T15:54:45.168 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:54:45.168 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:54:45.168 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:54:45.168 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:54:45.168 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:54:45.168 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:54:45.168 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:54:45.168 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:54:45.168 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:54:45.168 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:54:45.168 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:54:45.168 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000100", 2024-08-18T15:54:45.168 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:54:45.169 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:54:45.169 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:54:45.169 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:54:45.169 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:54:45.169 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000093", 2024-08-18T15:54:45.169 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:54:45.169 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000004", 2024-08-18T15:54:45.169 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000063", 2024-08-18T15:54:45.169 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000122", 2024-08-18T15:54:45.169 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:54:45.170 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:54:45.170 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:54:45.170 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000057", 2024-08-18T15:54:45.170 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:54:45.170 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:54:45.170 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:54:45.170 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:54:45.170 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000054", 2024-08-18T15:54:45.170 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:54:45.170 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000114", 2024-08-18T15:54:45.170 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:54:45.170 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:54:45.171 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:54:45.171 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.2", 2024-08-18T15:54:45.171 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:54:45.171 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:54:45.171 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0", 2024-08-18T15:54:45.171 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:54:45.171 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:54:45.171 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:54:45.171 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:54:45.171 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:54:45.171 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:54:45.171 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000012", 2024-08-18T15:54:45.171 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:54:45.172 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:54:45.172 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:54:45.172 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000084", 2024-08-18T15:54:45.172 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000043", 2024-08-18T15:54:45.172 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3", 2024-08-18T15:54:45.172 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:54:45.172 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:54:45.172 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.11", 2024-08-18T15:54:45.172 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:54:45.172 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:54:45.172 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:54:45.172 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:54:45.172 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:54:45.173 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:54:45.173 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:54:45.173 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:54:45.173 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000111", 2024-08-18T15:54:45.173 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:54:45.173 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000048", 2024-08-18T15:54:45.173 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:54:45.173 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000020", 2024-08-18T15:54:45.173 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:54:45.173 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:54:45.173 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:54:45.173 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:54:45.173 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:54:45.173 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000099", 2024-08-18T15:54:45.174 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000056", 2024-08-18T15:54:45.174 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:54:45.174 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:54:45.174 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.19", 2024-08-18T15:54:45.174 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:54:45.174 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.28", 2024-08-18T15:54:45.174 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:54:45.174 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:54:45.174 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:54:45.174 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:54:45.174 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:54:45.174 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000072", 2024-08-18T15:54:45.174 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:54:45.175 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:54:45.175 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000062", 2024-08-18T15:54:45.175 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:54:45.175 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:54:45.175 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:54:45.175 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000115", 2024-08-18T15:54:45.175 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:54:45.175 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.1", 2024-08-18T15:54:45.175 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2", 2024-08-18T15:54:45.175 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000033", 2024-08-18T15:54:45.175 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:54:45.175 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:54:45.175 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:54:45.175 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:54:45.176 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:54:45.176 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:54:45.176 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:54:45.176 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:54:45.176 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:54:45.176 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:54:45.176 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:54:45.176 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:54:45.176 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:54:45.176 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:54:45.176 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:54:45.176 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:54:45.176 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:54:45.176 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:54:45.177 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:54:45.177 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000025", 2024-08-18T15:54:45.177 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:54:45.177 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000123", 2024-08-18T15:54:45.177 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.31", 2024-08-18T15:54:45.177 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:54:45.177 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:54:45.177 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:54:45.177 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:54:45.177 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.18", 2024-08-18T15:54:45.177 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:54:45.177 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:54:45.177 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:54:45.177 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:54:45.178 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:54:45.178 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000108", 2024-08-18T15:54:45.178 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:54:45.178 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:54:45.178 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000094", 2024-08-18T15:54:45.178 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:54:45.178 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:54:45.178 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:54:45.178 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:54:45.178 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:54:45.178 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:54:45.178 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000060", 2024-08-18T15:54:45.178 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:54:45.179 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:54:45.179 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:54:45.179 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000109", 2024-08-18T15:54:45.179 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.15", 2024-08-18T15:54:45.179 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:54:45.179 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:54:45.179 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:54:45.179 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:54:45.179 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:54:45.179 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:54:45.179 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:54:45.179 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:54:45.179 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:54:45.180 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000013", 2024-08-18T15:54:45.180 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:54:45.180 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:54:45.180 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:54:45.180 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:54:45.180 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:54:45.180 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:54:45.180 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:54:45.180 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:54:45.180 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:54:45.180 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:54:45.180 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:54:45.180 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:54:45.180 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:54:45.181 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:54:45.181 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000051", 2024-08-18T15:54:45.181 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:54:45.181 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:54:45.181 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:54:45.181 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:54:45.181 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000106", 2024-08-18T15:54:45.181 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:54:45.181 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:54:45.181 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:54:45.181 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:54:45.181 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:54:45.181 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000017", 2024-08-18T15:54:45.181 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:54:45.182 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:54:45.182 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:54:45.182 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000023", 2024-08-18T15:54:45.182 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:54:45.182 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:54:45.182 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:54:45.182 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:54:45.182 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:54:45.182 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:54:45.182 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:54:45.182 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000107", 2024-08-18T15:54:45.182 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:54:45.182 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:54:45.182 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000046", 2024-08-18T15:54:45.183 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:54:45.185 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:54:45.186 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:54:45.186 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.26", 2024-08-18T15:54:45.186 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:54:45.186 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:54:45.186 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9", 2024-08-18T15:54:45.186 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.22", 2024-08-18T15:54:45.186 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:54:45.186 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:54:45.186 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:54:45.186 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:54:45.186 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:54:45.186 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:54:45.186 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:54:45.187 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2", 2024-08-18T15:54:45.187 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:54:45.187 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:54:45.187 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000067", 2024-08-18T15:54:45.187 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:54:45.187 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:54:45.187 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:54:45.187 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:54:45.187 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:54:45.187 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:54:45.187 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:54:45.187 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:54:45.187 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:54:45.188 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:54:45.188 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:54:45.188 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:54:45.188 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:54:45.188 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:54:45.188 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:54:45.188 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:54:45.188 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:54:45.188 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:54:45.188 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:54:45.188 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.9", 2024-08-18T15:54:45.188 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000086", 2024-08-18T15:54:45.188 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.5", 2024-08-18T15:54:45.188 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:54:45.189 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:54:45.189 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:54:45.189 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:54:45.189 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:54:45.189 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:54:45.189 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2.MFXJ583gCr1.0", 2024-08-18T15:54:45.189 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:54:45.189 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1.GPnO16liyZ+.0", 2024-08-18T15:54:45.189 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:54:45.189 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:54:45.189 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:54:45.189 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:54:45.189 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:54:45.189 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:54:45.190 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:54:45.190 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:54:45.190 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:54:45.190 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.20", 2024-08-18T15:54:45.190 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:54:45.190 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:54:45.190 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:54:45.190 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:54:45.190 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:54:45.190 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:54:45.190 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1.retry", 2024-08-18T15:54:45.190 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:54:45.191 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:54:45.191 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:54:45.191 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0.retry", 2024-08-18T15:54:45.191 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000050", 2024-08-18T15:54:45.191 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:54:45.191 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000102", 2024-08-18T15:54:45.191 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:54:45.191 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:54:45.191 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:54:45.191 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:54:45.191 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:54:45.191 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:54:45.191 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:54:45.191 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:54:45.192 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:54:45.192 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:54:45.192 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:54:45.192 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000124", 2024-08-18T15:54:45.192 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:54:45.192 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:54:45.192 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:54:45.192 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000001", 2024-08-18T15:54:45.192 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:54:45.192 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:54:45.192 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:54:45.192 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:54:45.192 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000126", 2024-08-18T15:54:45.193 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:54:45.193 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0.fZhctfLXnSt.0", 2024-08-18T15:54:45.193 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:54:45.193 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000055", 2024-08-18T15:54:45.193 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:54:45.193 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000061", 2024-08-18T15:54:45.193 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1", 2024-08-18T15:54:45.193 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:54:45.193 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:54:45.193 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:54:45.193 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000097", 2024-08-18T15:54:45.193 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:54:45.193 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:54:45.194 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:54:45.194 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:54:45.194 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:54:45.194 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:54:45.194 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:54:45.194 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:54:45.194 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000042", 2024-08-18T15:54:45.194 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:54:45.194 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:54:45.194 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:54:45.194 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1.retry", 2024-08-18T15:54:45.194 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:54:45.194 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:54:45.194 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:54:45.195 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:54:45.195 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:54:45.195 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:54:45.195 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:54:45.195 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000075", 2024-08-18T15:54:45.195 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:54:45.195 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000120", 2024-08-18T15:54:45.195 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000030", 2024-08-18T15:54:45.195 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:54:45.195 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:54:45.195 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:54:45.195 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:54:45.195 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:54:45.196 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:54:45.196 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9" 2024-08-18T15:54:45.196 INFO:teuthology.orchestra.run.smithi193.stdout:] 2024-08-18T15:54:45.196 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:54:50.198 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:54:50.198 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:54:50.198 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:54:50.250 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:54:50.251 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:54:50.314 INFO:teuthology.orchestra.run.smithi193.stdout:[ 2024-08-18T15:54:50.314 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:54:50.314 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:54:50.314 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.29", 2024-08-18T15:54:50.315 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000078", 2024-08-18T15:54:50.315 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.history", 2024-08-18T15:54:50.315 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:54:50.315 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:54:50.315 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:54:50.315 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:54:50.315 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.23", 2024-08-18T15:54:50.315 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000070", 2024-08-18T15:54:50.315 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000104", 2024-08-18T15:54:50.315 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:54:50.315 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2.retry", 2024-08-18T15:54:50.316 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:54:50.316 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:54:50.316 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:54:50.316 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:54:50.316 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000026", 2024-08-18T15:54:50.316 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:54:50.316 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:54:50.316 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:54:50.316 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:54:50.316 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:54:50.316 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:54:50.316 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.8", 2024-08-18T15:54:50.317 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:54:50.317 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:54:50.317 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:54:50.317 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:54:50.317 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000028", 2024-08-18T15:54:50.317 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:54:50.317 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:54:50.317 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:54:50.317 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:54:50.317 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:54:50.317 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:54:50.317 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:54:50.317 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2.retry", 2024-08-18T15:54:50.317 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:54:50.318 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.30", 2024-08-18T15:54:50.318 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:54:50.318 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:54:50.318 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:54:50.318 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:54:50.318 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000040", 2024-08-18T15:54:50.318 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:54:50.318 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.6", 2024-08-18T15:54:50.318 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:54:50.318 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:54:50.318 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:54:50.318 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:54:50.319 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:54:50.319 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:54:50.319 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000015", 2024-08-18T15:54:50.319 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:54:50.319 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:54:50.319 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000069", 2024-08-18T15:54:50.319 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:54:50.319 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:54:50.319 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:54:50.319 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000095", 2024-08-18T15:54:50.319 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0", 2024-08-18T15:54:50.319 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000003", 2024-08-18T15:54:50.319 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000047", 2024-08-18T15:54:50.320 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:54:50.320 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:54:50.320 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:54:50.320 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:54:50.320 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:54:50.320 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:54:50.320 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:54:50.320 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:54:50.320 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:54:50.320 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000035", 2024-08-18T15:54:50.320 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:54:50.320 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:54:50.320 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000037", 2024-08-18T15:54:50.320 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:54:50.321 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:54:50.321 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:54:50.321 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:54:50.321 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:54:50.321 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:54:50.321 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:54:50.321 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:54:50.321 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:54:50.321 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:54:50.321 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:54:50.321 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:54:50.321 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:54:50.321 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:54:50.322 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000024", 2024-08-18T15:54:50.322 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:54:50.322 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:54:50.322 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:54:50.322 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:54:50.322 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000000", 2024-08-18T15:54:50.322 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:54:50.322 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000031", 2024-08-18T15:54:50.322 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:54:50.322 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:54:50.322 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:54:50.322 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:54:50.322 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:54:50.323 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:54:50.323 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3", 2024-08-18T15:54:50.323 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:54:50.323 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:54:50.323 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:54:50.323 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:54:50.323 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:54:50.323 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:54:50.323 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:54:50.323 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:54:50.323 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:54:50.323 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:54:50.323 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:54:50.323 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:54:50.324 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:54:50.324 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:54:50.324 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:54:50.324 INFO:teuthology.orchestra.run.smithi193.stdout: "data_loggenerations_metadata", 2024-08-18T15:54:50.324 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2", 2024-08-18T15:54:50.324 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:54:50.324 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:54:50.324 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:54:50.324 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:54:50.324 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T15:54:50.324 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:54:50.324 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:54:50.324 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:54:50.324 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:54:50.325 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:54:50.325 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000076", 2024-08-18T15:54:50.325 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:54:50.325 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:54:50.325 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000113", 2024-08-18T15:54:50.325 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000005", 2024-08-18T15:54:50.325 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:54:50.325 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000011", 2024-08-18T15:54:50.325 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:54:50.325 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000058", 2024-08-18T15:54:50.325 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:54:50.325 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:54:50.325 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:54:50.325 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000052", 2024-08-18T15:54:50.326 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:54:50.326 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:54:50.326 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:54:50.326 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:54:50.326 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.21", 2024-08-18T15:54:50.326 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:54:50.326 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:54:50.326 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:54:50.326 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:54:50.326 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:54:50.326 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:54:50.326 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:54:50.327 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:54:50.327 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:54:50.327 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:54:50.327 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:54:50.327 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:54:50.327 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:54:50.327 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:54:50.327 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:54:50.327 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:54:50.327 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:54:50.327 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:54:50.327 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:54:50.327 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000088", 2024-08-18T15:54:50.328 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000080", 2024-08-18T15:54:50.328 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:54:50.328 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:54:50.328 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000090", 2024-08-18T15:54:50.328 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:54:50.328 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:54:50.328 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000110", 2024-08-18T15:54:50.328 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:54:50.328 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:54:50.328 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:54:50.328 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.0", 2024-08-18T15:54:50.328 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:54:50.328 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000096", 2024-08-18T15:54:50.329 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:54:50.329 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:54:50.329 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:54:50.329 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:54:50.329 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:54:50.329 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000087", 2024-08-18T15:54:50.329 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:54:50.329 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:54:50.329 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:54:50.329 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:54:50.329 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000008", 2024-08-18T15:54:50.329 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:54:50.329 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:54:50.330 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:54:50.330 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:54:50.330 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:54:50.330 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000006", 2024-08-18T15:54:50.330 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000029", 2024-08-18T15:54:50.330 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:54:50.330 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000089", 2024-08-18T15:54:50.330 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:54:50.330 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:54:50.330 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000022", 2024-08-18T15:54:50.330 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:54:50.330 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:54:50.330 INFO:teuthology.orchestra.run.smithi193.stdout: "bilog.trim", 2024-08-18T15:54:50.331 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000016", 2024-08-18T15:54:50.331 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:54:50.331 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:54:50.331 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:54:50.331 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:54:50.331 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:54:50.331 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000041", 2024-08-18T15:54:50.331 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:54:50.331 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:54:50.331 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.log.01050e30-179b-43be-bdbb-3bb23cf37ae8.3", 2024-08-18T15:54:50.331 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:54:50.331 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000018", 2024-08-18T15:54:50.332 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000092", 2024-08-18T15:54:50.332 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:54:50.332 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:54:50.332 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:54:50.332 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:54:50.332 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.27", 2024-08-18T15:54:50.332 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.4", 2024-08-18T15:54:50.332 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:54:50.332 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:54:50.332 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:54:50.332 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:54:50.332 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:54:50.332 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:54:50.332 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:54:50.333 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000014", 2024-08-18T15:54:50.333 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:54:50.333 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3.retry", 2024-08-18T15:54:50.333 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:54:50.333 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:54:50.333 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:54:50.333 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:54:50.333 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:54:50.333 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000112", 2024-08-18T15:54:50.333 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:54:50.333 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000007", 2024-08-18T15:54:50.333 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000021", 2024-08-18T15:54:50.333 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:54:50.334 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:54:50.334 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:54:50.334 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:54:50.334 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:54:50.334 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.16", 2024-08-18T15:54:50.334 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:54:50.334 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000064", 2024-08-18T15:54:50.334 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:54:50.334 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:54:50.334 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:54:50.334 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:54:50.334 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:54:50.334 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:54:50.335 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:54:50.335 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000071", 2024-08-18T15:54:50.335 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:54:50.335 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:54:50.335 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:54:50.335 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000074", 2024-08-18T15:54:50.335 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:54:50.335 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:54:50.335 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:54:50.335 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:54:50.335 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:54:50.335 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:54:50.335 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:54:50.336 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:54:50.336 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:54:50.336 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:54:50.336 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:54:50.336 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:54:50.336 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:54:50.336 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:54:50.336 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:54:50.336 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000081", 2024-08-18T15:54:50.336 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:54:50.336 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000009", 2024-08-18T15:54:50.336 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000121", 2024-08-18T15:54:50.336 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1", 2024-08-18T15:54:50.336 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:54:50.337 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:54:50.337 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:54:50.337 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:54:50.337 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:54:50.337 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:54:50.337 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.17", 2024-08-18T15:54:50.337 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000125", 2024-08-18T15:54:50.337 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:54:50.337 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000082", 2024-08-18T15:54:50.337 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:54:50.337 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:54:50.337 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:54:50.337 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:54:50.338 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:54:50.338 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:54:50.338 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:54:50.338 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:54:50.338 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:54:50.338 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:54:50.338 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:54:50.338 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000105", 2024-08-18T15:54:50.338 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000059", 2024-08-18T15:54:50.338 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:54:50.338 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:54:50.338 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:54:50.338 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:54:50.339 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:54:50.339 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:54:50.339 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:54:50.339 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:54:50.339 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:54:50.339 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:54:50.339 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:54:50.339 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000077", 2024-08-18T15:54:50.339 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000032", 2024-08-18T15:54:50.339 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:54:50.339 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:54:50.339 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:54:50.339 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:54:50.339 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:54:50.340 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:54:50.340 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.24", 2024-08-18T15:54:50.340 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:54:50.340 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:54:50.340 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:54:50.340 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:54:50.340 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3.retry", 2024-08-18T15:54:50.340 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:54:50.340 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:54:50.340 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:54:50.340 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:54:50.340 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:54:50.340 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:54:50.341 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000053", 2024-08-18T15:54:50.341 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000091", 2024-08-18T15:54:50.341 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:54:50.341 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:54:50.341 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:54:50.341 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000065", 2024-08-18T15:54:50.341 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:54:50.341 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:54:50.341 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:54:50.341 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:54:50.341 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000083", 2024-08-18T15:54:50.341 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000010", 2024-08-18T15:54:50.342 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:54:50.342 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:54:50.342 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:54:50.342 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:54:50.342 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:54:50.342 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:54:50.342 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:54:50.342 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:54:50.342 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:54:50.342 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:54:50.342 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:54:50.343 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.7", 2024-08-18T15:54:50.343 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.3", 2024-08-18T15:54:50.343 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:54:50.343 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:54:50.343 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:54:50.343 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:54:50.343 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:54:50.343 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:54:50.343 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:54:50.343 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:54:50.343 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.2", 2024-08-18T15:54:50.343 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:54:50.343 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:54:50.344 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:54:50.344 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:54:50.344 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:54:50.344 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:54:50.344 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:54:50.344 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000045", 2024-08-18T15:54:50.344 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000002", 2024-08-18T15:54:50.344 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:54:50.344 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3.2UbiLidwSTy.0", 2024-08-18T15:54:50.344 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:54:50.344 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.12", 2024-08-18T15:54:50.344 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:54:50.344 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:54:50.345 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:54:50.345 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:54:50.345 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:54:50.345 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:54:50.345 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000116", 2024-08-18T15:54:50.345 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.25", 2024-08-18T15:54:50.345 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:54:50.345 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:54:50.345 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:54:50.345 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:54:50.345 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000034", 2024-08-18T15:54:50.345 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:54:50.346 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:54:50.346 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000101", 2024-08-18T15:54:50.346 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:54:50.346 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:54:50.346 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.13", 2024-08-18T15:54:50.346 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:54:50.346 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:54:50.347 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.1", 2024-08-18T15:54:50.347 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0.retry", 2024-08-18T15:54:50.347 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:54:50.347 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:54:50.347 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000079", 2024-08-18T15:54:50.347 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000049", 2024-08-18T15:54:50.347 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:54:50.347 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0", 2024-08-18T15:54:50.347 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:54:50.347 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:54:50.347 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000117", 2024-08-18T15:54:50.347 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:54:50.348 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000044", 2024-08-18T15:54:50.348 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:54:50.348 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:54:50.348 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:54:50.348 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:54:50.348 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:54:50.348 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:54:50.348 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.14", 2024-08-18T15:54:50.348 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:54:50.348 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000066", 2024-08-18T15:54:50.348 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:54:50.348 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:54:50.348 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:54:50.349 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:54:50.349 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:54:50.349 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000068", 2024-08-18T15:54:50.349 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:54:50.349 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000085", 2024-08-18T15:54:50.349 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:54:50.349 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:54:50.349 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:54:50.349 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:54:50.349 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:54:50.349 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:54:50.349 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:54:50.349 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:54:50.350 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:54:50.350 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:54:50.350 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:54:50.350 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:54:50.350 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000073", 2024-08-18T15:54:50.350 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:54:50.350 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:54:50.350 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:54:50.350 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000038", 2024-08-18T15:54:50.350 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:54:50.350 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:54:50.350 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:54:50.351 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:54:50.351 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.10", 2024-08-18T15:54:50.351 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:54:50.351 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:54:50.351 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:54:50.351 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000118", 2024-08-18T15:54:50.351 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:54:50.351 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3", 2024-08-18T15:54:50.351 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:54:50.351 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000036", 2024-08-18T15:54:50.351 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:54:50.351 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:54:50.351 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:54:50.352 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:54:50.352 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:54:50.352 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000103", 2024-08-18T15:54:50.352 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.3", 2024-08-18T15:54:50.352 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:54:50.352 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000119", 2024-08-18T15:54:50.352 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000098", 2024-08-18T15:54:50.352 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:54:50.352 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:54:50.352 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:54:50.352 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:54:50.352 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:54:50.352 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:54:50.352 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:54:50.353 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:54:50.353 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.0", 2024-08-18T15:54:50.353 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:54:50.353 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:54:50.353 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:54:50.353 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:54:50.353 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:54:50.353 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:54:50.353 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:54:50.353 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:54:50.353 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000027", 2024-08-18T15:54:50.353 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:54:50.354 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status", 2024-08-18T15:54:50.354 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:54:50.354 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000019", 2024-08-18T15:54:50.354 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:54:50.354 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:54:50.354 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:54:50.354 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:54:50.354 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1", 2024-08-18T15:54:50.354 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:54:50.354 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:54:50.354 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:54:50.354 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:54:50.355 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000039", 2024-08-18T15:54:50.355 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:54:50.355 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:54:50.355 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:54:50.355 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:54:50.355 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:54:50.355 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:54:50.355 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:54:50.355 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:54:50.355 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:54:50.355 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:54:50.355 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:54:50.355 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000100", 2024-08-18T15:54:50.356 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:54:50.356 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:54:50.356 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:54:50.356 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:54:50.356 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:54:50.356 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000093", 2024-08-18T15:54:50.356 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:54:50.356 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000004", 2024-08-18T15:54:50.356 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000063", 2024-08-18T15:54:50.356 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000122", 2024-08-18T15:54:50.356 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:54:50.356 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:54:50.356 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:54:50.357 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000057", 2024-08-18T15:54:50.357 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:54:50.357 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:54:50.357 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:54:50.357 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:54:50.357 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000054", 2024-08-18T15:54:50.357 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:54:50.357 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000114", 2024-08-18T15:54:50.357 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:54:50.357 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:54:50.357 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:54:50.357 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.2", 2024-08-18T15:54:50.357 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:54:50.358 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:54:50.358 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0", 2024-08-18T15:54:50.358 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:54:50.358 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:54:50.358 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:54:50.358 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:54:50.358 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:54:50.358 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:54:50.358 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000012", 2024-08-18T15:54:50.358 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:54:50.358 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:54:50.358 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:54:50.358 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000084", 2024-08-18T15:54:50.359 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000043", 2024-08-18T15:54:50.359 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3", 2024-08-18T15:54:50.359 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:54:50.359 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:54:50.359 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.11", 2024-08-18T15:54:50.359 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:54:50.359 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:54:50.359 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:54:50.359 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:54:50.359 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:54:50.359 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:54:50.359 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:54:50.360 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:54:50.360 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000111", 2024-08-18T15:54:50.360 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:54:50.360 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000048", 2024-08-18T15:54:50.360 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:54:50.360 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000020", 2024-08-18T15:54:50.360 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:54:50.360 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:54:50.360 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:54:50.360 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:54:50.360 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:54:50.360 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000099", 2024-08-18T15:54:50.360 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000056", 2024-08-18T15:54:50.360 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:54:50.361 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:54:50.361 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.19", 2024-08-18T15:54:50.361 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:54:50.361 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.28", 2024-08-18T15:54:50.361 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:54:50.361 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:54:50.361 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:54:50.361 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:54:50.361 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:54:50.361 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000072", 2024-08-18T15:54:50.361 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:54:50.362 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:54:50.362 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000062", 2024-08-18T15:54:50.362 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:54:50.362 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:54:50.362 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:54:50.362 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000115", 2024-08-18T15:54:50.362 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:54:50.362 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.1", 2024-08-18T15:54:50.362 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2", 2024-08-18T15:54:50.362 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000033", 2024-08-18T15:54:50.362 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:54:50.362 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:54:50.362 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:54:50.363 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:54:50.363 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:54:50.363 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:54:50.363 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:54:50.363 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:54:50.363 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:54:50.363 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:54:50.363 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:54:50.363 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:54:50.363 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:54:50.363 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:54:50.363 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:54:50.363 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:54:50.364 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:54:50.364 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:54:50.364 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:54:50.364 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000025", 2024-08-18T15:54:50.364 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:54:50.364 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000123", 2024-08-18T15:54:50.364 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.31", 2024-08-18T15:54:50.364 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:54:50.364 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:54:50.364 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:54:50.364 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:54:50.364 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.18", 2024-08-18T15:54:50.364 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:54:50.364 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:54:50.365 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:54:50.365 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:54:50.365 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:54:50.365 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000108", 2024-08-18T15:54:50.365 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:54:50.365 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:54:50.365 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000094", 2024-08-18T15:54:50.365 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:54:50.365 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:54:50.365 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:54:50.365 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:54:50.365 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:54:50.365 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:54:50.366 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000060", 2024-08-18T15:54:50.366 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:54:50.366 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:54:50.366 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:54:50.366 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000109", 2024-08-18T15:54:50.366 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.15", 2024-08-18T15:54:50.366 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:54:50.366 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:54:50.366 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:54:50.366 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:54:50.366 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:54:50.366 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:54:50.366 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:54:50.367 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:54:50.367 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:54:50.367 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000013", 2024-08-18T15:54:50.367 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:54:50.367 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:54:50.367 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:54:50.367 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:54:50.367 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:54:50.367 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:54:50.367 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:54:50.367 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:54:50.367 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:54:50.367 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:54:50.368 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:54:50.368 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:54:50.368 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:54:50.368 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:54:50.368 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000051", 2024-08-18T15:54:50.368 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:54:50.368 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:54:50.368 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:54:50.368 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:54:50.368 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000106", 2024-08-18T15:54:50.368 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:54:50.368 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:54:50.369 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:54:50.369 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:54:50.369 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:54:50.369 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000017", 2024-08-18T15:54:50.369 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:54:50.369 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:54:50.369 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:54:50.369 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000023", 2024-08-18T15:54:50.369 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:54:50.369 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:54:50.369 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:54:50.369 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:54:50.369 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:54:50.369 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:54:50.370 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:54:50.370 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000107", 2024-08-18T15:54:50.370 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:54:50.370 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:54:50.370 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000046", 2024-08-18T15:54:50.370 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:54:50.370 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:54:50.370 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:54:50.370 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.26", 2024-08-18T15:54:50.370 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:54:50.370 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:54:50.370 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9", 2024-08-18T15:54:50.370 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.22", 2024-08-18T15:54:50.371 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:54:50.371 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:54:50.371 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:54:50.371 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:54:50.371 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:54:50.371 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:54:50.371 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:54:50.371 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2", 2024-08-18T15:54:50.371 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:54:50.371 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:54:50.371 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000067", 2024-08-18T15:54:50.371 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:54:50.371 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:54:50.372 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:54:50.372 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:54:50.372 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:54:50.372 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:54:50.372 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:54:50.372 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:54:50.372 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:54:50.372 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:54:50.372 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:54:50.372 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:54:50.372 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:54:50.372 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:54:50.372 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:54:50.372 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:54:50.373 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:54:50.373 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:54:50.373 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:54:50.373 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.9", 2024-08-18T15:54:50.373 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000086", 2024-08-18T15:54:50.373 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.5", 2024-08-18T15:54:50.373 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:54:50.373 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:54:50.373 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:54:50.373 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:54:50.373 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:54:50.373 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:54:50.373 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2.MFXJ583gCr1.0", 2024-08-18T15:54:50.374 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:54:50.374 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1.GPnO16liyZ+.0", 2024-08-18T15:54:50.374 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:54:50.374 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:54:50.374 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:54:50.374 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:54:50.374 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:54:50.374 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:54:50.374 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:54:50.374 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:54:50.374 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:54:50.374 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.20", 2024-08-18T15:54:50.374 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:54:50.375 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:54:50.375 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:54:50.375 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:54:50.375 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:54:50.375 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:54:50.375 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1.retry", 2024-08-18T15:54:50.375 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:54:50.375 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:54:50.375 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:54:50.375 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0.retry", 2024-08-18T15:54:50.375 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000050", 2024-08-18T15:54:50.375 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:54:50.375 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000102", 2024-08-18T15:54:50.376 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:54:50.376 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:54:50.376 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:54:50.376 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:54:50.376 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:54:50.376 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:54:50.376 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:54:50.376 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:54:50.376 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:54:50.376 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:54:50.376 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:54:50.376 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000124", 2024-08-18T15:54:50.376 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:54:50.377 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:54:50.377 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:54:50.377 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000001", 2024-08-18T15:54:50.377 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:54:50.377 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:54:50.377 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:54:50.377 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:54:50.377 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000126", 2024-08-18T15:54:50.377 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:54:50.377 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0.fZhctfLXnSt.0", 2024-08-18T15:54:50.377 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:54:50.377 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000055", 2024-08-18T15:54:50.377 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:54:50.378 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000061", 2024-08-18T15:54:50.378 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1", 2024-08-18T15:54:50.378 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:54:50.378 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:54:50.378 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:54:50.378 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000097", 2024-08-18T15:54:50.378 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:54:50.378 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:54:50.378 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:54:50.378 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:54:50.378 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:54:50.378 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:54:50.378 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:54:50.379 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:54:50.379 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000042", 2024-08-18T15:54:50.379 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:54:50.379 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:54:50.379 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:54:50.379 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1.retry", 2024-08-18T15:54:50.379 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:54:50.379 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:54:50.379 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:54:50.379 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:54:50.379 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:54:50.379 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:54:50.379 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:54:50.380 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000075", 2024-08-18T15:54:50.380 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:54:50.380 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000120", 2024-08-18T15:54:50.380 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000030", 2024-08-18T15:54:50.380 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:54:50.380 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:54:50.380 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:54:50.380 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:54:50.380 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:54:50.380 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:54:50.380 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9" 2024-08-18T15:54:50.380 INFO:teuthology.orchestra.run.smithi193.stdout:] 2024-08-18T15:54:50.380 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:54:55.382 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:54:55.383 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:54:55.383 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:54:55.436 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:54:55.436 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:54:55.499 INFO:teuthology.orchestra.run.smithi193.stdout:[ 2024-08-18T15:54:55.499 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:54:55.499 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:54:55.499 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.29", 2024-08-18T15:54:55.499 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000078", 2024-08-18T15:54:55.499 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.history", 2024-08-18T15:54:55.500 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:54:55.500 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:54:55.500 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:54:55.500 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:54:55.500 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.23", 2024-08-18T15:54:55.500 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000070", 2024-08-18T15:54:55.500 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000104", 2024-08-18T15:54:55.500 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:54:55.500 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2.retry", 2024-08-18T15:54:55.500 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:54:55.500 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:54:55.500 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:54:55.500 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:54:55.500 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000026", 2024-08-18T15:54:55.501 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:54:55.501 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:54:55.501 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:54:55.501 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:54:55.501 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:54:55.501 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:54:55.501 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.8", 2024-08-18T15:54:55.501 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:54:55.501 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:54:55.501 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:54:55.501 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:54:55.502 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000028", 2024-08-18T15:54:55.502 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:54:55.502 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:54:55.502 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:54:55.502 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:54:55.502 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:54:55.502 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:54:55.502 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:54:55.502 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2.retry", 2024-08-18T15:54:55.502 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:54:55.502 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.30", 2024-08-18T15:54:55.503 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:54:55.503 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:54:55.503 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:54:55.503 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:54:55.503 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000040", 2024-08-18T15:54:55.503 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:54:55.503 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.6", 2024-08-18T15:54:55.503 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:54:55.503 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:54:55.504 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:54:55.504 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:54:55.504 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:54:55.504 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:54:55.504 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000015", 2024-08-18T15:54:55.504 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:54:55.504 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:54:55.504 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000069", 2024-08-18T15:54:55.505 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:54:55.505 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:54:55.505 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:54:55.505 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000095", 2024-08-18T15:54:55.505 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0", 2024-08-18T15:54:55.505 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000003", 2024-08-18T15:54:55.505 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000047", 2024-08-18T15:54:55.505 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:54:55.505 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:54:55.505 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:54:55.505 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:54:55.505 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:54:55.506 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:54:55.506 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:54:55.506 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:54:55.506 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:54:55.506 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000035", 2024-08-18T15:54:55.506 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:54:55.506 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:54:55.506 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000037", 2024-08-18T15:54:55.506 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:54:55.506 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:54:55.506 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:54:55.506 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:54:55.507 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:54:55.507 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:54:55.507 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:54:55.507 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:54:55.507 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:54:55.507 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:54:55.507 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:54:55.507 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:54:55.507 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:54:55.507 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:54:55.507 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000024", 2024-08-18T15:54:55.507 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:54:55.507 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:54:55.508 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:54:55.508 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:54:55.508 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000000", 2024-08-18T15:54:55.508 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:54:55.508 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000031", 2024-08-18T15:54:55.508 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:54:55.508 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:54:55.508 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:54:55.508 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:54:55.508 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:54:55.509 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:54:55.509 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3", 2024-08-18T15:54:55.509 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:54:55.509 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:54:55.509 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:54:55.509 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:54:55.509 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:54:55.509 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:54:55.510 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:54:55.510 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:54:55.510 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:54:55.510 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:54:55.510 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:54:55.510 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:54:55.510 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:54:55.510 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:54:55.510 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:54:55.510 INFO:teuthology.orchestra.run.smithi193.stdout: "data_loggenerations_metadata", 2024-08-18T15:54:55.510 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2", 2024-08-18T15:54:55.511 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:54:55.511 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:54:55.511 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:54:55.511 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:54:55.511 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T15:54:55.511 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:54:55.511 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:54:55.511 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:54:55.511 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:54:55.511 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:54:55.511 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000076", 2024-08-18T15:54:55.512 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:54:55.512 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:54:55.512 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000113", 2024-08-18T15:54:55.512 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000005", 2024-08-18T15:54:55.512 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:54:55.512 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000011", 2024-08-18T15:54:55.512 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:54:55.512 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000058", 2024-08-18T15:54:55.512 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:54:55.512 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:54:55.512 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:54:55.512 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000052", 2024-08-18T15:54:55.512 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:54:55.512 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:54:55.513 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:54:55.513 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:54:55.513 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.21", 2024-08-18T15:54:55.513 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:54:55.513 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:54:55.513 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:54:55.513 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:54:55.513 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:54:55.513 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:54:55.513 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:54:55.513 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:54:55.513 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:54:55.514 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:54:55.514 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:54:55.514 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:54:55.514 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:54:55.514 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:54:55.514 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:54:55.514 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:54:55.514 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:54:55.514 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:54:55.515 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:54:55.515 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000088", 2024-08-18T15:54:55.515 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000080", 2024-08-18T15:54:55.515 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:54:55.515 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:54:55.515 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000090", 2024-08-18T15:54:55.515 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:54:55.515 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:54:55.515 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000110", 2024-08-18T15:54:55.515 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:54:55.515 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:54:55.516 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:54:55.516 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.0", 2024-08-18T15:54:55.516 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:54:55.516 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000096", 2024-08-18T15:54:55.516 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:54:55.516 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:54:55.516 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:54:55.516 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:54:55.516 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:54:55.516 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000087", 2024-08-18T15:54:55.516 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:54:55.516 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:54:55.516 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:54:55.517 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:54:55.517 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000008", 2024-08-18T15:54:55.517 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:54:55.517 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:54:55.517 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:54:55.517 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:54:55.517 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:54:55.517 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000006", 2024-08-18T15:54:55.517 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000029", 2024-08-18T15:54:55.517 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:54:55.517 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000089", 2024-08-18T15:54:55.517 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:54:55.517 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:54:55.517 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000022", 2024-08-18T15:54:55.518 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:54:55.518 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:54:55.518 INFO:teuthology.orchestra.run.smithi193.stdout: "bilog.trim", 2024-08-18T15:54:55.518 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000016", 2024-08-18T15:54:55.518 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:54:55.518 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:54:55.518 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:54:55.519 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:54:55.519 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:54:55.519 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000041", 2024-08-18T15:54:55.519 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:54:55.519 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:54:55.519 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.log.01050e30-179b-43be-bdbb-3bb23cf37ae8.3", 2024-08-18T15:54:55.519 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:54:55.519 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000018", 2024-08-18T15:54:55.519 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000092", 2024-08-18T15:54:55.519 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:54:55.519 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:54:55.519 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:54:55.520 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:54:55.520 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.27", 2024-08-18T15:54:55.520 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.4", 2024-08-18T15:54:55.520 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:54:55.520 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:54:55.520 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:54:55.520 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:54:55.520 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:54:55.520 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:54:55.520 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:54:55.520 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000014", 2024-08-18T15:54:55.520 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:54:55.520 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3.retry", 2024-08-18T15:54:55.520 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:54:55.521 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:54:55.521 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:54:55.521 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:54:55.521 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:54:55.521 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000112", 2024-08-18T15:54:55.521 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:54:55.521 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000007", 2024-08-18T15:54:55.521 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000021", 2024-08-18T15:54:55.521 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:54:55.521 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:54:55.521 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:54:55.521 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:54:55.521 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:54:55.522 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.16", 2024-08-18T15:54:55.522 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:54:55.522 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000064", 2024-08-18T15:54:55.522 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:54:55.522 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:54:55.522 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:54:55.522 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:54:55.522 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:54:55.522 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:54:55.522 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:54:55.522 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000071", 2024-08-18T15:54:55.522 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:54:55.522 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:54:55.523 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:54:55.523 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000074", 2024-08-18T15:54:55.523 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:54:55.523 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:54:55.523 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:54:55.523 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:54:55.523 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:54:55.523 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:54:55.523 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:54:55.523 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:54:55.523 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:54:55.523 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:54:55.523 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:54:55.523 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:54:55.524 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:54:55.524 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:54:55.524 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:54:55.524 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000081", 2024-08-18T15:54:55.524 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:54:55.524 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000009", 2024-08-18T15:54:55.524 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000121", 2024-08-18T15:54:55.524 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1", 2024-08-18T15:54:55.524 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:54:55.524 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:54:55.524 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:54:55.525 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:54:55.525 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:54:55.525 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:54:55.525 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.17", 2024-08-18T15:54:55.525 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000125", 2024-08-18T15:54:55.525 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:54:55.525 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000082", 2024-08-18T15:54:55.525 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:54:55.525 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:54:55.525 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:54:55.525 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:54:55.525 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:54:55.525 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:54:55.525 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:54:55.526 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:54:55.526 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:54:55.526 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:54:55.526 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:54:55.526 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000105", 2024-08-18T15:54:55.526 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000059", 2024-08-18T15:54:55.526 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:54:55.526 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:54:55.526 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:54:55.526 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:54:55.526 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:54:55.527 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:54:55.527 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:54:55.527 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:54:55.527 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:54:55.527 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:54:55.527 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:54:55.527 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000077", 2024-08-18T15:54:55.527 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000032", 2024-08-18T15:54:55.527 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:54:55.527 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:54:55.527 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:54:55.527 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:54:55.527 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:54:55.528 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:54:55.528 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.24", 2024-08-18T15:54:55.528 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:54:55.528 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:54:55.528 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:54:55.528 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:54:55.528 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3.retry", 2024-08-18T15:54:55.528 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:54:55.528 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:54:55.528 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:54:55.528 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:54:55.528 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:54:55.528 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:54:55.529 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000053", 2024-08-18T15:54:55.529 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000091", 2024-08-18T15:54:55.529 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:54:55.529 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:54:55.529 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:54:55.529 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000065", 2024-08-18T15:54:55.529 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:54:55.529 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:54:55.529 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:54:55.529 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:54:55.529 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000083", 2024-08-18T15:54:55.529 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000010", 2024-08-18T15:54:55.529 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:54:55.530 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:54:55.530 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:54:55.530 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:54:55.530 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:54:55.530 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:54:55.530 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:54:55.530 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:54:55.530 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:54:55.530 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:54:55.530 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:54:55.530 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.7", 2024-08-18T15:54:55.530 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.3", 2024-08-18T15:54:55.531 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:54:55.531 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:54:55.531 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:54:55.531 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:54:55.531 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:54:55.531 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:54:55.531 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:54:55.531 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:54:55.531 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.2", 2024-08-18T15:54:55.531 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:54:55.532 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:54:55.532 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:54:55.532 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:54:55.532 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:54:55.532 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:54:55.532 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:54:55.532 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000045", 2024-08-18T15:54:55.532 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000002", 2024-08-18T15:54:55.532 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:54:55.532 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3.2UbiLidwSTy.0", 2024-08-18T15:54:55.532 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:54:55.532 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.12", 2024-08-18T15:54:55.532 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:54:55.532 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:54:55.533 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:54:55.533 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:54:55.533 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:54:55.533 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:54:55.533 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000116", 2024-08-18T15:54:55.533 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.25", 2024-08-18T15:54:55.533 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:54:55.533 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:54:55.533 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:54:55.533 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:54:55.533 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000034", 2024-08-18T15:54:55.533 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:54:55.533 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:54:55.534 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000101", 2024-08-18T15:54:55.534 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:54:55.534 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:54:55.534 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.13", 2024-08-18T15:54:55.534 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:54:55.534 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:54:55.534 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.1", 2024-08-18T15:54:55.534 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0.retry", 2024-08-18T15:54:55.534 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:54:55.534 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:54:55.534 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000079", 2024-08-18T15:54:55.535 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000049", 2024-08-18T15:54:55.535 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:54:55.535 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0", 2024-08-18T15:54:55.535 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:54:55.535 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:54:55.535 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000117", 2024-08-18T15:54:55.535 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:54:55.535 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000044", 2024-08-18T15:54:55.535 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:54:55.535 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:54:55.535 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:54:55.535 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:54:55.535 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:54:55.536 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:54:55.536 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.14", 2024-08-18T15:54:55.536 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:54:55.536 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000066", 2024-08-18T15:54:55.536 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:54:55.536 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:54:55.536 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:54:55.536 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:54:55.536 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:54:55.536 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000068", 2024-08-18T15:54:55.536 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:54:55.536 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000085", 2024-08-18T15:54:55.536 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:54:55.536 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:54:55.537 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:54:55.537 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:54:55.537 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:54:55.537 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:54:55.537 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:54:55.537 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:54:55.537 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:54:55.537 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:54:55.537 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:54:55.537 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:54:55.537 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000073", 2024-08-18T15:54:55.537 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:54:55.538 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:54:55.538 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:54:55.538 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000038", 2024-08-18T15:54:55.538 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:54:55.538 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:54:55.538 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:54:55.538 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:54:55.538 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.10", 2024-08-18T15:54:55.538 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:54:55.538 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:54:55.538 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:54:55.538 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000118", 2024-08-18T15:54:55.538 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:54:55.539 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3", 2024-08-18T15:54:55.539 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:54:55.539 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000036", 2024-08-18T15:54:55.539 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:54:55.539 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:54:55.539 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:54:55.539 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:54:55.539 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:54:55.539 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000103", 2024-08-18T15:54:55.539 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.3", 2024-08-18T15:54:55.539 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:54:55.539 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000119", 2024-08-18T15:54:55.539 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000098", 2024-08-18T15:54:55.540 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:54:55.540 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:54:55.540 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:54:55.540 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:54:55.540 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:54:55.540 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:54:55.540 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:54:55.540 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:54:55.540 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.0", 2024-08-18T15:54:55.540 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:54:55.540 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:54:55.540 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:54:55.541 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:54:55.541 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:54:55.541 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:54:55.541 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:54:55.541 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:54:55.541 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000027", 2024-08-18T15:54:55.541 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:54:55.541 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status", 2024-08-18T15:54:55.541 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:54:55.541 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000019", 2024-08-18T15:54:55.541 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:54:55.541 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:54:55.541 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:54:55.541 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:54:55.542 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1", 2024-08-18T15:54:55.542 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:54:55.542 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:54:55.542 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:54:55.542 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:54:55.542 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000039", 2024-08-18T15:54:55.542 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:54:55.542 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:54:55.542 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:54:55.542 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:54:55.542 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:54:55.542 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:54:55.543 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:54:55.543 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:54:55.543 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:54:55.543 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:54:55.543 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:54:55.543 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000100", 2024-08-18T15:54:55.543 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:54:55.543 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:54:55.543 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:54:55.543 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:54:55.543 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:54:55.543 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000093", 2024-08-18T15:54:55.544 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:54:55.544 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000004", 2024-08-18T15:54:55.544 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000063", 2024-08-18T15:54:55.544 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000122", 2024-08-18T15:54:55.544 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:54:55.544 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:54:55.544 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:54:55.544 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000057", 2024-08-18T15:54:55.545 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:54:55.545 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:54:55.545 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:54:55.545 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:54:55.545 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000054", 2024-08-18T15:54:55.545 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:54:55.545 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000114", 2024-08-18T15:54:55.545 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:54:55.545 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:54:55.545 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:54:55.545 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.2", 2024-08-18T15:54:55.546 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:54:55.546 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:54:55.546 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0", 2024-08-18T15:54:55.546 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:54:55.546 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:54:55.546 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:54:55.546 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:54:55.546 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:54:55.546 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:54:55.546 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000012", 2024-08-18T15:54:55.546 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:54:55.546 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:54:55.547 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:54:55.547 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000084", 2024-08-18T15:54:55.547 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000043", 2024-08-18T15:54:55.547 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3", 2024-08-18T15:54:55.547 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:54:55.547 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:54:55.547 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.11", 2024-08-18T15:54:55.547 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:54:55.548 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:54:55.548 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:54:55.548 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:54:55.548 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:54:55.548 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:54:55.548 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:54:55.548 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:54:55.548 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000111", 2024-08-18T15:54:55.548 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:54:55.548 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000048", 2024-08-18T15:54:55.548 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:54:55.548 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000020", 2024-08-18T15:54:55.549 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:54:55.549 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:54:55.549 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:54:55.549 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:54:55.549 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:54:55.549 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000099", 2024-08-18T15:54:55.549 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000056", 2024-08-18T15:54:55.549 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:54:55.550 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:54:55.550 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.19", 2024-08-18T15:54:55.550 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:54:55.550 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.28", 2024-08-18T15:54:55.550 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:54:55.550 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:54:55.550 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:54:55.550 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:54:55.550 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:54:55.550 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000072", 2024-08-18T15:54:55.551 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:54:55.551 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:54:55.551 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000062", 2024-08-18T15:54:55.551 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:54:55.551 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:54:55.551 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:54:55.551 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000115", 2024-08-18T15:54:55.551 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:54:55.551 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.1", 2024-08-18T15:54:55.551 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2", 2024-08-18T15:54:55.551 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000033", 2024-08-18T15:54:55.551 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:54:55.552 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:54:55.552 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:54:55.552 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:54:55.552 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:54:55.552 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:54:55.552 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:54:55.552 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:54:55.552 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:54:55.552 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:54:55.552 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:54:55.552 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:54:55.552 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:54:55.552 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:54:55.552 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:54:55.553 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:54:55.553 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:54:55.553 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:54:55.553 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:54:55.553 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000025", 2024-08-18T15:54:55.553 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:54:55.553 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000123", 2024-08-18T15:54:55.553 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.31", 2024-08-18T15:54:55.553 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:54:55.553 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:54:55.553 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:54:55.553 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:54:55.553 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.18", 2024-08-18T15:54:55.554 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:54:55.554 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:54:55.554 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:54:55.554 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:54:55.554 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:54:55.554 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000108", 2024-08-18T15:54:55.554 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:54:55.554 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:54:55.554 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000094", 2024-08-18T15:54:55.554 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:54:55.554 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:54:55.554 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:54:55.555 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:54:55.555 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:54:55.555 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:54:55.555 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000060", 2024-08-18T15:54:55.555 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:54:55.555 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:54:55.555 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:54:55.555 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000109", 2024-08-18T15:54:55.555 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.15", 2024-08-18T15:54:55.555 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:54:55.555 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:54:55.555 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:54:55.555 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:54:55.556 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:54:55.556 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:54:55.556 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:54:55.556 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:54:55.556 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:54:55.556 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000013", 2024-08-18T15:54:55.556 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:54:55.556 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:54:55.556 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:54:55.556 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:54:55.556 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:54:55.556 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:54:55.557 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:54:55.557 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:54:55.557 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:54:55.557 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:54:55.557 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:54:55.557 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:54:55.557 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:54:55.557 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:54:55.557 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000051", 2024-08-18T15:54:55.557 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:54:55.557 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:54:55.557 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:54:55.557 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:54:55.557 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000106", 2024-08-18T15:54:55.558 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:54:55.558 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:54:55.558 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:54:55.558 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:54:55.558 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:54:55.558 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000017", 2024-08-18T15:54:55.558 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:54:55.558 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:54:55.558 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:54:55.558 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000023", 2024-08-18T15:54:55.558 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:54:55.558 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:54:55.559 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:54:55.559 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:54:55.559 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:54:55.559 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:54:55.559 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:54:55.559 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000107", 2024-08-18T15:54:55.559 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:54:55.559 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:54:55.559 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000046", 2024-08-18T15:54:55.559 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:54:55.559 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:54:55.559 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:54:55.559 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.26", 2024-08-18T15:54:55.560 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:54:55.560 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:54:55.560 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9", 2024-08-18T15:54:55.560 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.22", 2024-08-18T15:54:55.560 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:54:55.560 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:54:55.560 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:54:55.560 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:54:55.560 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:54:55.560 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:54:55.560 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:54:55.560 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2", 2024-08-18T15:54:55.560 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:54:55.560 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:54:55.561 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000067", 2024-08-18T15:54:55.561 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:54:55.561 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:54:55.561 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:54:55.561 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:54:55.561 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:54:55.561 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:54:55.561 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:54:55.561 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:54:55.561 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:54:55.561 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:54:55.561 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:54:55.561 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:54:55.562 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:54:55.562 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:54:55.562 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:54:55.562 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:54:55.562 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:54:55.562 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:54:55.562 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:54:55.562 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.9", 2024-08-18T15:54:55.562 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000086", 2024-08-18T15:54:55.562 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.5", 2024-08-18T15:54:55.562 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:54:55.562 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:54:55.562 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:54:55.563 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:54:55.563 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:54:55.563 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:54:55.563 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2.MFXJ583gCr1.0", 2024-08-18T15:54:55.563 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:54:55.563 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1.GPnO16liyZ+.0", 2024-08-18T15:54:55.563 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:54:55.563 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:54:55.563 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:54:55.563 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:54:55.563 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:54:55.563 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:54:55.563 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:54:55.564 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:54:55.564 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:54:55.564 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.20", 2024-08-18T15:54:55.564 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:54:55.564 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:54:55.564 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:54:55.564 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:54:55.564 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:54:55.564 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:54:55.564 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1.retry", 2024-08-18T15:54:55.564 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:54:55.564 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:54:55.564 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:54:55.565 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0.retry", 2024-08-18T15:54:55.565 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000050", 2024-08-18T15:54:55.565 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:54:55.565 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000102", 2024-08-18T15:54:55.565 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:54:55.565 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:54:55.565 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:54:55.565 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:54:55.565 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:54:55.565 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:54:55.565 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:54:55.565 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:54:55.565 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:54:55.566 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:54:55.566 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:54:55.566 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000124", 2024-08-18T15:54:55.566 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:54:55.566 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:54:55.566 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:54:55.566 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000001", 2024-08-18T15:54:55.566 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:54:55.566 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:54:55.566 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:54:55.566 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:54:55.566 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000126", 2024-08-18T15:54:55.566 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:54:55.567 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0.fZhctfLXnSt.0", 2024-08-18T15:54:55.567 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:54:55.567 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000055", 2024-08-18T15:54:55.567 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:54:55.567 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000061", 2024-08-18T15:54:55.567 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1", 2024-08-18T15:54:55.567 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:54:55.567 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:54:55.567 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:54:55.568 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000097", 2024-08-18T15:54:55.568 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:54:55.568 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:54:55.568 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:54:55.568 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:54:55.568 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:54:55.568 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:54:55.568 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:54:55.569 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:54:55.569 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000042", 2024-08-18T15:54:55.569 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:54:55.569 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:54:55.569 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:54:55.569 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1.retry", 2024-08-18T15:54:55.569 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:54:55.569 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:54:55.569 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:54:55.569 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:54:55.569 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:54:55.569 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:54:55.570 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:54:55.570 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000075", 2024-08-18T15:54:55.570 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:54:55.570 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000120", 2024-08-18T15:54:55.570 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000030", 2024-08-18T15:54:55.570 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:54:55.570 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:54:55.570 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:54:55.570 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:54:55.570 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:54:55.570 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:54:55.570 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9" 2024-08-18T15:54:55.571 INFO:teuthology.orchestra.run.smithi193.stdout:] 2024-08-18T15:54:55.571 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:55:00.573 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:55:00.573 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:55:00.573 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:55:00.626 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:55:00.626 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:55:00.690 INFO:teuthology.orchestra.run.smithi193.stdout:[ 2024-08-18T15:55:00.690 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:55:00.690 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:55:00.690 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.29", 2024-08-18T15:55:00.690 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000078", 2024-08-18T15:55:00.690 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.history", 2024-08-18T15:55:00.690 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:55:00.691 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:55:00.691 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:55:00.691 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:55:00.691 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.23", 2024-08-18T15:55:00.691 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000070", 2024-08-18T15:55:00.691 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000104", 2024-08-18T15:55:00.691 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:55:00.691 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2.retry", 2024-08-18T15:55:00.691 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:55:00.691 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:55:00.691 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:55:00.691 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:55:00.691 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000026", 2024-08-18T15:55:00.691 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:55:00.692 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:55:00.692 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:55:00.692 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:55:00.692 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:55:00.692 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:55:00.692 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.8", 2024-08-18T15:55:00.692 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:55:00.692 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:55:00.692 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:55:00.692 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:55:00.692 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000028", 2024-08-18T15:55:00.692 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:55:00.693 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:55:00.693 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:55:00.693 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:55:00.693 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:55:00.693 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:55:00.693 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:55:00.693 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2.retry", 2024-08-18T15:55:00.693 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:55:00.693 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.30", 2024-08-18T15:55:00.693 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:55:00.693 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:55:00.693 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:55:00.693 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:55:00.694 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000040", 2024-08-18T15:55:00.694 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:55:00.694 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.6", 2024-08-18T15:55:00.694 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:55:00.694 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:55:00.694 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:55:00.694 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:55:00.694 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:55:00.694 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:55:00.694 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000015", 2024-08-18T15:55:00.694 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:55:00.694 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:55:00.694 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000069", 2024-08-18T15:55:00.695 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:55:00.695 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:55:00.695 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:55:00.695 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000095", 2024-08-18T15:55:00.695 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0", 2024-08-18T15:55:00.695 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000003", 2024-08-18T15:55:00.695 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000047", 2024-08-18T15:55:00.695 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:55:00.696 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:55:00.696 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:55:00.696 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:55:00.696 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:55:00.696 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:55:00.696 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:55:00.696 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:55:00.696 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:55:00.696 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000035", 2024-08-18T15:55:00.696 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:55:00.697 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:55:00.697 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000037", 2024-08-18T15:55:00.697 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:55:00.697 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:55:00.697 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:55:00.697 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:55:00.697 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:55:00.697 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:55:00.697 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:55:00.697 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:55:00.697 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:55:00.697 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:55:00.698 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:55:00.698 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:55:00.698 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:55:00.698 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:55:00.698 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000024", 2024-08-18T15:55:00.698 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:55:00.698 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:55:00.698 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:55:00.698 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:55:00.699 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000000", 2024-08-18T15:55:00.699 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:55:00.699 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000031", 2024-08-18T15:55:00.699 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:55:00.699 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:55:00.699 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:55:00.699 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:55:00.699 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:55:00.699 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:55:00.699 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3", 2024-08-18T15:55:00.699 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:55:00.699 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:55:00.699 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:55:00.700 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:55:00.700 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:55:00.700 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:55:00.700 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:55:00.700 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:55:00.700 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:55:00.700 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:55:00.700 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:55:00.700 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:55:00.700 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:55:00.700 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:55:00.700 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:55:00.700 INFO:teuthology.orchestra.run.smithi193.stdout: "data_loggenerations_metadata", 2024-08-18T15:55:00.701 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2", 2024-08-18T15:55:00.701 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:55:00.701 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:55:00.701 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:55:00.701 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:55:00.701 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T15:55:00.701 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:55:00.701 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:55:00.701 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:55:00.701 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:55:00.701 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:55:00.701 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000076", 2024-08-18T15:55:00.701 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:55:00.702 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:55:00.702 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000113", 2024-08-18T15:55:00.702 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000005", 2024-08-18T15:55:00.702 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:55:00.702 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000011", 2024-08-18T15:55:00.702 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:55:00.702 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000058", 2024-08-18T15:55:00.702 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:55:00.702 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:55:00.702 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:55:00.702 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000052", 2024-08-18T15:55:00.702 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:55:00.702 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:55:00.703 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:55:00.703 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:55:00.703 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.21", 2024-08-18T15:55:00.703 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:55:00.703 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:55:00.703 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:55:00.703 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:55:00.703 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:55:00.703 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:55:00.703 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:55:00.703 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:55:00.704 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:55:00.704 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:55:00.704 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:55:00.704 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:55:00.704 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:55:00.704 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:55:00.704 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:55:00.704 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:55:00.704 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:55:00.704 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:55:00.704 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:55:00.705 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000088", 2024-08-18T15:55:00.705 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000080", 2024-08-18T15:55:00.705 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:55:00.705 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:55:00.705 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000090", 2024-08-18T15:55:00.705 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:55:00.705 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:55:00.705 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000110", 2024-08-18T15:55:00.705 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:55:00.705 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:55:00.705 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:55:00.705 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.0", 2024-08-18T15:55:00.705 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:55:00.705 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000096", 2024-08-18T15:55:00.706 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:55:00.706 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:55:00.706 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:55:00.706 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:55:00.706 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:55:00.706 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000087", 2024-08-18T15:55:00.706 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:55:00.706 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:55:00.706 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:55:00.706 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:55:00.706 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000008", 2024-08-18T15:55:00.706 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:55:00.706 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:55:00.707 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:55:00.707 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:55:00.707 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:55:00.707 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000006", 2024-08-18T15:55:00.707 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000029", 2024-08-18T15:55:00.707 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:55:00.707 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000089", 2024-08-18T15:55:00.707 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:55:00.707 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:55:00.707 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000022", 2024-08-18T15:55:00.707 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:55:00.707 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:55:00.707 INFO:teuthology.orchestra.run.smithi193.stdout: "bilog.trim", 2024-08-18T15:55:00.707 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000016", 2024-08-18T15:55:00.708 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:55:00.708 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:55:00.708 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:55:00.708 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:55:00.708 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:55:00.708 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000041", 2024-08-18T15:55:00.708 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:55:00.708 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:55:00.708 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.log.01050e30-179b-43be-bdbb-3bb23cf37ae8.3", 2024-08-18T15:55:00.708 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:55:00.708 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000018", 2024-08-18T15:55:00.708 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000092", 2024-08-18T15:55:00.708 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:55:00.709 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:55:00.709 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:55:00.709 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:55:00.709 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.27", 2024-08-18T15:55:00.709 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.4", 2024-08-18T15:55:00.709 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:55:00.709 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:55:00.709 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:55:00.709 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:55:00.709 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:55:00.709 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:55:00.709 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:55:00.709 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000014", 2024-08-18T15:55:00.709 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:55:00.709 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3.retry", 2024-08-18T15:55:00.710 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:55:00.710 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:55:00.710 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:55:00.710 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:55:00.710 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:55:00.710 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000112", 2024-08-18T15:55:00.710 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:55:00.710 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000007", 2024-08-18T15:55:00.710 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000021", 2024-08-18T15:55:00.710 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:55:00.710 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:55:00.710 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:55:00.710 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:55:00.711 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:55:00.711 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.16", 2024-08-18T15:55:00.711 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:55:00.711 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000064", 2024-08-18T15:55:00.711 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:55:00.711 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:55:00.711 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:55:00.711 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:55:00.711 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:55:00.711 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:55:00.711 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:55:00.711 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000071", 2024-08-18T15:55:00.712 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:55:00.712 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:55:00.712 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:55:00.712 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000074", 2024-08-18T15:55:00.712 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:55:00.712 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:55:00.712 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:55:00.712 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:55:00.712 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:55:00.712 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:55:00.712 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:55:00.712 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:55:00.712 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:55:00.712 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:55:00.713 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:55:00.713 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:55:00.713 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:55:00.713 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:55:00.713 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:55:00.713 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000081", 2024-08-18T15:55:00.713 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:55:00.713 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000009", 2024-08-18T15:55:00.713 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000121", 2024-08-18T15:55:00.713 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1", 2024-08-18T15:55:00.714 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:55:00.714 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:55:00.714 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:55:00.714 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:55:00.714 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:55:00.714 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:55:00.714 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.17", 2024-08-18T15:55:00.714 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000125", 2024-08-18T15:55:00.714 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:55:00.714 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000082", 2024-08-18T15:55:00.714 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:55:00.715 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:55:00.715 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:55:00.715 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:55:00.715 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:55:00.715 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:55:00.715 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:55:00.715 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:55:00.715 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:55:00.715 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:55:00.715 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:55:00.716 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000105", 2024-08-18T15:55:00.716 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000059", 2024-08-18T15:55:00.716 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:55:00.716 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:55:00.716 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:55:00.716 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:55:00.716 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:55:00.716 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:55:00.716 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:55:00.716 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:55:00.716 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:55:00.716 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:55:00.716 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:55:00.717 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000077", 2024-08-18T15:55:00.717 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000032", 2024-08-18T15:55:00.717 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:55:00.717 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:55:00.717 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:55:00.717 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:55:00.717 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:55:00.717 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:55:00.717 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.24", 2024-08-18T15:55:00.717 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:55:00.718 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:55:00.718 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:55:00.718 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:55:00.718 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3.retry", 2024-08-18T15:55:00.718 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:55:00.718 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:55:00.718 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:55:00.718 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:55:00.718 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:55:00.718 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:55:00.718 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000053", 2024-08-18T15:55:00.718 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000091", 2024-08-18T15:55:00.719 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:55:00.719 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:55:00.719 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:55:00.719 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000065", 2024-08-18T15:55:00.719 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:55:00.719 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:55:00.719 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:55:00.719 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:55:00.719 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000083", 2024-08-18T15:55:00.719 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000010", 2024-08-18T15:55:00.719 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:55:00.719 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:55:00.719 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:55:00.719 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:55:00.720 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:55:00.720 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:55:00.720 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:55:00.720 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:55:00.720 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:55:00.720 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:55:00.720 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:55:00.720 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.7", 2024-08-18T15:55:00.720 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.3", 2024-08-18T15:55:00.720 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:55:00.720 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:55:00.720 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:55:00.721 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:55:00.721 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:55:00.721 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:55:00.721 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:55:00.721 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:55:00.721 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.2", 2024-08-18T15:55:00.721 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:55:00.721 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:55:00.721 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:55:00.721 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:55:00.721 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:55:00.721 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:55:00.721 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:55:00.721 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000045", 2024-08-18T15:55:00.722 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000002", 2024-08-18T15:55:00.722 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:55:00.722 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3.2UbiLidwSTy.0", 2024-08-18T15:55:00.722 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:55:00.722 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.12", 2024-08-18T15:55:00.722 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:55:00.722 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:55:00.722 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:55:00.722 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:55:00.722 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:55:00.722 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:55:00.722 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000116", 2024-08-18T15:55:00.722 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.25", 2024-08-18T15:55:00.723 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:55:00.723 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:55:00.723 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:55:00.723 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:55:00.723 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000034", 2024-08-18T15:55:00.723 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:55:00.723 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:55:00.723 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000101", 2024-08-18T15:55:00.723 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:55:00.723 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:55:00.723 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.13", 2024-08-18T15:55:00.723 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:55:00.723 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:55:00.724 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.1", 2024-08-18T15:55:00.724 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0.retry", 2024-08-18T15:55:00.724 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:55:00.724 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:55:00.724 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000079", 2024-08-18T15:55:00.724 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000049", 2024-08-18T15:55:00.724 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:55:00.724 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0", 2024-08-18T15:55:00.724 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:55:00.724 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:55:00.724 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000117", 2024-08-18T15:55:00.724 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:55:00.725 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000044", 2024-08-18T15:55:00.725 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:55:00.725 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:55:00.725 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:55:00.725 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:55:00.725 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:55:00.725 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:55:00.725 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.14", 2024-08-18T15:55:00.725 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:55:00.725 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000066", 2024-08-18T15:55:00.725 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:55:00.725 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:55:00.725 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:55:00.726 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:55:00.726 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:55:00.726 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000068", 2024-08-18T15:55:00.726 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:55:00.726 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000085", 2024-08-18T15:55:00.726 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:55:00.726 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:55:00.726 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:55:00.726 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:55:00.726 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:55:00.726 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:55:00.726 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:55:00.727 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:55:00.727 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:55:00.727 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:55:00.727 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:55:00.727 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:55:00.727 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000073", 2024-08-18T15:55:00.727 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:55:00.727 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:55:00.727 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:55:00.727 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000038", 2024-08-18T15:55:00.727 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:55:00.727 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:55:00.727 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:55:00.727 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:55:00.728 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.10", 2024-08-18T15:55:00.728 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:55:00.728 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:55:00.728 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:55:00.728 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000118", 2024-08-18T15:55:00.728 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:55:00.728 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3", 2024-08-18T15:55:00.728 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:55:00.728 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000036", 2024-08-18T15:55:00.728 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:55:00.728 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:55:00.728 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:55:00.728 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:55:00.728 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:55:00.729 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000103", 2024-08-18T15:55:00.729 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.3", 2024-08-18T15:55:00.729 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:55:00.729 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000119", 2024-08-18T15:55:00.729 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000098", 2024-08-18T15:55:00.729 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:55:00.729 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:55:00.729 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:55:00.729 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:55:00.729 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:55:00.729 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:55:00.729 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:55:00.729 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:55:00.730 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.0", 2024-08-18T15:55:00.730 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:55:00.730 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:55:00.730 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:55:00.730 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:55:00.730 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:55:00.730 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:55:00.730 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:55:00.730 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:55:00.730 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000027", 2024-08-18T15:55:00.730 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:55:00.730 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status", 2024-08-18T15:55:00.730 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:55:00.731 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000019", 2024-08-18T15:55:00.731 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:55:00.731 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:55:00.731 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:55:00.731 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:55:00.731 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1", 2024-08-18T15:55:00.731 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:55:00.731 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:55:00.731 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:55:00.731 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:55:00.731 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000039", 2024-08-18T15:55:00.731 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:55:00.731 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:55:00.732 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:55:00.732 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:55:00.732 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:55:00.732 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:55:00.732 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:55:00.732 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:55:00.732 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:55:00.732 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:55:00.732 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:55:00.732 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000100", 2024-08-18T15:55:00.732 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:55:00.732 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:55:00.732 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:55:00.732 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:55:00.733 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:55:00.733 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000093", 2024-08-18T15:55:00.733 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:55:00.733 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000004", 2024-08-18T15:55:00.733 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000063", 2024-08-18T15:55:00.733 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000122", 2024-08-18T15:55:00.733 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:55:00.733 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:55:00.733 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:55:00.733 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000057", 2024-08-18T15:55:00.733 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:55:00.733 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:55:00.733 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:55:00.734 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:55:00.734 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000054", 2024-08-18T15:55:00.734 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:55:00.734 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000114", 2024-08-18T15:55:00.734 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:55:00.734 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:55:00.734 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:55:00.734 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.2", 2024-08-18T15:55:00.734 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:55:00.734 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:55:00.734 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0", 2024-08-18T15:55:00.734 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:55:00.734 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:55:00.735 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:55:00.735 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:55:00.735 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:55:00.735 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:55:00.735 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000012", 2024-08-18T15:55:00.735 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:55:00.735 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:55:00.735 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:55:00.735 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000084", 2024-08-18T15:55:00.735 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000043", 2024-08-18T15:55:00.735 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3", 2024-08-18T15:55:00.735 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:55:00.735 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:55:00.735 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.11", 2024-08-18T15:55:00.736 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:55:00.736 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:55:00.736 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:55:00.736 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:55:00.736 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:55:00.736 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:55:00.736 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:55:00.736 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:55:00.736 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000111", 2024-08-18T15:55:00.737 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:55:00.737 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000048", 2024-08-18T15:55:00.737 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:55:00.737 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000020", 2024-08-18T15:55:00.737 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:55:00.737 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:55:00.737 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:55:00.737 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:55:00.737 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:55:00.737 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000099", 2024-08-18T15:55:00.737 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000056", 2024-08-18T15:55:00.737 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:55:00.737 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:55:00.738 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.19", 2024-08-18T15:55:00.738 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:55:00.738 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.28", 2024-08-18T15:55:00.738 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:55:00.738 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:55:00.738 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:55:00.738 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:55:00.738 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:55:00.738 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000072", 2024-08-18T15:55:00.738 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:55:00.738 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:55:00.738 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000062", 2024-08-18T15:55:00.739 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:55:00.739 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:55:00.739 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:55:00.739 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000115", 2024-08-18T15:55:00.739 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:55:00.739 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.1", 2024-08-18T15:55:00.739 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2", 2024-08-18T15:55:00.739 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000033", 2024-08-18T15:55:00.739 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:55:00.739 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:55:00.739 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:55:00.739 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:55:00.739 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:55:00.739 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:55:00.740 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:55:00.740 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:55:00.740 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:55:00.740 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:55:00.740 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:55:00.740 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:55:00.740 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:55:00.740 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:55:00.740 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:55:00.740 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:55:00.740 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:55:00.740 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:55:00.740 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:55:00.740 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000025", 2024-08-18T15:55:00.740 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:55:00.741 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000123", 2024-08-18T15:55:00.741 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.31", 2024-08-18T15:55:00.741 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:55:00.741 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:55:00.741 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:55:00.741 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:55:00.741 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.18", 2024-08-18T15:55:00.741 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:55:00.741 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:55:00.741 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:55:00.741 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:55:00.741 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:55:00.741 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000108", 2024-08-18T15:55:00.742 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:55:00.742 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:55:00.742 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000094", 2024-08-18T15:55:00.742 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:55:00.742 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:55:00.742 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:55:00.742 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:55:00.742 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:55:00.742 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:55:00.742 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000060", 2024-08-18T15:55:00.742 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:55:00.742 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:55:00.742 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:55:00.743 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000109", 2024-08-18T15:55:00.743 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.15", 2024-08-18T15:55:00.743 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:55:00.743 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:55:00.743 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:55:00.743 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:55:00.743 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:55:00.743 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:55:00.743 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:55:00.743 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:55:00.743 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:55:00.743 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000013", 2024-08-18T15:55:00.743 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:55:00.744 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:55:00.744 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:55:00.744 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:55:00.744 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:55:00.744 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:55:00.744 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:55:00.744 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:55:00.744 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:55:00.744 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:55:00.744 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:55:00.744 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:55:00.744 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:55:00.744 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:55:00.744 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000051", 2024-08-18T15:55:00.745 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:55:00.745 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:55:00.745 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:55:00.745 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:55:00.745 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000106", 2024-08-18T15:55:00.745 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:55:00.745 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:55:00.745 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:55:00.745 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:55:00.745 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:55:00.745 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000017", 2024-08-18T15:55:00.745 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:55:00.745 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:55:00.745 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:55:00.745 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000023", 2024-08-18T15:55:00.745 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:55:00.746 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:55:00.746 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:55:00.746 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:55:00.746 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:55:00.746 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:55:00.746 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:55:00.746 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000107", 2024-08-18T15:55:00.746 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:55:00.746 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:55:00.746 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000046", 2024-08-18T15:55:00.746 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:55:00.746 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:55:00.746 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:55:00.747 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.26", 2024-08-18T15:55:00.747 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:55:00.747 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:55:00.747 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9", 2024-08-18T15:55:00.747 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.22", 2024-08-18T15:55:00.747 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:55:00.747 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:55:00.747 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:55:00.747 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:55:00.747 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:55:00.747 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:55:00.747 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:55:00.747 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2", 2024-08-18T15:55:00.747 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:55:00.747 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:55:00.748 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000067", 2024-08-18T15:55:00.748 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:55:00.748 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:55:00.748 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:55:00.748 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:55:00.748 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:55:00.748 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:55:00.748 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:55:00.748 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:55:00.748 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:55:00.748 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:55:00.748 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:55:00.748 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:55:00.748 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:55:00.749 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:55:00.749 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:55:00.749 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:55:00.749 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:55:00.749 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:55:00.749 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:55:00.749 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.9", 2024-08-18T15:55:00.749 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000086", 2024-08-18T15:55:00.749 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.5", 2024-08-18T15:55:00.749 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:55:00.749 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:55:00.749 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:55:00.749 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:55:00.749 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:55:00.749 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:55:00.750 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2.MFXJ583gCr1.0", 2024-08-18T15:55:00.750 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:55:00.750 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1.GPnO16liyZ+.0", 2024-08-18T15:55:00.750 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:55:00.750 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:55:00.750 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:55:00.750 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:55:00.750 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:55:00.750 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:55:00.750 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:55:00.750 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:55:00.750 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:55:00.750 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.20", 2024-08-18T15:55:00.751 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:55:00.751 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:55:00.751 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:55:00.751 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:55:00.751 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:55:00.751 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:55:00.751 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1.retry", 2024-08-18T15:55:00.751 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:55:00.751 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:55:00.751 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:55:00.751 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0.retry", 2024-08-18T15:55:00.751 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000050", 2024-08-18T15:55:00.751 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:55:00.753 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000102", 2024-08-18T15:55:00.753 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:55:00.753 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:55:00.753 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:55:00.754 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:55:00.754 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:55:00.754 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:55:00.754 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:55:00.754 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:55:00.754 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:55:00.754 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:55:00.754 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:55:00.754 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000124", 2024-08-18T15:55:00.754 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:55:00.755 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:55:00.755 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:55:00.755 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000001", 2024-08-18T15:55:00.755 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:55:00.755 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:55:00.755 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:55:00.755 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:55:00.755 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000126", 2024-08-18T15:55:00.755 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:55:00.755 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0.fZhctfLXnSt.0", 2024-08-18T15:55:00.755 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:55:00.755 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000055", 2024-08-18T15:55:00.755 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:55:00.755 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000061", 2024-08-18T15:55:00.755 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1", 2024-08-18T15:55:00.756 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:55:00.756 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:55:00.756 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:55:00.756 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000097", 2024-08-18T15:55:00.756 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:55:00.756 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:55:00.756 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:55:00.756 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:55:00.756 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:55:00.756 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:55:00.756 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:55:00.756 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:55:00.756 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000042", 2024-08-18T15:55:00.756 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:55:00.757 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:55:00.757 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:55:00.757 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1.retry", 2024-08-18T15:55:00.757 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:55:00.757 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:55:00.757 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:55:00.757 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:55:00.757 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:55:00.757 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:55:00.757 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:55:00.757 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000075", 2024-08-18T15:55:00.757 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:55:00.757 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000120", 2024-08-18T15:55:00.757 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000030", 2024-08-18T15:55:00.757 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:55:00.757 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:55:00.758 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:55:00.758 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:55:00.758 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:55:00.758 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:55:00.758 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9" 2024-08-18T15:55:00.758 INFO:teuthology.orchestra.run.smithi193.stdout:] 2024-08-18T15:55:00.758 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:55:05.759 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:55:05.760 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:55:05.760 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:55:05.812 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:55:05.812 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:55:05.875 INFO:teuthology.orchestra.run.smithi193.stdout:[ 2024-08-18T15:55:05.875 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:55:05.875 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:55:05.875 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.29", 2024-08-18T15:55:05.876 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000078", 2024-08-18T15:55:05.876 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.history", 2024-08-18T15:55:05.876 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:55:05.876 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:55:05.876 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:55:05.876 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:55:05.876 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.23", 2024-08-18T15:55:05.876 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000070", 2024-08-18T15:55:05.876 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000104", 2024-08-18T15:55:05.876 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:55:05.876 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2.retry", 2024-08-18T15:55:05.876 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:55:05.877 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:55:05.877 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:55:05.877 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:55:05.877 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000026", 2024-08-18T15:55:05.877 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:55:05.877 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:55:05.877 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:55:05.877 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:55:05.877 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:55:05.877 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:55:05.877 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.8", 2024-08-18T15:55:05.877 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:55:05.877 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:55:05.878 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:55:05.878 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:55:05.878 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000028", 2024-08-18T15:55:05.878 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:55:05.878 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:55:05.878 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:55:05.878 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:55:05.878 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:55:05.878 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:55:05.878 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:55:05.878 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2.retry", 2024-08-18T15:55:05.878 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:55:05.878 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.30", 2024-08-18T15:55:05.879 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:55:05.879 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:55:05.879 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:55:05.879 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:55:05.879 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000040", 2024-08-18T15:55:05.879 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:55:05.879 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.6", 2024-08-18T15:55:05.879 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:55:05.879 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:55:05.879 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:55:05.879 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:55:05.879 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:55:05.879 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:55:05.880 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000015", 2024-08-18T15:55:05.880 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:55:05.880 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:55:05.880 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000069", 2024-08-18T15:55:05.880 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:55:05.880 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:55:05.880 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:55:05.880 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000095", 2024-08-18T15:55:05.880 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0", 2024-08-18T15:55:05.880 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000003", 2024-08-18T15:55:05.881 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000047", 2024-08-18T15:55:05.881 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:55:05.881 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:55:05.881 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:55:05.881 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:55:05.881 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:55:05.881 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:55:05.881 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:55:05.881 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:55:05.881 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:55:05.881 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000035", 2024-08-18T15:55:05.881 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:55:05.881 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:55:05.882 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000037", 2024-08-18T15:55:05.882 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:55:05.882 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:55:05.882 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:55:05.882 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:55:05.882 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:55:05.882 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:55:05.882 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:55:05.882 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:55:05.882 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:55:05.882 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:55:05.882 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:55:05.883 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:55:05.883 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:55:05.883 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:55:05.883 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000024", 2024-08-18T15:55:05.883 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:55:05.883 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:55:05.883 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:55:05.883 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:55:05.883 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000000", 2024-08-18T15:55:05.883 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:55:05.883 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000031", 2024-08-18T15:55:05.883 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:55:05.883 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:55:05.883 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:55:05.884 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:55:05.884 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:55:05.884 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:55:05.884 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3", 2024-08-18T15:55:05.884 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:55:05.884 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:55:05.884 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:55:05.884 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:55:05.884 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:55:05.884 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:55:05.884 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:55:05.884 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:55:05.884 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:55:05.885 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:55:05.885 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:55:05.885 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:55:05.885 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:55:05.885 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:55:05.885 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:55:05.885 INFO:teuthology.orchestra.run.smithi193.stdout: "data_loggenerations_metadata", 2024-08-18T15:55:05.885 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2", 2024-08-18T15:55:05.885 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:55:05.885 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:55:05.886 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:55:05.886 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:55:05.886 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T15:55:05.886 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:55:05.886 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:55:05.886 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:55:05.886 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:55:05.886 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:55:05.886 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000076", 2024-08-18T15:55:05.886 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:55:05.886 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:55:05.886 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000113", 2024-08-18T15:55:05.886 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000005", 2024-08-18T15:55:05.887 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:55:05.887 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000011", 2024-08-18T15:55:05.887 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:55:05.887 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000058", 2024-08-18T15:55:05.887 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:55:05.887 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:55:05.887 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:55:05.887 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000052", 2024-08-18T15:55:05.887 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:55:05.887 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:55:05.887 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:55:05.887 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:55:05.888 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.21", 2024-08-18T15:55:05.888 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:55:05.888 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:55:05.888 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:55:05.888 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:55:05.888 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:55:05.888 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:55:05.888 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:55:05.888 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:55:05.888 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:55:05.888 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:55:05.888 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:55:05.888 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:55:05.889 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:55:05.889 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:55:05.889 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:55:05.889 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:55:05.889 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:55:05.889 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:55:05.889 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:55:05.889 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000088", 2024-08-18T15:55:05.889 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000080", 2024-08-18T15:55:05.889 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:55:05.889 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:55:05.889 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000090", 2024-08-18T15:55:05.889 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:55:05.890 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:55:05.890 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000110", 2024-08-18T15:55:05.890 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:55:05.890 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:55:05.890 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:55:05.890 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.0", 2024-08-18T15:55:05.890 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:55:05.890 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000096", 2024-08-18T15:55:05.890 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:55:05.890 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:55:05.890 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:55:05.890 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:55:05.890 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:55:05.891 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000087", 2024-08-18T15:55:05.891 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:55:05.891 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:55:05.891 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:55:05.891 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:55:05.891 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000008", 2024-08-18T15:55:05.891 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:55:05.891 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:55:05.891 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:55:05.891 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:55:05.891 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:55:05.891 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000006", 2024-08-18T15:55:05.891 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000029", 2024-08-18T15:55:05.892 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:55:05.892 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000089", 2024-08-18T15:55:05.892 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:55:05.892 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:55:05.892 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000022", 2024-08-18T15:55:05.892 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:55:05.892 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:55:05.892 INFO:teuthology.orchestra.run.smithi193.stdout: "bilog.trim", 2024-08-18T15:55:05.892 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000016", 2024-08-18T15:55:05.892 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:55:05.892 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:55:05.892 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:55:05.892 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:55:05.892 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:55:05.893 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000041", 2024-08-18T15:55:05.893 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:55:05.893 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:55:05.893 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.log.01050e30-179b-43be-bdbb-3bb23cf37ae8.3", 2024-08-18T15:55:05.893 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:55:05.893 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000018", 2024-08-18T15:55:05.893 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000092", 2024-08-18T15:55:05.893 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:55:05.893 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:55:05.893 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:55:05.893 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:55:05.893 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.27", 2024-08-18T15:55:05.894 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.4", 2024-08-18T15:55:05.894 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:55:05.894 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:55:05.894 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:55:05.894 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:55:05.894 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:55:05.894 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:55:05.894 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:55:05.894 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000014", 2024-08-18T15:55:05.894 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:55:05.894 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3.retry", 2024-08-18T15:55:05.894 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:55:05.894 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:55:05.895 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:55:05.895 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:55:05.895 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:55:05.895 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000112", 2024-08-18T15:55:05.895 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:55:05.895 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000007", 2024-08-18T15:55:05.895 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000021", 2024-08-18T15:55:05.895 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:55:05.895 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:55:05.895 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:55:05.895 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:55:05.895 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:55:05.896 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.16", 2024-08-18T15:55:05.896 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:55:05.896 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000064", 2024-08-18T15:55:05.896 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:55:05.896 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:55:05.896 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:55:05.896 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:55:05.897 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:55:05.897 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:55:05.897 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:55:05.897 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000071", 2024-08-18T15:55:05.897 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:55:05.897 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:55:05.897 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:55:05.897 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000074", 2024-08-18T15:55:05.897 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:55:05.898 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:55:05.898 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:55:05.898 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:55:05.898 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:55:05.898 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:55:05.898 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:55:05.898 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:55:05.898 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:55:05.898 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:55:05.898 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:55:05.898 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:55:05.898 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:55:05.898 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:55:05.899 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:55:05.899 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000081", 2024-08-18T15:55:05.899 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:55:05.899 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000009", 2024-08-18T15:55:05.899 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000121", 2024-08-18T15:55:05.899 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1", 2024-08-18T15:55:05.899 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:55:05.899 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:55:05.899 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:55:05.899 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:55:05.899 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:55:05.899 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:55:05.899 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.17", 2024-08-18T15:55:05.900 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000125", 2024-08-18T15:55:05.900 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:55:05.900 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000082", 2024-08-18T15:55:05.900 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:55:05.900 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:55:05.900 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:55:05.900 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:55:05.900 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:55:05.900 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:55:05.900 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:55:05.900 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:55:05.900 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:55:05.900 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:55:05.901 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:55:05.901 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000105", 2024-08-18T15:55:05.901 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000059", 2024-08-18T15:55:05.901 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:55:05.901 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:55:05.901 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:55:05.901 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:55:05.901 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:55:05.901 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:55:05.901 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:55:05.901 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:55:05.901 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:55:05.901 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:55:05.902 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:55:05.902 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000077", 2024-08-18T15:55:05.902 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000032", 2024-08-18T15:55:05.902 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:55:05.902 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:55:05.902 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:55:05.902 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:55:05.902 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:55:05.902 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:55:05.902 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.24", 2024-08-18T15:55:05.902 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:55:05.902 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:55:05.903 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:55:05.903 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:55:05.903 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3.retry", 2024-08-18T15:55:05.903 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:55:05.903 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:55:05.903 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:55:05.903 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:55:05.903 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:55:05.903 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:55:05.903 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000053", 2024-08-18T15:55:05.903 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000091", 2024-08-18T15:55:05.903 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:55:05.903 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:55:05.904 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:55:05.904 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000065", 2024-08-18T15:55:05.904 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:55:05.904 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:55:05.904 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:55:05.904 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:55:05.904 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000083", 2024-08-18T15:55:05.904 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000010", 2024-08-18T15:55:05.904 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:55:05.904 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:55:05.904 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:55:05.904 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:55:05.904 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:55:05.905 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:55:05.905 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:55:05.905 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:55:05.905 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:55:05.905 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:55:05.905 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:55:05.905 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.7", 2024-08-18T15:55:05.905 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.3", 2024-08-18T15:55:05.905 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:55:05.905 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:55:05.905 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:55:05.905 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:55:05.905 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:55:05.905 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:55:05.906 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:55:05.906 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:55:05.906 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.2", 2024-08-18T15:55:05.906 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:55:05.906 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:55:05.906 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:55:05.906 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:55:05.906 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:55:05.906 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:55:05.906 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:55:05.906 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000045", 2024-08-18T15:55:05.906 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000002", 2024-08-18T15:55:05.906 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:55:05.907 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3.2UbiLidwSTy.0", 2024-08-18T15:55:05.907 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:55:05.907 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.12", 2024-08-18T15:55:05.907 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:55:05.907 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:55:05.907 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:55:05.907 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:55:05.907 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:55:05.907 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:55:05.907 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000116", 2024-08-18T15:55:05.907 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.25", 2024-08-18T15:55:05.907 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:55:05.907 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:55:05.908 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:55:05.908 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:55:05.908 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000034", 2024-08-18T15:55:05.908 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:55:05.908 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:55:05.908 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000101", 2024-08-18T15:55:05.908 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:55:05.908 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:55:05.908 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.13", 2024-08-18T15:55:05.908 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:55:05.908 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:55:05.908 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.1", 2024-08-18T15:55:05.909 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0.retry", 2024-08-18T15:55:05.909 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:55:05.909 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:55:05.909 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000079", 2024-08-18T15:55:05.909 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000049", 2024-08-18T15:55:05.909 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:55:05.909 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0", 2024-08-18T15:55:05.909 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:55:05.909 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:55:05.909 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000117", 2024-08-18T15:55:05.909 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:55:05.909 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000044", 2024-08-18T15:55:05.909 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:55:05.910 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:55:05.910 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:55:05.910 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:55:05.910 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:55:05.910 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:55:05.910 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.14", 2024-08-18T15:55:05.910 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:55:05.910 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000066", 2024-08-18T15:55:05.910 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:55:05.910 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:55:05.910 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:55:05.910 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:55:05.911 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:55:05.911 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000068", 2024-08-18T15:55:05.911 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:55:05.911 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000085", 2024-08-18T15:55:05.911 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:55:05.911 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:55:05.911 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:55:05.911 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:55:05.911 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:55:05.911 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:55:05.911 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:55:05.911 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:55:05.911 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:55:05.912 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:55:05.912 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:55:05.912 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:55:05.912 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000073", 2024-08-18T15:55:05.912 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:55:05.912 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:55:05.912 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:55:05.912 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000038", 2024-08-18T15:55:05.912 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:55:05.912 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:55:05.912 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:55:05.912 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:55:05.912 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.10", 2024-08-18T15:55:05.912 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:55:05.913 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:55:05.913 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:55:05.913 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000118", 2024-08-18T15:55:05.913 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:55:05.913 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3", 2024-08-18T15:55:05.913 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:55:05.913 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000036", 2024-08-18T15:55:05.913 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:55:05.913 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:55:05.913 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:55:05.913 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:55:05.913 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:55:05.913 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000103", 2024-08-18T15:55:05.914 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.3", 2024-08-18T15:55:05.914 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:55:05.914 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000119", 2024-08-18T15:55:05.914 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000098", 2024-08-18T15:55:05.914 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:55:05.914 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:55:05.914 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:55:05.914 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:55:05.914 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:55:05.914 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:55:05.914 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:55:05.914 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:55:05.914 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.0", 2024-08-18T15:55:05.915 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:55:05.915 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:55:05.915 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:55:05.915 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:55:05.915 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:55:05.915 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:55:05.915 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:55:05.915 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:55:05.915 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000027", 2024-08-18T15:55:05.915 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:55:05.915 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status", 2024-08-18T15:55:05.915 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:55:05.915 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000019", 2024-08-18T15:55:05.916 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:55:05.916 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:55:05.916 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:55:05.916 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:55:05.916 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1", 2024-08-18T15:55:05.916 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:55:05.916 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:55:05.916 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:55:05.916 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:55:05.916 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000039", 2024-08-18T15:55:05.916 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:55:05.916 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:55:05.916 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:55:05.916 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:55:05.917 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:55:05.917 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:55:05.917 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:55:05.917 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:55:05.917 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:55:05.917 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:55:05.917 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:55:05.917 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000100", 2024-08-18T15:55:05.917 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:55:05.917 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:55:05.917 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:55:05.917 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:55:05.917 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:55:05.918 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000093", 2024-08-18T15:55:05.918 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:55:05.918 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000004", 2024-08-18T15:55:05.918 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000063", 2024-08-18T15:55:05.918 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000122", 2024-08-18T15:55:05.918 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:55:05.918 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:55:05.918 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:55:05.918 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000057", 2024-08-18T15:55:05.918 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:55:05.918 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:55:05.918 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:55:05.918 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:55:05.919 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000054", 2024-08-18T15:55:05.919 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:55:05.919 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000114", 2024-08-18T15:55:05.919 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:55:05.919 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:55:05.919 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:55:05.919 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.2", 2024-08-18T15:55:05.919 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:55:05.919 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:55:05.919 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0", 2024-08-18T15:55:05.919 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:55:05.919 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:55:05.919 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:55:05.919 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:55:05.920 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:55:05.920 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:55:05.920 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000012", 2024-08-18T15:55:05.920 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:55:05.920 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:55:05.920 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:55:05.920 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000084", 2024-08-18T15:55:05.920 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000043", 2024-08-18T15:55:05.920 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3", 2024-08-18T15:55:05.920 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:55:05.920 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:55:05.920 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.11", 2024-08-18T15:55:05.920 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:55:05.921 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:55:05.921 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:55:05.921 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:55:05.921 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:55:05.921 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:55:05.921 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:55:05.921 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:55:05.921 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000111", 2024-08-18T15:55:05.921 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:55:05.921 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000048", 2024-08-18T15:55:05.921 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:55:05.921 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000020", 2024-08-18T15:55:05.921 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:55:05.922 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:55:05.922 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:55:05.922 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:55:05.922 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:55:05.922 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000099", 2024-08-18T15:55:05.922 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000056", 2024-08-18T15:55:05.922 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:55:05.922 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:55:05.922 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.19", 2024-08-18T15:55:05.922 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:55:05.922 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.28", 2024-08-18T15:55:05.922 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:55:05.922 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:55:05.923 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:55:05.923 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:55:05.923 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:55:05.923 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000072", 2024-08-18T15:55:05.923 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:55:05.923 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:55:05.923 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000062", 2024-08-18T15:55:05.923 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:55:05.923 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:55:05.923 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:55:05.923 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000115", 2024-08-18T15:55:05.923 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:55:05.923 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.1", 2024-08-18T15:55:05.924 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2", 2024-08-18T15:55:05.924 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000033", 2024-08-18T15:55:05.924 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:55:05.924 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:55:05.924 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:55:05.924 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:55:05.924 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:55:05.924 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:55:05.924 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:55:05.924 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:55:05.924 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:55:05.924 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:55:05.924 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:55:05.925 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:55:05.925 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:55:05.925 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:55:05.925 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:55:05.925 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:55:05.925 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:55:05.925 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:55:05.925 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:55:05.925 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000025", 2024-08-18T15:55:05.925 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:55:05.925 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000123", 2024-08-18T15:55:05.925 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.31", 2024-08-18T15:55:05.925 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:55:05.926 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:55:05.926 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:55:05.926 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:55:05.926 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.18", 2024-08-18T15:55:05.926 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:55:05.926 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:55:05.926 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:55:05.926 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:55:05.926 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:55:05.926 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000108", 2024-08-18T15:55:05.926 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:55:05.926 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:55:05.926 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000094", 2024-08-18T15:55:05.927 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:55:05.927 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:55:05.927 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:55:05.927 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:55:05.927 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:55:05.927 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:55:05.927 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000060", 2024-08-18T15:55:05.927 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:55:05.927 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:55:05.927 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:55:05.927 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000109", 2024-08-18T15:55:05.927 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.15", 2024-08-18T15:55:05.927 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:55:05.928 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:55:05.928 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:55:05.928 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:55:05.928 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:55:05.928 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:55:05.928 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:55:05.928 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:55:05.928 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:55:05.928 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000013", 2024-08-18T15:55:05.928 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:55:05.928 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:55:05.928 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:55:05.928 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:55:05.929 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:55:05.929 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:55:05.929 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:55:05.929 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:55:05.929 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:55:05.929 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:55:05.929 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:55:05.929 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:55:05.929 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:55:05.929 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:55:05.929 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000051", 2024-08-18T15:55:05.929 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:55:05.929 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:55:05.929 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:55:05.930 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:55:05.930 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000106", 2024-08-18T15:55:05.930 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:55:05.930 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:55:05.930 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:55:05.930 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:55:05.930 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:55:05.930 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000017", 2024-08-18T15:55:05.930 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:55:05.930 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:55:05.930 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:55:05.930 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000023", 2024-08-18T15:55:05.930 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:55:05.931 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:55:05.931 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:55:05.931 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:55:05.931 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:55:05.931 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:55:05.931 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:55:05.931 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000107", 2024-08-18T15:55:05.931 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:55:05.931 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:55:05.931 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000046", 2024-08-18T15:55:05.932 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:55:05.932 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:55:05.932 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:55:05.932 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.26", 2024-08-18T15:55:05.932 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:55:05.932 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:55:05.932 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9", 2024-08-18T15:55:05.932 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.22", 2024-08-18T15:55:05.932 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:55:05.932 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:55:05.932 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:55:05.932 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:55:05.932 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:55:05.933 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:55:05.933 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:55:05.933 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2", 2024-08-18T15:55:05.933 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:55:05.933 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:55:05.933 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000067", 2024-08-18T15:55:05.933 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:55:05.933 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:55:05.933 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:55:05.933 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:55:05.933 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:55:05.933 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:55:05.934 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:55:05.934 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:55:05.934 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:55:05.934 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:55:05.934 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:55:05.934 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:55:05.934 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:55:05.934 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:55:05.934 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:55:05.934 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:55:05.934 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:55:05.934 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:55:05.934 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:55:05.935 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.9", 2024-08-18T15:55:05.935 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000086", 2024-08-18T15:55:05.935 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.5", 2024-08-18T15:55:05.935 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:55:05.935 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:55:05.935 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:55:05.935 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:55:05.935 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:55:05.935 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:55:05.935 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2.MFXJ583gCr1.0", 2024-08-18T15:55:05.935 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:55:05.935 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1.GPnO16liyZ+.0", 2024-08-18T15:55:05.935 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:55:05.936 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:55:05.936 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:55:05.936 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:55:05.936 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:55:05.936 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:55:05.936 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:55:05.936 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:55:05.936 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:55:05.936 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.20", 2024-08-18T15:55:05.936 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:55:05.936 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:55:05.936 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:55:05.936 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:55:05.936 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:55:05.937 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:55:05.937 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1.retry", 2024-08-18T15:55:05.937 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:55:05.937 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:55:05.937 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:55:05.937 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0.retry", 2024-08-18T15:55:05.937 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000050", 2024-08-18T15:55:05.937 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:55:05.937 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000102", 2024-08-18T15:55:05.937 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:55:05.937 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:55:05.937 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:55:05.937 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:55:05.938 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:55:05.938 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:55:05.938 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:55:05.938 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:55:05.938 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:55:05.938 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:55:05.938 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:55:05.938 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000124", 2024-08-18T15:55:05.938 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:55:05.938 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:55:05.938 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:55:05.938 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000001", 2024-08-18T15:55:05.938 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:55:05.939 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:55:05.939 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:55:05.939 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:55:05.939 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000126", 2024-08-18T15:55:05.939 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:55:05.939 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0.fZhctfLXnSt.0", 2024-08-18T15:55:05.939 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:55:05.939 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000055", 2024-08-18T15:55:05.939 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:55:05.939 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000061", 2024-08-18T15:55:05.939 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1", 2024-08-18T15:55:05.939 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:55:05.940 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:55:05.940 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:55:05.940 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000097", 2024-08-18T15:55:05.940 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:55:05.940 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:55:05.940 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:55:05.940 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:55:05.940 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:55:05.940 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:55:05.940 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:55:05.940 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:55:05.940 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000042", 2024-08-18T15:55:05.940 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:55:05.941 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:55:05.941 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:55:05.941 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1.retry", 2024-08-18T15:55:05.941 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:55:05.941 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:55:05.941 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:55:05.941 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:55:05.941 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:55:05.941 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:55:05.941 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:55:05.941 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000075", 2024-08-18T15:55:05.941 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:55:05.941 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000120", 2024-08-18T15:55:05.941 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000030", 2024-08-18T15:55:05.942 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:55:05.942 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:55:05.942 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:55:05.942 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:55:05.942 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:55:05.942 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:55:05.942 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9" 2024-08-18T15:55:05.942 INFO:teuthology.orchestra.run.smithi193.stdout:] 2024-08-18T15:55:05.942 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:55:10.944 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:55:10.944 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:55:10.945 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:55:11.000 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:55:11.000 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:55:11.064 INFO:teuthology.orchestra.run.smithi193.stdout:[ 2024-08-18T15:55:11.064 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:55:11.064 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:55:11.064 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.29", 2024-08-18T15:55:11.065 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000078", 2024-08-18T15:55:11.065 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.history", 2024-08-18T15:55:11.065 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:55:11.065 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:55:11.065 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:55:11.065 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:55:11.065 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.23", 2024-08-18T15:55:11.065 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000070", 2024-08-18T15:55:11.065 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000104", 2024-08-18T15:55:11.065 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:55:11.065 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2.retry", 2024-08-18T15:55:11.065 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:55:11.066 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:55:11.066 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:55:11.066 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:55:11.066 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000026", 2024-08-18T15:55:11.066 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:55:11.066 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:55:11.066 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:55:11.066 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:55:11.066 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:55:11.066 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:55:11.066 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.8", 2024-08-18T15:55:11.067 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:55:11.067 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:55:11.067 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:55:11.067 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:55:11.067 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000028", 2024-08-18T15:55:11.067 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:55:11.067 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:55:11.067 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:55:11.067 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:55:11.067 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:55:11.067 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:55:11.067 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:55:11.067 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2.retry", 2024-08-18T15:55:11.068 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:55:11.068 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.30", 2024-08-18T15:55:11.068 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:55:11.068 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:55:11.068 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:55:11.068 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:55:11.068 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000040", 2024-08-18T15:55:11.068 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:55:11.068 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.6", 2024-08-18T15:55:11.068 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:55:11.068 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:55:11.068 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:55:11.068 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:55:11.069 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:55:11.069 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:55:11.069 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000015", 2024-08-18T15:55:11.069 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:55:11.069 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:55:11.069 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000069", 2024-08-18T15:55:11.069 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:55:11.069 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:55:11.069 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:55:11.069 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000095", 2024-08-18T15:55:11.069 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0", 2024-08-18T15:55:11.069 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000003", 2024-08-18T15:55:11.070 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000047", 2024-08-18T15:55:11.070 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:55:11.070 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:55:11.070 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:55:11.070 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:55:11.070 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:55:11.070 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:55:11.070 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:55:11.070 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:55:11.070 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:55:11.070 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000035", 2024-08-18T15:55:11.070 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:55:11.071 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:55:11.071 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000037", 2024-08-18T15:55:11.071 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:55:11.071 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:55:11.071 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:55:11.071 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:55:11.071 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:55:11.071 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:55:11.071 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:55:11.071 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:55:11.071 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:55:11.071 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:55:11.071 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:55:11.072 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:55:11.072 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:55:11.072 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:55:11.072 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000024", 2024-08-18T15:55:11.072 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:55:11.072 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:55:11.072 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:55:11.072 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:55:11.072 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000000", 2024-08-18T15:55:11.072 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:55:11.072 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000031", 2024-08-18T15:55:11.072 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:55:11.073 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:55:11.073 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:55:11.073 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:55:11.073 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:55:11.073 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:55:11.073 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3", 2024-08-18T15:55:11.073 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:55:11.073 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:55:11.073 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:55:11.073 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:55:11.073 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:55:11.073 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:55:11.074 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:55:11.074 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:55:11.074 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:55:11.074 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:55:11.074 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:55:11.074 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:55:11.074 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:55:11.074 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:55:11.074 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:55:11.074 INFO:teuthology.orchestra.run.smithi193.stdout: "data_loggenerations_metadata", 2024-08-18T15:55:11.074 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2", 2024-08-18T15:55:11.075 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:55:11.075 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:55:11.075 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:55:11.075 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:55:11.075 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T15:55:11.075 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:55:11.075 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:55:11.075 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:55:11.075 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:55:11.075 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:55:11.075 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000076", 2024-08-18T15:55:11.075 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:55:11.075 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:55:11.076 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000113", 2024-08-18T15:55:11.076 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000005", 2024-08-18T15:55:11.076 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:55:11.076 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000011", 2024-08-18T15:55:11.076 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:55:11.076 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000058", 2024-08-18T15:55:11.076 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:55:11.076 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:55:11.076 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:55:11.076 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000052", 2024-08-18T15:55:11.076 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:55:11.076 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:55:11.076 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:55:11.077 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:55:11.077 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.21", 2024-08-18T15:55:11.077 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:55:11.077 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:55:11.077 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:55:11.077 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:55:11.077 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:55:11.077 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:55:11.077 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:55:11.077 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:55:11.077 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:55:11.077 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:55:11.078 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:55:11.078 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:55:11.078 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:55:11.078 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:55:11.078 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:55:11.078 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:55:11.078 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:55:11.078 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:55:11.078 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:55:11.078 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000088", 2024-08-18T15:55:11.078 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000080", 2024-08-18T15:55:11.078 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:55:11.079 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:55:11.079 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000090", 2024-08-18T15:55:11.079 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:55:11.079 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:55:11.079 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000110", 2024-08-18T15:55:11.079 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:55:11.079 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:55:11.079 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:55:11.079 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.0", 2024-08-18T15:55:11.079 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:55:11.079 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000096", 2024-08-18T15:55:11.079 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:55:11.080 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:55:11.080 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:55:11.080 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:55:11.080 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:55:11.080 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000087", 2024-08-18T15:55:11.080 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:55:11.080 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:55:11.080 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:55:11.080 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:55:11.080 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000008", 2024-08-18T15:55:11.080 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:55:11.080 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:55:11.080 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:55:11.080 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:55:11.081 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:55:11.081 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000006", 2024-08-18T15:55:11.081 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000029", 2024-08-18T15:55:11.081 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:55:11.081 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000089", 2024-08-18T15:55:11.081 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:55:11.081 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:55:11.081 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000022", 2024-08-18T15:55:11.081 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:55:11.081 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:55:11.081 INFO:teuthology.orchestra.run.smithi193.stdout: "bilog.trim", 2024-08-18T15:55:11.081 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000016", 2024-08-18T15:55:11.081 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:55:11.082 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:55:11.082 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:55:11.082 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:55:11.082 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:55:11.082 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000041", 2024-08-18T15:55:11.082 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:55:11.082 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:55:11.082 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.log.01050e30-179b-43be-bdbb-3bb23cf37ae8.3", 2024-08-18T15:55:11.082 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:55:11.082 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000018", 2024-08-18T15:55:11.082 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000092", 2024-08-18T15:55:11.082 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:55:11.082 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:55:11.083 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:55:11.083 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:55:11.083 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.27", 2024-08-18T15:55:11.083 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.4", 2024-08-18T15:55:11.083 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:55:11.083 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:55:11.083 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:55:11.083 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:55:11.083 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:55:11.083 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:55:11.083 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:55:11.083 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000014", 2024-08-18T15:55:11.083 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:55:11.084 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3.retry", 2024-08-18T15:55:11.084 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:55:11.084 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:55:11.084 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:55:11.084 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:55:11.084 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:55:11.084 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000112", 2024-08-18T15:55:11.084 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:55:11.084 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000007", 2024-08-18T15:55:11.084 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000021", 2024-08-18T15:55:11.084 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:55:11.084 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:55:11.085 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:55:11.085 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:55:11.085 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:55:11.085 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.16", 2024-08-18T15:55:11.085 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:55:11.085 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000064", 2024-08-18T15:55:11.085 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:55:11.085 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:55:11.085 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:55:11.085 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:55:11.085 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:55:11.085 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:55:11.086 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:55:11.086 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000071", 2024-08-18T15:55:11.086 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:55:11.086 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:55:11.086 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:55:11.086 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000074", 2024-08-18T15:55:11.086 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:55:11.086 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:55:11.086 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:55:11.086 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:55:11.086 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:55:11.086 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:55:11.087 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:55:11.087 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:55:11.087 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:55:11.087 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:55:11.087 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:55:11.087 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:55:11.087 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:55:11.087 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:55:11.087 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:55:11.087 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000081", 2024-08-18T15:55:11.087 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:55:11.087 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000009", 2024-08-18T15:55:11.088 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000121", 2024-08-18T15:55:11.088 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1", 2024-08-18T15:55:11.088 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:55:11.088 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:55:11.088 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:55:11.088 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:55:11.088 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:55:11.088 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:55:11.088 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.17", 2024-08-18T15:55:11.088 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000125", 2024-08-18T15:55:11.088 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:55:11.088 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000082", 2024-08-18T15:55:11.088 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:55:11.089 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:55:11.089 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:55:11.089 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:55:11.089 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:55:11.089 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:55:11.089 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:55:11.089 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:55:11.089 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:55:11.089 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:55:11.089 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:55:11.089 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000105", 2024-08-18T15:55:11.089 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000059", 2024-08-18T15:55:11.090 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:55:11.090 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:55:11.090 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:55:11.090 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:55:11.090 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:55:11.090 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:55:11.090 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:55:11.090 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:55:11.090 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:55:11.090 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:55:11.090 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:55:11.090 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000077", 2024-08-18T15:55:11.091 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000032", 2024-08-18T15:55:11.091 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:55:11.091 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:55:11.091 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:55:11.091 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:55:11.091 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:55:11.091 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:55:11.091 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.24", 2024-08-18T15:55:11.091 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:55:11.091 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:55:11.091 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:55:11.091 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:55:11.091 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3.retry", 2024-08-18T15:55:11.092 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:55:11.092 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:55:11.092 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:55:11.092 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:55:11.092 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:55:11.092 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:55:11.092 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000053", 2024-08-18T15:55:11.092 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000091", 2024-08-18T15:55:11.092 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:55:11.092 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:55:11.092 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:55:11.092 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000065", 2024-08-18T15:55:11.092 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:55:11.093 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:55:11.093 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:55:11.093 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:55:11.093 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000083", 2024-08-18T15:55:11.093 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000010", 2024-08-18T15:55:11.093 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:55:11.093 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:55:11.093 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:55:11.093 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:55:11.093 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:55:11.093 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:55:11.093 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:55:11.094 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:55:11.094 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:55:11.094 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:55:11.094 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:55:11.094 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.7", 2024-08-18T15:55:11.094 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.3", 2024-08-18T15:55:11.094 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:55:11.094 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:55:11.094 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:55:11.094 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:55:11.094 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:55:11.094 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:55:11.095 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:55:11.095 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:55:11.095 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.2", 2024-08-18T15:55:11.095 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:55:11.095 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:55:11.095 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:55:11.095 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:55:11.095 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:55:11.095 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:55:11.095 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:55:11.095 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000045", 2024-08-18T15:55:11.095 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000002", 2024-08-18T15:55:11.095 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:55:11.096 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3.2UbiLidwSTy.0", 2024-08-18T15:55:11.096 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:55:11.096 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.12", 2024-08-18T15:55:11.096 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:55:11.096 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:55:11.096 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:55:11.096 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:55:11.096 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:55:11.096 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:55:11.096 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000116", 2024-08-18T15:55:11.096 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.25", 2024-08-18T15:55:11.096 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:55:11.097 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:55:11.097 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:55:11.097 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:55:11.097 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000034", 2024-08-18T15:55:11.097 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:55:11.097 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:55:11.097 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000101", 2024-08-18T15:55:11.097 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:55:11.097 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:55:11.097 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.13", 2024-08-18T15:55:11.097 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:55:11.097 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:55:11.098 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.1", 2024-08-18T15:55:11.098 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0.retry", 2024-08-18T15:55:11.098 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:55:11.098 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:55:11.098 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000079", 2024-08-18T15:55:11.098 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000049", 2024-08-18T15:55:11.098 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:55:11.098 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0", 2024-08-18T15:55:11.098 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:55:11.098 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:55:11.098 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000117", 2024-08-18T15:55:11.098 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:55:11.098 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000044", 2024-08-18T15:55:11.099 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:55:11.099 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:55:11.099 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:55:11.099 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:55:11.099 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:55:11.099 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:55:11.099 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.14", 2024-08-18T15:55:11.099 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:55:11.099 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000066", 2024-08-18T15:55:11.099 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:55:11.099 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:55:11.099 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:55:11.099 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:55:11.100 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:55:11.100 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000068", 2024-08-18T15:55:11.100 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:55:11.100 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000085", 2024-08-18T15:55:11.100 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:55:11.100 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:55:11.100 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:55:11.100 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:55:11.100 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:55:11.100 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:55:11.100 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:55:11.100 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:55:11.100 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:55:11.101 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:55:11.101 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:55:11.101 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:55:11.101 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000073", 2024-08-18T15:55:11.101 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:55:11.101 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:55:11.101 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:55:11.101 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000038", 2024-08-18T15:55:11.101 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:55:11.101 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:55:11.101 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:55:11.101 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:55:11.101 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.10", 2024-08-18T15:55:11.102 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:55:11.102 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:55:11.102 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:55:11.102 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000118", 2024-08-18T15:55:11.102 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:55:11.102 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3", 2024-08-18T15:55:11.102 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:55:11.102 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000036", 2024-08-18T15:55:11.102 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:55:11.102 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:55:11.102 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:55:11.102 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:55:11.102 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:55:11.103 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000103", 2024-08-18T15:55:11.103 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.3", 2024-08-18T15:55:11.103 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:55:11.103 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000119", 2024-08-18T15:55:11.103 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000098", 2024-08-18T15:55:11.103 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:55:11.103 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:55:11.103 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:55:11.103 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:55:11.103 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:55:11.103 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:55:11.103 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:55:11.104 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:55:11.104 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.0", 2024-08-18T15:55:11.104 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:55:11.104 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:55:11.104 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:55:11.104 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:55:11.104 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:55:11.104 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:55:11.104 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:55:11.104 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:55:11.104 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000027", 2024-08-18T15:55:11.104 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:55:11.104 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status", 2024-08-18T15:55:11.105 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:55:11.105 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000019", 2024-08-18T15:55:11.105 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:55:11.105 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:55:11.105 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:55:11.105 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:55:11.105 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1", 2024-08-18T15:55:11.105 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:55:11.105 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:55:11.105 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:55:11.105 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:55:11.105 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000039", 2024-08-18T15:55:11.106 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:55:11.106 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:55:11.106 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:55:11.106 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:55:11.106 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:55:11.106 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:55:11.106 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:55:11.106 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:55:11.106 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:55:11.106 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:55:11.106 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:55:11.106 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000100", 2024-08-18T15:55:11.106 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:55:11.107 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:55:11.107 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:55:11.107 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:55:11.107 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:55:11.107 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000093", 2024-08-18T15:55:11.107 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:55:11.107 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000004", 2024-08-18T15:55:11.107 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000063", 2024-08-18T15:55:11.107 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000122", 2024-08-18T15:55:11.107 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:55:11.107 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:55:11.107 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:55:11.107 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000057", 2024-08-18T15:55:11.108 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:55:11.108 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:55:11.108 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:55:11.108 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:55:11.108 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000054", 2024-08-18T15:55:11.108 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:55:11.108 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000114", 2024-08-18T15:55:11.108 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:55:11.108 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:55:11.108 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:55:11.108 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.2", 2024-08-18T15:55:11.108 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:55:11.108 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:55:11.109 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0", 2024-08-18T15:55:11.109 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:55:11.109 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:55:11.109 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:55:11.109 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:55:11.109 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:55:11.109 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:55:11.109 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000012", 2024-08-18T15:55:11.109 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:55:11.109 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:55:11.109 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:55:11.109 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000084", 2024-08-18T15:55:11.109 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000043", 2024-08-18T15:55:11.110 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3", 2024-08-18T15:55:11.110 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:55:11.110 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:55:11.110 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.11", 2024-08-18T15:55:11.110 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:55:11.110 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:55:11.110 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:55:11.110 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:55:11.110 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:55:11.110 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:55:11.110 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:55:11.110 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:55:11.111 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000111", 2024-08-18T15:55:11.111 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:55:11.111 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000048", 2024-08-18T15:55:11.111 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:55:11.111 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000020", 2024-08-18T15:55:11.111 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:55:11.111 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:55:11.111 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:55:11.111 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:55:11.111 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:55:11.111 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000099", 2024-08-18T15:55:11.111 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000056", 2024-08-18T15:55:11.111 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:55:11.112 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:55:11.112 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.19", 2024-08-18T15:55:11.112 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:55:11.112 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.28", 2024-08-18T15:55:11.112 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:55:11.112 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:55:11.112 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:55:11.112 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:55:11.112 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:55:11.112 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000072", 2024-08-18T15:55:11.112 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:55:11.112 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:55:11.112 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000062", 2024-08-18T15:55:11.113 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:55:11.113 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:55:11.113 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:55:11.113 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000115", 2024-08-18T15:55:11.113 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:55:11.113 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.1", 2024-08-18T15:55:11.113 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2", 2024-08-18T15:55:11.113 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000033", 2024-08-18T15:55:11.113 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:55:11.113 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:55:11.113 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:55:11.113 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:55:11.113 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:55:11.113 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:55:11.114 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:55:11.114 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:55:11.114 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:55:11.114 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:55:11.114 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:55:11.114 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:55:11.114 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:55:11.114 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:55:11.114 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:55:11.114 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:55:11.114 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:55:11.115 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:55:11.115 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:55:11.115 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000025", 2024-08-18T15:55:11.115 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:55:11.115 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000123", 2024-08-18T15:55:11.115 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.31", 2024-08-18T15:55:11.115 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:55:11.115 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:55:11.115 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:55:11.115 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:55:11.115 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.18", 2024-08-18T15:55:11.116 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:55:11.116 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:55:11.116 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:55:11.116 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:55:11.116 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:55:11.116 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000108", 2024-08-18T15:55:11.116 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:55:11.116 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:55:11.116 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000094", 2024-08-18T15:55:11.117 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:55:11.117 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:55:11.117 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:55:11.117 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:55:11.117 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:55:11.117 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:55:11.117 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000060", 2024-08-18T15:55:11.117 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:55:11.117 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:55:11.117 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:55:11.117 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000109", 2024-08-18T15:55:11.117 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.15", 2024-08-18T15:55:11.117 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:55:11.118 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:55:11.118 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:55:11.118 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:55:11.118 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:55:11.118 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:55:11.118 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:55:11.118 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:55:11.118 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:55:11.118 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000013", 2024-08-18T15:55:11.118 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:55:11.118 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:55:11.118 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:55:11.119 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:55:11.119 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:55:11.119 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:55:11.119 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:55:11.119 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:55:11.119 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:55:11.119 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:55:11.119 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:55:11.119 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:55:11.119 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:55:11.119 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:55:11.119 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000051", 2024-08-18T15:55:11.119 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:55:11.120 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:55:11.120 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:55:11.120 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:55:11.120 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000106", 2024-08-18T15:55:11.120 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:55:11.120 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:55:11.120 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:55:11.120 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:55:11.120 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:55:11.120 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000017", 2024-08-18T15:55:11.120 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:55:11.120 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:55:11.120 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:55:11.121 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000023", 2024-08-18T15:55:11.121 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:55:11.121 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:55:11.121 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:55:11.121 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:55:11.121 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:55:11.121 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:55:11.121 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:55:11.121 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000107", 2024-08-18T15:55:11.121 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:55:11.121 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:55:11.121 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000046", 2024-08-18T15:55:11.121 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:55:11.121 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:55:11.122 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:55:11.122 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.26", 2024-08-18T15:55:11.122 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:55:11.122 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:55:11.122 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9", 2024-08-18T15:55:11.122 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.22", 2024-08-18T15:55:11.122 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:55:11.122 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:55:11.122 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:55:11.122 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:55:11.122 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:55:11.122 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:55:11.122 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:55:11.123 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2", 2024-08-18T15:55:11.123 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:55:11.123 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:55:11.123 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000067", 2024-08-18T15:55:11.123 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:55:11.123 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:55:11.123 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:55:11.123 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:55:11.123 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:55:11.123 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:55:11.123 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:55:11.123 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:55:11.123 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:55:11.124 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:55:11.124 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:55:11.124 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:55:11.124 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:55:11.124 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:55:11.124 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:55:11.124 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:55:11.124 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:55:11.124 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:55:11.124 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:55:11.124 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.9", 2024-08-18T15:55:11.124 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000086", 2024-08-18T15:55:11.125 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.5", 2024-08-18T15:55:11.125 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:55:11.125 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:55:11.125 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:55:11.125 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:55:11.125 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:55:11.125 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:55:11.125 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2.MFXJ583gCr1.0", 2024-08-18T15:55:11.125 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:55:11.125 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1.GPnO16liyZ+.0", 2024-08-18T15:55:11.125 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:55:11.125 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:55:11.125 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:55:11.126 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:55:11.126 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:55:11.126 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:55:11.126 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:55:11.126 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:55:11.126 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:55:11.126 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.20", 2024-08-18T15:55:11.126 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:55:11.126 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:55:11.126 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:55:11.126 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:55:11.126 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:55:11.126 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:55:11.127 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1.retry", 2024-08-18T15:55:11.127 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:55:11.127 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:55:11.127 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:55:11.127 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0.retry", 2024-08-18T15:55:11.127 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000050", 2024-08-18T15:55:11.127 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:55:11.127 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000102", 2024-08-18T15:55:11.127 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:55:11.127 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:55:11.127 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:55:11.127 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:55:11.127 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:55:11.128 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:55:11.128 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:55:11.128 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:55:11.128 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:55:11.128 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:55:11.128 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:55:11.128 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000124", 2024-08-18T15:55:11.128 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:55:11.128 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:55:11.128 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:55:11.128 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000001", 2024-08-18T15:55:11.128 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:55:11.128 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:55:11.129 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:55:11.129 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:55:11.129 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000126", 2024-08-18T15:55:11.129 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:55:11.129 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0.fZhctfLXnSt.0", 2024-08-18T15:55:11.129 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:55:11.129 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000055", 2024-08-18T15:55:11.129 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:55:11.129 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000061", 2024-08-18T15:55:11.129 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1", 2024-08-18T15:55:11.129 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:55:11.129 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:55:11.130 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:55:11.130 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000097", 2024-08-18T15:55:11.130 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:55:11.130 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:55:11.130 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:55:11.130 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:55:11.130 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:55:11.130 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:55:11.130 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:55:11.130 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:55:11.130 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000042", 2024-08-18T15:55:11.130 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:55:11.131 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:55:11.131 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:55:11.131 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1.retry", 2024-08-18T15:55:11.131 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:55:11.131 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:55:11.131 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:55:11.131 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:55:11.131 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:55:11.131 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:55:11.131 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:55:11.131 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000075", 2024-08-18T15:55:11.131 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:55:11.131 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000120", 2024-08-18T15:55:11.132 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000030", 2024-08-18T15:55:11.132 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:55:11.132 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:55:11.132 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:55:11.132 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:55:11.132 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:55:11.132 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:55:11.132 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9" 2024-08-18T15:55:11.132 INFO:teuthology.orchestra.run.smithi193.stdout:] 2024-08-18T15:55:11.132 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:55:16.134 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:55:16.134 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:55:16.135 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:55:16.185 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:55:16.186 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:55:16.249 INFO:teuthology.orchestra.run.smithi193.stdout:[ 2024-08-18T15:55:16.249 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:55:16.249 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:55:16.249 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.29", 2024-08-18T15:55:16.249 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000078", 2024-08-18T15:55:16.249 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.history", 2024-08-18T15:55:16.249 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:55:16.250 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:55:16.250 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:55:16.250 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:55:16.250 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.23", 2024-08-18T15:55:16.250 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000070", 2024-08-18T15:55:16.250 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000104", 2024-08-18T15:55:16.250 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:55:16.250 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2.retry", 2024-08-18T15:55:16.250 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:55:16.250 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:55:16.250 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:55:16.250 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:55:16.251 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000026", 2024-08-18T15:55:16.251 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:55:16.251 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:55:16.251 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:55:16.251 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:55:16.251 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:55:16.251 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:55:16.251 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.8", 2024-08-18T15:55:16.251 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:55:16.251 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:55:16.251 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:55:16.251 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:55:16.251 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000028", 2024-08-18T15:55:16.251 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:55:16.252 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:55:16.252 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:55:16.252 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:55:16.252 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:55:16.252 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:55:16.252 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:55:16.252 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2.retry", 2024-08-18T15:55:16.252 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:55:16.252 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.30", 2024-08-18T15:55:16.252 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:55:16.252 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:55:16.252 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:55:16.252 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:55:16.252 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000040", 2024-08-18T15:55:16.253 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:55:16.253 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.6", 2024-08-18T15:55:16.253 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:55:16.253 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:55:16.253 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:55:16.253 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:55:16.253 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:55:16.253 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:55:16.253 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000015", 2024-08-18T15:55:16.253 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:55:16.253 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:55:16.253 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000069", 2024-08-18T15:55:16.254 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:55:16.254 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:55:16.254 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:55:16.254 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000095", 2024-08-18T15:55:16.254 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0", 2024-08-18T15:55:16.254 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000003", 2024-08-18T15:55:16.254 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000047", 2024-08-18T15:55:16.254 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:55:16.254 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:55:16.255 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:55:16.255 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:55:16.255 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:55:16.255 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:55:16.255 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:55:16.255 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:55:16.255 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:55:16.255 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000035", 2024-08-18T15:55:16.255 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:55:16.255 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:55:16.256 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000037", 2024-08-18T15:55:16.256 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:55:16.256 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:55:16.256 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:55:16.256 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:55:16.256 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:55:16.256 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:55:16.256 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:55:16.256 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:55:16.256 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:55:16.256 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:55:16.256 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:55:16.257 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:55:16.257 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:55:16.257 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:55:16.257 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000024", 2024-08-18T15:55:16.257 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:55:16.257 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:55:16.257 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:55:16.257 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:55:16.257 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000000", 2024-08-18T15:55:16.258 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:55:16.258 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000031", 2024-08-18T15:55:16.258 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:55:16.258 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:55:16.258 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:55:16.258 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:55:16.258 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:55:16.258 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:55:16.258 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3", 2024-08-18T15:55:16.258 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:55:16.258 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:55:16.258 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:55:16.259 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:55:16.259 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:55:16.259 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:55:16.259 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:55:16.259 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:55:16.259 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:55:16.259 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:55:16.259 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:55:16.259 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:55:16.259 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:55:16.259 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:55:16.259 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:55:16.259 INFO:teuthology.orchestra.run.smithi193.stdout: "data_loggenerations_metadata", 2024-08-18T15:55:16.259 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2", 2024-08-18T15:55:16.260 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:55:16.260 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:55:16.260 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:55:16.260 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:55:16.260 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T15:55:16.260 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:55:16.261 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:55:16.261 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:55:16.261 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:55:16.262 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:55:16.262 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000076", 2024-08-18T15:55:16.262 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:55:16.262 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:55:16.262 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000113", 2024-08-18T15:55:16.262 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000005", 2024-08-18T15:55:16.263 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:55:16.263 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000011", 2024-08-18T15:55:16.263 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:55:16.263 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000058", 2024-08-18T15:55:16.263 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:55:16.263 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:55:16.263 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:55:16.264 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000052", 2024-08-18T15:55:16.264 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:55:16.264 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:55:16.264 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:55:16.264 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:55:16.264 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.21", 2024-08-18T15:55:16.265 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:55:16.265 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:55:16.265 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:55:16.265 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:55:16.265 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:55:16.265 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:55:16.266 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:55:16.266 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:55:16.266 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:55:16.266 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:55:16.266 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:55:16.267 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:55:16.267 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:55:16.267 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:55:16.267 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:55:16.267 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:55:16.268 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:55:16.268 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:55:16.268 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:55:16.268 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000088", 2024-08-18T15:55:16.268 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000080", 2024-08-18T15:55:16.268 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:55:16.268 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:55:16.269 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000090", 2024-08-18T15:55:16.269 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:55:16.269 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:55:16.269 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000110", 2024-08-18T15:55:16.269 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:55:16.269 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:55:16.269 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:55:16.269 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.0", 2024-08-18T15:55:16.270 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:55:16.270 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000096", 2024-08-18T15:55:16.270 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:55:16.270 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:55:16.270 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:55:16.270 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:55:16.270 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:55:16.270 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000087", 2024-08-18T15:55:16.270 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:55:16.270 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:55:16.271 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:55:16.271 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:55:16.271 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000008", 2024-08-18T15:55:16.271 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:55:16.271 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:55:16.271 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:55:16.271 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:55:16.271 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:55:16.271 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000006", 2024-08-18T15:55:16.271 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000029", 2024-08-18T15:55:16.271 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:55:16.272 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000089", 2024-08-18T15:55:16.272 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:55:16.272 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:55:16.272 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000022", 2024-08-18T15:55:16.272 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:55:16.272 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:55:16.272 INFO:teuthology.orchestra.run.smithi193.stdout: "bilog.trim", 2024-08-18T15:55:16.272 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000016", 2024-08-18T15:55:16.272 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:55:16.272 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:55:16.272 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:55:16.272 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:55:16.273 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:55:16.273 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000041", 2024-08-18T15:55:16.273 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:55:16.273 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:55:16.273 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.log.01050e30-179b-43be-bdbb-3bb23cf37ae8.3", 2024-08-18T15:55:16.273 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:55:16.273 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000018", 2024-08-18T15:55:16.273 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000092", 2024-08-18T15:55:16.273 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:55:16.273 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:55:16.273 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:55:16.273 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:55:16.274 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.27", 2024-08-18T15:55:16.274 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.4", 2024-08-18T15:55:16.274 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:55:16.274 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:55:16.274 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:55:16.274 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:55:16.274 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:55:16.274 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:55:16.274 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:55:16.274 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000014", 2024-08-18T15:55:16.274 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:55:16.275 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3.retry", 2024-08-18T15:55:16.275 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:55:16.275 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:55:16.275 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:55:16.275 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:55:16.275 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:55:16.275 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000112", 2024-08-18T15:55:16.275 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:55:16.275 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000007", 2024-08-18T15:55:16.275 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000021", 2024-08-18T15:55:16.275 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:55:16.275 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:55:16.275 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:55:16.276 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:55:16.276 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:55:16.276 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.16", 2024-08-18T15:55:16.276 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:55:16.276 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000064", 2024-08-18T15:55:16.276 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:55:16.276 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:55:16.276 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:55:16.276 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:55:16.276 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:55:16.276 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:55:16.276 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:55:16.276 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000071", 2024-08-18T15:55:16.277 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:55:16.277 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:55:16.277 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:55:16.277 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000074", 2024-08-18T15:55:16.277 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:55:16.277 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:55:16.277 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:55:16.277 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:55:16.277 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:55:16.277 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:55:16.277 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:55:16.277 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:55:16.277 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:55:16.278 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:55:16.278 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:55:16.278 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:55:16.278 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:55:16.278 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:55:16.278 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:55:16.278 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000081", 2024-08-18T15:55:16.278 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:55:16.278 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000009", 2024-08-18T15:55:16.278 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000121", 2024-08-18T15:55:16.278 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1", 2024-08-18T15:55:16.279 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:55:16.279 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:55:16.279 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:55:16.279 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:55:16.279 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:55:16.279 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:55:16.279 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.17", 2024-08-18T15:55:16.279 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000125", 2024-08-18T15:55:16.279 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:55:16.279 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000082", 2024-08-18T15:55:16.279 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:55:16.279 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:55:16.279 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:55:16.279 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:55:16.280 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:55:16.280 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:55:16.280 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:55:16.280 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:55:16.280 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:55:16.280 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:55:16.280 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:55:16.280 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000105", 2024-08-18T15:55:16.280 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000059", 2024-08-18T15:55:16.280 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:55:16.280 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:55:16.280 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:55:16.280 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:55:16.281 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:55:16.281 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:55:16.281 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:55:16.281 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:55:16.281 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:55:16.281 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:55:16.281 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:55:16.281 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000077", 2024-08-18T15:55:16.282 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000032", 2024-08-18T15:55:16.282 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:55:16.282 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:55:16.282 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:55:16.282 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:55:16.282 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:55:16.282 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:55:16.282 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.24", 2024-08-18T15:55:16.282 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:55:16.282 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:55:16.282 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:55:16.283 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:55:16.283 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3.retry", 2024-08-18T15:55:16.283 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:55:16.283 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:55:16.283 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:55:16.283 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:55:16.283 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:55:16.283 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:55:16.283 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000053", 2024-08-18T15:55:16.283 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000091", 2024-08-18T15:55:16.283 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:55:16.283 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:55:16.283 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:55:16.283 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000065", 2024-08-18T15:55:16.284 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:55:16.284 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:55:16.284 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:55:16.284 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:55:16.284 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000083", 2024-08-18T15:55:16.284 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000010", 2024-08-18T15:55:16.284 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:55:16.284 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:55:16.284 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:55:16.284 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:55:16.284 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:55:16.284 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:55:16.284 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:55:16.285 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:55:16.285 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:55:16.285 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:55:16.285 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:55:16.285 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.7", 2024-08-18T15:55:16.285 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.3", 2024-08-18T15:55:16.285 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:55:16.285 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:55:16.285 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:55:16.285 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:55:16.285 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:55:16.285 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:55:16.285 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:55:16.285 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:55:16.286 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.2", 2024-08-18T15:55:16.286 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:55:16.286 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:55:16.286 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:55:16.286 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:55:16.286 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:55:16.286 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:55:16.286 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:55:16.286 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000045", 2024-08-18T15:55:16.286 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000002", 2024-08-18T15:55:16.286 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:55:16.286 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3.2UbiLidwSTy.0", 2024-08-18T15:55:16.287 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:55:16.287 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.12", 2024-08-18T15:55:16.287 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:55:16.287 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:55:16.287 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:55:16.287 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:55:16.287 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:55:16.287 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:55:16.287 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000116", 2024-08-18T15:55:16.287 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.25", 2024-08-18T15:55:16.287 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:55:16.287 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:55:16.287 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:55:16.287 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:55:16.288 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000034", 2024-08-18T15:55:16.288 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:55:16.288 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:55:16.288 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000101", 2024-08-18T15:55:16.288 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:55:16.288 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:55:16.288 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.13", 2024-08-18T15:55:16.288 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:55:16.288 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:55:16.288 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.1", 2024-08-18T15:55:16.288 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0.retry", 2024-08-18T15:55:16.288 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:55:16.289 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:55:16.289 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000079", 2024-08-18T15:55:16.289 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000049", 2024-08-18T15:55:16.289 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:55:16.289 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0", 2024-08-18T15:55:16.289 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:55:16.289 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:55:16.289 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000117", 2024-08-18T15:55:16.289 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:55:16.289 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000044", 2024-08-18T15:55:16.289 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:55:16.290 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:55:16.290 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:55:16.290 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:55:16.290 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:55:16.290 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:55:16.290 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.14", 2024-08-18T15:55:16.290 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:55:16.290 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000066", 2024-08-18T15:55:16.290 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:55:16.290 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:55:16.290 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:55:16.290 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:55:16.291 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:55:16.291 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000068", 2024-08-18T15:55:16.291 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:55:16.291 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000085", 2024-08-18T15:55:16.291 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:55:16.291 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:55:16.291 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:55:16.291 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:55:16.291 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:55:16.291 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:55:16.291 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:55:16.292 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:55:16.292 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:55:16.292 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:55:16.292 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:55:16.292 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:55:16.292 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000073", 2024-08-18T15:55:16.292 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:55:16.292 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:55:16.292 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:55:16.292 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000038", 2024-08-18T15:55:16.292 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:55:16.292 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:55:16.293 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:55:16.293 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:55:16.293 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.10", 2024-08-18T15:55:16.293 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:55:16.294 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:55:16.294 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:55:16.294 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000118", 2024-08-18T15:55:16.294 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:55:16.294 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3", 2024-08-18T15:55:16.294 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:55:16.294 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000036", 2024-08-18T15:55:16.294 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:55:16.294 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:55:16.294 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:55:16.294 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:55:16.295 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:55:16.295 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000103", 2024-08-18T15:55:16.295 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.3", 2024-08-18T15:55:16.295 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:55:16.295 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000119", 2024-08-18T15:55:16.295 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000098", 2024-08-18T15:55:16.295 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:55:16.295 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:55:16.295 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:55:16.295 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:55:16.295 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:55:16.295 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:55:16.295 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:55:16.296 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:55:16.296 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.0", 2024-08-18T15:55:16.296 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:55:16.296 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:55:16.296 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:55:16.296 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:55:16.296 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:55:16.296 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:55:16.296 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:55:16.296 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:55:16.296 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000027", 2024-08-18T15:55:16.296 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:55:16.296 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status", 2024-08-18T15:55:16.297 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:55:16.297 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000019", 2024-08-18T15:55:16.297 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:55:16.297 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:55:16.297 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:55:16.297 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:55:16.297 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1", 2024-08-18T15:55:16.297 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:55:16.297 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:55:16.297 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:55:16.297 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:55:16.297 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000039", 2024-08-18T15:55:16.297 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:55:16.297 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:55:16.298 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:55:16.298 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:55:16.298 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:55:16.298 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:55:16.298 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:55:16.298 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:55:16.298 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:55:16.298 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:55:16.298 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:55:16.298 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000100", 2024-08-18T15:55:16.298 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:55:16.298 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:55:16.299 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:55:16.299 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:55:16.299 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:55:16.299 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000093", 2024-08-18T15:55:16.299 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:55:16.299 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000004", 2024-08-18T15:55:16.299 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000063", 2024-08-18T15:55:16.299 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000122", 2024-08-18T15:55:16.299 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:55:16.299 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:55:16.299 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:55:16.299 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000057", 2024-08-18T15:55:16.299 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:55:16.300 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:55:16.300 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:55:16.300 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:55:16.300 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000054", 2024-08-18T15:55:16.300 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:55:16.300 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000114", 2024-08-18T15:55:16.300 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:55:16.300 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:55:16.300 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:55:16.300 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.2", 2024-08-18T15:55:16.300 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:55:16.300 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:55:16.300 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0", 2024-08-18T15:55:16.300 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:55:16.301 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:55:16.301 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:55:16.301 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:55:16.301 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:55:16.301 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:55:16.301 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000012", 2024-08-18T15:55:16.301 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:55:16.301 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:55:16.301 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:55:16.301 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000084", 2024-08-18T15:55:16.301 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000043", 2024-08-18T15:55:16.301 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3", 2024-08-18T15:55:16.301 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:55:16.301 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:55:16.302 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.11", 2024-08-18T15:55:16.302 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:55:16.302 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:55:16.302 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:55:16.302 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:55:16.302 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:55:16.302 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:55:16.302 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:55:16.302 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:55:16.302 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000111", 2024-08-18T15:55:16.302 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:55:16.303 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000048", 2024-08-18T15:55:16.303 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:55:16.303 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000020", 2024-08-18T15:55:16.303 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:55:16.303 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:55:16.303 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:55:16.303 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:55:16.303 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:55:16.303 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000099", 2024-08-18T15:55:16.303 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000056", 2024-08-18T15:55:16.304 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:55:16.304 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:55:16.304 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.19", 2024-08-18T15:55:16.304 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:55:16.304 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.28", 2024-08-18T15:55:16.304 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:55:16.304 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:55:16.304 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:55:16.304 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:55:16.304 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:55:16.304 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000072", 2024-08-18T15:55:16.304 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:55:16.304 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:55:16.304 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000062", 2024-08-18T15:55:16.305 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:55:16.305 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:55:16.305 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:55:16.305 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000115", 2024-08-18T15:55:16.305 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:55:16.305 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.1", 2024-08-18T15:55:16.305 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2", 2024-08-18T15:55:16.305 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000033", 2024-08-18T15:55:16.305 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:55:16.305 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:55:16.305 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:55:16.305 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:55:16.305 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:55:16.305 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:55:16.306 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:55:16.306 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:55:16.306 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:55:16.306 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:55:16.306 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:55:16.306 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:55:16.306 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:55:16.306 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:55:16.306 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:55:16.306 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:55:16.306 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:55:16.306 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:55:16.307 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:55:16.307 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000025", 2024-08-18T15:55:16.307 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:55:16.307 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000123", 2024-08-18T15:55:16.307 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.31", 2024-08-18T15:55:16.307 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:55:16.307 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:55:16.307 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:55:16.307 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:55:16.307 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.18", 2024-08-18T15:55:16.307 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:55:16.307 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:55:16.307 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:55:16.308 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:55:16.308 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:55:16.308 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000108", 2024-08-18T15:55:16.308 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:55:16.308 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:55:16.308 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000094", 2024-08-18T15:55:16.308 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:55:16.308 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:55:16.308 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:55:16.308 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:55:16.308 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:55:16.308 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:55:16.308 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000060", 2024-08-18T15:55:16.309 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:55:16.309 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:55:16.309 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:55:16.309 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000109", 2024-08-18T15:55:16.309 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.15", 2024-08-18T15:55:16.309 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:55:16.309 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:55:16.309 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:55:16.309 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:55:16.310 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:55:16.310 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:55:16.310 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:55:16.310 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:55:16.310 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:55:16.310 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000013", 2024-08-18T15:55:16.310 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:55:16.310 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:55:16.311 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:55:16.311 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:55:16.311 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:55:16.311 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:55:16.311 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:55:16.311 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:55:16.311 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:55:16.311 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:55:16.311 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:55:16.312 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:55:16.312 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:55:16.312 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:55:16.312 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000051", 2024-08-18T15:55:16.312 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:55:16.312 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:55:16.312 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:55:16.312 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:55:16.312 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000106", 2024-08-18T15:55:16.312 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:55:16.312 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:55:16.312 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:55:16.312 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:55:16.313 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:55:16.313 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000017", 2024-08-18T15:55:16.313 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:55:16.313 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:55:16.313 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:55:16.313 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000023", 2024-08-18T15:55:16.313 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:55:16.313 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:55:16.313 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:55:16.313 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:55:16.313 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:55:16.313 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:55:16.313 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:55:16.314 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000107", 2024-08-18T15:55:16.314 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:55:16.314 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:55:16.314 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000046", 2024-08-18T15:55:16.314 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:55:16.314 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:55:16.314 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:55:16.314 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.26", 2024-08-18T15:55:16.314 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:55:16.314 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:55:16.314 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9", 2024-08-18T15:55:16.314 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.22", 2024-08-18T15:55:16.314 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:55:16.315 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:55:16.315 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:55:16.315 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:55:16.315 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:55:16.315 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:55:16.315 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:55:16.315 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2", 2024-08-18T15:55:16.315 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:55:16.315 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:55:16.315 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000067", 2024-08-18T15:55:16.315 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:55:16.315 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:55:16.315 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:55:16.316 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:55:16.316 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:55:16.316 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:55:16.316 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:55:16.316 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:55:16.316 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:55:16.316 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:55:16.316 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:55:16.316 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:55:16.316 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:55:16.316 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:55:16.316 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:55:16.316 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:55:16.316 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:55:16.317 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:55:16.317 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:55:16.317 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.9", 2024-08-18T15:55:16.317 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000086", 2024-08-18T15:55:16.317 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.5", 2024-08-18T15:55:16.317 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:55:16.317 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:55:16.317 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:55:16.317 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:55:16.317 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:55:16.317 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:55:16.317 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2.MFXJ583gCr1.0", 2024-08-18T15:55:16.317 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:55:16.318 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1.GPnO16liyZ+.0", 2024-08-18T15:55:16.318 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:55:16.318 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:55:16.318 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:55:16.318 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:55:16.318 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:55:16.318 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:55:16.318 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:55:16.318 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:55:16.318 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:55:16.318 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.20", 2024-08-18T15:55:16.318 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:55:16.319 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:55:16.319 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:55:16.319 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:55:16.319 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:55:16.319 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:55:16.319 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1.retry", 2024-08-18T15:55:16.319 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:55:16.319 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:55:16.319 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:55:16.319 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0.retry", 2024-08-18T15:55:16.319 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000050", 2024-08-18T15:55:16.319 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:55:16.319 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000102", 2024-08-18T15:55:16.319 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:55:16.320 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:55:16.320 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:55:16.320 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:55:16.320 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:55:16.320 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:55:16.320 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:55:16.320 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:55:16.320 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:55:16.320 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:55:16.320 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:55:16.320 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000124", 2024-08-18T15:55:16.320 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:55:16.320 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:55:16.321 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:55:16.321 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000001", 2024-08-18T15:55:16.321 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:55:16.321 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:55:16.321 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:55:16.321 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:55:16.321 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000126", 2024-08-18T15:55:16.321 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:55:16.321 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0.fZhctfLXnSt.0", 2024-08-18T15:55:16.321 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:55:16.321 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000055", 2024-08-18T15:55:16.321 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:55:16.321 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000061", 2024-08-18T15:55:16.322 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1", 2024-08-18T15:55:16.322 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:55:16.322 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:55:16.322 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:55:16.322 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000097", 2024-08-18T15:55:16.322 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:55:16.322 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:55:16.322 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:55:16.322 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:55:16.322 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:55:16.322 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:55:16.322 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:55:16.323 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:55:16.323 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000042", 2024-08-18T15:55:16.323 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:55:16.323 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:55:16.323 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:55:16.323 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1.retry", 2024-08-18T15:55:16.323 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:55:16.323 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:55:16.323 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:55:16.323 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:55:16.323 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:55:16.323 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:55:16.323 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:55:16.323 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000075", 2024-08-18T15:55:16.324 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:55:16.324 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000120", 2024-08-18T15:55:16.324 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000030", 2024-08-18T15:55:16.324 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:55:16.324 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:55:16.324 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:55:16.324 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:55:16.324 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:55:16.324 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:55:16.324 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9" 2024-08-18T15:55:16.324 INFO:teuthology.orchestra.run.smithi193.stdout:] 2024-08-18T15:55:16.324 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:55:21.327 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:55:21.327 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:55:21.327 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:55:21.377 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:55:21.377 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:55:21.439 INFO:teuthology.orchestra.run.smithi193.stdout:[ 2024-08-18T15:55:21.439 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:55:21.439 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:55:21.439 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.29", 2024-08-18T15:55:21.439 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000078", 2024-08-18T15:55:21.439 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.history", 2024-08-18T15:55:21.439 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:55:21.439 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:55:21.439 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:55:21.439 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:55:21.440 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.23", 2024-08-18T15:55:21.440 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000070", 2024-08-18T15:55:21.440 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000104", 2024-08-18T15:55:21.440 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:55:21.440 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2.retry", 2024-08-18T15:55:21.440 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:55:21.440 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:55:21.440 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:55:21.440 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:55:21.440 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000026", 2024-08-18T15:55:21.440 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:55:21.440 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:55:21.440 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:55:21.440 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:55:21.441 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:55:21.441 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:55:21.441 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.8", 2024-08-18T15:55:21.441 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:55:21.441 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:55:21.441 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:55:21.441 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:55:21.441 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000028", 2024-08-18T15:55:21.441 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:55:21.441 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:55:21.441 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:55:21.441 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:55:21.441 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:55:21.442 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:55:21.442 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:55:21.442 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2.retry", 2024-08-18T15:55:21.442 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:55:21.442 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.30", 2024-08-18T15:55:21.442 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:55:21.442 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:55:21.442 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:55:21.442 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:55:21.442 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000040", 2024-08-18T15:55:21.442 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:55:21.442 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.6", 2024-08-18T15:55:21.442 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:55:21.442 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:55:21.443 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:55:21.443 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:55:21.443 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:55:21.443 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:55:21.443 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000015", 2024-08-18T15:55:21.443 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:55:21.443 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:55:21.443 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000069", 2024-08-18T15:55:21.443 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:55:21.443 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:55:21.443 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:55:21.443 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000095", 2024-08-18T15:55:21.444 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0", 2024-08-18T15:55:21.444 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000003", 2024-08-18T15:55:21.444 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000047", 2024-08-18T15:55:21.444 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:55:21.444 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:55:21.444 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:55:21.444 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:55:21.444 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:55:21.444 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:55:21.444 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:55:21.444 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:55:21.444 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:55:21.444 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000035", 2024-08-18T15:55:21.444 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:55:21.444 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:55:21.445 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000037", 2024-08-18T15:55:21.445 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:55:21.445 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:55:21.445 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:55:21.445 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:55:21.445 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:55:21.445 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:55:21.445 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:55:21.445 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:55:21.445 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:55:21.445 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:55:21.445 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:55:21.445 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:55:21.446 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:55:21.446 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:55:21.446 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000024", 2024-08-18T15:55:21.446 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:55:21.446 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:55:21.446 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:55:21.446 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:55:21.446 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000000", 2024-08-18T15:55:21.446 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:55:21.446 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000031", 2024-08-18T15:55:21.446 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:55:21.446 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:55:21.446 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:55:21.447 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:55:21.447 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:55:21.447 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:55:21.447 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3", 2024-08-18T15:55:21.447 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:55:21.447 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:55:21.447 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:55:21.447 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:55:21.447 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:55:21.447 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:55:21.447 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:55:21.447 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:55:21.447 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:55:21.448 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:55:21.448 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:55:21.448 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:55:21.448 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:55:21.448 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:55:21.448 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:55:21.448 INFO:teuthology.orchestra.run.smithi193.stdout: "data_loggenerations_metadata", 2024-08-18T15:55:21.448 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2", 2024-08-18T15:55:21.448 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:55:21.448 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:55:21.448 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:55:21.448 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:55:21.448 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T15:55:21.449 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:55:21.449 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:55:21.449 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:55:21.449 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:55:21.449 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:55:21.449 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000076", 2024-08-18T15:55:21.449 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:55:21.449 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:55:21.449 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000113", 2024-08-18T15:55:21.449 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000005", 2024-08-18T15:55:21.449 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:55:21.449 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000011", 2024-08-18T15:55:21.449 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:55:21.449 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000058", 2024-08-18T15:55:21.450 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:55:21.450 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:55:21.450 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:55:21.450 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000052", 2024-08-18T15:55:21.450 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:55:21.450 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:55:21.450 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:55:21.450 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:55:21.450 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.21", 2024-08-18T15:55:21.450 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:55:21.450 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:55:21.450 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:55:21.450 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:55:21.450 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:55:21.451 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:55:21.451 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:55:21.451 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:55:21.451 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:55:21.451 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:55:21.451 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:55:21.451 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:55:21.451 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:55:21.451 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:55:21.451 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:55:21.451 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:55:21.451 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:55:21.451 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:55:21.452 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:55:21.452 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000088", 2024-08-18T15:55:21.452 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000080", 2024-08-18T15:55:21.452 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:55:21.452 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:55:21.452 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000090", 2024-08-18T15:55:21.452 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:55:21.452 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:55:21.452 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000110", 2024-08-18T15:55:21.452 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:55:21.452 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:55:21.452 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:55:21.452 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.0", 2024-08-18T15:55:21.453 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:55:21.453 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000096", 2024-08-18T15:55:21.453 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:55:21.453 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:55:21.453 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:55:21.453 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:55:21.453 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:55:21.453 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000087", 2024-08-18T15:55:21.453 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:55:21.453 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:55:21.453 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:55:21.453 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:55:21.454 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000008", 2024-08-18T15:55:21.454 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:55:21.454 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:55:21.454 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:55:21.454 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:55:21.454 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:55:21.454 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000006", 2024-08-18T15:55:21.454 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000029", 2024-08-18T15:55:21.454 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:55:21.454 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000089", 2024-08-18T15:55:21.454 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:55:21.454 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:55:21.454 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000022", 2024-08-18T15:55:21.455 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:55:21.455 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:55:21.455 INFO:teuthology.orchestra.run.smithi193.stdout: "bilog.trim", 2024-08-18T15:55:21.455 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000016", 2024-08-18T15:55:21.455 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:55:21.455 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:55:21.455 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:55:21.455 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:55:21.455 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:55:21.455 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000041", 2024-08-18T15:55:21.455 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:55:21.455 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:55:21.456 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.log.01050e30-179b-43be-bdbb-3bb23cf37ae8.3", 2024-08-18T15:55:21.456 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:55:21.456 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000018", 2024-08-18T15:55:21.456 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000092", 2024-08-18T15:55:21.456 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:55:21.456 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:55:21.456 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:55:21.456 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:55:21.456 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.27", 2024-08-18T15:55:21.456 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.4", 2024-08-18T15:55:21.456 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:55:21.456 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:55:21.456 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:55:21.457 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:55:21.457 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:55:21.457 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:55:21.457 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:55:21.457 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000014", 2024-08-18T15:55:21.457 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:55:21.457 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3.retry", 2024-08-18T15:55:21.457 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:55:21.457 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:55:21.457 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:55:21.457 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:55:21.457 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:55:21.458 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000112", 2024-08-18T15:55:21.458 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:55:21.458 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000007", 2024-08-18T15:55:21.458 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000021", 2024-08-18T15:55:21.458 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:55:21.458 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:55:21.458 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:55:21.458 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:55:21.458 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:55:21.458 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.16", 2024-08-18T15:55:21.458 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:55:21.458 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000064", 2024-08-18T15:55:21.458 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:55:21.459 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:55:21.459 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:55:21.459 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:55:21.459 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:55:21.459 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:55:21.459 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:55:21.459 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000071", 2024-08-18T15:55:21.459 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:55:21.459 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:55:21.459 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:55:21.459 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000074", 2024-08-18T15:55:21.459 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:55:21.459 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:55:21.460 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:55:21.460 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:55:21.460 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:55:21.460 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:55:21.460 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:55:21.460 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:55:21.460 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:55:21.460 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:55:21.460 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:55:21.460 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:55:21.460 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:55:21.460 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:55:21.460 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:55:21.460 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000081", 2024-08-18T15:55:21.461 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:55:21.461 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000009", 2024-08-18T15:55:21.461 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000121", 2024-08-18T15:55:21.461 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1", 2024-08-18T15:55:21.461 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:55:21.461 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:55:21.461 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:55:21.461 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:55:21.461 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:55:21.461 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:55:21.461 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.17", 2024-08-18T15:55:21.461 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000125", 2024-08-18T15:55:21.462 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:55:21.462 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000082", 2024-08-18T15:55:21.462 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:55:21.462 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:55:21.462 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:55:21.462 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:55:21.462 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:55:21.462 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:55:21.462 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:55:21.463 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:55:21.463 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:55:21.463 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:55:21.463 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:55:21.463 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000105", 2024-08-18T15:55:21.463 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000059", 2024-08-18T15:55:21.463 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:55:21.463 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:55:21.463 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:55:21.463 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:55:21.463 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:55:21.463 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:55:21.464 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:55:21.464 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:55:21.464 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:55:21.464 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:55:21.464 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:55:21.464 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000077", 2024-08-18T15:55:21.464 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000032", 2024-08-18T15:55:21.464 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:55:21.464 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:55:21.464 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:55:21.464 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:55:21.464 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:55:21.464 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:55:21.465 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.24", 2024-08-18T15:55:21.465 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:55:21.465 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:55:21.465 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:55:21.465 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:55:21.465 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3.retry", 2024-08-18T15:55:21.465 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:55:21.465 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:55:21.465 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:55:21.465 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:55:21.466 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:55:21.466 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:55:21.466 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000053", 2024-08-18T15:55:21.466 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000091", 2024-08-18T15:55:21.466 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:55:21.466 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:55:21.466 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:55:21.466 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000065", 2024-08-18T15:55:21.466 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:55:21.466 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:55:21.466 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:55:21.467 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:55:21.467 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000083", 2024-08-18T15:55:21.467 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000010", 2024-08-18T15:55:21.467 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:55:21.467 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:55:21.467 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:55:21.467 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:55:21.467 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:55:21.467 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:55:21.467 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:55:21.467 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:55:21.467 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:55:21.467 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:55:21.468 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:55:21.468 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.7", 2024-08-18T15:55:21.468 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.3", 2024-08-18T15:55:21.468 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:55:21.468 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:55:21.468 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:55:21.468 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:55:21.468 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:55:21.468 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:55:21.468 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:55:21.468 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:55:21.468 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.2", 2024-08-18T15:55:21.468 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:55:21.469 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:55:21.469 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:55:21.469 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:55:21.469 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:55:21.469 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:55:21.469 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:55:21.469 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000045", 2024-08-18T15:55:21.469 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000002", 2024-08-18T15:55:21.469 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:55:21.469 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3.2UbiLidwSTy.0", 2024-08-18T15:55:21.469 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:55:21.469 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.12", 2024-08-18T15:55:21.470 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:55:21.470 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:55:21.470 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:55:21.470 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:55:21.470 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:55:21.470 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:55:21.470 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000116", 2024-08-18T15:55:21.470 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.25", 2024-08-18T15:55:21.470 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:55:21.470 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:55:21.470 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:55:21.470 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:55:21.471 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000034", 2024-08-18T15:55:21.471 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:55:21.471 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:55:21.471 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000101", 2024-08-18T15:55:21.471 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:55:21.471 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:55:21.471 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.13", 2024-08-18T15:55:21.471 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:55:21.471 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:55:21.471 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.1", 2024-08-18T15:55:21.471 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0.retry", 2024-08-18T15:55:21.471 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:55:21.471 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:55:21.472 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000079", 2024-08-18T15:55:21.472 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000049", 2024-08-18T15:55:21.472 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:55:21.472 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0", 2024-08-18T15:55:21.472 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:55:21.472 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:55:21.472 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000117", 2024-08-18T15:55:21.472 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:55:21.472 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000044", 2024-08-18T15:55:21.472 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:55:21.472 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:55:21.472 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:55:21.472 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:55:21.473 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:55:21.473 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:55:21.473 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.14", 2024-08-18T15:55:21.473 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:55:21.473 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000066", 2024-08-18T15:55:21.473 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:55:21.473 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:55:21.473 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:55:21.473 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:55:21.473 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:55:21.473 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000068", 2024-08-18T15:55:21.473 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:55:21.473 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000085", 2024-08-18T15:55:21.474 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:55:21.474 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:55:21.474 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:55:21.474 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:55:21.474 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:55:21.474 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:55:21.474 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:55:21.474 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:55:21.474 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:55:21.474 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:55:21.474 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:55:21.474 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:55:21.474 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000073", 2024-08-18T15:55:21.475 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:55:21.475 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:55:21.475 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:55:21.475 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000038", 2024-08-18T15:55:21.475 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:55:21.475 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:55:21.475 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:55:21.475 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:55:21.475 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.10", 2024-08-18T15:55:21.475 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:55:21.475 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:55:21.476 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:55:21.476 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000118", 2024-08-18T15:55:21.476 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:55:21.476 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3", 2024-08-18T15:55:21.476 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:55:21.476 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000036", 2024-08-18T15:55:21.476 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:55:21.476 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:55:21.476 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:55:21.476 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:55:21.476 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:55:21.476 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000103", 2024-08-18T15:55:21.476 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.3", 2024-08-18T15:55:21.477 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:55:21.477 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000119", 2024-08-18T15:55:21.477 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000098", 2024-08-18T15:55:21.477 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:55:21.477 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:55:21.477 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:55:21.477 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:55:21.477 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:55:21.477 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:55:21.477 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:55:21.477 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:55:21.477 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.0", 2024-08-18T15:55:21.477 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:55:21.478 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:55:21.478 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:55:21.478 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:55:21.478 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:55:21.478 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:55:21.478 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:55:21.478 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:55:21.478 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000027", 2024-08-18T15:55:21.478 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:55:21.478 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status", 2024-08-18T15:55:21.478 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:55:21.478 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000019", 2024-08-18T15:55:21.479 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:55:21.479 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:55:21.479 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:55:21.479 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:55:21.479 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1", 2024-08-18T15:55:21.479 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:55:21.479 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:55:21.479 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:55:21.479 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:55:21.479 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000039", 2024-08-18T15:55:21.479 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:55:21.479 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:55:21.479 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:55:21.480 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:55:21.480 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:55:21.480 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:55:21.480 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:55:21.480 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:55:21.480 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:55:21.480 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:55:21.480 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:55:21.480 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000100", 2024-08-18T15:55:21.480 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:55:21.480 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:55:21.480 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:55:21.480 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:55:21.480 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:55:21.481 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000093", 2024-08-18T15:55:21.481 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:55:21.481 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000004", 2024-08-18T15:55:21.481 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000063", 2024-08-18T15:55:21.481 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000122", 2024-08-18T15:55:21.481 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:55:21.481 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:55:21.481 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:55:21.481 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000057", 2024-08-18T15:55:21.481 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:55:21.481 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:55:21.481 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:55:21.481 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:55:21.481 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000054", 2024-08-18T15:55:21.482 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:55:21.482 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000114", 2024-08-18T15:55:21.482 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:55:21.482 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:55:21.482 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:55:21.482 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.2", 2024-08-18T15:55:21.482 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:55:21.482 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:55:21.482 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0", 2024-08-18T15:55:21.482 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:55:21.482 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:55:21.482 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:55:21.482 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:55:21.483 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:55:21.483 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:55:21.483 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000012", 2024-08-18T15:55:21.483 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:55:21.483 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:55:21.483 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:55:21.483 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000084", 2024-08-18T15:55:21.483 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000043", 2024-08-18T15:55:21.483 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3", 2024-08-18T15:55:21.483 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:55:21.483 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:55:21.483 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.11", 2024-08-18T15:55:21.483 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:55:21.483 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:55:21.484 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:55:21.484 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:55:21.484 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:55:21.484 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:55:21.484 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:55:21.484 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:55:21.484 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000111", 2024-08-18T15:55:21.484 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:55:21.484 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000048", 2024-08-18T15:55:21.484 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:55:21.484 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000020", 2024-08-18T15:55:21.484 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:55:21.484 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:55:21.484 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:55:21.485 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:55:21.485 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:55:21.485 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000099", 2024-08-18T15:55:21.485 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000056", 2024-08-18T15:55:21.485 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:55:21.485 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:55:21.485 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.19", 2024-08-18T15:55:21.485 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:55:21.485 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.28", 2024-08-18T15:55:21.485 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:55:21.485 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:55:21.485 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:55:21.485 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:55:21.485 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:55:21.486 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000072", 2024-08-18T15:55:21.486 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:55:21.486 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:55:21.486 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000062", 2024-08-18T15:55:21.486 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:55:21.486 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:55:21.486 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:55:21.486 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000115", 2024-08-18T15:55:21.486 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:55:21.486 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.1", 2024-08-18T15:55:21.486 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2", 2024-08-18T15:55:21.486 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000033", 2024-08-18T15:55:21.486 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:55:21.487 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:55:21.487 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:55:21.487 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:55:21.487 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:55:21.487 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:55:21.487 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:55:21.487 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:55:21.487 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:55:21.487 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:55:21.487 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:55:21.487 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:55:21.487 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:55:21.487 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:55:21.487 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:55:21.488 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:55:21.488 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:55:21.488 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:55:21.488 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:55:21.488 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000025", 2024-08-18T15:55:21.488 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:55:21.488 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000123", 2024-08-18T15:55:21.488 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.31", 2024-08-18T15:55:21.488 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:55:21.488 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:55:21.488 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:55:21.488 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:55:21.488 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.18", 2024-08-18T15:55:21.489 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:55:21.489 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:55:21.489 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:55:21.489 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:55:21.489 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:55:21.489 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000108", 2024-08-18T15:55:21.489 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:55:21.489 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:55:21.489 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000094", 2024-08-18T15:55:21.489 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:55:21.489 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:55:21.489 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:55:21.489 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:55:21.490 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:55:21.490 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:55:21.490 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000060", 2024-08-18T15:55:21.490 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:55:21.490 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:55:21.490 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:55:21.490 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000109", 2024-08-18T15:55:21.490 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.15", 2024-08-18T15:55:21.490 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:55:21.490 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:55:21.490 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:55:21.490 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:55:21.490 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:55:21.491 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:55:21.491 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:55:21.491 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:55:21.491 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:55:21.491 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000013", 2024-08-18T15:55:21.491 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:55:21.491 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:55:21.491 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:55:21.491 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:55:21.491 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:55:21.491 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:55:21.491 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:55:21.491 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:55:21.491 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:55:21.492 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:55:21.492 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:55:21.492 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:55:21.492 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:55:21.492 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:55:21.492 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000051", 2024-08-18T15:55:21.492 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:55:21.492 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:55:21.492 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:55:21.492 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:55:21.492 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000106", 2024-08-18T15:55:21.492 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:55:21.492 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:55:21.493 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:55:21.493 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:55:21.493 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:55:21.493 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000017", 2024-08-18T15:55:21.493 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:55:21.493 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:55:21.493 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:55:21.493 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000023", 2024-08-18T15:55:21.493 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:55:21.493 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:55:21.493 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:55:21.493 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:55:21.493 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:55:21.494 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:55:21.494 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:55:21.494 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000107", 2024-08-18T15:55:21.494 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:55:21.494 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:55:21.494 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000046", 2024-08-18T15:55:21.494 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:55:21.494 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:55:21.494 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:55:21.494 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.26", 2024-08-18T15:55:21.494 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:55:21.494 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:55:21.494 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9", 2024-08-18T15:55:21.495 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.22", 2024-08-18T15:55:21.495 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:55:21.495 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:55:21.495 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:55:21.495 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:55:21.495 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:55:21.495 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:55:21.495 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:55:21.495 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2", 2024-08-18T15:55:21.495 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:55:21.495 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:55:21.495 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000067", 2024-08-18T15:55:21.495 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:55:21.496 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:55:21.496 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:55:21.496 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:55:21.496 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:55:21.496 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:55:21.496 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:55:21.496 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:55:21.496 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:55:21.496 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:55:21.496 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:55:21.496 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:55:21.496 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:55:21.496 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:55:21.497 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:55:21.497 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:55:21.497 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:55:21.497 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:55:21.497 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:55:21.497 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.9", 2024-08-18T15:55:21.497 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000086", 2024-08-18T15:55:21.497 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.5", 2024-08-18T15:55:21.497 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:55:21.497 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:55:21.497 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:55:21.497 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:55:21.497 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:55:21.497 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:55:21.497 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2.MFXJ583gCr1.0", 2024-08-18T15:55:21.498 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:55:21.498 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1.GPnO16liyZ+.0", 2024-08-18T15:55:21.498 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:55:21.498 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:55:21.498 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:55:21.498 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:55:21.498 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:55:21.498 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:55:21.498 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:55:21.498 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:55:21.498 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:55:21.498 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.20", 2024-08-18T15:55:21.498 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:55:21.499 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:55:21.499 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:55:21.499 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:55:21.499 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:55:21.499 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:55:21.499 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1.retry", 2024-08-18T15:55:21.499 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:55:21.499 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:55:21.499 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:55:21.499 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0.retry", 2024-08-18T15:55:21.499 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000050", 2024-08-18T15:55:21.499 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:55:21.499 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000102", 2024-08-18T15:55:21.499 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:55:21.500 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:55:21.500 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:55:21.500 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:55:21.500 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:55:21.500 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:55:21.500 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:55:21.500 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:55:21.500 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:55:21.500 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:55:21.500 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:55:21.500 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000124", 2024-08-18T15:55:21.500 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:55:21.500 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:55:21.501 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:55:21.501 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000001", 2024-08-18T15:55:21.501 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:55:21.501 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:55:21.501 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:55:21.501 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:55:21.501 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000126", 2024-08-18T15:55:21.501 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:55:21.501 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0.fZhctfLXnSt.0", 2024-08-18T15:55:21.501 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:55:21.501 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000055", 2024-08-18T15:55:21.501 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:55:21.501 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000061", 2024-08-18T15:55:21.501 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1", 2024-08-18T15:55:21.502 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:55:21.502 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:55:21.502 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:55:21.502 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000097", 2024-08-18T15:55:21.502 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:55:21.502 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:55:21.502 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:55:21.502 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:55:21.502 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:55:21.502 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:55:21.502 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:55:21.502 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:55:21.502 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000042", 2024-08-18T15:55:21.503 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:55:21.503 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:55:21.503 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:55:21.503 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1.retry", 2024-08-18T15:55:21.503 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:55:21.503 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:55:21.503 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:55:21.503 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:55:21.503 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:55:21.503 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:55:21.503 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:55:21.503 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000075", 2024-08-18T15:55:21.503 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:55:21.503 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000120", 2024-08-18T15:55:21.504 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000030", 2024-08-18T15:55:21.504 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:55:21.504 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:55:21.504 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:55:21.504 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:55:21.504 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:55:21.504 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:55:21.504 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9" 2024-08-18T15:55:21.504 INFO:teuthology.orchestra.run.smithi193.stdout:] 2024-08-18T15:55:21.504 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:55:26.506 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:55:26.506 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:55:26.506 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:55:26.559 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:55:26.559 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:55:26.622 INFO:teuthology.orchestra.run.smithi193.stdout:[ 2024-08-18T15:55:26.622 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:55:26.622 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:55:26.622 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.29", 2024-08-18T15:55:26.622 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000078", 2024-08-18T15:55:26.622 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.history", 2024-08-18T15:55:26.622 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:55:26.622 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:55:26.622 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:55:26.622 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:55:26.623 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.23", 2024-08-18T15:55:26.623 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000070", 2024-08-18T15:55:26.623 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000104", 2024-08-18T15:55:26.623 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:55:26.623 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2.retry", 2024-08-18T15:55:26.623 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:55:26.623 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:55:26.623 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:55:26.623 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:55:26.623 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000026", 2024-08-18T15:55:26.623 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:55:26.623 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:55:26.623 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:55:26.624 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:55:26.624 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:55:26.624 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:55:26.624 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.8", 2024-08-18T15:55:26.624 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:55:26.624 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:55:26.624 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:55:26.624 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:55:26.624 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000028", 2024-08-18T15:55:26.624 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:55:26.624 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:55:26.624 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:55:26.624 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:55:26.625 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:55:26.625 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:55:26.625 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:55:26.625 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2.retry", 2024-08-18T15:55:26.625 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:55:26.625 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.30", 2024-08-18T15:55:26.625 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:55:26.625 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:55:26.625 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:55:26.625 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:55:26.625 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000040", 2024-08-18T15:55:26.625 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:55:26.625 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.6", 2024-08-18T15:55:26.626 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:55:26.626 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:55:26.626 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:55:26.626 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:55:26.626 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:55:26.626 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:55:26.626 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000015", 2024-08-18T15:55:26.626 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:55:26.626 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:55:26.626 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000069", 2024-08-18T15:55:26.626 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:55:26.627 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:55:26.627 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:55:26.627 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000095", 2024-08-18T15:55:26.627 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0", 2024-08-18T15:55:26.627 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000003", 2024-08-18T15:55:26.627 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000047", 2024-08-18T15:55:26.627 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:55:26.627 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:55:26.627 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:55:26.627 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:55:26.627 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:55:26.627 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:55:26.628 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:55:26.628 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:55:26.628 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:55:26.628 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000035", 2024-08-18T15:55:26.628 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:55:26.628 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:55:26.628 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000037", 2024-08-18T15:55:26.628 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:55:26.628 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:55:26.628 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:55:26.628 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:55:26.628 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:55:26.629 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:55:26.629 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:55:26.629 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:55:26.629 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:55:26.629 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:55:26.629 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:55:26.629 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:55:26.629 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:55:26.629 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:55:26.629 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000024", 2024-08-18T15:55:26.629 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:55:26.629 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:55:26.629 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:55:26.629 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:55:26.630 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000000", 2024-08-18T15:55:26.630 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:55:26.630 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000031", 2024-08-18T15:55:26.630 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:55:26.630 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:55:26.630 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:55:26.630 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:55:26.630 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:55:26.630 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:55:26.630 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3", 2024-08-18T15:55:26.630 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:55:26.630 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:55:26.630 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:55:26.631 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:55:26.631 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:55:26.631 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:55:26.631 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:55:26.631 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:55:26.631 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:55:26.631 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:55:26.631 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:55:26.631 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:55:26.631 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:55:26.631 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:55:26.631 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:55:26.631 INFO:teuthology.orchestra.run.smithi193.stdout: "data_loggenerations_metadata", 2024-08-18T15:55:26.631 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2", 2024-08-18T15:55:26.632 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:55:26.632 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:55:26.632 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:55:26.632 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:55:26.632 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T15:55:26.632 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:55:26.632 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:55:26.632 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:55:26.632 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:55:26.632 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:55:26.632 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000076", 2024-08-18T15:55:26.632 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:55:26.632 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:55:26.633 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000113", 2024-08-18T15:55:26.633 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000005", 2024-08-18T15:55:26.633 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:55:26.633 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000011", 2024-08-18T15:55:26.633 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:55:26.633 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000058", 2024-08-18T15:55:26.633 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:55:26.633 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:55:26.633 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:55:26.633 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000052", 2024-08-18T15:55:26.633 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:55:26.633 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:55:26.633 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:55:26.634 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:55:26.634 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.21", 2024-08-18T15:55:26.634 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:55:26.634 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:55:26.634 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:55:26.634 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:55:26.634 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:55:26.634 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:55:26.634 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:55:26.634 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:55:26.634 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:55:26.634 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:55:26.635 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:55:26.635 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:55:26.635 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:55:26.635 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:55:26.635 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:55:26.635 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:55:26.635 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:55:26.635 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:55:26.635 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:55:26.635 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000088", 2024-08-18T15:55:26.635 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000080", 2024-08-18T15:55:26.635 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:55:26.635 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:55:26.635 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000090", 2024-08-18T15:55:26.635 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:55:26.636 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:55:26.636 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000110", 2024-08-18T15:55:26.636 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:55:26.636 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:55:26.636 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:55:26.636 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.0", 2024-08-18T15:55:26.636 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:55:26.636 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000096", 2024-08-18T15:55:26.636 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:55:26.636 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:55:26.636 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:55:26.636 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:55:26.636 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:55:26.636 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000087", 2024-08-18T15:55:26.637 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:55:26.637 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:55:26.637 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:55:26.637 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:55:26.637 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000008", 2024-08-18T15:55:26.637 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:55:26.637 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:55:26.637 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:55:26.637 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:55:26.637 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:55:26.637 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000006", 2024-08-18T15:55:26.637 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000029", 2024-08-18T15:55:26.637 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:55:26.637 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000089", 2024-08-18T15:55:26.637 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:55:26.638 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:55:26.638 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000022", 2024-08-18T15:55:26.638 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:55:26.638 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:55:26.638 INFO:teuthology.orchestra.run.smithi193.stdout: "bilog.trim", 2024-08-18T15:55:26.638 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000016", 2024-08-18T15:55:26.638 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:55:26.638 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:55:26.638 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:55:26.638 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:55:26.638 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:55:26.638 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000041", 2024-08-18T15:55:26.638 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:55:26.639 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:55:26.639 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.log.01050e30-179b-43be-bdbb-3bb23cf37ae8.3", 2024-08-18T15:55:26.639 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:55:26.639 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000018", 2024-08-18T15:55:26.639 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000092", 2024-08-18T15:55:26.639 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:55:26.639 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:55:26.639 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:55:26.639 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:55:26.639 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.27", 2024-08-18T15:55:26.639 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.4", 2024-08-18T15:55:26.639 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:55:26.639 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:55:26.639 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:55:26.639 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:55:26.640 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:55:26.640 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:55:26.640 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:55:26.640 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000014", 2024-08-18T15:55:26.640 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:55:26.640 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3.retry", 2024-08-18T15:55:26.640 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:55:26.640 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:55:26.640 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:55:26.640 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:55:26.640 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:55:26.640 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000112", 2024-08-18T15:55:26.640 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:55:26.640 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000007", 2024-08-18T15:55:26.640 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000021", 2024-08-18T15:55:26.641 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:55:26.641 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:55:26.641 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:55:26.641 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:55:26.641 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:55:26.641 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.16", 2024-08-18T15:55:26.641 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:55:26.641 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000064", 2024-08-18T15:55:26.641 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:55:26.641 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:55:26.641 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:55:26.641 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:55:26.641 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:55:26.641 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:55:26.641 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:55:26.642 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000071", 2024-08-18T15:55:26.642 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:55:26.642 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:55:26.642 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:55:26.642 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000074", 2024-08-18T15:55:26.642 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:55:26.642 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:55:26.642 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:55:26.642 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:55:26.642 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:55:26.642 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:55:26.642 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:55:26.642 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:55:26.642 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:55:26.643 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:55:26.643 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:55:26.643 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:55:26.643 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:55:26.643 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:55:26.643 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:55:26.643 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000081", 2024-08-18T15:55:26.643 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:55:26.643 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000009", 2024-08-18T15:55:26.643 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000121", 2024-08-18T15:55:26.643 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1", 2024-08-18T15:55:26.643 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:55:26.643 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:55:26.643 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:55:26.643 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:55:26.644 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:55:26.644 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:55:26.644 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.17", 2024-08-18T15:55:26.644 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000125", 2024-08-18T15:55:26.644 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:55:26.644 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000082", 2024-08-18T15:55:26.644 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:55:26.644 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:55:26.644 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:55:26.644 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:55:26.644 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:55:26.644 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:55:26.644 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:55:26.644 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:55:26.645 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:55:26.645 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:55:26.645 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:55:26.645 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000105", 2024-08-18T15:55:26.645 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000059", 2024-08-18T15:55:26.645 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:55:26.645 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:55:26.645 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:55:26.645 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:55:26.645 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:55:26.645 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:55:26.645 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:55:26.645 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:55:26.645 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:55:26.645 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:55:26.645 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:55:26.646 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000077", 2024-08-18T15:55:26.646 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000032", 2024-08-18T15:55:26.646 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:55:26.646 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:55:26.646 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:55:26.646 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:55:26.646 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:55:26.646 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:55:26.646 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.24", 2024-08-18T15:55:26.646 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:55:26.646 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:55:26.646 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:55:26.646 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:55:26.647 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3.retry", 2024-08-18T15:55:26.647 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:55:26.647 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:55:26.647 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:55:26.647 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:55:26.647 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:55:26.647 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:55:26.647 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000053", 2024-08-18T15:55:26.647 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000091", 2024-08-18T15:55:26.647 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:55:26.647 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:55:26.647 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:55:26.647 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000065", 2024-08-18T15:55:26.648 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:55:26.648 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:55:26.648 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:55:26.648 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:55:26.648 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000083", 2024-08-18T15:55:26.648 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000010", 2024-08-18T15:55:26.648 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:55:26.648 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:55:26.648 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:55:26.648 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:55:26.648 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:55:26.648 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:55:26.648 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:55:26.648 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:55:26.649 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:55:26.649 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:55:26.649 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:55:26.649 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.7", 2024-08-18T15:55:26.649 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.3", 2024-08-18T15:55:26.649 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:55:26.649 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:55:26.649 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:55:26.649 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:55:26.649 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:55:26.649 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:55:26.649 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:55:26.649 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:55:26.649 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.2", 2024-08-18T15:55:26.649 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:55:26.650 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:55:26.650 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:55:26.650 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:55:26.650 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:55:26.650 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:55:26.650 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:55:26.650 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000045", 2024-08-18T15:55:26.650 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000002", 2024-08-18T15:55:26.650 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:55:26.650 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3.2UbiLidwSTy.0", 2024-08-18T15:55:26.650 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:55:26.650 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.12", 2024-08-18T15:55:26.651 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:55:26.651 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:55:26.651 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:55:26.651 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:55:26.651 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:55:26.651 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:55:26.651 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000116", 2024-08-18T15:55:26.651 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.25", 2024-08-18T15:55:26.651 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:55:26.651 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:55:26.651 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:55:26.652 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:55:26.652 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000034", 2024-08-18T15:55:26.652 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:55:26.652 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:55:26.652 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000101", 2024-08-18T15:55:26.652 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:55:26.652 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:55:26.652 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.13", 2024-08-18T15:55:26.652 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:55:26.652 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:55:26.652 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.1", 2024-08-18T15:55:26.652 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0.retry", 2024-08-18T15:55:26.652 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:55:26.653 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:55:26.653 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000079", 2024-08-18T15:55:26.653 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000049", 2024-08-18T15:55:26.653 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:55:26.653 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0", 2024-08-18T15:55:26.653 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:55:26.653 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:55:26.653 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000117", 2024-08-18T15:55:26.653 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:55:26.653 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000044", 2024-08-18T15:55:26.653 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:55:26.653 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:55:26.653 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:55:26.654 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:55:26.654 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:55:26.654 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:55:26.654 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.14", 2024-08-18T15:55:26.654 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:55:26.654 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000066", 2024-08-18T15:55:26.654 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:55:26.654 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:55:26.654 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:55:26.654 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:55:26.654 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:55:26.654 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000068", 2024-08-18T15:55:26.654 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:55:26.655 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000085", 2024-08-18T15:55:26.655 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:55:26.655 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:55:26.655 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:55:26.655 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:55:26.655 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:55:26.655 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:55:26.655 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:55:26.655 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:55:26.655 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:55:26.655 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:55:26.655 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:55:26.655 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:55:26.656 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000073", 2024-08-18T15:55:26.656 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:55:26.656 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:55:26.656 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:55:26.656 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000038", 2024-08-18T15:55:26.656 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:55:26.656 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:55:26.656 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:55:26.656 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:55:26.656 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.10", 2024-08-18T15:55:26.656 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:55:26.656 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:55:26.656 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:55:26.656 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000118", 2024-08-18T15:55:26.657 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:55:26.657 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3", 2024-08-18T15:55:26.657 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:55:26.657 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000036", 2024-08-18T15:55:26.657 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:55:26.657 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:55:26.657 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:55:26.657 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:55:26.657 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:55:26.657 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000103", 2024-08-18T15:55:26.657 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.3", 2024-08-18T15:55:26.657 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:55:26.657 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000119", 2024-08-18T15:55:26.657 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000098", 2024-08-18T15:55:26.658 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:55:26.658 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:55:26.658 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:55:26.658 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:55:26.658 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:55:26.658 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:55:26.658 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:55:26.658 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:55:26.658 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.0", 2024-08-18T15:55:26.659 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:55:26.659 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:55:26.659 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:55:26.659 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:55:26.659 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:55:26.659 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:55:26.659 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:55:26.659 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:55:26.659 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000027", 2024-08-18T15:55:26.659 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:55:26.659 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status", 2024-08-18T15:55:26.659 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:55:26.659 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000019", 2024-08-18T15:55:26.659 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:55:26.659 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:55:26.660 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:55:26.660 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:55:26.660 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1", 2024-08-18T15:55:26.660 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:55:26.660 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:55:26.660 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:55:26.660 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:55:26.660 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000039", 2024-08-18T15:55:26.660 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:55:26.660 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:55:26.660 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:55:26.660 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:55:26.660 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:55:26.660 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:55:26.661 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:55:26.661 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:55:26.661 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:55:26.661 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:55:26.661 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:55:26.661 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000100", 2024-08-18T15:55:26.661 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:55:26.661 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:55:26.661 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:55:26.661 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:55:26.661 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:55:26.661 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000093", 2024-08-18T15:55:26.661 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:55:26.661 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000004", 2024-08-18T15:55:26.662 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000063", 2024-08-18T15:55:26.662 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000122", 2024-08-18T15:55:26.662 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:55:26.662 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:55:26.662 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:55:26.662 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000057", 2024-08-18T15:55:26.662 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:55:26.662 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:55:26.662 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:55:26.662 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:55:26.662 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000054", 2024-08-18T15:55:26.662 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:55:26.662 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000114", 2024-08-18T15:55:26.662 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:55:26.663 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:55:26.663 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:55:26.663 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.2", 2024-08-18T15:55:26.663 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:55:26.663 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:55:26.663 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0", 2024-08-18T15:55:26.663 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:55:26.663 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:55:26.663 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:55:26.663 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:55:26.663 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:55:26.663 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:55:26.664 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000012", 2024-08-18T15:55:26.664 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:55:26.664 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:55:26.664 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:55:26.664 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000084", 2024-08-18T15:55:26.664 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000043", 2024-08-18T15:55:26.664 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3", 2024-08-18T15:55:26.664 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:55:26.664 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:55:26.664 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.11", 2024-08-18T15:55:26.664 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:55:26.664 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:55:26.664 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:55:26.664 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:55:26.665 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:55:26.665 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:55:26.665 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:55:26.665 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:55:26.665 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000111", 2024-08-18T15:55:26.665 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:55:26.665 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000048", 2024-08-18T15:55:26.665 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:55:26.665 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000020", 2024-08-18T15:55:26.665 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:55:26.665 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:55:26.665 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:55:26.665 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:55:26.665 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:55:26.666 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000099", 2024-08-18T15:55:26.666 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000056", 2024-08-18T15:55:26.666 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:55:26.666 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:55:26.666 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.19", 2024-08-18T15:55:26.666 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:55:26.666 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.28", 2024-08-18T15:55:26.666 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:55:26.666 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:55:26.666 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:55:26.666 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:55:26.666 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:55:26.666 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000072", 2024-08-18T15:55:26.667 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:55:26.667 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:55:26.667 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000062", 2024-08-18T15:55:26.667 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:55:26.667 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:55:26.667 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:55:26.667 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000115", 2024-08-18T15:55:26.667 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:55:26.667 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.1", 2024-08-18T15:55:26.667 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2", 2024-08-18T15:55:26.667 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000033", 2024-08-18T15:55:26.667 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:55:26.667 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:55:26.667 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:55:26.667 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:55:26.668 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:55:26.668 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:55:26.668 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:55:26.668 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:55:26.668 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:55:26.668 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:55:26.668 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:55:26.668 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:55:26.668 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:55:26.668 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:55:26.668 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:55:26.668 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:55:26.668 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:55:26.668 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:55:26.669 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:55:26.669 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000025", 2024-08-18T15:55:26.669 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:55:26.669 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000123", 2024-08-18T15:55:26.669 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.31", 2024-08-18T15:55:26.669 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:55:26.669 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:55:26.669 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:55:26.669 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:55:26.669 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.18", 2024-08-18T15:55:26.669 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:55:26.669 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:55:26.669 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:55:26.669 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:55:26.670 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:55:26.670 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000108", 2024-08-18T15:55:26.670 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:55:26.670 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:55:26.670 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000094", 2024-08-18T15:55:26.670 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:55:26.670 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:55:26.670 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:55:26.670 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:55:26.670 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:55:26.670 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:55:26.670 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000060", 2024-08-18T15:55:26.670 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:55:26.671 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:55:26.671 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:55:26.671 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000109", 2024-08-18T15:55:26.671 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.15", 2024-08-18T15:55:26.671 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:55:26.671 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:55:26.671 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:55:26.671 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:55:26.671 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:55:26.671 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:55:26.671 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:55:26.671 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:55:26.671 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:55:26.671 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000013", 2024-08-18T15:55:26.672 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:55:26.672 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:55:26.672 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:55:26.672 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:55:26.672 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:55:26.672 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:55:26.672 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:55:26.672 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:55:26.673 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:55:26.673 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:55:26.673 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:55:26.673 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:55:26.673 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:55:26.673 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:55:26.674 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000051", 2024-08-18T15:55:26.674 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:55:26.674 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:55:26.674 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:55:26.674 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:55:26.674 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000106", 2024-08-18T15:55:26.674 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:55:26.674 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:55:26.674 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:55:26.674 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:55:26.674 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:55:26.674 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000017", 2024-08-18T15:55:26.675 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:55:26.675 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:55:26.675 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:55:26.675 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000023", 2024-08-18T15:55:26.675 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:55:26.675 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:55:26.675 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:55:26.675 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:55:26.675 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:55:26.675 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:55:26.675 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:55:26.675 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000107", 2024-08-18T15:55:26.675 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:55:26.676 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:55:26.676 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000046", 2024-08-18T15:55:26.676 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:55:26.676 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:55:26.676 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:55:26.676 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.26", 2024-08-18T15:55:26.676 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:55:26.676 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:55:26.676 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9", 2024-08-18T15:55:26.676 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.22", 2024-08-18T15:55:26.676 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:55:26.676 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:55:26.676 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:55:26.676 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:55:26.677 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:55:26.677 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:55:26.677 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:55:26.677 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2", 2024-08-18T15:55:26.677 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:55:26.677 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:55:26.677 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000067", 2024-08-18T15:55:26.677 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:55:26.677 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:55:26.677 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:55:26.677 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:55:26.677 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:55:26.678 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:55:26.678 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:55:26.678 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:55:26.678 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:55:26.678 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:55:26.678 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:55:26.678 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:55:26.678 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:55:26.678 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:55:26.678 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:55:26.678 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:55:26.678 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:55:26.678 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:55:26.679 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:55:26.679 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.9", 2024-08-18T15:55:26.679 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000086", 2024-08-18T15:55:26.679 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.5", 2024-08-18T15:55:26.679 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:55:26.679 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:55:26.679 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:55:26.679 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:55:26.679 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:55:26.679 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:55:26.679 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2.MFXJ583gCr1.0", 2024-08-18T15:55:26.679 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:55:26.679 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1.GPnO16liyZ+.0", 2024-08-18T15:55:26.679 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:55:26.680 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:55:26.680 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:55:26.680 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:55:26.680 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:55:26.680 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:55:26.680 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:55:26.680 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:55:26.680 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:55:26.680 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.20", 2024-08-18T15:55:26.680 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:55:26.680 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:55:26.680 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:55:26.680 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:55:26.681 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:55:26.681 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:55:26.681 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1.retry", 2024-08-18T15:55:26.681 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:55:26.681 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:55:26.681 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:55:26.681 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0.retry", 2024-08-18T15:55:26.681 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000050", 2024-08-18T15:55:26.681 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:55:26.681 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000102", 2024-08-18T15:55:26.681 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:55:26.681 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:55:26.681 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:55:26.681 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:55:26.682 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:55:26.682 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:55:26.682 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:55:26.682 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:55:26.682 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:55:26.682 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:55:26.682 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:55:26.682 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000124", 2024-08-18T15:55:26.682 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:55:26.682 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:55:26.682 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:55:26.682 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000001", 2024-08-18T15:55:26.682 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:55:26.682 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:55:26.683 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:55:26.683 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:55:26.683 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000126", 2024-08-18T15:55:26.683 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:55:26.683 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0.fZhctfLXnSt.0", 2024-08-18T15:55:26.683 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:55:26.683 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000055", 2024-08-18T15:55:26.683 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:55:26.683 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000061", 2024-08-18T15:55:26.683 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1", 2024-08-18T15:55:26.683 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:55:26.683 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:55:26.683 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:55:26.684 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000097", 2024-08-18T15:55:26.684 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:55:26.684 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:55:26.684 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:55:26.684 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:55:26.684 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:55:26.684 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:55:26.684 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:55:26.684 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:55:26.684 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000042", 2024-08-18T15:55:26.684 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:55:26.684 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:55:26.684 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:55:26.685 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1.retry", 2024-08-18T15:55:26.685 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:55:26.685 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:55:26.685 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:55:26.685 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:55:26.685 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:55:26.685 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:55:26.685 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:55:26.685 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000075", 2024-08-18T15:55:26.685 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:55:26.685 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000120", 2024-08-18T15:55:26.685 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000030", 2024-08-18T15:55:26.685 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:55:26.685 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:55:26.686 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:55:26.686 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:55:26.686 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:55:26.686 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:55:26.686 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9" 2024-08-18T15:55:26.686 INFO:teuthology.orchestra.run.smithi193.stdout:] 2024-08-18T15:55:26.686 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:55:31.688 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:55:31.688 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:55:31.688 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:55:31.741 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:55:31.742 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:55:31.806 INFO:teuthology.orchestra.run.smithi193.stdout:[ 2024-08-18T15:55:31.806 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:55:31.806 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:55:31.807 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.29", 2024-08-18T15:55:31.807 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000078", 2024-08-18T15:55:31.807 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.history", 2024-08-18T15:55:31.807 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:55:31.807 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:55:31.807 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:55:31.807 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:55:31.807 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.23", 2024-08-18T15:55:31.807 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000070", 2024-08-18T15:55:31.807 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000104", 2024-08-18T15:55:31.807 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:55:31.807 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2.retry", 2024-08-18T15:55:31.807 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:55:31.808 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:55:31.808 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:55:31.808 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:55:31.808 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000026", 2024-08-18T15:55:31.808 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:55:31.808 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:55:31.808 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:55:31.808 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:55:31.808 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:55:31.808 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:55:31.808 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.8", 2024-08-18T15:55:31.808 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:55:31.808 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:55:31.808 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:55:31.808 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:55:31.809 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000028", 2024-08-18T15:55:31.809 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:55:31.809 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:55:31.809 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:55:31.809 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:55:31.809 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:55:31.809 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:55:31.809 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:55:31.809 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2.retry", 2024-08-18T15:55:31.809 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:55:31.809 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.30", 2024-08-18T15:55:31.809 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:55:31.809 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:55:31.809 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:55:31.809 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:55:31.810 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000040", 2024-08-18T15:55:31.810 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:55:31.810 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.6", 2024-08-18T15:55:31.810 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:55:31.810 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:55:31.810 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:55:31.810 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:55:31.810 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:55:31.810 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:55:31.810 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000015", 2024-08-18T15:55:31.810 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:55:31.810 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:55:31.810 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000069", 2024-08-18T15:55:31.811 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:55:31.811 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:55:31.811 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:55:31.811 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000095", 2024-08-18T15:55:31.811 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0", 2024-08-18T15:55:31.811 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000003", 2024-08-18T15:55:31.811 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000047", 2024-08-18T15:55:31.811 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:55:31.811 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:55:31.811 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:55:31.811 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:55:31.811 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:55:31.811 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:55:31.811 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:55:31.812 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:55:31.812 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:55:31.812 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000035", 2024-08-18T15:55:31.812 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:55:31.812 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:55:31.812 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000037", 2024-08-18T15:55:31.812 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:55:31.812 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:55:31.812 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:55:31.812 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:55:31.812 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:55:31.812 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:55:31.812 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:55:31.812 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:55:31.813 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:55:31.813 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:55:31.813 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:55:31.813 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:55:31.813 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:55:31.813 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:55:31.813 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000024", 2024-08-18T15:55:31.813 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:55:31.813 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:55:31.813 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:55:31.813 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:55:31.813 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000000", 2024-08-18T15:55:31.813 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:55:31.813 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000031", 2024-08-18T15:55:31.814 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:55:31.814 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:55:31.814 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:55:31.814 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:55:31.814 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:55:31.814 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:55:31.814 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3", 2024-08-18T15:55:31.814 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:55:31.814 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:55:31.814 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:55:31.814 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:55:31.814 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:55:31.814 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:55:31.815 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:55:31.815 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:55:31.815 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:55:31.815 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:55:31.815 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:55:31.815 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:55:31.815 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:55:31.815 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:55:31.815 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:55:31.815 INFO:teuthology.orchestra.run.smithi193.stdout: "data_loggenerations_metadata", 2024-08-18T15:55:31.815 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2", 2024-08-18T15:55:31.815 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:55:31.815 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:55:31.815 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:55:31.816 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:55:31.816 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T15:55:31.816 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:55:31.816 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:55:31.816 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:55:31.816 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:55:31.816 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:55:31.816 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000076", 2024-08-18T15:55:31.816 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:55:31.816 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:55:31.816 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000113", 2024-08-18T15:55:31.816 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000005", 2024-08-18T15:55:31.816 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:55:31.816 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000011", 2024-08-18T15:55:31.817 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:55:31.817 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000058", 2024-08-18T15:55:31.817 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:55:31.817 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:55:31.817 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:55:31.817 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000052", 2024-08-18T15:55:31.817 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:55:31.817 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:55:31.817 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:55:31.817 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:55:31.817 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.21", 2024-08-18T15:55:31.817 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:55:31.817 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:55:31.817 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:55:31.818 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:55:31.818 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:55:31.818 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:55:31.818 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:55:31.818 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:55:31.818 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:55:31.818 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:55:31.818 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:55:31.818 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:55:31.818 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:55:31.818 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:55:31.818 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:55:31.818 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:55:31.819 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:55:31.819 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:55:31.819 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:55:31.819 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000088", 2024-08-18T15:55:31.819 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000080", 2024-08-18T15:55:31.819 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:55:31.819 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:55:31.819 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000090", 2024-08-18T15:55:31.819 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:55:31.819 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:55:31.819 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000110", 2024-08-18T15:55:31.819 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:55:31.819 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:55:31.820 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:55:31.820 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.0", 2024-08-18T15:55:31.820 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:55:31.820 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000096", 2024-08-18T15:55:31.820 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:55:31.820 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:55:31.820 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:55:31.820 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:55:31.820 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:55:31.820 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000087", 2024-08-18T15:55:31.820 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:55:31.820 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:55:31.820 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:55:31.820 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:55:31.821 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000008", 2024-08-18T15:55:31.821 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:55:31.821 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:55:31.821 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:55:31.821 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:55:31.821 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:55:31.821 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000006", 2024-08-18T15:55:31.821 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000029", 2024-08-18T15:55:31.821 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:55:31.821 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000089", 2024-08-18T15:55:31.821 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:55:31.821 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:55:31.821 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000022", 2024-08-18T15:55:31.822 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:55:31.822 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:55:31.822 INFO:teuthology.orchestra.run.smithi193.stdout: "bilog.trim", 2024-08-18T15:55:31.822 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000016", 2024-08-18T15:55:31.822 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:55:31.822 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:55:31.822 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:55:31.822 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:55:31.822 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:55:31.822 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000041", 2024-08-18T15:55:31.822 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:55:31.822 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:55:31.823 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.log.01050e30-179b-43be-bdbb-3bb23cf37ae8.3", 2024-08-18T15:55:31.823 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:55:31.823 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000018", 2024-08-18T15:55:31.823 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000092", 2024-08-18T15:55:31.823 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:55:31.823 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:55:31.823 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:55:31.823 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:55:31.823 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.27", 2024-08-18T15:55:31.823 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.4", 2024-08-18T15:55:31.823 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:55:31.823 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:55:31.824 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:55:31.824 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:55:31.824 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:55:31.824 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:55:31.824 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:55:31.824 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000014", 2024-08-18T15:55:31.824 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:55:31.824 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3.retry", 2024-08-18T15:55:31.824 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:55:31.824 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:55:31.824 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:55:31.824 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:55:31.824 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:55:31.825 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000112", 2024-08-18T15:55:31.825 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:55:31.825 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000007", 2024-08-18T15:55:31.825 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000021", 2024-08-18T15:55:31.825 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:55:31.825 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:55:31.825 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:55:31.825 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:55:31.825 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:55:31.825 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.16", 2024-08-18T15:55:31.825 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:55:31.825 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000064", 2024-08-18T15:55:31.826 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:55:31.826 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:55:31.826 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:55:31.826 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:55:31.826 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:55:31.826 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:55:31.826 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:55:31.826 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000071", 2024-08-18T15:55:31.826 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:55:31.826 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:55:31.826 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:55:31.826 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000074", 2024-08-18T15:55:31.827 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:55:31.827 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:55:31.827 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:55:31.827 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:55:31.827 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:55:31.827 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:55:31.827 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:55:31.827 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:55:31.827 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:55:31.827 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:55:31.827 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:55:31.827 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:55:31.827 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:55:31.828 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:55:31.828 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:55:31.828 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000081", 2024-08-18T15:55:31.828 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:55:31.828 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000009", 2024-08-18T15:55:31.828 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000121", 2024-08-18T15:55:31.828 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1", 2024-08-18T15:55:31.828 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:55:31.828 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:55:31.828 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:55:31.828 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:55:31.828 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:55:31.828 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:55:31.828 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.17", 2024-08-18T15:55:31.829 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000125", 2024-08-18T15:55:31.829 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:55:31.829 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000082", 2024-08-18T15:55:31.829 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:55:31.829 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:55:31.829 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:55:31.829 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:55:31.829 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:55:31.829 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:55:31.829 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:55:31.829 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:55:31.829 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:55:31.829 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:55:31.830 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:55:31.830 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000105", 2024-08-18T15:55:31.830 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000059", 2024-08-18T15:55:31.830 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:55:31.830 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:55:31.830 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:55:31.830 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:55:31.830 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:55:31.830 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:55:31.830 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:55:31.830 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:55:31.831 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:55:31.831 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:55:31.831 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:55:31.831 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000077", 2024-08-18T15:55:31.831 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000032", 2024-08-18T15:55:31.831 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:55:31.831 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:55:31.831 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:55:31.831 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:55:31.831 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:55:31.831 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:55:31.831 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.24", 2024-08-18T15:55:31.832 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:55:31.832 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:55:31.832 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:55:31.832 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:55:31.832 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3.retry", 2024-08-18T15:55:31.832 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:55:31.832 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:55:31.832 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:55:31.832 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:55:31.832 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:55:31.832 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:55:31.832 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000053", 2024-08-18T15:55:31.832 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000091", 2024-08-18T15:55:31.833 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:55:31.833 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:55:31.833 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:55:31.833 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000065", 2024-08-18T15:55:31.833 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:55:31.833 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:55:31.833 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:55:31.833 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:55:31.833 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000083", 2024-08-18T15:55:31.833 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000010", 2024-08-18T15:55:31.833 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:55:31.833 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:55:31.833 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:55:31.834 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:55:31.834 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:55:31.834 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:55:31.834 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:55:31.834 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:55:31.834 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:55:31.834 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:55:31.834 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:55:31.834 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.7", 2024-08-18T15:55:31.834 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.3", 2024-08-18T15:55:31.834 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:55:31.835 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:55:31.835 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:55:31.835 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:55:31.835 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:55:31.835 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:55:31.835 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:55:31.835 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:55:31.835 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.2", 2024-08-18T15:55:31.835 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:55:31.835 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:55:31.835 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:55:31.835 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:55:31.835 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:55:31.835 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:55:31.836 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:55:31.836 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000045", 2024-08-18T15:55:31.836 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000002", 2024-08-18T15:55:31.836 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:55:31.836 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3.2UbiLidwSTy.0", 2024-08-18T15:55:31.836 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:55:31.836 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.12", 2024-08-18T15:55:31.836 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:55:31.836 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:55:31.836 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:55:31.836 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:55:31.836 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:55:31.836 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:55:31.836 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000116", 2024-08-18T15:55:31.837 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.25", 2024-08-18T15:55:31.837 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:55:31.837 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:55:31.837 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:55:31.837 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:55:31.837 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000034", 2024-08-18T15:55:31.837 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:55:31.837 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:55:31.837 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000101", 2024-08-18T15:55:31.837 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:55:31.837 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:55:31.837 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.13", 2024-08-18T15:55:31.838 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:55:31.838 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:55:31.838 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.1", 2024-08-18T15:55:31.838 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0.retry", 2024-08-18T15:55:31.838 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:55:31.838 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:55:31.838 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000079", 2024-08-18T15:55:31.838 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000049", 2024-08-18T15:55:31.838 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:55:31.838 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0", 2024-08-18T15:55:31.838 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:55:31.838 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:55:31.838 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000117", 2024-08-18T15:55:31.839 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:55:31.839 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000044", 2024-08-18T15:55:31.839 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:55:31.839 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:55:31.839 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:55:31.839 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:55:31.839 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:55:31.839 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:55:31.839 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.14", 2024-08-18T15:55:31.839 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:55:31.839 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000066", 2024-08-18T15:55:31.839 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:55:31.839 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:55:31.839 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:55:31.840 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:55:31.840 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:55:31.840 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000068", 2024-08-18T15:55:31.840 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:55:31.840 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000085", 2024-08-18T15:55:31.840 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:55:31.840 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:55:31.840 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:55:31.840 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:55:31.840 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:55:31.840 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:55:31.840 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:55:31.840 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:55:31.841 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:55:31.841 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:55:31.841 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:55:31.841 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:55:31.841 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000073", 2024-08-18T15:55:31.841 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:55:31.841 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:55:31.841 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:55:31.841 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000038", 2024-08-18T15:55:31.841 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:55:31.841 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:55:31.841 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:55:31.841 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:55:31.841 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.10", 2024-08-18T15:55:31.842 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:55:31.842 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:55:31.842 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:55:31.842 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000118", 2024-08-18T15:55:31.842 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:55:31.842 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3", 2024-08-18T15:55:31.842 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:55:31.842 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000036", 2024-08-18T15:55:31.842 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:55:31.842 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:55:31.842 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:55:31.842 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:55:31.843 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:55:31.843 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000103", 2024-08-18T15:55:31.843 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.3", 2024-08-18T15:55:31.843 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:55:31.843 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000119", 2024-08-18T15:55:31.843 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000098", 2024-08-18T15:55:31.843 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:55:31.843 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:55:31.843 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:55:31.843 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:55:31.843 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:55:31.843 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:55:31.843 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:55:31.843 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:55:31.844 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.0", 2024-08-18T15:55:31.844 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:55:31.844 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:55:31.844 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:55:31.844 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:55:31.844 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:55:31.844 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:55:31.844 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:55:31.844 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:55:31.844 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000027", 2024-08-18T15:55:31.844 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:55:31.844 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status", 2024-08-18T15:55:31.844 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:55:31.845 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000019", 2024-08-18T15:55:31.845 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:55:31.845 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:55:31.845 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:55:31.845 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:55:31.845 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1", 2024-08-18T15:55:31.845 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:55:31.845 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:55:31.845 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:55:31.845 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:55:31.845 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000039", 2024-08-18T15:55:31.845 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:55:31.845 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:55:31.846 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:55:31.846 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:55:31.846 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:55:31.846 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:55:31.846 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:55:31.846 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:55:31.846 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:55:31.846 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:55:31.846 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:55:31.846 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000100", 2024-08-18T15:55:31.846 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:55:31.846 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:55:31.846 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:55:31.847 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:55:31.847 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:55:31.847 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000093", 2024-08-18T15:55:31.847 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:55:31.847 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000004", 2024-08-18T15:55:31.847 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000063", 2024-08-18T15:55:31.847 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000122", 2024-08-18T15:55:31.847 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:55:31.847 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:55:31.847 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:55:31.847 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000057", 2024-08-18T15:55:31.847 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:55:31.847 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:55:31.848 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:55:31.848 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:55:31.848 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000054", 2024-08-18T15:55:31.848 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:55:31.848 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000114", 2024-08-18T15:55:31.848 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:55:31.848 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:55:31.848 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:55:31.848 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.2", 2024-08-18T15:55:31.848 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:55:31.848 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:55:31.848 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0", 2024-08-18T15:55:31.848 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:55:31.848 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:55:31.849 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:55:31.849 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:55:31.849 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:55:31.849 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:55:31.849 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000012", 2024-08-18T15:55:31.849 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:55:31.849 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:55:31.849 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:55:31.849 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000084", 2024-08-18T15:55:31.849 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000043", 2024-08-18T15:55:31.849 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3", 2024-08-18T15:55:31.849 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:55:31.849 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:55:31.849 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.11", 2024-08-18T15:55:31.850 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:55:31.850 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:55:31.850 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:55:31.850 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:55:31.850 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:55:31.850 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:55:31.850 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:55:31.850 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:55:31.850 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000111", 2024-08-18T15:55:31.850 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:55:31.850 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000048", 2024-08-18T15:55:31.850 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:55:31.850 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000020", 2024-08-18T15:55:31.851 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:55:31.851 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:55:31.851 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:55:31.851 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:55:31.851 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:55:31.851 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000099", 2024-08-18T15:55:31.851 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000056", 2024-08-18T15:55:31.851 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:55:31.851 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:55:31.851 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.19", 2024-08-18T15:55:31.851 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:55:31.853 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.28", 2024-08-18T15:55:31.853 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:55:31.853 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:55:31.853 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:55:31.854 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:55:31.854 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:55:31.854 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000072", 2024-08-18T15:55:31.854 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:55:31.854 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:55:31.854 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000062", 2024-08-18T15:55:31.854 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:55:31.854 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:55:31.854 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:55:31.854 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000115", 2024-08-18T15:55:31.854 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:55:31.854 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.1", 2024-08-18T15:55:31.855 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2", 2024-08-18T15:55:31.855 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000033", 2024-08-18T15:55:31.855 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:55:31.855 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:55:31.855 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:55:31.855 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:55:31.855 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:55:31.855 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:55:31.855 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:55:31.855 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:55:31.855 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:55:31.855 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:55:31.855 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:55:31.856 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:55:31.856 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:55:31.856 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:55:31.856 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:55:31.856 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:55:31.856 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:55:31.856 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:55:31.856 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:55:31.856 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000025", 2024-08-18T15:55:31.856 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:55:31.856 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000123", 2024-08-18T15:55:31.856 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.31", 2024-08-18T15:55:31.856 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:55:31.857 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:55:31.857 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:55:31.857 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:55:31.857 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.18", 2024-08-18T15:55:31.857 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:55:31.857 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:55:31.857 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:55:31.857 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:55:31.857 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:55:31.857 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000108", 2024-08-18T15:55:31.857 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:55:31.857 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:55:31.857 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000094", 2024-08-18T15:55:31.858 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:55:31.858 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:55:31.858 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:55:31.858 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:55:31.858 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:55:31.858 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:55:31.858 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000060", 2024-08-18T15:55:31.858 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:55:31.858 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:55:31.858 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:55:31.858 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000109", 2024-08-18T15:55:31.858 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.15", 2024-08-18T15:55:31.859 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:55:31.859 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:55:31.859 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:55:31.859 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:55:31.859 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:55:31.859 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:55:31.859 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:55:31.859 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:55:31.859 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:55:31.859 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000013", 2024-08-18T15:55:31.859 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:55:31.859 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:55:31.859 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:55:31.859 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:55:31.860 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:55:31.860 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:55:31.860 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:55:31.860 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:55:31.860 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:55:31.860 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:55:31.860 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:55:31.860 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:55:31.860 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:55:31.860 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:55:31.860 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000051", 2024-08-18T15:55:31.860 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:55:31.860 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:55:31.860 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:55:31.861 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:55:31.861 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000106", 2024-08-18T15:55:31.861 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:55:31.861 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:55:31.861 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:55:31.861 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:55:31.861 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:55:31.861 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000017", 2024-08-18T15:55:31.861 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:55:31.861 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:55:31.861 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:55:31.861 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000023", 2024-08-18T15:55:31.861 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:55:31.861 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:55:31.862 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:55:31.862 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:55:31.862 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:55:31.862 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:55:31.862 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:55:31.862 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000107", 2024-08-18T15:55:31.862 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:55:31.862 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:55:31.862 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000046", 2024-08-18T15:55:31.862 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:55:31.862 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:55:31.862 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:55:31.862 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.26", 2024-08-18T15:55:31.863 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:55:31.863 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:55:31.863 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9", 2024-08-18T15:55:31.863 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.22", 2024-08-18T15:55:31.863 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:55:31.863 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:55:31.863 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:55:31.863 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:55:31.863 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:55:31.863 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:55:31.863 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:55:31.863 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2", 2024-08-18T15:55:31.863 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:55:31.864 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:55:31.864 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000067", 2024-08-18T15:55:31.864 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:55:31.864 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:55:31.864 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:55:31.864 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:55:31.864 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:55:31.864 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:55:31.864 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:55:31.864 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:55:31.864 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:55:31.864 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:55:31.864 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:55:31.865 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:55:31.865 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:55:31.865 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:55:31.865 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:55:31.865 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:55:31.865 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:55:31.865 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:55:31.865 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:55:31.865 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.9", 2024-08-18T15:55:31.865 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000086", 2024-08-18T15:55:31.865 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.5", 2024-08-18T15:55:31.865 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:55:31.865 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:55:31.866 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:55:31.866 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:55:31.866 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:55:31.866 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:55:31.866 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2.MFXJ583gCr1.0", 2024-08-18T15:55:31.866 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:55:31.866 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1.GPnO16liyZ+.0", 2024-08-18T15:55:31.866 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:55:31.866 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:55:31.866 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:55:31.866 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:55:31.866 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:55:31.866 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:55:31.867 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:55:31.867 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:55:31.867 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:55:31.867 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.20", 2024-08-18T15:55:31.867 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:55:31.867 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:55:31.867 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:55:31.867 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:55:31.867 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:55:31.867 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:55:31.867 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1.retry", 2024-08-18T15:55:31.867 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:55:31.867 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:55:31.867 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:55:31.868 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0.retry", 2024-08-18T15:55:31.868 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000050", 2024-08-18T15:55:31.868 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:55:31.868 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000102", 2024-08-18T15:55:31.868 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:55:31.868 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:55:31.868 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:55:31.868 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:55:31.868 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:55:31.868 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:55:31.868 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:55:31.868 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:55:31.868 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:55:31.869 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:55:31.869 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:55:31.869 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000124", 2024-08-18T15:55:31.869 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:55:31.869 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:55:31.869 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:55:31.869 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000001", 2024-08-18T15:55:31.869 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:55:31.869 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:55:31.869 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:55:31.869 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:55:31.869 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000126", 2024-08-18T15:55:31.869 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:55:31.869 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0.fZhctfLXnSt.0", 2024-08-18T15:55:31.870 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:55:31.870 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000055", 2024-08-18T15:55:31.870 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:55:31.870 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000061", 2024-08-18T15:55:31.870 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1", 2024-08-18T15:55:31.870 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:55:31.870 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:55:31.870 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:55:31.870 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000097", 2024-08-18T15:55:31.870 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:55:31.870 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:55:31.870 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:55:31.870 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:55:31.871 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:55:31.871 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:55:31.871 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:55:31.871 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:55:31.871 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000042", 2024-08-18T15:55:31.871 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:55:31.871 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:55:31.871 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:55:31.871 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1.retry", 2024-08-18T15:55:31.871 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:55:31.871 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:55:31.871 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:55:31.871 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:55:31.871 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:55:31.872 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:55:31.872 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:55:31.872 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000075", 2024-08-18T15:55:31.872 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:55:31.872 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000120", 2024-08-18T15:55:31.872 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000030", 2024-08-18T15:55:31.872 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:55:31.872 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:55:31.872 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:55:31.872 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:55:31.872 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:55:31.872 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:55:31.872 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9" 2024-08-18T15:55:31.872 INFO:teuthology.orchestra.run.smithi193.stdout:] 2024-08-18T15:55:31.873 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:55:36.874 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:55:36.874 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:55:36.875 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:55:36.927 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:55:36.927 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:55:36.990 INFO:teuthology.orchestra.run.smithi193.stdout:[ 2024-08-18T15:55:36.990 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:55:36.990 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:55:36.990 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.29", 2024-08-18T15:55:36.990 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000078", 2024-08-18T15:55:36.990 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.history", 2024-08-18T15:55:36.990 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:55:36.991 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:55:36.991 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:55:36.991 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:55:36.991 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.23", 2024-08-18T15:55:36.991 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000070", 2024-08-18T15:55:36.991 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000104", 2024-08-18T15:55:36.991 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:55:36.991 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2.retry", 2024-08-18T15:55:36.991 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:55:36.991 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:55:36.991 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:55:36.991 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:55:36.991 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000026", 2024-08-18T15:55:36.991 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:55:36.992 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:55:36.992 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:55:36.992 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:55:36.992 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:55:36.992 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:55:36.992 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.8", 2024-08-18T15:55:36.992 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:55:36.992 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:55:36.992 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:55:36.992 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:55:36.992 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000028", 2024-08-18T15:55:36.992 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:55:36.992 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:55:36.992 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:55:36.992 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:55:36.993 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:55:36.993 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:55:36.993 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:55:36.993 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2.retry", 2024-08-18T15:55:36.993 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:55:36.993 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.30", 2024-08-18T15:55:36.993 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:55:36.993 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:55:36.993 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:55:36.993 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:55:36.993 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000040", 2024-08-18T15:55:36.993 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:55:36.993 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.6", 2024-08-18T15:55:36.993 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:55:36.993 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:55:36.994 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:55:36.994 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:55:36.994 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:55:36.994 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:55:36.994 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000015", 2024-08-18T15:55:36.994 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:55:36.994 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:55:36.994 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000069", 2024-08-18T15:55:36.994 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:55:36.994 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:55:36.994 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:55:36.994 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000095", 2024-08-18T15:55:36.994 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0", 2024-08-18T15:55:36.995 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000003", 2024-08-18T15:55:36.995 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000047", 2024-08-18T15:55:36.995 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:55:36.995 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:55:36.995 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:55:36.995 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:55:36.995 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:55:36.995 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:55:36.995 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:55:36.995 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:55:36.995 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:55:36.995 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000035", 2024-08-18T15:55:36.995 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:55:36.995 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:55:36.996 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000037", 2024-08-18T15:55:36.996 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:55:36.996 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:55:36.996 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:55:36.996 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:55:36.996 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:55:36.996 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:55:36.996 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:55:36.996 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:55:36.996 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:55:36.996 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:55:36.996 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:55:36.996 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:55:36.997 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:55:36.997 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:55:36.997 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000024", 2024-08-18T15:55:36.997 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:55:36.997 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:55:36.997 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:55:36.997 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:55:36.997 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000000", 2024-08-18T15:55:36.997 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:55:36.997 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000031", 2024-08-18T15:55:36.997 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:55:36.997 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:55:36.997 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:55:36.997 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:55:36.998 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:55:36.998 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:55:36.998 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3", 2024-08-18T15:55:36.998 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:55:36.998 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:55:36.998 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:55:36.998 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:55:36.998 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:55:36.998 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:55:36.998 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:55:36.998 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:55:36.998 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:55:36.998 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:55:36.999 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:55:36.999 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:55:36.999 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:55:36.999 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:55:36.999 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:55:36.999 INFO:teuthology.orchestra.run.smithi193.stdout: "data_loggenerations_metadata", 2024-08-18T15:55:36.999 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2", 2024-08-18T15:55:36.999 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:55:36.999 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:55:36.999 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:55:36.999 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:55:36.999 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T15:55:36.999 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:55:36.999 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:55:36.999 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:55:37.000 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:55:37.000 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:55:37.000 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000076", 2024-08-18T15:55:37.000 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:55:37.000 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:55:37.000 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000113", 2024-08-18T15:55:37.000 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000005", 2024-08-18T15:55:37.000 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:55:37.000 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000011", 2024-08-18T15:55:37.000 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:55:37.000 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000058", 2024-08-18T15:55:37.000 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:55:37.000 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:55:37.000 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:55:37.000 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000052", 2024-08-18T15:55:37.001 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:55:37.001 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:55:37.001 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:55:37.001 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:55:37.001 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.21", 2024-08-18T15:55:37.001 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:55:37.001 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:55:37.001 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:55:37.001 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:55:37.001 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:55:37.001 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:55:37.001 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:55:37.001 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:55:37.001 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:55:37.001 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:55:37.002 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:55:37.002 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:55:37.002 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:55:37.002 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:55:37.002 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:55:37.002 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:55:37.002 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:55:37.002 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:55:37.002 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:55:37.002 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000088", 2024-08-18T15:55:37.002 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000080", 2024-08-18T15:55:37.002 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:55:37.002 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:55:37.003 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000090", 2024-08-18T15:55:37.003 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:55:37.003 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:55:37.003 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000110", 2024-08-18T15:55:37.003 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:55:37.003 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:55:37.003 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:55:37.003 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.0", 2024-08-18T15:55:37.003 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:55:37.003 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000096", 2024-08-18T15:55:37.003 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:55:37.003 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:55:37.003 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:55:37.003 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:55:37.003 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:55:37.004 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000087", 2024-08-18T15:55:37.004 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:55:37.004 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:55:37.004 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:55:37.004 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:55:37.004 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000008", 2024-08-18T15:55:37.004 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:55:37.004 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:55:37.004 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:55:37.004 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:55:37.004 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:55:37.004 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000006", 2024-08-18T15:55:37.004 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000029", 2024-08-18T15:55:37.004 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:55:37.004 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000089", 2024-08-18T15:55:37.005 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:55:37.005 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:55:37.005 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000022", 2024-08-18T15:55:37.005 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:55:37.005 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:55:37.005 INFO:teuthology.orchestra.run.smithi193.stdout: "bilog.trim", 2024-08-18T15:55:37.005 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000016", 2024-08-18T15:55:37.005 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:55:37.005 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:55:37.005 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:55:37.005 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:55:37.005 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:55:37.005 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000041", 2024-08-18T15:55:37.005 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:55:37.006 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:55:37.006 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.log.01050e30-179b-43be-bdbb-3bb23cf37ae8.3", 2024-08-18T15:55:37.006 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:55:37.006 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000018", 2024-08-18T15:55:37.006 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000092", 2024-08-18T15:55:37.006 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:55:37.006 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:55:37.006 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:55:37.006 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:55:37.006 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.27", 2024-08-18T15:55:37.006 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.4", 2024-08-18T15:55:37.006 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:55:37.007 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:55:37.007 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:55:37.007 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:55:37.007 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:55:37.007 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:55:37.007 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:55:37.007 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000014", 2024-08-18T15:55:37.007 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:55:37.007 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3.retry", 2024-08-18T15:55:37.007 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:55:37.007 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:55:37.007 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:55:37.007 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:55:37.008 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:55:37.008 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000112", 2024-08-18T15:55:37.008 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:55:37.008 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000007", 2024-08-18T15:55:37.008 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000021", 2024-08-18T15:55:37.008 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:55:37.008 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:55:37.008 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:55:37.008 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:55:37.008 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:55:37.008 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.16", 2024-08-18T15:55:37.008 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:55:37.008 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000064", 2024-08-18T15:55:37.008 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:55:37.008 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:55:37.009 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:55:37.009 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:55:37.009 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:55:37.009 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:55:37.009 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:55:37.009 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000071", 2024-08-18T15:55:37.009 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:55:37.009 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:55:37.009 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:55:37.009 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000074", 2024-08-18T15:55:37.009 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:55:37.009 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:55:37.009 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:55:37.009 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:55:37.009 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:55:37.010 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:55:37.010 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:55:37.010 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:55:37.010 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:55:37.010 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:55:37.010 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:55:37.010 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:55:37.010 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:55:37.010 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:55:37.010 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:55:37.010 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000081", 2024-08-18T15:55:37.010 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:55:37.010 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000009", 2024-08-18T15:55:37.011 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000121", 2024-08-18T15:55:37.011 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1", 2024-08-18T15:55:37.011 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:55:37.011 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:55:37.011 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:55:37.011 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:55:37.011 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:55:37.011 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:55:37.011 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.17", 2024-08-18T15:55:37.011 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000125", 2024-08-18T15:55:37.011 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:55:37.011 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000082", 2024-08-18T15:55:37.011 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:55:37.011 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:55:37.011 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:55:37.012 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:55:37.012 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:55:37.012 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:55:37.012 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:55:37.012 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:55:37.012 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:55:37.012 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:55:37.012 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:55:37.012 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000105", 2024-08-18T15:55:37.012 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000059", 2024-08-18T15:55:37.012 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:55:37.012 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:55:37.012 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:55:37.012 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:55:37.012 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:55:37.013 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:55:37.013 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:55:37.013 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:55:37.013 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:55:37.013 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:55:37.013 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:55:37.013 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000077", 2024-08-18T15:55:37.013 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000032", 2024-08-18T15:55:37.013 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:55:37.013 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:55:37.013 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:55:37.013 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:55:37.013 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:55:37.013 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:55:37.013 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.24", 2024-08-18T15:55:37.013 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:55:37.014 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:55:37.014 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:55:37.014 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:55:37.014 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3.retry", 2024-08-18T15:55:37.014 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:55:37.014 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:55:37.014 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:55:37.014 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:55:37.014 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:55:37.014 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:55:37.014 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000053", 2024-08-18T15:55:37.014 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000091", 2024-08-18T15:55:37.015 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:55:37.015 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:55:37.015 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:55:37.015 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000065", 2024-08-18T15:55:37.015 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:55:37.015 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:55:37.015 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:55:37.015 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:55:37.015 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000083", 2024-08-18T15:55:37.015 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000010", 2024-08-18T15:55:37.015 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:55:37.015 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:55:37.015 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:55:37.015 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:55:37.016 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:55:37.016 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:55:37.016 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:55:37.016 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:55:37.016 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:55:37.016 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:55:37.016 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:55:37.016 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.7", 2024-08-18T15:55:37.016 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.3", 2024-08-18T15:55:37.016 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:55:37.016 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:55:37.016 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:55:37.016 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:55:37.016 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:55:37.016 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:55:37.017 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:55:37.017 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:55:37.017 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.2", 2024-08-18T15:55:37.017 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:55:37.017 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:55:37.017 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:55:37.017 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:55:37.017 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:55:37.017 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:55:37.017 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:55:37.017 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000045", 2024-08-18T15:55:37.017 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000002", 2024-08-18T15:55:37.017 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:55:37.018 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3.2UbiLidwSTy.0", 2024-08-18T15:55:37.018 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:55:37.018 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.12", 2024-08-18T15:55:37.018 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:55:37.018 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:55:37.018 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:55:37.018 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:55:37.018 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:55:37.018 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:55:37.018 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000116", 2024-08-18T15:55:37.018 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.25", 2024-08-18T15:55:37.018 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:55:37.018 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:55:37.019 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:55:37.019 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:55:37.019 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000034", 2024-08-18T15:55:37.019 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:55:37.019 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:55:37.019 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000101", 2024-08-18T15:55:37.019 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:55:37.019 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:55:37.019 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.13", 2024-08-18T15:55:37.019 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:55:37.019 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:55:37.019 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.1", 2024-08-18T15:55:37.019 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0.retry", 2024-08-18T15:55:37.020 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:55:37.020 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:55:37.020 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000079", 2024-08-18T15:55:37.020 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000049", 2024-08-18T15:55:37.020 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:55:37.020 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0", 2024-08-18T15:55:37.020 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:55:37.020 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:55:37.020 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000117", 2024-08-18T15:55:37.020 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:55:37.020 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000044", 2024-08-18T15:55:37.020 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:55:37.021 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:55:37.021 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:55:37.021 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:55:37.021 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:55:37.021 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:55:37.021 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.14", 2024-08-18T15:55:37.021 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:55:37.021 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000066", 2024-08-18T15:55:37.021 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:55:37.021 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:55:37.021 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:55:37.021 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:55:37.022 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:55:37.022 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000068", 2024-08-18T15:55:37.022 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:55:37.022 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000085", 2024-08-18T15:55:37.022 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:55:37.022 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:55:37.022 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:55:37.022 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:55:37.022 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:55:37.022 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:55:37.022 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:55:37.022 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:55:37.023 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:55:37.023 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:55:37.023 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:55:37.023 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:55:37.023 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000073", 2024-08-18T15:55:37.023 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:55:37.023 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:55:37.023 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:55:37.023 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000038", 2024-08-18T15:55:37.023 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:55:37.023 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:55:37.023 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:55:37.023 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:55:37.024 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.10", 2024-08-18T15:55:37.024 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:55:37.024 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:55:37.024 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:55:37.024 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000118", 2024-08-18T15:55:37.024 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:55:37.024 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3", 2024-08-18T15:55:37.024 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:55:37.024 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000036", 2024-08-18T15:55:37.024 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:55:37.024 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:55:37.024 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:55:37.024 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:55:37.024 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:55:37.025 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000103", 2024-08-18T15:55:37.025 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.3", 2024-08-18T15:55:37.025 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:55:37.025 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000119", 2024-08-18T15:55:37.025 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000098", 2024-08-18T15:55:37.025 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:55:37.025 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:55:37.025 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:55:37.025 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:55:37.025 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:55:37.026 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:55:37.026 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:55:37.026 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:55:37.026 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.0", 2024-08-18T15:55:37.026 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:55:37.026 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:55:37.026 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:55:37.026 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:55:37.026 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:55:37.026 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:55:37.026 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:55:37.027 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:55:37.027 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000027", 2024-08-18T15:55:37.027 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:55:37.027 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status", 2024-08-18T15:55:37.027 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:55:37.027 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000019", 2024-08-18T15:55:37.027 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:55:37.027 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:55:37.027 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:55:37.027 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:55:37.027 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1", 2024-08-18T15:55:37.027 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:55:37.027 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:55:37.028 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:55:37.028 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:55:37.028 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000039", 2024-08-18T15:55:37.028 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:55:37.028 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:55:37.028 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:55:37.028 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:55:37.028 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:55:37.028 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:55:37.028 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:55:37.028 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:55:37.028 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:55:37.028 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:55:37.028 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:55:37.028 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000100", 2024-08-18T15:55:37.029 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:55:37.029 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:55:37.029 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:55:37.029 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:55:37.029 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:55:37.029 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000093", 2024-08-18T15:55:37.029 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:55:37.029 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000004", 2024-08-18T15:55:37.029 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000063", 2024-08-18T15:55:37.029 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000122", 2024-08-18T15:55:37.029 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:55:37.029 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:55:37.029 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:55:37.029 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000057", 2024-08-18T15:55:37.030 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:55:37.030 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:55:37.030 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:55:37.030 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:55:37.030 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000054", 2024-08-18T15:55:37.030 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:55:37.030 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000114", 2024-08-18T15:55:37.030 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:55:37.030 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:55:37.030 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:55:37.030 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.2", 2024-08-18T15:55:37.030 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:55:37.030 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:55:37.031 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0", 2024-08-18T15:55:37.031 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:55:37.031 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:55:37.031 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:55:37.031 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:55:37.031 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:55:37.031 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:55:37.031 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000012", 2024-08-18T15:55:37.031 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:55:37.031 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:55:37.031 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:55:37.031 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000084", 2024-08-18T15:55:37.031 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000043", 2024-08-18T15:55:37.032 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3", 2024-08-18T15:55:37.032 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:55:37.032 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:55:37.032 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.11", 2024-08-18T15:55:37.032 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:55:37.032 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:55:37.032 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:55:37.032 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:55:37.032 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:55:37.032 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:55:37.032 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:55:37.032 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:55:37.032 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000111", 2024-08-18T15:55:37.033 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:55:37.033 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000048", 2024-08-18T15:55:37.033 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:55:37.033 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000020", 2024-08-18T15:55:37.033 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:55:37.033 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:55:37.033 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:55:37.033 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:55:37.033 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:55:37.033 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000099", 2024-08-18T15:55:37.033 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000056", 2024-08-18T15:55:37.033 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:55:37.033 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:55:37.033 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.19", 2024-08-18T15:55:37.034 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:55:37.034 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.28", 2024-08-18T15:55:37.034 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:55:37.034 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:55:37.034 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:55:37.034 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:55:37.034 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:55:37.034 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000072", 2024-08-18T15:55:37.034 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:55:37.034 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:55:37.034 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000062", 2024-08-18T15:55:37.035 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:55:37.035 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:55:37.035 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:55:37.035 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000115", 2024-08-18T15:55:37.035 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:55:37.035 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.1", 2024-08-18T15:55:37.035 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2", 2024-08-18T15:55:37.035 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000033", 2024-08-18T15:55:37.035 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:55:37.035 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:55:37.035 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:55:37.035 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:55:37.035 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:55:37.036 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:55:37.036 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:55:37.036 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:55:37.036 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:55:37.036 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:55:37.036 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:55:37.036 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:55:37.036 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:55:37.036 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:55:37.036 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:55:37.036 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:55:37.036 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:55:37.036 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:55:37.037 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:55:37.037 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000025", 2024-08-18T15:55:37.037 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:55:37.037 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000123", 2024-08-18T15:55:37.037 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.31", 2024-08-18T15:55:37.037 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:55:37.037 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:55:37.037 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:55:37.037 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:55:37.037 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.18", 2024-08-18T15:55:37.037 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:55:37.038 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:55:37.038 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:55:37.038 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:55:37.038 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:55:37.038 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000108", 2024-08-18T15:55:37.038 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:55:37.038 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:55:37.038 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000094", 2024-08-18T15:55:37.038 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:55:37.038 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:55:37.038 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:55:37.038 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:55:37.039 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:55:37.039 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:55:37.039 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000060", 2024-08-18T15:55:37.039 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:55:37.039 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:55:37.039 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:55:37.039 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000109", 2024-08-18T15:55:37.039 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.15", 2024-08-18T15:55:37.039 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:55:37.039 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:55:37.039 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:55:37.040 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:55:37.040 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:55:37.040 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:55:37.040 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:55:37.040 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:55:37.040 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:55:37.040 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000013", 2024-08-18T15:55:37.040 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:55:37.040 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:55:37.040 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:55:37.040 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:55:37.040 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:55:37.041 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:55:37.041 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:55:37.041 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:55:37.041 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:55:37.041 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:55:37.041 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:55:37.041 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:55:37.041 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:55:37.041 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:55:37.041 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000051", 2024-08-18T15:55:37.041 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:55:37.041 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:55:37.041 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:55:37.041 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:55:37.042 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000106", 2024-08-18T15:55:37.042 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:55:37.042 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:55:37.042 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:55:37.042 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:55:37.042 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:55:37.042 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000017", 2024-08-18T15:55:37.042 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:55:37.042 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:55:37.042 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:55:37.042 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000023", 2024-08-18T15:55:37.042 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:55:37.042 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:55:37.043 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:55:37.043 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:55:37.043 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:55:37.043 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:55:37.043 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:55:37.043 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000107", 2024-08-18T15:55:37.043 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:55:37.043 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:55:37.043 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000046", 2024-08-18T15:55:37.043 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:55:37.043 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:55:37.043 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:55:37.043 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.26", 2024-08-18T15:55:37.043 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:55:37.044 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:55:37.044 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9", 2024-08-18T15:55:37.044 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.22", 2024-08-18T15:55:37.044 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:55:37.044 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:55:37.044 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:55:37.044 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:55:37.044 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:55:37.044 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:55:37.044 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:55:37.044 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2", 2024-08-18T15:55:37.044 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:55:37.044 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:55:37.045 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000067", 2024-08-18T15:55:37.045 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:55:37.045 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:55:37.045 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:55:37.045 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:55:37.045 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:55:37.045 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:55:37.045 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:55:37.045 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:55:37.045 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:55:37.045 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:55:37.045 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:55:37.046 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:55:37.046 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:55:37.046 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:55:37.046 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:55:37.046 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:55:37.046 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:55:37.046 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:55:37.046 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:55:37.046 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.9", 2024-08-18T15:55:37.046 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000086", 2024-08-18T15:55:37.046 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.5", 2024-08-18T15:55:37.046 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:55:37.047 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:55:37.047 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:55:37.047 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:55:37.047 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:55:37.047 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:55:37.047 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2.MFXJ583gCr1.0", 2024-08-18T15:55:37.047 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:55:37.047 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1.GPnO16liyZ+.0", 2024-08-18T15:55:37.047 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:55:37.047 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:55:37.047 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:55:37.047 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:55:37.047 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:55:37.048 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:55:37.048 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:55:37.048 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:55:37.048 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:55:37.048 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.20", 2024-08-18T15:55:37.048 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:55:37.048 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:55:37.048 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:55:37.048 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:55:37.048 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:55:37.048 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:55:37.048 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1.retry", 2024-08-18T15:55:37.048 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:55:37.049 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:55:37.049 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:55:37.049 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0.retry", 2024-08-18T15:55:37.049 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000050", 2024-08-18T15:55:37.049 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:55:37.049 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000102", 2024-08-18T15:55:37.049 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:55:37.049 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:55:37.049 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:55:37.049 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:55:37.049 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:55:37.049 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:55:37.049 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:55:37.049 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:55:37.050 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:55:37.050 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:55:37.050 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:55:37.050 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000124", 2024-08-18T15:55:37.050 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:55:37.050 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:55:37.050 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:55:37.050 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000001", 2024-08-18T15:55:37.050 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:55:37.050 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:55:37.050 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:55:37.050 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:55:37.050 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000126", 2024-08-18T15:55:37.051 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:55:37.051 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0.fZhctfLXnSt.0", 2024-08-18T15:55:37.051 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:55:37.051 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000055", 2024-08-18T15:55:37.051 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:55:37.051 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000061", 2024-08-18T15:55:37.051 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1", 2024-08-18T15:55:37.051 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:55:37.051 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:55:37.051 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:55:37.051 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000097", 2024-08-18T15:55:37.051 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:55:37.051 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:55:37.051 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:55:37.052 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:55:37.052 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:55:37.052 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:55:37.052 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:55:37.052 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:55:37.052 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000042", 2024-08-18T15:55:37.052 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:55:37.052 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:55:37.052 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:55:37.052 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1.retry", 2024-08-18T15:55:37.052 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:55:37.052 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:55:37.052 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:55:37.052 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:55:37.052 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:55:37.053 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:55:37.053 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:55:37.053 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000075", 2024-08-18T15:55:37.053 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:55:37.053 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000120", 2024-08-18T15:55:37.053 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000030", 2024-08-18T15:55:37.053 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:55:37.053 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:55:37.053 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:55:37.053 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:55:37.053 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:55:37.053 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:55:37.053 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9" 2024-08-18T15:55:37.053 INFO:teuthology.orchestra.run.smithi193.stdout:] 2024-08-18T15:55:37.054 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:55:42.056 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:55:42.056 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:55:42.057 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:55:42.109 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:55:42.109 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:55:42.172 INFO:teuthology.orchestra.run.smithi193.stdout:[ 2024-08-18T15:55:42.172 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:55:42.172 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:55:42.172 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.29", 2024-08-18T15:55:42.172 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000078", 2024-08-18T15:55:42.172 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.history", 2024-08-18T15:55:42.173 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:55:42.173 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:55:42.173 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:55:42.173 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:55:42.173 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.23", 2024-08-18T15:55:42.173 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000070", 2024-08-18T15:55:42.173 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000104", 2024-08-18T15:55:42.173 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:55:42.173 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2.retry", 2024-08-18T15:55:42.173 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:55:42.173 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:55:42.173 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:55:42.173 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:55:42.173 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000026", 2024-08-18T15:55:42.174 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:55:42.174 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:55:42.174 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:55:42.174 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:55:42.174 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:55:42.174 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:55:42.174 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.8", 2024-08-18T15:55:42.174 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:55:42.174 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:55:42.174 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:55:42.174 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:55:42.174 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000028", 2024-08-18T15:55:42.174 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:55:42.175 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:55:42.175 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:55:42.175 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:55:42.175 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:55:42.175 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:55:42.175 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:55:42.175 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2.retry", 2024-08-18T15:55:42.175 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:55:42.175 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.30", 2024-08-18T15:55:42.175 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:55:42.175 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:55:42.175 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:55:42.175 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:55:42.175 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000040", 2024-08-18T15:55:42.175 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:55:42.176 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.6", 2024-08-18T15:55:42.176 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:55:42.176 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:55:42.176 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:55:42.176 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:55:42.176 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:55:42.176 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:55:42.176 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000015", 2024-08-18T15:55:42.176 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:55:42.176 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:55:42.176 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000069", 2024-08-18T15:55:42.176 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:55:42.176 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:55:42.176 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:55:42.177 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000095", 2024-08-18T15:55:42.177 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0", 2024-08-18T15:55:42.177 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000003", 2024-08-18T15:55:42.177 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000047", 2024-08-18T15:55:42.177 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:55:42.177 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:55:42.177 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:55:42.177 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:55:42.177 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:55:42.177 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:55:42.177 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:55:42.177 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:55:42.177 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:55:42.178 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000035", 2024-08-18T15:55:42.178 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:55:42.178 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:55:42.178 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000037", 2024-08-18T15:55:42.178 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:55:42.178 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:55:42.178 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:55:42.178 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:55:42.178 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:55:42.178 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:55:42.178 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:55:42.178 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:55:42.178 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:55:42.179 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:55:42.179 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:55:42.179 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:55:42.179 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:55:42.179 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:55:42.179 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000024", 2024-08-18T15:55:42.179 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:55:42.179 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:55:42.179 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:55:42.179 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:55:42.179 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000000", 2024-08-18T15:55:42.179 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:55:42.179 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000031", 2024-08-18T15:55:42.179 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:55:42.180 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:55:42.180 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:55:42.180 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:55:42.180 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:55:42.180 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:55:42.180 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3", 2024-08-18T15:55:42.180 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:55:42.180 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:55:42.180 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:55:42.180 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:55:42.180 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:55:42.180 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:55:42.180 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:55:42.181 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:55:42.181 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:55:42.181 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:55:42.181 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:55:42.181 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:55:42.181 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:55:42.181 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:55:42.181 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:55:42.181 INFO:teuthology.orchestra.run.smithi193.stdout: "data_loggenerations_metadata", 2024-08-18T15:55:42.181 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2", 2024-08-18T15:55:42.181 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:55:42.181 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:55:42.181 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:55:42.181 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:55:42.182 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T15:55:42.182 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:55:42.182 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:55:42.182 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:55:42.182 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:55:42.182 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:55:42.182 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000076", 2024-08-18T15:55:42.182 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:55:42.182 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:55:42.182 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000113", 2024-08-18T15:55:42.182 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000005", 2024-08-18T15:55:42.182 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:55:42.182 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000011", 2024-08-18T15:55:42.183 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:55:42.183 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000058", 2024-08-18T15:55:42.183 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:55:42.183 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:55:42.183 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:55:42.183 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000052", 2024-08-18T15:55:42.183 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:55:42.183 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:55:42.183 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:55:42.183 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:55:42.183 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.21", 2024-08-18T15:55:42.183 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:55:42.183 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:55:42.184 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:55:42.184 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:55:42.184 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:55:42.184 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:55:42.184 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:55:42.184 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:55:42.184 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:55:42.184 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:55:42.184 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:55:42.184 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:55:42.184 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:55:42.184 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:55:42.185 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:55:42.185 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:55:42.185 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:55:42.185 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:55:42.185 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:55:42.185 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000088", 2024-08-18T15:55:42.185 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000080", 2024-08-18T15:55:42.185 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:55:42.185 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:55:42.185 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000090", 2024-08-18T15:55:42.185 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:55:42.185 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:55:42.186 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000110", 2024-08-18T15:55:42.186 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:55:42.186 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:55:42.186 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:55:42.186 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.0", 2024-08-18T15:55:42.186 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:55:42.186 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000096", 2024-08-18T15:55:42.186 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:55:42.186 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:55:42.186 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:55:42.186 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:55:42.186 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:55:42.186 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000087", 2024-08-18T15:55:42.187 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:55:42.187 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:55:42.187 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:55:42.187 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:55:42.187 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000008", 2024-08-18T15:55:42.187 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:55:42.187 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:55:42.187 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:55:42.187 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:55:42.187 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:55:42.187 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000006", 2024-08-18T15:55:42.187 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000029", 2024-08-18T15:55:42.187 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:55:42.188 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000089", 2024-08-18T15:55:42.188 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:55:42.188 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:55:42.188 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000022", 2024-08-18T15:55:42.188 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:55:42.188 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:55:42.188 INFO:teuthology.orchestra.run.smithi193.stdout: "bilog.trim", 2024-08-18T15:55:42.188 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000016", 2024-08-18T15:55:42.188 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:55:42.188 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:55:42.188 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:55:42.188 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:55:42.189 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:55:42.189 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000041", 2024-08-18T15:55:42.189 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:55:42.189 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:55:42.189 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.log.01050e30-179b-43be-bdbb-3bb23cf37ae8.3", 2024-08-18T15:55:42.189 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:55:42.189 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000018", 2024-08-18T15:55:42.189 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000092", 2024-08-18T15:55:42.189 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:55:42.189 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:55:42.189 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:55:42.189 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:55:42.189 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.27", 2024-08-18T15:55:42.189 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.4", 2024-08-18T15:55:42.190 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:55:42.190 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:55:42.190 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:55:42.190 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:55:42.190 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:55:42.190 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:55:42.190 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:55:42.190 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000014", 2024-08-18T15:55:42.190 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:55:42.190 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3.retry", 2024-08-18T15:55:42.190 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:55:42.190 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:55:42.190 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:55:42.191 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:55:42.191 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:55:42.191 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000112", 2024-08-18T15:55:42.191 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:55:42.191 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000007", 2024-08-18T15:55:42.191 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000021", 2024-08-18T15:55:42.191 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:55:42.191 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:55:42.191 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:55:42.191 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:55:42.191 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:55:42.191 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.16", 2024-08-18T15:55:42.191 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:55:42.191 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000064", 2024-08-18T15:55:42.192 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:55:42.192 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:55:42.192 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:55:42.192 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:55:42.192 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:55:42.192 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:55:42.192 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:55:42.192 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000071", 2024-08-18T15:55:42.192 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:55:42.192 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:55:42.192 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:55:42.192 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000074", 2024-08-18T15:55:42.192 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:55:42.193 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:55:42.193 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:55:42.193 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:55:42.193 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:55:42.193 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:55:42.193 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:55:42.193 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:55:42.193 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:55:42.193 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:55:42.193 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:55:42.193 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:55:42.193 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:55:42.193 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:55:42.194 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:55:42.194 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000081", 2024-08-18T15:55:42.194 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:55:42.194 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000009", 2024-08-18T15:55:42.194 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000121", 2024-08-18T15:55:42.194 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1", 2024-08-18T15:55:42.194 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:55:42.194 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:55:42.194 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:55:42.194 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:55:42.194 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:55:42.194 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:55:42.195 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.17", 2024-08-18T15:55:42.195 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000125", 2024-08-18T15:55:42.195 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:55:42.195 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000082", 2024-08-18T15:55:42.195 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:55:42.195 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:55:42.195 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:55:42.195 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:55:42.195 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:55:42.195 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:55:42.195 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:55:42.195 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:55:42.195 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:55:42.195 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:55:42.196 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:55:42.196 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000105", 2024-08-18T15:55:42.196 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000059", 2024-08-18T15:55:42.196 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:55:42.196 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:55:42.196 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:55:42.196 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:55:42.196 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:55:42.196 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:55:42.196 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:55:42.196 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:55:42.196 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:55:42.196 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:55:42.197 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:55:42.197 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000077", 2024-08-18T15:55:42.197 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000032", 2024-08-18T15:55:42.197 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:55:42.197 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:55:42.197 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:55:42.197 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:55:42.197 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:55:42.197 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:55:42.197 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.24", 2024-08-18T15:55:42.197 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:55:42.197 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:55:42.197 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:55:42.198 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:55:42.198 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3.retry", 2024-08-18T15:55:42.198 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:55:42.198 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:55:42.198 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:55:42.198 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:55:42.198 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:55:42.198 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:55:42.198 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000053", 2024-08-18T15:55:42.198 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000091", 2024-08-18T15:55:42.198 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:55:42.198 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:55:42.198 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:55:42.199 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000065", 2024-08-18T15:55:42.199 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:55:42.199 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:55:42.199 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:55:42.199 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:55:42.199 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000083", 2024-08-18T15:55:42.199 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000010", 2024-08-18T15:55:42.199 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:55:42.199 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:55:42.199 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:55:42.199 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:55:42.199 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:55:42.199 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:55:42.200 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:55:42.200 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:55:42.200 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:55:42.200 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:55:42.200 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:55:42.200 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.7", 2024-08-18T15:55:42.200 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.3", 2024-08-18T15:55:42.200 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:55:42.200 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:55:42.200 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:55:42.200 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:55:42.200 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:55:42.200 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:55:42.200 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:55:42.201 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:55:42.201 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.2", 2024-08-18T15:55:42.201 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:55:42.201 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:55:42.201 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:55:42.201 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:55:42.201 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:55:42.201 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:55:42.201 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:55:42.201 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000045", 2024-08-18T15:55:42.201 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000002", 2024-08-18T15:55:42.201 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:55:42.201 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3.2UbiLidwSTy.0", 2024-08-18T15:55:42.202 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:55:42.202 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.12", 2024-08-18T15:55:42.202 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:55:42.202 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:55:42.202 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:55:42.202 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:55:42.202 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:55:42.202 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:55:42.202 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000116", 2024-08-18T15:55:42.202 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.25", 2024-08-18T15:55:42.202 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:55:42.202 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:55:42.202 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:55:42.203 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:55:42.203 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000034", 2024-08-18T15:55:42.203 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:55:42.203 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:55:42.203 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000101", 2024-08-18T15:55:42.203 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:55:42.203 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:55:42.203 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.13", 2024-08-18T15:55:42.203 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:55:42.203 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:55:42.203 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.1", 2024-08-18T15:55:42.203 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0.retry", 2024-08-18T15:55:42.203 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:55:42.204 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:55:42.204 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000079", 2024-08-18T15:55:42.204 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000049", 2024-08-18T15:55:42.204 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:55:42.204 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0", 2024-08-18T15:55:42.204 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:55:42.204 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:55:42.204 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000117", 2024-08-18T15:55:42.204 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:55:42.204 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000044", 2024-08-18T15:55:42.204 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:55:42.204 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:55:42.204 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:55:42.204 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:55:42.204 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:55:42.205 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:55:42.205 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.14", 2024-08-18T15:55:42.205 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:55:42.205 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000066", 2024-08-18T15:55:42.205 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:55:42.205 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:55:42.205 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:55:42.205 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:55:42.205 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:55:42.205 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000068", 2024-08-18T15:55:42.205 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:55:42.205 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000085", 2024-08-18T15:55:42.205 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:55:42.205 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:55:42.206 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:55:42.206 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:55:42.206 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:55:42.206 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:55:42.206 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:55:42.206 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:55:42.206 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:55:42.206 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:55:42.206 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:55:42.206 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:55:42.206 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000073", 2024-08-18T15:55:42.206 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:55:42.206 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:55:42.207 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:55:42.207 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000038", 2024-08-18T15:55:42.207 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:55:42.207 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:55:42.207 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:55:42.207 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:55:42.207 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.10", 2024-08-18T15:55:42.207 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:55:42.207 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:55:42.207 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:55:42.207 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000118", 2024-08-18T15:55:42.207 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:55:42.207 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3", 2024-08-18T15:55:42.207 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:55:42.208 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000036", 2024-08-18T15:55:42.208 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:55:42.208 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:55:42.208 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:55:42.208 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:55:42.208 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:55:42.208 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000103", 2024-08-18T15:55:42.208 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.3", 2024-08-18T15:55:42.208 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:55:42.208 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000119", 2024-08-18T15:55:42.208 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000098", 2024-08-18T15:55:42.208 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:55:42.209 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:55:42.209 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:55:42.209 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:55:42.209 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:55:42.209 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:55:42.209 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:55:42.209 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:55:42.209 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.0", 2024-08-18T15:55:42.209 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:55:42.210 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:55:42.210 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:55:42.210 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:55:42.210 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:55:42.210 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:55:42.210 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:55:42.210 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:55:42.210 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000027", 2024-08-18T15:55:42.210 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:55:42.210 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status", 2024-08-18T15:55:42.210 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:55:42.210 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000019", 2024-08-18T15:55:42.211 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:55:42.211 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:55:42.211 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:55:42.211 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:55:42.211 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1", 2024-08-18T15:55:42.211 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:55:42.211 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:55:42.211 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:55:42.211 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:55:42.211 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000039", 2024-08-18T15:55:42.211 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:55:42.211 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:55:42.211 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:55:42.212 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:55:42.212 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:55:42.212 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:55:42.212 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:55:42.212 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:55:42.212 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:55:42.212 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:55:42.212 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:55:42.212 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000100", 2024-08-18T15:55:42.212 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:55:42.212 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:55:42.212 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:55:42.212 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:55:42.212 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:55:42.213 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000093", 2024-08-18T15:55:42.213 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:55:42.213 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000004", 2024-08-18T15:55:42.213 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000063", 2024-08-18T15:55:42.213 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000122", 2024-08-18T15:55:42.213 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:55:42.213 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:55:42.213 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:55:42.213 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000057", 2024-08-18T15:55:42.213 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:55:42.213 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:55:42.213 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:55:42.213 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:55:42.214 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000054", 2024-08-18T15:55:42.214 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:55:42.214 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000114", 2024-08-18T15:55:42.214 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:55:42.214 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:55:42.214 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:55:42.214 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.2", 2024-08-18T15:55:42.214 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:55:42.214 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:55:42.214 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0", 2024-08-18T15:55:42.214 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:55:42.214 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:55:42.215 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:55:42.215 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:55:42.215 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:55:42.215 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:55:42.215 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000012", 2024-08-18T15:55:42.215 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:55:42.215 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:55:42.215 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:55:42.215 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000084", 2024-08-18T15:55:42.215 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000043", 2024-08-18T15:55:42.215 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3", 2024-08-18T15:55:42.215 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:55:42.216 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:55:42.216 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.11", 2024-08-18T15:55:42.216 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:55:42.216 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:55:42.216 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:55:42.216 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:55:42.216 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:55:42.216 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:55:42.216 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:55:42.216 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:55:42.216 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000111", 2024-08-18T15:55:42.216 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:55:42.216 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000048", 2024-08-18T15:55:42.217 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:55:42.217 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000020", 2024-08-18T15:55:42.217 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:55:42.217 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:55:42.217 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:55:42.217 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:55:42.217 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:55:42.217 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000099", 2024-08-18T15:55:42.217 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000056", 2024-08-18T15:55:42.217 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:55:42.217 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:55:42.217 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.19", 2024-08-18T15:55:42.218 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:55:42.218 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.28", 2024-08-18T15:55:42.218 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:55:42.218 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:55:42.218 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:55:42.218 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:55:42.218 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:55:42.218 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000072", 2024-08-18T15:55:42.218 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:55:42.218 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:55:42.218 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000062", 2024-08-18T15:55:42.218 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:55:42.219 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:55:42.219 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:55:42.219 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000115", 2024-08-18T15:55:42.219 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:55:42.219 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.1", 2024-08-18T15:55:42.219 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2", 2024-08-18T15:55:42.219 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000033", 2024-08-18T15:55:42.219 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:55:42.219 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:55:42.219 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:55:42.219 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:55:42.219 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:55:42.219 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:55:42.220 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:55:42.220 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:55:42.220 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:55:42.220 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:55:42.220 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:55:42.220 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:55:42.220 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:55:42.220 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:55:42.220 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:55:42.220 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:55:42.220 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:55:42.220 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:55:42.220 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:55:42.220 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000025", 2024-08-18T15:55:42.221 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:55:42.221 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000123", 2024-08-18T15:55:42.221 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.31", 2024-08-18T15:55:42.221 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:55:42.221 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:55:42.221 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:55:42.221 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:55:42.221 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.18", 2024-08-18T15:55:42.221 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:55:42.221 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:55:42.221 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:55:42.221 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:55:42.221 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:55:42.221 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000108", 2024-08-18T15:55:42.222 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:55:42.222 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:55:42.222 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000094", 2024-08-18T15:55:42.222 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:55:42.222 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:55:42.222 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:55:42.222 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:55:42.222 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:55:42.222 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:55:42.222 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000060", 2024-08-18T15:55:42.222 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:55:42.222 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:55:42.223 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:55:42.223 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000109", 2024-08-18T15:55:42.223 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.15", 2024-08-18T15:55:42.223 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:55:42.223 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:55:42.223 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:55:42.223 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:55:42.223 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:55:42.223 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:55:42.223 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:55:42.223 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:55:42.223 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:55:42.224 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000013", 2024-08-18T15:55:42.224 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:55:42.224 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:55:42.224 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:55:42.224 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:55:42.224 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:55:42.224 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:55:42.224 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:55:42.224 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:55:42.224 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:55:42.224 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:55:42.224 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:55:42.224 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:55:42.224 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:55:42.225 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:55:42.225 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000051", 2024-08-18T15:55:42.225 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:55:42.225 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:55:42.225 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:55:42.225 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:55:42.225 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000106", 2024-08-18T15:55:42.225 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:55:42.225 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:55:42.225 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:55:42.225 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:55:42.225 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:55:42.225 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000017", 2024-08-18T15:55:42.225 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:55:42.226 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:55:42.226 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:55:42.226 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000023", 2024-08-18T15:55:42.226 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:55:42.226 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:55:42.226 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:55:42.226 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:55:42.226 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:55:42.226 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:55:42.226 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:55:42.226 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000107", 2024-08-18T15:55:42.226 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:55:42.226 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:55:42.227 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000046", 2024-08-18T15:55:42.227 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:55:42.227 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:55:42.227 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:55:42.227 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.26", 2024-08-18T15:55:42.227 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:55:42.227 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:55:42.227 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9", 2024-08-18T15:55:42.227 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.22", 2024-08-18T15:55:42.227 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:55:42.227 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:55:42.227 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:55:42.227 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:55:42.227 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:55:42.227 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:55:42.228 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:55:42.228 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2", 2024-08-18T15:55:42.228 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:55:42.228 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:55:42.228 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000067", 2024-08-18T15:55:42.228 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:55:42.228 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:55:42.228 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:55:42.228 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:55:42.228 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:55:42.228 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:55:42.228 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:55:42.228 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:55:42.228 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:55:42.229 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:55:42.229 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:55:42.229 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:55:42.229 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:55:42.229 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:55:42.229 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:55:42.229 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:55:42.229 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:55:42.229 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:55:42.229 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:55:42.229 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.9", 2024-08-18T15:55:42.229 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000086", 2024-08-18T15:55:42.229 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.5", 2024-08-18T15:55:42.229 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:55:42.230 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:55:42.230 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:55:42.230 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:55:42.230 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:55:42.230 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:55:42.230 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2.MFXJ583gCr1.0", 2024-08-18T15:55:42.230 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:55:42.230 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1.GPnO16liyZ+.0", 2024-08-18T15:55:42.230 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:55:42.230 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:55:42.230 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:55:42.230 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:55:42.231 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:55:42.231 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:55:42.231 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:55:42.231 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:55:42.231 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:55:42.231 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.20", 2024-08-18T15:55:42.231 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:55:42.231 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:55:42.231 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:55:42.231 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:55:42.231 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:55:42.231 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:55:42.231 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1.retry", 2024-08-18T15:55:42.231 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:55:42.232 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:55:42.232 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:55:42.232 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0.retry", 2024-08-18T15:55:42.232 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000050", 2024-08-18T15:55:42.232 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:55:42.232 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000102", 2024-08-18T15:55:42.232 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:55:42.232 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:55:42.232 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:55:42.232 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:55:42.232 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:55:42.232 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:55:42.232 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:55:42.232 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:55:42.233 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:55:42.233 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:55:42.233 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:55:42.233 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000124", 2024-08-18T15:55:42.233 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:55:42.233 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:55:42.233 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:55:42.233 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000001", 2024-08-18T15:55:42.233 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:55:42.233 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:55:42.233 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:55:42.233 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:55:42.233 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000126", 2024-08-18T15:55:42.233 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:55:42.233 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0.fZhctfLXnSt.0", 2024-08-18T15:55:42.234 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:55:42.234 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000055", 2024-08-18T15:55:42.234 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:55:42.234 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000061", 2024-08-18T15:55:42.234 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1", 2024-08-18T15:55:42.234 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:55:42.234 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:55:42.234 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:55:42.234 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000097", 2024-08-18T15:55:42.234 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:55:42.234 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:55:42.234 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:55:42.235 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:55:42.235 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:55:42.235 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:55:42.235 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:55:42.235 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:55:42.235 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000042", 2024-08-18T15:55:42.235 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:55:42.235 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:55:42.235 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:55:42.235 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1.retry", 2024-08-18T15:55:42.235 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:55:42.235 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:55:42.235 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:55:42.236 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:55:42.236 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:55:42.236 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:55:42.236 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:55:42.236 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000075", 2024-08-18T15:55:42.236 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:55:42.236 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000120", 2024-08-18T15:55:42.236 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000030", 2024-08-18T15:55:42.236 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:55:42.237 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:55:42.237 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:55:42.237 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:55:42.237 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:55:42.237 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:55:42.237 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9" 2024-08-18T15:55:42.237 INFO:teuthology.orchestra.run.smithi193.stdout:] 2024-08-18T15:55:42.237 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:55:47.239 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:55:47.239 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:55:47.240 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:55:47.293 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:55:47.293 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:55:47.357 INFO:teuthology.orchestra.run.smithi193.stdout:[ 2024-08-18T15:55:47.357 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:55:47.357 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:55:47.357 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.29", 2024-08-18T15:55:47.358 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000078", 2024-08-18T15:55:47.358 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.history", 2024-08-18T15:55:47.358 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:55:47.358 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:55:47.358 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:55:47.358 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:55:47.358 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.23", 2024-08-18T15:55:47.358 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000070", 2024-08-18T15:55:47.358 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000104", 2024-08-18T15:55:47.358 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:55:47.358 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2.retry", 2024-08-18T15:55:47.358 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:55:47.359 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:55:47.359 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:55:47.359 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:55:47.359 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000026", 2024-08-18T15:55:47.359 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:55:47.359 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:55:47.359 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:55:47.359 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:55:47.360 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:55:47.360 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:55:47.360 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.8", 2024-08-18T15:55:47.360 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:55:47.360 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:55:47.360 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:55:47.360 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:55:47.360 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000028", 2024-08-18T15:55:47.360 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:55:47.360 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:55:47.360 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:55:47.360 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:55:47.360 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:55:47.361 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:55:47.361 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:55:47.361 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2.retry", 2024-08-18T15:55:47.361 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:55:47.361 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.30", 2024-08-18T15:55:47.361 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:55:47.361 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:55:47.361 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:55:47.361 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:55:47.361 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000040", 2024-08-18T15:55:47.361 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:55:47.361 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.6", 2024-08-18T15:55:47.361 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:55:47.361 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:55:47.362 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:55:47.362 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:55:47.362 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:55:47.362 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:55:47.362 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000015", 2024-08-18T15:55:47.362 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:55:47.362 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:55:47.362 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000069", 2024-08-18T15:55:47.362 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:55:47.362 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:55:47.362 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:55:47.362 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000095", 2024-08-18T15:55:47.362 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0", 2024-08-18T15:55:47.363 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000003", 2024-08-18T15:55:47.363 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000047", 2024-08-18T15:55:47.363 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:55:47.363 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:55:47.363 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:55:47.363 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:55:47.363 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:55:47.363 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:55:47.363 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:55:47.363 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:55:47.363 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:55:47.363 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000035", 2024-08-18T15:55:47.363 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:55:47.364 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:55:47.364 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000037", 2024-08-18T15:55:47.364 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:55:47.364 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:55:47.364 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:55:47.364 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:55:47.364 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:55:47.364 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:55:47.364 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:55:47.365 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:55:47.365 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:55:47.365 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:55:47.365 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:55:47.365 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:55:47.365 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:55:47.365 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:55:47.365 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000024", 2024-08-18T15:55:47.365 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:55:47.365 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:55:47.365 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:55:47.366 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:55:47.366 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000000", 2024-08-18T15:55:47.366 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:55:47.366 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000031", 2024-08-18T15:55:47.366 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:55:47.366 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:55:47.366 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:55:47.366 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:55:47.366 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:55:47.366 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:55:47.366 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3", 2024-08-18T15:55:47.366 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:55:47.366 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:55:47.367 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:55:47.367 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:55:47.367 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:55:47.367 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:55:47.367 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:55:47.367 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:55:47.367 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:55:47.367 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:55:47.367 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:55:47.367 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:55:47.367 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:55:47.367 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:55:47.367 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:55:47.368 INFO:teuthology.orchestra.run.smithi193.stdout: "data_loggenerations_metadata", 2024-08-18T15:55:47.368 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2", 2024-08-18T15:55:47.368 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:55:47.368 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:55:47.368 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:55:47.368 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:55:47.368 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T15:55:47.368 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:55:47.368 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:55:47.368 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:55:47.368 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:55:47.368 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:55:47.369 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000076", 2024-08-18T15:55:47.369 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:55:47.369 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:55:47.369 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000113", 2024-08-18T15:55:47.369 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000005", 2024-08-18T15:55:47.369 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:55:47.369 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000011", 2024-08-18T15:55:47.369 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:55:47.369 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000058", 2024-08-18T15:55:47.369 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:55:47.370 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:55:47.370 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:55:47.370 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000052", 2024-08-18T15:55:47.370 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:55:47.370 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:55:47.370 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:55:47.370 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:55:47.370 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.21", 2024-08-18T15:55:47.370 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:55:47.370 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:55:47.371 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:55:47.371 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:55:47.371 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:55:47.371 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:55:47.371 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:55:47.371 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:55:47.371 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:55:47.371 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:55:47.371 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:55:47.371 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:55:47.371 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:55:47.371 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:55:47.371 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:55:47.372 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:55:47.372 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:55:47.372 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:55:47.372 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:55:47.372 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000088", 2024-08-18T15:55:47.372 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000080", 2024-08-18T15:55:47.372 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:55:47.372 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:55:47.372 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000090", 2024-08-18T15:55:47.372 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:55:47.372 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:55:47.372 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000110", 2024-08-18T15:55:47.373 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:55:47.373 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:55:47.373 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:55:47.373 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.0", 2024-08-18T15:55:47.373 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:55:47.373 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000096", 2024-08-18T15:55:47.373 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:55:47.373 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:55:47.373 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:55:47.373 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:55:47.373 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:55:47.373 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000087", 2024-08-18T15:55:47.373 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:55:47.373 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:55:47.374 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:55:47.374 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:55:47.374 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000008", 2024-08-18T15:55:47.374 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:55:47.374 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:55:47.374 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:55:47.374 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:55:47.374 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:55:47.374 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000006", 2024-08-18T15:55:47.374 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000029", 2024-08-18T15:55:47.375 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:55:47.375 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000089", 2024-08-18T15:55:47.375 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:55:47.375 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:55:47.375 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000022", 2024-08-18T15:55:47.375 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:55:47.375 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:55:47.375 INFO:teuthology.orchestra.run.smithi193.stdout: "bilog.trim", 2024-08-18T15:55:47.375 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000016", 2024-08-18T15:55:47.375 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:55:47.375 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:55:47.375 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:55:47.375 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:55:47.375 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:55:47.376 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000041", 2024-08-18T15:55:47.376 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:55:47.376 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:55:47.376 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.log.01050e30-179b-43be-bdbb-3bb23cf37ae8.3", 2024-08-18T15:55:47.376 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:55:47.376 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000018", 2024-08-18T15:55:47.376 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000092", 2024-08-18T15:55:47.376 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:55:47.376 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:55:47.376 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:55:47.376 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:55:47.376 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.27", 2024-08-18T15:55:47.376 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.4", 2024-08-18T15:55:47.377 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:55:47.377 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:55:47.377 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:55:47.377 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:55:47.377 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:55:47.377 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:55:47.377 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:55:47.377 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000014", 2024-08-18T15:55:47.377 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:55:47.377 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3.retry", 2024-08-18T15:55:47.377 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:55:47.377 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:55:47.378 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:55:47.378 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:55:47.378 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:55:47.378 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000112", 2024-08-18T15:55:47.378 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:55:47.378 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000007", 2024-08-18T15:55:47.378 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000021", 2024-08-18T15:55:47.378 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:55:47.378 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:55:47.378 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:55:47.378 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:55:47.378 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:55:47.378 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.16", 2024-08-18T15:55:47.379 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:55:47.379 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000064", 2024-08-18T15:55:47.379 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:55:47.379 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:55:47.379 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:55:47.379 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:55:47.379 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:55:47.379 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:55:47.379 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:55:47.379 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000071", 2024-08-18T15:55:47.379 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:55:47.379 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:55:47.379 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:55:47.379 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000074", 2024-08-18T15:55:47.380 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:55:47.380 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:55:47.380 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:55:47.380 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:55:47.380 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:55:47.380 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:55:47.380 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:55:47.380 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:55:47.381 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:55:47.381 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:55:47.381 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:55:47.381 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:55:47.381 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:55:47.381 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:55:47.381 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:55:47.381 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000081", 2024-08-18T15:55:47.381 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:55:47.381 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000009", 2024-08-18T15:55:47.381 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000121", 2024-08-18T15:55:47.381 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1", 2024-08-18T15:55:47.381 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:55:47.381 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:55:47.381 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:55:47.382 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:55:47.382 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:55:47.382 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:55:47.382 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.17", 2024-08-18T15:55:47.382 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000125", 2024-08-18T15:55:47.382 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:55:47.382 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000082", 2024-08-18T15:55:47.382 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:55:47.382 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:55:47.382 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:55:47.382 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:55:47.382 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:55:47.382 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:55:47.383 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:55:47.383 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:55:47.383 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:55:47.383 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:55:47.383 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:55:47.383 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000105", 2024-08-18T15:55:47.383 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000059", 2024-08-18T15:55:47.383 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:55:47.383 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:55:47.383 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:55:47.383 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:55:47.383 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:55:47.383 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:55:47.384 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:55:47.384 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:55:47.384 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:55:47.384 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:55:47.384 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:55:47.384 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000077", 2024-08-18T15:55:47.384 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000032", 2024-08-18T15:55:47.384 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:55:47.384 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:55:47.384 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:55:47.384 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:55:47.384 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:55:47.384 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:55:47.385 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.24", 2024-08-18T15:55:47.385 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:55:47.385 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:55:47.385 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:55:47.385 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:55:47.385 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3.retry", 2024-08-18T15:55:47.385 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:55:47.385 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:55:47.385 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:55:47.385 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:55:47.385 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:55:47.386 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:55:47.386 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000053", 2024-08-18T15:55:47.386 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000091", 2024-08-18T15:55:47.386 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:55:47.386 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:55:47.386 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:55:47.386 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000065", 2024-08-18T15:55:47.386 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:55:47.386 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:55:47.387 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:55:47.387 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:55:47.387 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000083", 2024-08-18T15:55:47.387 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000010", 2024-08-18T15:55:47.387 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:55:47.387 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:55:47.387 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:55:47.387 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:55:47.387 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:55:47.387 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:55:47.388 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:55:47.388 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:55:47.388 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:55:47.388 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:55:47.388 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:55:47.388 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.7", 2024-08-18T15:55:47.388 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.3", 2024-08-18T15:55:47.388 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:55:47.389 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:55:47.389 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:55:47.389 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:55:47.389 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:55:47.389 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:55:47.389 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:55:47.389 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:55:47.389 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.2", 2024-08-18T15:55:47.389 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:55:47.389 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:55:47.389 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:55:47.389 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:55:47.389 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:55:47.390 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:55:47.390 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:55:47.390 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000045", 2024-08-18T15:55:47.390 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000002", 2024-08-18T15:55:47.390 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:55:47.390 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3.2UbiLidwSTy.0", 2024-08-18T15:55:47.390 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:55:47.390 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.12", 2024-08-18T15:55:47.390 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:55:47.390 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:55:47.390 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:55:47.391 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:55:47.391 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:55:47.391 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:55:47.391 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000116", 2024-08-18T15:55:47.391 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.25", 2024-08-18T15:55:47.391 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:55:47.391 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:55:47.391 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:55:47.391 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:55:47.392 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000034", 2024-08-18T15:55:47.392 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:55:47.392 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:55:47.392 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000101", 2024-08-18T15:55:47.392 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:55:47.392 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:55:47.392 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.13", 2024-08-18T15:55:47.392 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:55:47.392 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:55:47.392 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.1", 2024-08-18T15:55:47.392 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0.retry", 2024-08-18T15:55:47.392 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:55:47.392 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:55:47.392 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000079", 2024-08-18T15:55:47.392 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000049", 2024-08-18T15:55:47.393 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:55:47.393 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0", 2024-08-18T15:55:47.393 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:55:47.393 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:55:47.393 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000117", 2024-08-18T15:55:47.393 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:55:47.393 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000044", 2024-08-18T15:55:47.393 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:55:47.393 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:55:47.393 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:55:47.393 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:55:47.393 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:55:47.393 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:55:47.393 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.14", 2024-08-18T15:55:47.393 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:55:47.394 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000066", 2024-08-18T15:55:47.394 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:55:47.394 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:55:47.394 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:55:47.394 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:55:47.394 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:55:47.394 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000068", 2024-08-18T15:55:47.394 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:55:47.394 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000085", 2024-08-18T15:55:47.394 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:55:47.394 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:55:47.394 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:55:47.394 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:55:47.394 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:55:47.395 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:55:47.395 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:55:47.395 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:55:47.395 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:55:47.395 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:55:47.395 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:55:47.395 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:55:47.395 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000073", 2024-08-18T15:55:47.395 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:55:47.395 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:55:47.395 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:55:47.395 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000038", 2024-08-18T15:55:47.395 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:55:47.395 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:55:47.395 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:55:47.396 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:55:47.396 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.10", 2024-08-18T15:55:47.396 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:55:47.396 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:55:47.396 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:55:47.396 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000118", 2024-08-18T15:55:47.396 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:55:47.396 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3", 2024-08-18T15:55:47.396 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:55:47.396 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000036", 2024-08-18T15:55:47.396 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:55:47.396 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:55:47.397 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:55:47.397 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:55:47.397 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:55:47.397 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000103", 2024-08-18T15:55:47.397 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.3", 2024-08-18T15:55:47.397 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:55:47.397 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000119", 2024-08-18T15:55:47.397 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000098", 2024-08-18T15:55:47.397 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:55:47.397 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:55:47.397 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:55:47.397 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:55:47.397 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:55:47.398 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:55:47.398 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:55:47.398 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:55:47.398 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.0", 2024-08-18T15:55:47.398 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:55:47.398 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:55:47.398 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:55:47.398 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:55:47.398 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:55:47.398 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:55:47.398 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:55:47.398 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:55:47.398 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000027", 2024-08-18T15:55:47.399 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:55:47.399 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status", 2024-08-18T15:55:47.399 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:55:47.399 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000019", 2024-08-18T15:55:47.399 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:55:47.399 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:55:47.399 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:55:47.399 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:55:47.399 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1", 2024-08-18T15:55:47.399 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:55:47.399 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:55:47.399 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:55:47.399 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:55:47.399 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000039", 2024-08-18T15:55:47.400 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:55:47.400 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:55:47.400 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:55:47.400 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:55:47.400 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:55:47.400 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:55:47.400 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:55:47.400 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:55:47.400 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:55:47.400 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:55:47.400 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:55:47.400 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000100", 2024-08-18T15:55:47.400 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:55:47.400 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:55:47.401 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:55:47.401 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:55:47.401 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:55:47.401 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000093", 2024-08-18T15:55:47.401 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:55:47.401 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000004", 2024-08-18T15:55:47.401 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000063", 2024-08-18T15:55:47.401 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000122", 2024-08-18T15:55:47.401 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:55:47.401 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:55:47.402 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:55:47.402 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000057", 2024-08-18T15:55:47.402 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:55:47.402 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:55:47.402 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:55:47.402 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:55:47.402 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000054", 2024-08-18T15:55:47.402 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:55:47.402 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000114", 2024-08-18T15:55:47.402 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:55:47.402 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:55:47.402 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:55:47.402 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.2", 2024-08-18T15:55:47.402 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:55:47.403 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:55:47.403 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0", 2024-08-18T15:55:47.403 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:55:47.403 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:55:47.403 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:55:47.403 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:55:47.403 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:55:47.403 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:55:47.403 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000012", 2024-08-18T15:55:47.403 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:55:47.403 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:55:47.403 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:55:47.403 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000084", 2024-08-18T15:55:47.403 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000043", 2024-08-18T15:55:47.404 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3", 2024-08-18T15:55:47.404 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:55:47.404 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:55:47.404 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.11", 2024-08-18T15:55:47.404 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:55:47.404 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:55:47.404 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:55:47.404 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:55:47.404 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:55:47.404 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:55:47.404 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:55:47.404 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:55:47.404 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000111", 2024-08-18T15:55:47.404 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:55:47.405 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000048", 2024-08-18T15:55:47.405 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:55:47.405 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000020", 2024-08-18T15:55:47.405 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:55:47.405 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:55:47.405 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:55:47.405 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:55:47.405 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:55:47.405 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000099", 2024-08-18T15:55:47.405 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000056", 2024-08-18T15:55:47.405 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:55:47.405 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:55:47.405 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.19", 2024-08-18T15:55:47.405 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:55:47.405 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.28", 2024-08-18T15:55:47.406 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:55:47.406 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:55:47.406 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:55:47.406 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:55:47.406 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:55:47.406 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000072", 2024-08-18T15:55:47.406 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:55:47.406 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:55:47.406 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000062", 2024-08-18T15:55:47.406 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:55:47.406 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:55:47.406 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:55:47.406 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000115", 2024-08-18T15:55:47.406 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:55:47.407 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.1", 2024-08-18T15:55:47.407 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2", 2024-08-18T15:55:47.407 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000033", 2024-08-18T15:55:47.407 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:55:47.407 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:55:47.407 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:55:47.407 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:55:47.407 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:55:47.407 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:55:47.407 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:55:47.408 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:55:47.408 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:55:47.408 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:55:47.408 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:55:47.408 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:55:47.408 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:55:47.408 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:55:47.408 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:55:47.408 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:55:47.408 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:55:47.408 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:55:47.408 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:55:47.408 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000025", 2024-08-18T15:55:47.408 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:55:47.409 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000123", 2024-08-18T15:55:47.409 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.31", 2024-08-18T15:55:47.409 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:55:47.409 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:55:47.409 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:55:47.409 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:55:47.409 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.18", 2024-08-18T15:55:47.409 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:55:47.409 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:55:47.409 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:55:47.409 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:55:47.409 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:55:47.409 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000108", 2024-08-18T15:55:47.409 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:55:47.409 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:55:47.410 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000094", 2024-08-18T15:55:47.410 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:55:47.410 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:55:47.410 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:55:47.410 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:55:47.410 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:55:47.410 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:55:47.410 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000060", 2024-08-18T15:55:47.413 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:55:47.414 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:55:47.414 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:55:47.414 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000109", 2024-08-18T15:55:47.414 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.15", 2024-08-18T15:55:47.414 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:55:47.414 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:55:47.415 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:55:47.415 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:55:47.415 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:55:47.415 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:55:47.415 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:55:47.415 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:55:47.415 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:55:47.415 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000013", 2024-08-18T15:55:47.415 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:55:47.415 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:55:47.415 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:55:47.415 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:55:47.415 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:55:47.415 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:55:47.415 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:55:47.416 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:55:47.416 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:55:47.416 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:55:47.416 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:55:47.416 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:55:47.416 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:55:47.416 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:55:47.416 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000051", 2024-08-18T15:55:47.416 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:55:47.416 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:55:47.416 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:55:47.416 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:55:47.416 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000106", 2024-08-18T15:55:47.416 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:55:47.416 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:55:47.417 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:55:47.417 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:55:47.417 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:55:47.417 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000017", 2024-08-18T15:55:47.417 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:55:47.417 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:55:47.417 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:55:47.417 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000023", 2024-08-18T15:55:47.417 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:55:47.417 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:55:47.417 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:55:47.417 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:55:47.418 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:55:47.418 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:55:47.418 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:55:47.418 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000107", 2024-08-18T15:55:47.418 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:55:47.418 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:55:47.418 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000046", 2024-08-18T15:55:47.418 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:55:47.418 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:55:47.418 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:55:47.418 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.26", 2024-08-18T15:55:47.418 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:55:47.418 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:55:47.419 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9", 2024-08-18T15:55:47.419 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.22", 2024-08-18T15:55:47.419 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:55:47.419 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:55:47.419 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:55:47.419 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:55:47.419 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:55:47.419 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:55:47.419 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:55:47.419 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2", 2024-08-18T15:55:47.419 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:55:47.419 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:55:47.419 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000067", 2024-08-18T15:55:47.419 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:55:47.420 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:55:47.420 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:55:47.420 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:55:47.420 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:55:47.420 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:55:47.420 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:55:47.420 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:55:47.420 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:55:47.420 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:55:47.420 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:55:47.420 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:55:47.420 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:55:47.420 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:55:47.420 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:55:47.421 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:55:47.421 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:55:47.421 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:55:47.421 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:55:47.421 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.9", 2024-08-18T15:55:47.421 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000086", 2024-08-18T15:55:47.421 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.5", 2024-08-18T15:55:47.421 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:55:47.421 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:55:47.421 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:55:47.421 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:55:47.421 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:55:47.421 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:55:47.422 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2.MFXJ583gCr1.0", 2024-08-18T15:55:47.422 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:55:47.422 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1.GPnO16liyZ+.0", 2024-08-18T15:55:47.422 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:55:47.422 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:55:47.422 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:55:47.422 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:55:47.422 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:55:47.422 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:55:47.422 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:55:47.422 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:55:47.423 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:55:47.423 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.20", 2024-08-18T15:55:47.423 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:55:47.423 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:55:47.423 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:55:47.423 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:55:47.423 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:55:47.423 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:55:47.423 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1.retry", 2024-08-18T15:55:47.423 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:55:47.423 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:55:47.423 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:55:47.423 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0.retry", 2024-08-18T15:55:47.423 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000050", 2024-08-18T15:55:47.423 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:55:47.424 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000102", 2024-08-18T15:55:47.424 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:55:47.424 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:55:47.424 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:55:47.424 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:55:47.424 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:55:47.424 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:55:47.424 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:55:47.424 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:55:47.424 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:55:47.424 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:55:47.424 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:55:47.425 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000124", 2024-08-18T15:55:47.425 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:55:47.425 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:55:47.425 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:55:47.425 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000001", 2024-08-18T15:55:47.425 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:55:47.425 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:55:47.425 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:55:47.425 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:55:47.425 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000126", 2024-08-18T15:55:47.425 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:55:47.426 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0.fZhctfLXnSt.0", 2024-08-18T15:55:47.426 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:55:47.426 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000055", 2024-08-18T15:55:47.426 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:55:47.426 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000061", 2024-08-18T15:55:47.426 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1", 2024-08-18T15:55:47.426 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:55:47.426 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:55:47.426 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:55:47.426 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000097", 2024-08-18T15:55:47.426 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:55:47.426 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:55:47.426 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:55:47.426 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:55:47.426 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:55:47.427 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:55:47.427 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:55:47.427 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:55:47.427 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000042", 2024-08-18T15:55:47.427 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:55:47.427 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:55:47.427 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:55:47.427 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1.retry", 2024-08-18T15:55:47.427 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:55:47.427 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:55:47.427 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:55:47.427 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:55:47.427 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:55:47.427 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:55:47.427 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:55:47.428 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000075", 2024-08-18T15:55:47.428 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:55:47.428 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000120", 2024-08-18T15:55:47.428 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000030", 2024-08-18T15:55:47.428 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:55:47.428 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:55:47.428 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:55:47.428 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:55:47.428 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:55:47.428 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:55:47.428 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9" 2024-08-18T15:55:47.429 INFO:teuthology.orchestra.run.smithi193.stdout:] 2024-08-18T15:55:47.429 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:55:52.430 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:55:52.431 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:55:52.431 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:55:52.483 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:55:52.483 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:55:52.546 INFO:teuthology.orchestra.run.smithi193.stdout:[ 2024-08-18T15:55:52.546 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:55:52.547 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:55:52.547 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.29", 2024-08-18T15:55:52.547 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000078", 2024-08-18T15:55:52.547 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.history", 2024-08-18T15:55:52.547 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:55:52.547 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:55:52.547 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:55:52.547 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:55:52.547 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.23", 2024-08-18T15:55:52.548 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000070", 2024-08-18T15:55:52.548 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000104", 2024-08-18T15:55:52.548 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:55:52.548 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2.retry", 2024-08-18T15:55:52.548 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:55:52.548 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:55:52.548 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:55:52.548 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:55:52.548 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000026", 2024-08-18T15:55:52.548 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:55:52.548 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:55:52.549 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:55:52.549 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:55:52.549 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:55:52.549 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:55:52.549 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.8", 2024-08-18T15:55:52.549 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:55:52.549 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:55:52.549 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:55:52.549 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:55:52.549 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000028", 2024-08-18T15:55:52.549 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:55:52.549 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:55:52.550 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:55:52.550 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:55:52.550 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:55:52.550 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:55:52.550 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:55:52.550 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2.retry", 2024-08-18T15:55:52.550 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:55:52.550 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.30", 2024-08-18T15:55:52.550 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:55:52.550 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:55:52.550 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:55:52.550 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:55:52.551 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000040", 2024-08-18T15:55:52.551 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:55:52.551 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.6", 2024-08-18T15:55:52.551 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:55:52.551 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:55:52.551 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:55:52.551 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:55:52.551 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:55:52.551 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:55:52.551 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000015", 2024-08-18T15:55:52.551 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:55:52.551 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:55:52.551 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000069", 2024-08-18T15:55:52.552 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:55:52.552 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:55:52.552 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:55:52.552 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000095", 2024-08-18T15:55:52.552 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0", 2024-08-18T15:55:52.552 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000003", 2024-08-18T15:55:52.552 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000047", 2024-08-18T15:55:52.552 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:55:52.552 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:55:52.552 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:55:52.552 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:55:52.552 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:55:52.552 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:55:52.553 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:55:52.553 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:55:52.553 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:55:52.553 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000035", 2024-08-18T15:55:52.553 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:55:52.553 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:55:52.553 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000037", 2024-08-18T15:55:52.553 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:55:52.553 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:55:52.553 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:55:52.553 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:55:52.553 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:55:52.553 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:55:52.554 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:55:52.554 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:55:52.554 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:55:52.554 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:55:52.554 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:55:52.554 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:55:52.554 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:55:52.554 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:55:52.554 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000024", 2024-08-18T15:55:52.554 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:55:52.554 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:55:52.554 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:55:52.555 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:55:52.555 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000000", 2024-08-18T15:55:52.555 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:55:52.555 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000031", 2024-08-18T15:55:52.555 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:55:52.555 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:55:52.555 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:55:52.555 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:55:52.555 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:55:52.555 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:55:52.555 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3", 2024-08-18T15:55:52.555 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:55:52.555 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:55:52.556 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:55:52.556 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:55:52.556 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:55:52.556 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:55:52.556 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:55:52.556 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:55:52.556 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:55:52.556 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:55:52.556 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:55:52.556 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:55:52.556 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:55:52.556 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:55:52.556 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:55:52.557 INFO:teuthology.orchestra.run.smithi193.stdout: "data_loggenerations_metadata", 2024-08-18T15:55:52.557 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2", 2024-08-18T15:55:52.557 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:55:52.557 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:55:52.557 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:55:52.557 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:55:52.557 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T15:55:52.557 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:55:52.557 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:55:52.557 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:55:52.557 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:55:52.557 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:55:52.557 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000076", 2024-08-18T15:55:52.557 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:55:52.558 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:55:52.558 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000113", 2024-08-18T15:55:52.558 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000005", 2024-08-18T15:55:52.558 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:55:52.558 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000011", 2024-08-18T15:55:52.558 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:55:52.558 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000058", 2024-08-18T15:55:52.558 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:55:52.558 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:55:52.558 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:55:52.558 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000052", 2024-08-18T15:55:52.558 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:55:52.558 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:55:52.559 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:55:52.559 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:55:52.559 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.21", 2024-08-18T15:55:52.559 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:55:52.559 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:55:52.559 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:55:52.559 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:55:52.559 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:55:52.559 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:55:52.559 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:55:52.559 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:55:52.560 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:55:52.560 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:55:52.560 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:55:52.560 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:55:52.560 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:55:52.560 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:55:52.560 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:55:52.560 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:55:52.560 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:55:52.560 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:55:52.560 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:55:52.560 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000088", 2024-08-18T15:55:52.560 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000080", 2024-08-18T15:55:52.561 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:55:52.561 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:55:52.561 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000090", 2024-08-18T15:55:52.561 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:55:52.561 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:55:52.561 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000110", 2024-08-18T15:55:52.561 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:55:52.561 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:55:52.561 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:55:52.561 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.0", 2024-08-18T15:55:52.561 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:55:52.561 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000096", 2024-08-18T15:55:52.561 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:55:52.561 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:55:52.562 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:55:52.562 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:55:52.562 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:55:52.562 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000087", 2024-08-18T15:55:52.562 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:55:52.562 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:55:52.562 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:55:52.562 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:55:52.562 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000008", 2024-08-18T15:55:52.562 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:55:52.562 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:55:52.562 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:55:52.563 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:55:52.563 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:55:52.563 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000006", 2024-08-18T15:55:52.563 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000029", 2024-08-18T15:55:52.563 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:55:52.563 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000089", 2024-08-18T15:55:52.563 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:55:52.563 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:55:52.563 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000022", 2024-08-18T15:55:52.563 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:55:52.563 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:55:52.563 INFO:teuthology.orchestra.run.smithi193.stdout: "bilog.trim", 2024-08-18T15:55:52.563 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000016", 2024-08-18T15:55:52.563 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:55:52.564 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:55:52.564 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:55:52.564 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:55:52.564 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:55:52.564 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000041", 2024-08-18T15:55:52.564 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:55:52.564 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:55:52.564 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.log.01050e30-179b-43be-bdbb-3bb23cf37ae8.3", 2024-08-18T15:55:52.564 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:55:52.564 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000018", 2024-08-18T15:55:52.564 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000092", 2024-08-18T15:55:52.564 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:55:52.564 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:55:52.565 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:55:52.565 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:55:52.565 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.27", 2024-08-18T15:55:52.565 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.4", 2024-08-18T15:55:52.565 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:55:52.565 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:55:52.565 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:55:52.565 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:55:52.565 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:55:52.565 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:55:52.565 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:55:52.566 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000014", 2024-08-18T15:55:52.566 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:55:52.566 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3.retry", 2024-08-18T15:55:52.566 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:55:52.566 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:55:52.566 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:55:52.566 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:55:52.566 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:55:52.566 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000112", 2024-08-18T15:55:52.567 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:55:52.567 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000007", 2024-08-18T15:55:52.567 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000021", 2024-08-18T15:55:52.567 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:55:52.567 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:55:52.567 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:55:52.567 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:55:52.567 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:55:52.567 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.16", 2024-08-18T15:55:52.567 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:55:52.567 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000064", 2024-08-18T15:55:52.567 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:55:52.568 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:55:52.568 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:55:52.568 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:55:52.568 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:55:52.568 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:55:52.568 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:55:52.568 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000071", 2024-08-18T15:55:52.568 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:55:52.568 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:55:52.568 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:55:52.568 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000074", 2024-08-18T15:55:52.568 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:55:52.568 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:55:52.569 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:55:52.569 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:55:52.569 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:55:52.569 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:55:52.569 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:55:52.569 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:55:52.569 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:55:52.569 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:55:52.569 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:55:52.569 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:55:52.569 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:55:52.569 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:55:52.569 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:55:52.569 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000081", 2024-08-18T15:55:52.570 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:55:52.570 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000009", 2024-08-18T15:55:52.570 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000121", 2024-08-18T15:55:52.570 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1", 2024-08-18T15:55:52.570 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:55:52.570 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:55:52.570 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:55:52.570 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:55:52.570 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:55:52.570 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:55:52.570 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.17", 2024-08-18T15:55:52.570 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000125", 2024-08-18T15:55:52.571 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:55:52.571 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000082", 2024-08-18T15:55:52.571 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:55:52.571 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:55:52.571 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:55:52.571 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:55:52.571 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:55:52.571 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:55:52.571 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:55:52.571 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:55:52.571 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:55:52.571 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:55:52.571 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:55:52.571 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000105", 2024-08-18T15:55:52.572 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000059", 2024-08-18T15:55:52.572 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:55:52.572 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:55:52.572 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:55:52.572 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:55:52.572 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:55:52.572 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:55:52.572 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:55:52.572 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:55:52.572 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:55:52.572 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:55:52.573 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:55:52.573 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000077", 2024-08-18T15:55:52.573 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000032", 2024-08-18T15:55:52.573 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:55:52.573 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:55:52.573 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:55:52.573 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:55:52.573 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:55:52.573 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:55:52.573 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.24", 2024-08-18T15:55:52.573 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:55:52.573 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:55:52.573 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:55:52.573 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:55:52.574 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3.retry", 2024-08-18T15:55:52.574 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:55:52.574 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:55:52.574 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:55:52.574 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:55:52.574 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:55:52.574 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:55:52.574 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000053", 2024-08-18T15:55:52.574 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000091", 2024-08-18T15:55:52.574 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:55:52.574 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:55:52.574 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:55:52.575 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000065", 2024-08-18T15:55:52.575 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:55:52.575 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:55:52.575 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:55:52.575 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:55:52.575 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000083", 2024-08-18T15:55:52.575 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000010", 2024-08-18T15:55:52.575 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:55:52.575 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:55:52.575 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:55:52.576 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:55:52.576 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:55:52.576 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:55:52.576 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:55:52.576 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:55:52.576 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:55:52.576 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:55:52.576 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:55:52.576 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.7", 2024-08-18T15:55:52.576 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.3", 2024-08-18T15:55:52.576 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:55:52.576 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:55:52.576 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:55:52.577 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:55:52.577 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:55:52.577 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:55:52.577 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:55:52.577 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:55:52.577 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.2", 2024-08-18T15:55:52.577 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:55:52.577 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:55:52.577 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:55:52.577 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:55:52.577 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:55:52.577 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:55:52.577 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:55:52.578 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000045", 2024-08-18T15:55:52.578 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000002", 2024-08-18T15:55:52.578 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:55:52.578 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3.2UbiLidwSTy.0", 2024-08-18T15:55:52.578 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:55:52.578 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.12", 2024-08-18T15:55:52.578 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:55:52.578 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:55:52.578 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:55:52.578 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:55:52.578 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:55:52.578 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:55:52.579 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000116", 2024-08-18T15:55:52.579 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.25", 2024-08-18T15:55:52.579 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:55:52.579 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:55:52.579 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:55:52.579 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:55:52.579 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000034", 2024-08-18T15:55:52.579 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:55:52.579 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:55:52.579 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000101", 2024-08-18T15:55:52.580 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:55:52.580 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:55:52.580 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.13", 2024-08-18T15:55:52.580 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:55:52.580 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:55:52.580 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.1", 2024-08-18T15:55:52.580 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0.retry", 2024-08-18T15:55:52.580 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:55:52.580 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:55:52.580 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000079", 2024-08-18T15:55:52.580 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000049", 2024-08-18T15:55:52.580 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:55:52.580 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0", 2024-08-18T15:55:52.581 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:55:52.581 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:55:52.581 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000117", 2024-08-18T15:55:52.581 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:55:52.581 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000044", 2024-08-18T15:55:52.581 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:55:52.581 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:55:52.581 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:55:52.581 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:55:52.581 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:55:52.581 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:55:52.581 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.14", 2024-08-18T15:55:52.582 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:55:52.582 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000066", 2024-08-18T15:55:52.582 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:55:52.582 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:55:52.582 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:55:52.582 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:55:52.582 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:55:52.582 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000068", 2024-08-18T15:55:52.582 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:55:52.582 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000085", 2024-08-18T15:55:52.582 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:55:52.582 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:55:52.583 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:55:52.583 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:55:52.583 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:55:52.583 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:55:52.583 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:55:52.583 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:55:52.583 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:55:52.583 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:55:52.583 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:55:52.584 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:55:52.584 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000073", 2024-08-18T15:55:52.584 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:55:52.584 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:55:52.584 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:55:52.584 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000038", 2024-08-18T15:55:52.584 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:55:52.584 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:55:52.584 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:55:52.584 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:55:52.584 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.10", 2024-08-18T15:55:52.584 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:55:52.585 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:55:52.585 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:55:52.585 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000118", 2024-08-18T15:55:52.585 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:55:52.585 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3", 2024-08-18T15:55:52.585 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:55:52.585 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000036", 2024-08-18T15:55:52.585 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:55:52.585 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:55:52.585 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:55:52.585 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:55:52.585 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:55:52.586 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000103", 2024-08-18T15:55:52.586 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.3", 2024-08-18T15:55:52.586 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:55:52.586 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000119", 2024-08-18T15:55:52.586 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000098", 2024-08-18T15:55:52.586 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:55:52.586 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:55:52.586 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:55:52.586 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:55:52.586 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:55:52.586 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:55:52.586 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:55:52.587 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:55:52.587 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.0", 2024-08-18T15:55:52.587 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:55:52.587 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:55:52.587 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:55:52.587 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:55:52.587 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:55:52.587 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:55:52.587 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:55:52.588 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:55:52.588 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000027", 2024-08-18T15:55:52.588 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:55:52.588 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status", 2024-08-18T15:55:52.588 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:55:52.588 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000019", 2024-08-18T15:55:52.588 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:55:52.588 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:55:52.588 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:55:52.588 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:55:52.588 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1", 2024-08-18T15:55:52.589 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:55:52.589 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:55:52.589 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:55:52.589 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:55:52.589 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000039", 2024-08-18T15:55:52.589 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:55:52.589 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:55:52.589 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:55:52.590 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:55:52.590 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:55:52.590 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:55:52.590 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:55:52.590 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:55:52.590 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:55:52.590 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:55:52.590 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:55:52.590 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000100", 2024-08-18T15:55:52.590 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:55:52.590 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:55:52.591 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:55:52.591 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:55:52.591 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:55:52.591 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000093", 2024-08-18T15:55:52.591 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:55:52.591 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000004", 2024-08-18T15:55:52.591 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000063", 2024-08-18T15:55:52.591 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000122", 2024-08-18T15:55:52.592 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:55:52.592 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:55:52.592 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:55:52.592 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000057", 2024-08-18T15:55:52.592 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:55:52.592 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:55:52.592 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:55:52.592 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:55:52.592 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000054", 2024-08-18T15:55:52.592 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:55:52.592 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000114", 2024-08-18T15:55:52.592 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:55:52.592 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:55:52.592 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:55:52.593 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.2", 2024-08-18T15:55:52.593 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:55:52.593 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:55:52.593 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0", 2024-08-18T15:55:52.593 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:55:52.593 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:55:52.593 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:55:52.593 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:55:52.593 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:55:52.593 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:55:52.593 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000012", 2024-08-18T15:55:52.594 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:55:52.594 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:55:52.594 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:55:52.594 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000084", 2024-08-18T15:55:52.594 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000043", 2024-08-18T15:55:52.594 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3", 2024-08-18T15:55:52.594 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:55:52.594 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:55:52.594 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.11", 2024-08-18T15:55:52.594 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:55:52.594 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:55:52.595 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:55:52.595 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:55:52.595 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:55:52.595 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:55:52.595 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:55:52.595 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:55:52.595 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000111", 2024-08-18T15:55:52.595 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:55:52.595 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000048", 2024-08-18T15:55:52.595 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:55:52.595 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000020", 2024-08-18T15:55:52.595 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:55:52.595 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:55:52.596 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:55:52.596 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:55:52.596 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:55:52.596 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000099", 2024-08-18T15:55:52.596 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000056", 2024-08-18T15:55:52.596 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:55:52.596 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:55:52.596 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.19", 2024-08-18T15:55:52.596 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:55:52.596 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.28", 2024-08-18T15:55:52.596 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:55:52.596 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:55:52.596 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:55:52.597 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:55:52.597 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:55:52.597 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000072", 2024-08-18T15:55:52.597 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:55:52.597 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:55:52.597 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000062", 2024-08-18T15:55:52.597 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:55:52.597 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:55:52.597 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:55:52.597 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000115", 2024-08-18T15:55:52.597 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:55:52.598 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.1", 2024-08-18T15:55:52.598 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2", 2024-08-18T15:55:52.598 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000033", 2024-08-18T15:55:52.598 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:55:52.598 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:55:52.598 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:55:52.598 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:55:52.598 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:55:52.598 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:55:52.598 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:55:52.599 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:55:52.599 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:55:52.599 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:55:52.599 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:55:52.599 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:55:52.599 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:55:52.599 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:55:52.599 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:55:52.599 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:55:52.599 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:55:52.599 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:55:52.599 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:55:52.599 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000025", 2024-08-18T15:55:52.600 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:55:52.600 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000123", 2024-08-18T15:55:52.600 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.31", 2024-08-18T15:55:52.600 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:55:52.600 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:55:52.600 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:55:52.600 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:55:52.600 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.18", 2024-08-18T15:55:52.600 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:55:52.600 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:55:52.600 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:55:52.600 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:55:52.600 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:55:52.601 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000108", 2024-08-18T15:55:52.601 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:55:52.601 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:55:52.601 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000094", 2024-08-18T15:55:52.601 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:55:52.601 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:55:52.601 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:55:52.601 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:55:52.601 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:55:52.601 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:55:52.601 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000060", 2024-08-18T15:55:52.601 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:55:52.601 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:55:52.602 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:55:52.602 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000109", 2024-08-18T15:55:52.602 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.15", 2024-08-18T15:55:52.602 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:55:52.602 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:55:52.602 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:55:52.602 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:55:52.602 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:55:52.602 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:55:52.602 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:55:52.602 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:55:52.602 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:55:52.602 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000013", 2024-08-18T15:55:52.603 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:55:52.603 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:55:52.603 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:55:52.603 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:55:52.603 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:55:52.603 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:55:52.603 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:55:52.603 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:55:52.603 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:55:52.603 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:55:52.603 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:55:52.603 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:55:52.603 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:55:52.604 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:55:52.604 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000051", 2024-08-18T15:55:52.604 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:55:52.604 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:55:52.604 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:55:52.604 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:55:52.604 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000106", 2024-08-18T15:55:52.604 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:55:52.604 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:55:52.604 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:55:52.604 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:55:52.604 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:55:52.604 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000017", 2024-08-18T15:55:52.605 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:55:52.605 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:55:52.605 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:55:52.605 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000023", 2024-08-18T15:55:52.605 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:55:52.605 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:55:52.605 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:55:52.605 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:55:52.605 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:55:52.605 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:55:52.605 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:55:52.605 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000107", 2024-08-18T15:55:52.605 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:55:52.606 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:55:52.606 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000046", 2024-08-18T15:55:52.606 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:55:52.606 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:55:52.606 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:55:52.606 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.26", 2024-08-18T15:55:52.606 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:55:52.606 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:55:52.606 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9", 2024-08-18T15:55:52.606 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.22", 2024-08-18T15:55:52.606 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:55:52.607 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:55:52.607 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:55:52.607 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:55:52.607 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:55:52.607 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:55:52.607 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:55:52.607 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2", 2024-08-18T15:55:52.607 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:55:52.608 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:55:52.608 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000067", 2024-08-18T15:55:52.608 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:55:52.608 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:55:52.608 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:55:52.608 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:55:52.608 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:55:52.608 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:55:52.608 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:55:52.608 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:55:52.608 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:55:52.608 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:55:52.608 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:55:52.608 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:55:52.609 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:55:52.609 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:55:52.609 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:55:52.609 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:55:52.609 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:55:52.609 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:55:52.609 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:55:52.609 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.9", 2024-08-18T15:55:52.609 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000086", 2024-08-18T15:55:52.609 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.5", 2024-08-18T15:55:52.609 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:55:52.609 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:55:52.609 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:55:52.610 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:55:52.610 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:55:52.610 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:55:52.610 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2.MFXJ583gCr1.0", 2024-08-18T15:55:52.610 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:55:52.610 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1.GPnO16liyZ+.0", 2024-08-18T15:55:52.610 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:55:52.610 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:55:52.610 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:55:52.610 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:55:52.610 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:55:52.610 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:55:52.610 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:55:52.611 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:55:52.611 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:55:52.611 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.20", 2024-08-18T15:55:52.611 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:55:52.611 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:55:52.611 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:55:52.611 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:55:52.611 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:55:52.611 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:55:52.611 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1.retry", 2024-08-18T15:55:52.611 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:55:52.611 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:55:52.611 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:55:52.611 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0.retry", 2024-08-18T15:55:52.612 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000050", 2024-08-18T15:55:52.612 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:55:52.612 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000102", 2024-08-18T15:55:52.612 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:55:52.612 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:55:52.612 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:55:52.612 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:55:52.612 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:55:52.612 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:55:52.612 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:55:52.612 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:55:52.612 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:55:52.612 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:55:52.613 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:55:52.613 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000124", 2024-08-18T15:55:52.613 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:55:52.613 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:55:52.613 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:55:52.613 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000001", 2024-08-18T15:55:52.613 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:55:52.613 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:55:52.613 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:55:52.613 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:55:52.613 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000126", 2024-08-18T15:55:52.613 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:55:52.613 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0.fZhctfLXnSt.0", 2024-08-18T15:55:52.613 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:55:52.614 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000055", 2024-08-18T15:55:52.614 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:55:52.614 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000061", 2024-08-18T15:55:52.614 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1", 2024-08-18T15:55:52.614 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:55:52.614 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:55:52.614 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:55:52.614 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000097", 2024-08-18T15:55:52.614 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:55:52.614 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:55:52.614 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:55:52.615 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:55:52.615 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:55:52.615 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:55:52.615 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:55:52.615 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:55:52.615 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000042", 2024-08-18T15:55:52.615 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:55:52.615 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:55:52.615 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:55:52.615 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1.retry", 2024-08-18T15:55:52.615 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:55:52.615 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:55:52.615 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:55:52.616 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:55:52.616 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:55:52.616 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:55:52.616 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:55:52.616 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000075", 2024-08-18T15:55:52.616 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:55:52.616 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000120", 2024-08-18T15:55:52.616 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000030", 2024-08-18T15:55:52.616 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:55:52.616 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:55:52.616 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:55:52.616 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:55:52.616 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:55:52.617 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:55:52.617 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9" 2024-08-18T15:55:52.617 INFO:teuthology.orchestra.run.smithi193.stdout:] 2024-08-18T15:55:52.617 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:55:57.618 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:55:57.619 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:55:57.619 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:55:57.672 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:55:57.672 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:55:57.736 INFO:teuthology.orchestra.run.smithi193.stdout:[ 2024-08-18T15:55:57.736 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:55:57.737 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:55:57.737 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.29", 2024-08-18T15:55:57.737 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000078", 2024-08-18T15:55:57.737 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.history", 2024-08-18T15:55:57.737 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:55:57.737 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:55:57.738 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:55:57.738 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:55:57.738 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.23", 2024-08-18T15:55:57.738 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000070", 2024-08-18T15:55:57.738 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000104", 2024-08-18T15:55:57.739 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:55:57.739 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2.retry", 2024-08-18T15:55:57.739 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:55:57.739 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:55:57.739 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:55:57.739 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:55:57.740 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000026", 2024-08-18T15:55:57.740 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:55:57.740 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:55:57.740 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:55:57.740 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:55:57.740 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:55:57.741 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:55:57.741 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.8", 2024-08-18T15:55:57.741 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:55:57.741 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:55:57.741 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:55:57.742 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:55:57.742 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000028", 2024-08-18T15:55:57.742 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:55:57.742 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:55:57.742 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:55:57.743 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:55:57.743 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:55:57.743 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:55:57.743 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:55:57.743 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2.retry", 2024-08-18T15:55:57.743 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:55:57.744 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.30", 2024-08-18T15:55:57.744 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:55:57.744 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:55:57.744 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:55:57.744 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:55:57.745 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000040", 2024-08-18T15:55:57.745 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:55:57.745 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.6", 2024-08-18T15:55:57.745 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:55:57.745 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:55:57.745 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:55:57.746 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:55:57.746 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:55:57.746 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:55:57.746 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000015", 2024-08-18T15:55:57.746 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:55:57.746 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:55:57.747 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000069", 2024-08-18T15:55:57.747 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:55:57.747 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:55:57.747 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:55:57.747 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000095", 2024-08-18T15:55:57.748 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0", 2024-08-18T15:55:57.748 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000003", 2024-08-18T15:55:57.748 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000047", 2024-08-18T15:55:57.748 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:55:57.748 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:55:57.748 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:55:57.749 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:55:57.749 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:55:57.749 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:55:57.749 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:55:57.749 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:55:57.750 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:55:57.750 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000035", 2024-08-18T15:55:57.750 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:55:57.750 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:55:57.750 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000037", 2024-08-18T15:55:57.750 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:55:57.751 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:55:57.751 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:55:57.751 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:55:57.751 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:55:57.751 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:55:57.752 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:55:57.752 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:55:57.752 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:55:57.752 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:55:57.752 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:55:57.752 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:55:57.753 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:55:57.753 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:55:57.753 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000024", 2024-08-18T15:55:57.753 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:55:57.753 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:55:57.753 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:55:57.754 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:55:57.754 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000000", 2024-08-18T15:55:57.754 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:55:57.754 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000031", 2024-08-18T15:55:57.754 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:55:57.755 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:55:57.755 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:55:57.755 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:55:57.755 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:55:57.755 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:55:57.755 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3", 2024-08-18T15:55:57.756 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:55:57.756 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:55:57.756 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:55:57.756 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:55:57.756 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:55:57.756 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:55:57.757 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:55:57.757 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:55:57.757 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:55:57.757 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:55:57.757 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:55:57.758 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:55:57.758 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:55:57.758 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:55:57.758 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:55:57.758 INFO:teuthology.orchestra.run.smithi193.stdout: "data_loggenerations_metadata", 2024-08-18T15:55:57.759 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2", 2024-08-18T15:55:57.759 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:55:57.759 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:55:57.759 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:55:57.759 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:55:57.759 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T15:55:57.760 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:55:57.760 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:55:57.760 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:55:57.760 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:55:57.760 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:55:57.760 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000076", 2024-08-18T15:55:57.761 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:55:57.761 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:55:57.761 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000113", 2024-08-18T15:55:57.761 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000005", 2024-08-18T15:55:57.761 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:55:57.762 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000011", 2024-08-18T15:55:57.762 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:55:57.762 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000058", 2024-08-18T15:55:57.762 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:55:57.762 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:55:57.762 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:55:57.763 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000052", 2024-08-18T15:55:57.763 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:55:57.763 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:55:57.763 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:55:57.763 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:55:57.764 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.21", 2024-08-18T15:55:57.764 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:55:57.764 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:55:57.764 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:55:57.764 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:55:57.764 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:55:57.765 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:55:57.765 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:55:57.765 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:55:57.765 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:55:57.765 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:55:57.766 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:55:57.766 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:55:57.766 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:55:57.766 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:55:57.766 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:55:57.766 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:55:57.767 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:55:57.767 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:55:57.767 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:55:57.767 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000088", 2024-08-18T15:55:57.767 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000080", 2024-08-18T15:55:57.768 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:55:57.768 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:55:57.768 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000090", 2024-08-18T15:55:57.768 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:55:57.768 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:55:57.768 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000110", 2024-08-18T15:55:57.768 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:55:57.769 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:55:57.769 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:55:57.769 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.0", 2024-08-18T15:55:57.769 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:55:57.769 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000096", 2024-08-18T15:55:57.769 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:55:57.769 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:55:57.769 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:55:57.769 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:55:57.770 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:55:57.770 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000087", 2024-08-18T15:55:57.770 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:55:57.770 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:55:57.770 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:55:57.770 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:55:57.770 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000008", 2024-08-18T15:55:57.770 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:55:57.770 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:55:57.771 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:55:57.771 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:55:57.771 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:55:57.771 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000006", 2024-08-18T15:55:57.771 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000029", 2024-08-18T15:55:57.771 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:55:57.771 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000089", 2024-08-18T15:55:57.771 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:55:57.771 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:55:57.771 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000022", 2024-08-18T15:55:57.772 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:55:57.772 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:55:57.772 INFO:teuthology.orchestra.run.smithi193.stdout: "bilog.trim", 2024-08-18T15:55:57.772 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000016", 2024-08-18T15:55:57.772 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:55:57.772 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:55:57.772 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:55:57.772 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:55:57.772 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:55:57.772 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000041", 2024-08-18T15:55:57.773 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:55:57.773 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:55:57.773 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.log.01050e30-179b-43be-bdbb-3bb23cf37ae8.3", 2024-08-18T15:55:57.773 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:55:57.773 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000018", 2024-08-18T15:55:57.773 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000092", 2024-08-18T15:55:57.773 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:55:57.773 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:55:57.773 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:55:57.774 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:55:57.774 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.27", 2024-08-18T15:55:57.774 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.4", 2024-08-18T15:55:57.774 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:55:57.774 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:55:57.774 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:55:57.774 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:55:57.774 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:55:57.774 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:55:57.774 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:55:57.775 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000014", 2024-08-18T15:55:57.775 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:55:57.775 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3.retry", 2024-08-18T15:55:57.775 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:55:57.775 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:55:57.775 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:55:57.775 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:55:57.775 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:55:57.775 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000112", 2024-08-18T15:55:57.776 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:55:57.776 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000007", 2024-08-18T15:55:57.776 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000021", 2024-08-18T15:55:57.776 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:55:57.776 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:55:57.776 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:55:57.776 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:55:57.776 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:55:57.776 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.16", 2024-08-18T15:55:57.777 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:55:57.777 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000064", 2024-08-18T15:55:57.777 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:55:57.777 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:55:57.777 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:55:57.777 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:55:57.777 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:55:57.777 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:55:57.777 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:55:57.778 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000071", 2024-08-18T15:55:57.778 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:55:57.778 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:55:57.778 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:55:57.778 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000074", 2024-08-18T15:55:57.778 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:55:57.778 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:55:57.778 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:55:57.779 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:55:57.779 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:55:57.779 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:55:57.779 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:55:57.779 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:55:57.779 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:55:57.779 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:55:57.779 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:55:57.779 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:55:57.779 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:55:57.780 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:55:57.780 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:55:57.780 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000081", 2024-08-18T15:55:57.780 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:55:57.780 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000009", 2024-08-18T15:55:57.781 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000121", 2024-08-18T15:55:57.781 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1", 2024-08-18T15:55:57.781 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:55:57.781 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:55:57.781 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:55:57.781 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:55:57.782 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:55:57.782 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:55:57.782 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.17", 2024-08-18T15:55:57.782 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000125", 2024-08-18T15:55:57.782 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:55:57.782 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000082", 2024-08-18T15:55:57.783 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:55:57.783 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:55:57.783 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:55:57.783 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:55:57.783 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:55:57.784 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:55:57.784 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:55:57.784 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:55:57.784 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:55:57.784 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:55:57.784 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:55:57.785 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000105", 2024-08-18T15:55:57.785 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000059", 2024-08-18T15:55:57.785 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:55:57.785 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:55:57.785 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:55:57.786 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:55:57.786 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:55:57.786 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:55:57.786 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:55:57.786 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:55:57.787 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:55:57.787 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:55:57.787 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:55:57.787 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000077", 2024-08-18T15:55:57.787 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000032", 2024-08-18T15:55:57.787 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:55:57.788 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:55:57.788 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:55:57.788 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:55:57.788 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:55:57.788 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:55:57.789 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.24", 2024-08-18T15:55:57.789 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:55:57.789 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:55:57.789 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:55:57.789 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:55:57.789 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3.retry", 2024-08-18T15:55:57.790 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:55:57.790 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:55:57.790 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:55:57.790 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:55:57.790 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:55:57.790 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:55:57.791 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000053", 2024-08-18T15:55:57.791 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000091", 2024-08-18T15:55:57.791 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:55:57.791 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:55:57.791 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:55:57.792 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000065", 2024-08-18T15:55:57.792 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:55:57.792 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:55:57.792 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:55:57.792 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:55:57.792 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000083", 2024-08-18T15:55:57.793 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000010", 2024-08-18T15:55:57.793 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:55:57.793 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:55:57.793 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:55:57.793 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:55:57.793 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:55:57.794 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:55:57.794 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:55:57.794 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:55:57.794 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:55:57.794 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:55:57.795 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:55:57.795 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.7", 2024-08-18T15:55:57.795 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.3", 2024-08-18T15:55:57.795 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:55:57.795 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:55:57.795 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:55:57.796 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:55:57.796 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:55:57.796 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:55:57.796 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:55:57.796 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:55:57.797 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.2", 2024-08-18T15:55:57.797 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:55:57.797 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:55:57.797 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:55:57.797 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:55:57.798 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:55:57.798 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:55:57.798 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:55:57.798 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000045", 2024-08-18T15:55:57.798 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000002", 2024-08-18T15:55:57.799 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:55:57.799 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3.2UbiLidwSTy.0", 2024-08-18T15:55:57.799 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:55:57.799 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.12", 2024-08-18T15:55:57.799 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:55:57.799 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:55:57.800 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:55:57.800 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:55:57.800 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:55:57.800 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:55:57.800 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000116", 2024-08-18T15:55:57.801 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.25", 2024-08-18T15:55:57.801 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:55:57.801 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:55:57.801 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:55:57.801 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:55:57.801 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000034", 2024-08-18T15:55:57.802 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:55:57.802 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:55:57.802 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000101", 2024-08-18T15:55:57.802 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:55:57.802 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:55:57.802 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.13", 2024-08-18T15:55:57.803 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:55:57.803 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:55:57.803 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.1", 2024-08-18T15:55:57.803 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0.retry", 2024-08-18T15:55:57.803 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:55:57.803 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:55:57.803 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000079", 2024-08-18T15:55:57.803 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000049", 2024-08-18T15:55:57.803 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:55:57.803 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0", 2024-08-18T15:55:57.803 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:55:57.803 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:55:57.803 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000117", 2024-08-18T15:55:57.804 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:55:57.804 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000044", 2024-08-18T15:55:57.804 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:55:57.804 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:55:57.804 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:55:57.804 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:55:57.804 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:55:57.804 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:55:57.804 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.14", 2024-08-18T15:55:57.804 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:55:57.804 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000066", 2024-08-18T15:55:57.804 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:55:57.804 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:55:57.804 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:55:57.805 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:55:57.805 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:55:57.805 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000068", 2024-08-18T15:55:57.805 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:55:57.805 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000085", 2024-08-18T15:55:57.805 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:55:57.805 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:55:57.805 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:55:57.805 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:55:57.805 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:55:57.805 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:55:57.805 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:55:57.805 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:55:57.806 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:55:57.806 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:55:57.806 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:55:57.806 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:55:57.806 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000073", 2024-08-18T15:55:57.806 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:55:57.806 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:55:57.806 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:55:57.806 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000038", 2024-08-18T15:55:57.806 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:55:57.806 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:55:57.807 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:55:57.807 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:55:57.807 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.10", 2024-08-18T15:55:57.807 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:55:57.807 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:55:57.807 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:55:57.807 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000118", 2024-08-18T15:55:57.807 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:55:57.807 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3", 2024-08-18T15:55:57.807 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:55:57.807 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000036", 2024-08-18T15:55:57.807 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:55:57.808 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:55:57.808 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:55:57.808 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:55:57.808 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:55:57.808 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000103", 2024-08-18T15:55:57.808 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.3", 2024-08-18T15:55:57.808 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:55:57.808 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000119", 2024-08-18T15:55:57.808 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000098", 2024-08-18T15:55:57.809 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:55:57.809 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:55:57.809 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:55:57.809 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:55:57.809 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:55:57.809 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:55:57.809 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:55:57.809 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:55:57.810 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.0", 2024-08-18T15:55:57.810 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:55:57.810 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:55:57.810 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:55:57.810 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:55:57.810 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:55:57.810 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:55:57.810 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:55:57.810 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:55:57.810 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000027", 2024-08-18T15:55:57.810 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:55:57.810 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status", 2024-08-18T15:55:57.811 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:55:57.811 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000019", 2024-08-18T15:55:57.811 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:55:57.811 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:55:57.811 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:55:57.811 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:55:57.811 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1", 2024-08-18T15:55:57.811 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:55:57.811 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:55:57.811 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:55:57.811 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:55:57.811 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000039", 2024-08-18T15:55:57.811 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:55:57.812 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:55:57.812 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:55:57.812 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:55:57.812 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:55:57.812 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:55:57.812 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:55:57.812 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:55:57.812 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:55:57.812 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:55:57.812 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:55:57.812 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000100", 2024-08-18T15:55:57.812 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:55:57.812 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:55:57.812 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:55:57.813 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:55:57.813 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:55:57.813 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000093", 2024-08-18T15:55:57.813 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:55:57.813 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000004", 2024-08-18T15:55:57.813 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000063", 2024-08-18T15:55:57.813 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000122", 2024-08-18T15:55:57.813 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:55:57.813 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:55:57.813 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:55:57.813 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000057", 2024-08-18T15:55:57.813 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:55:57.813 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:55:57.814 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:55:57.814 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:55:57.814 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000054", 2024-08-18T15:55:57.814 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:55:57.814 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000114", 2024-08-18T15:55:57.814 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:55:57.814 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:55:57.814 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:55:57.814 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.2", 2024-08-18T15:55:57.814 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:55:57.814 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:55:57.814 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0", 2024-08-18T15:55:57.814 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:55:57.814 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:55:57.815 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:55:57.815 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:55:57.815 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:55:57.815 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:55:57.815 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000012", 2024-08-18T15:55:57.815 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:55:57.815 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:55:57.815 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:55:57.815 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000084", 2024-08-18T15:55:57.815 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000043", 2024-08-18T15:55:57.815 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3", 2024-08-18T15:55:57.815 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:55:57.815 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:55:57.816 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.11", 2024-08-18T15:55:57.816 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:55:57.816 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:55:57.816 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:55:57.816 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:55:57.816 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:55:57.816 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:55:57.816 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:55:57.816 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:55:57.816 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000111", 2024-08-18T15:55:57.816 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:55:57.816 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000048", 2024-08-18T15:55:57.816 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:55:57.817 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000020", 2024-08-18T15:55:57.817 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:55:57.817 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:55:57.817 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:55:57.817 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:55:57.817 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:55:57.817 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000099", 2024-08-18T15:55:57.817 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000056", 2024-08-18T15:55:57.817 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:55:57.817 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:55:57.817 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.19", 2024-08-18T15:55:57.817 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:55:57.817 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.28", 2024-08-18T15:55:57.818 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:55:57.818 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:55:57.818 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:55:57.818 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:55:57.818 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:55:57.818 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000072", 2024-08-18T15:55:57.818 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:55:57.818 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:55:57.818 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000062", 2024-08-18T15:55:57.818 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:55:57.818 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:55:57.818 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:55:57.818 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000115", 2024-08-18T15:55:57.818 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:55:57.819 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.1", 2024-08-18T15:55:57.819 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2", 2024-08-18T15:55:57.819 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000033", 2024-08-18T15:55:57.819 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:55:57.819 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:55:57.819 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:55:57.819 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:55:57.819 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:55:57.819 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:55:57.819 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:55:57.819 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:55:57.819 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:55:57.819 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:55:57.820 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:55:57.820 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:55:57.820 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:55:57.820 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:55:57.820 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:55:57.820 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:55:57.820 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:55:57.820 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:55:57.820 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:55:57.820 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000025", 2024-08-18T15:55:57.820 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:55:57.820 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000123", 2024-08-18T15:55:57.820 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.31", 2024-08-18T15:55:57.820 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:55:57.821 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:55:57.821 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:55:57.821 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:55:57.821 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.18", 2024-08-18T15:55:57.821 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:55:57.821 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:55:57.821 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:55:57.821 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:55:57.821 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:55:57.821 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000108", 2024-08-18T15:55:57.821 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:55:57.821 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:55:57.821 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000094", 2024-08-18T15:55:57.822 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:55:57.822 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:55:57.822 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:55:57.822 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:55:57.822 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:55:57.822 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:55:57.822 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000060", 2024-08-18T15:55:57.822 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:55:57.822 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:55:57.822 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:55:57.822 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000109", 2024-08-18T15:55:57.822 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.15", 2024-08-18T15:55:57.822 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:55:57.823 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:55:57.823 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:55:57.823 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:55:57.823 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:55:57.823 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:55:57.823 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:55:57.823 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:55:57.823 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:55:57.823 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000013", 2024-08-18T15:55:57.823 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:55:57.823 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:55:57.823 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:55:57.824 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:55:57.824 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:55:57.824 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:55:57.824 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:55:57.824 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:55:57.824 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:55:57.824 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:55:57.824 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:55:57.824 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:55:57.824 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:55:57.824 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:55:57.824 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000051", 2024-08-18T15:55:57.824 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:55:57.824 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:55:57.825 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:55:57.825 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:55:57.825 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000106", 2024-08-18T15:55:57.825 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:55:57.825 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:55:57.825 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:55:57.825 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:55:57.825 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:55:57.825 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000017", 2024-08-18T15:55:57.825 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:55:57.825 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:55:57.825 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:55:57.825 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000023", 2024-08-18T15:55:57.825 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:55:57.826 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:55:57.826 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:55:57.826 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:55:57.826 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:55:57.826 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:55:57.826 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:55:57.826 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000107", 2024-08-18T15:55:57.826 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:55:57.826 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:55:57.826 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000046", 2024-08-18T15:55:57.826 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:55:57.826 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:55:57.826 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:55:57.827 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.26", 2024-08-18T15:55:57.827 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:55:57.827 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:55:57.827 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9", 2024-08-18T15:55:57.827 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.22", 2024-08-18T15:55:57.827 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:55:57.827 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:55:57.827 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:55:57.827 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:55:57.827 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:55:57.827 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:55:57.827 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:55:57.827 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2", 2024-08-18T15:55:57.828 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:55:57.828 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:55:57.828 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000067", 2024-08-18T15:55:57.828 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:55:57.828 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:55:57.828 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:55:57.828 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:55:57.828 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:55:57.828 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:55:57.828 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:55:57.828 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:55:57.828 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:55:57.828 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:55:57.829 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:55:57.829 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:55:57.829 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:55:57.829 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:55:57.829 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:55:57.829 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:55:57.829 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:55:57.829 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:55:57.829 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:55:57.829 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.9", 2024-08-18T15:55:57.829 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000086", 2024-08-18T15:55:57.829 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.5", 2024-08-18T15:55:57.829 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:55:57.829 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:55:57.830 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:55:57.830 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:55:57.830 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:55:57.830 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:55:57.830 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2.MFXJ583gCr1.0", 2024-08-18T15:55:57.830 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:55:57.830 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1.GPnO16liyZ+.0", 2024-08-18T15:55:57.830 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:55:57.830 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:55:57.830 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:55:57.830 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:55:57.830 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:55:57.830 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:55:57.831 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:55:57.831 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:55:57.831 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:55:57.831 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.20", 2024-08-18T15:55:57.831 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:55:57.831 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:55:57.831 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:55:57.831 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:55:57.831 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:55:57.831 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:55:57.831 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1.retry", 2024-08-18T15:55:57.831 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:55:57.831 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:55:57.832 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:55:57.832 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0.retry", 2024-08-18T15:55:57.832 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000050", 2024-08-18T15:55:57.832 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:55:57.832 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000102", 2024-08-18T15:55:57.832 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:55:57.832 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:55:57.832 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:55:57.832 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:55:57.832 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:55:57.832 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:55:57.832 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:55:57.832 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:55:57.833 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:55:57.833 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:55:57.833 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:55:57.833 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000124", 2024-08-18T15:55:57.833 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:55:57.833 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:55:57.833 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:55:57.833 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000001", 2024-08-18T15:55:57.833 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:55:57.833 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:55:57.833 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:55:57.833 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:55:57.833 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000126", 2024-08-18T15:55:57.833 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:55:57.834 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0.fZhctfLXnSt.0", 2024-08-18T15:55:57.834 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:55:57.834 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000055", 2024-08-18T15:55:57.834 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:55:57.834 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000061", 2024-08-18T15:55:57.834 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1", 2024-08-18T15:55:57.834 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:55:57.834 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:55:57.834 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:55:57.834 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000097", 2024-08-18T15:55:57.834 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:55:57.834 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:55:57.835 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:55:57.835 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:55:57.835 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:55:57.835 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:55:57.835 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:55:57.835 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:55:57.835 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000042", 2024-08-18T15:55:57.835 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:55:57.835 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:55:57.835 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:55:57.835 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1.retry", 2024-08-18T15:55:57.835 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:55:57.835 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:55:57.835 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:55:57.836 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:55:57.836 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:55:57.836 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:55:57.836 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:55:57.836 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000075", 2024-08-18T15:55:57.836 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:55:57.836 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000120", 2024-08-18T15:55:57.836 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000030", 2024-08-18T15:55:57.836 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:55:57.836 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:55:57.836 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:55:57.836 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:55:57.836 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:55:57.836 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:55:57.837 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9" 2024-08-18T15:55:57.837 INFO:teuthology.orchestra.run.smithi193.stdout:] 2024-08-18T15:55:57.837 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:56:02.838 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:56:02.838 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:56:02.839 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:56:02.891 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:56:02.891 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:56:02.954 INFO:teuthology.orchestra.run.smithi193.stdout:[ 2024-08-18T15:56:02.954 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:56:02.955 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:56:02.955 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.29", 2024-08-18T15:56:02.955 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000078", 2024-08-18T15:56:02.955 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.history", 2024-08-18T15:56:02.955 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:56:02.955 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:56:02.955 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:56:02.955 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:56:02.955 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.23", 2024-08-18T15:56:02.955 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000070", 2024-08-18T15:56:02.955 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000104", 2024-08-18T15:56:02.955 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:56:02.956 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2.retry", 2024-08-18T15:56:02.956 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:56:02.956 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:56:02.956 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:56:02.956 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:56:02.956 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000026", 2024-08-18T15:56:02.956 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:56:02.956 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:56:02.956 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:56:02.956 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:56:02.956 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:56:02.956 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:56:02.956 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.8", 2024-08-18T15:56:02.957 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:56:02.957 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:56:02.957 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:56:02.957 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:56:02.957 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000028", 2024-08-18T15:56:02.957 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:56:02.957 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:56:02.957 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:56:02.957 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:56:02.957 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:56:02.957 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:56:02.957 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:56:02.957 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2.retry", 2024-08-18T15:56:02.958 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:56:02.958 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.30", 2024-08-18T15:56:02.958 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:56:02.958 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:56:02.958 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:56:02.958 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:56:02.958 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000040", 2024-08-18T15:56:02.958 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:56:02.958 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.6", 2024-08-18T15:56:02.958 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:56:02.958 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:56:02.958 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:56:02.959 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:56:02.959 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:56:02.959 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:56:02.959 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000015", 2024-08-18T15:56:02.959 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:56:02.959 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:56:02.959 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000069", 2024-08-18T15:56:02.959 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:56:02.959 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:56:02.959 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:56:02.959 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000095", 2024-08-18T15:56:02.959 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0", 2024-08-18T15:56:02.959 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000003", 2024-08-18T15:56:02.960 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000047", 2024-08-18T15:56:02.961 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:56:02.961 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:56:02.961 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:56:02.961 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:56:02.961 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:56:02.961 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:56:02.961 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:56:02.962 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:56:02.962 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:56:02.962 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000035", 2024-08-18T15:56:02.962 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:56:02.962 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:56:02.962 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000037", 2024-08-18T15:56:02.962 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:56:02.962 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:56:02.962 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:56:02.962 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:56:02.962 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:56:02.962 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:56:02.962 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:56:02.963 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:56:02.963 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:56:02.963 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:56:02.963 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:56:02.963 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:56:02.963 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:56:02.963 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:56:02.963 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000024", 2024-08-18T15:56:02.963 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:56:02.963 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:56:02.963 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:56:02.963 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:56:02.963 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000000", 2024-08-18T15:56:02.964 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:56:02.964 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000031", 2024-08-18T15:56:02.964 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:56:02.964 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:56:02.964 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:56:02.964 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:56:02.964 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:56:02.964 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:56:02.964 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3", 2024-08-18T15:56:02.964 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:56:02.964 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:56:02.964 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:56:02.964 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:56:02.964 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:56:02.965 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:56:02.965 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:56:02.965 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:56:02.965 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:56:02.965 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:56:02.965 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:56:02.965 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:56:02.965 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:56:02.965 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:56:02.965 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:56:02.965 INFO:teuthology.orchestra.run.smithi193.stdout: "data_loggenerations_metadata", 2024-08-18T15:56:02.965 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2", 2024-08-18T15:56:02.965 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:56:02.965 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:56:02.966 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:56:02.966 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:56:02.966 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T15:56:02.966 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:56:02.966 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:56:02.966 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:56:02.966 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:56:02.966 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:56:02.966 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000076", 2024-08-18T15:56:02.966 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:56:02.966 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:56:02.966 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000113", 2024-08-18T15:56:02.966 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000005", 2024-08-18T15:56:02.967 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:56:02.967 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000011", 2024-08-18T15:56:02.967 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:56:02.967 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000058", 2024-08-18T15:56:02.967 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:56:02.967 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:56:02.967 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:56:02.967 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000052", 2024-08-18T15:56:02.967 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:56:02.967 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:56:02.967 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:56:02.967 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:56:02.968 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.21", 2024-08-18T15:56:02.968 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:56:02.968 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:56:02.968 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:56:02.968 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:56:02.968 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:56:02.968 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:56:02.968 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:56:02.968 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:56:02.968 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:56:02.968 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:56:02.968 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:56:02.968 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:56:02.968 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:56:02.969 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:56:02.969 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:56:02.969 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:56:02.969 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:56:02.969 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:56:02.969 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:56:02.969 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000088", 2024-08-18T15:56:02.969 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000080", 2024-08-18T15:56:02.969 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:56:02.969 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:56:02.970 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000090", 2024-08-18T15:56:02.970 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:56:02.970 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:56:02.970 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000110", 2024-08-18T15:56:02.970 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:56:02.970 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:56:02.970 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:56:02.970 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.0", 2024-08-18T15:56:02.970 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:56:02.970 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000096", 2024-08-18T15:56:02.970 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:56:02.971 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:56:02.971 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:56:02.971 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:56:02.971 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:56:02.971 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000087", 2024-08-18T15:56:02.971 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:56:02.971 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:56:02.971 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:56:02.971 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:56:02.971 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000008", 2024-08-18T15:56:02.971 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:56:02.971 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:56:02.971 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:56:02.971 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:56:02.971 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:56:02.972 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000006", 2024-08-18T15:56:02.972 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000029", 2024-08-18T15:56:02.972 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:56:02.972 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000089", 2024-08-18T15:56:02.972 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:56:02.972 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:56:02.972 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000022", 2024-08-18T15:56:02.972 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:56:02.972 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:56:02.972 INFO:teuthology.orchestra.run.smithi193.stdout: "bilog.trim", 2024-08-18T15:56:02.972 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000016", 2024-08-18T15:56:02.972 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:56:02.972 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:56:02.972 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:56:02.972 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:56:02.973 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:56:02.973 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000041", 2024-08-18T15:56:02.973 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:56:02.973 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:56:02.973 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.log.01050e30-179b-43be-bdbb-3bb23cf37ae8.3", 2024-08-18T15:56:02.973 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:56:02.973 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000018", 2024-08-18T15:56:02.973 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000092", 2024-08-18T15:56:02.973 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:56:02.973 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:56:02.973 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:56:02.973 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:56:02.974 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.27", 2024-08-18T15:56:02.974 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.4", 2024-08-18T15:56:02.974 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:56:02.974 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:56:02.974 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:56:02.974 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:56:02.974 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:56:02.974 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:56:02.974 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:56:02.974 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000014", 2024-08-18T15:56:02.974 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:56:02.974 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3.retry", 2024-08-18T15:56:02.974 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:56:02.974 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:56:02.975 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:56:02.975 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:56:02.975 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:56:02.975 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000112", 2024-08-18T15:56:02.975 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:56:02.975 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000007", 2024-08-18T15:56:02.975 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000021", 2024-08-18T15:56:02.975 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:56:02.975 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:56:02.975 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:56:02.975 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:56:02.975 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:56:02.975 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.16", 2024-08-18T15:56:02.975 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:56:02.976 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000064", 2024-08-18T15:56:02.976 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:56:02.976 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:56:02.976 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:56:02.976 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:56:02.976 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:56:02.976 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:56:02.976 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:56:02.976 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000071", 2024-08-18T15:56:02.976 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:56:02.976 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:56:02.976 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:56:02.976 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000074", 2024-08-18T15:56:02.976 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:56:02.977 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:56:02.977 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:56:02.977 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:56:02.977 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:56:02.977 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:56:02.977 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:56:02.977 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:56:02.977 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:56:02.977 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:56:02.977 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:56:02.977 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:56:02.977 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:56:02.978 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:56:02.978 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:56:02.978 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000081", 2024-08-18T15:56:02.978 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:56:02.978 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000009", 2024-08-18T15:56:02.978 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000121", 2024-08-18T15:56:02.978 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1", 2024-08-18T15:56:02.978 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:56:02.978 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:56:02.978 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:56:02.978 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:56:02.979 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:56:02.979 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:56:02.979 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.17", 2024-08-18T15:56:02.979 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000125", 2024-08-18T15:56:02.979 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:56:02.979 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000082", 2024-08-18T15:56:02.979 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:56:02.979 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:56:02.980 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:56:02.980 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:56:02.980 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:56:02.980 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:56:02.980 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:56:02.980 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:56:02.980 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:56:02.980 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:56:02.980 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:56:02.980 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000105", 2024-08-18T15:56:02.980 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000059", 2024-08-18T15:56:02.980 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:56:02.980 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:56:02.981 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:56:02.981 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:56:02.981 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:56:02.981 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:56:02.981 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:56:02.981 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:56:02.981 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:56:02.981 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:56:02.981 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:56:02.981 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000077", 2024-08-18T15:56:02.981 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000032", 2024-08-18T15:56:02.981 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:56:02.981 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:56:02.981 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:56:02.982 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:56:02.982 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:56:02.982 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:56:02.982 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.24", 2024-08-18T15:56:02.982 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:56:02.982 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:56:02.982 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:56:02.982 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:56:02.982 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3.retry", 2024-08-18T15:56:02.982 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:56:02.982 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:56:02.982 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:56:02.982 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:56:02.983 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:56:02.983 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:56:02.983 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000053", 2024-08-18T15:56:02.983 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000091", 2024-08-18T15:56:02.983 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:56:02.983 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:56:02.983 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:56:02.983 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000065", 2024-08-18T15:56:02.983 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:56:02.983 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:56:02.983 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:56:02.983 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:56:02.983 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000083", 2024-08-18T15:56:02.984 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000010", 2024-08-18T15:56:02.984 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:56:02.984 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:56:02.984 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:56:02.984 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:56:02.984 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:56:02.984 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:56:02.984 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:56:02.984 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:56:02.984 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:56:02.984 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:56:02.984 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:56:02.984 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.7", 2024-08-18T15:56:02.984 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.3", 2024-08-18T15:56:02.985 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:56:02.985 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:56:02.985 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:56:02.985 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:56:02.985 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:56:02.985 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:56:02.985 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:56:02.985 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:56:02.985 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.2", 2024-08-18T15:56:02.985 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:56:02.985 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:56:02.985 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:56:02.985 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:56:02.986 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:56:02.986 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:56:02.986 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:56:02.986 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000045", 2024-08-18T15:56:02.986 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000002", 2024-08-18T15:56:02.986 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:56:02.986 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3.2UbiLidwSTy.0", 2024-08-18T15:56:02.986 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:56:02.986 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.12", 2024-08-18T15:56:02.986 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:56:02.986 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:56:02.986 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:56:02.987 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:56:02.987 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:56:02.987 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:56:02.987 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000116", 2024-08-18T15:56:02.987 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.25", 2024-08-18T15:56:02.987 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:56:02.987 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:56:02.987 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:56:02.987 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:56:02.987 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000034", 2024-08-18T15:56:02.987 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:56:02.987 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:56:02.987 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000101", 2024-08-18T15:56:02.988 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:56:02.988 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:56:02.988 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.13", 2024-08-18T15:56:02.988 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:56:02.988 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:56:02.988 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.1", 2024-08-18T15:56:02.988 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0.retry", 2024-08-18T15:56:02.988 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:56:02.988 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:56:02.988 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000079", 2024-08-18T15:56:02.989 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000049", 2024-08-18T15:56:02.990 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:56:02.990 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0", 2024-08-18T15:56:02.990 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:56:02.990 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:56:02.991 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000117", 2024-08-18T15:56:02.991 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:56:02.991 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000044", 2024-08-18T15:56:02.991 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:56:02.991 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:56:02.991 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:56:02.991 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:56:02.992 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:56:02.992 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:56:02.992 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.14", 2024-08-18T15:56:02.993 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:56:02.993 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000066", 2024-08-18T15:56:02.993 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:56:02.993 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:56:02.993 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:56:02.993 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:56:02.993 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:56:02.993 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000068", 2024-08-18T15:56:02.994 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:56:02.994 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000085", 2024-08-18T15:56:02.994 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:56:02.994 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:56:02.994 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:56:02.995 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:56:02.995 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:56:02.995 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:56:02.995 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:56:02.995 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:56:02.995 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:56:02.995 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:56:02.995 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:56:02.995 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:56:02.995 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000073", 2024-08-18T15:56:02.995 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:56:02.995 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:56:02.995 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:56:02.996 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000038", 2024-08-18T15:56:02.996 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:56:02.996 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:56:02.996 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:56:02.996 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:56:02.996 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.10", 2024-08-18T15:56:02.996 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:56:02.996 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:56:02.996 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:56:02.996 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000118", 2024-08-18T15:56:02.996 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:56:02.996 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3", 2024-08-18T15:56:02.997 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:56:02.997 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000036", 2024-08-18T15:56:02.997 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:56:02.997 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:56:02.997 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:56:02.997 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:56:02.997 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:56:02.997 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000103", 2024-08-18T15:56:02.997 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.3", 2024-08-18T15:56:02.997 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:56:02.997 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000119", 2024-08-18T15:56:02.998 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000098", 2024-08-18T15:56:02.998 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:56:02.998 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:56:02.998 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:56:02.998 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:56:02.998 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:56:02.998 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:56:02.998 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:56:02.999 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:56:02.999 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.0", 2024-08-18T15:56:02.999 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:56:02.999 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:56:02.999 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:56:02.999 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:56:02.999 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:56:02.999 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:56:02.999 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:56:02.999 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:56:02.999 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000027", 2024-08-18T15:56:02.999 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:56:02.999 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status", 2024-08-18T15:56:03.000 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:56:03.000 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000019", 2024-08-18T15:56:03.000 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:56:03.000 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:56:03.000 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:56:03.000 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:56:03.000 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1", 2024-08-18T15:56:03.000 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:56:03.000 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:56:03.000 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:56:03.000 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:56:03.000 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000039", 2024-08-18T15:56:03.000 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:56:03.001 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:56:03.001 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:56:03.001 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:56:03.001 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:56:03.001 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:56:03.001 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:56:03.001 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:56:03.001 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:56:03.001 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:56:03.001 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:56:03.001 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000100", 2024-08-18T15:56:03.001 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:56:03.001 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:56:03.002 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:56:03.002 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:56:03.002 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:56:03.002 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000093", 2024-08-18T15:56:03.002 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:56:03.002 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000004", 2024-08-18T15:56:03.002 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000063", 2024-08-18T15:56:03.002 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000122", 2024-08-18T15:56:03.002 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:56:03.003 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:56:03.003 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:56:03.003 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000057", 2024-08-18T15:56:03.003 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:56:03.003 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:56:03.003 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:56:03.003 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:56:03.003 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000054", 2024-08-18T15:56:03.003 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:56:03.003 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000114", 2024-08-18T15:56:03.003 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:56:03.003 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:56:03.003 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:56:03.004 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.2", 2024-08-18T15:56:03.004 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:56:03.004 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:56:03.004 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0", 2024-08-18T15:56:03.004 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:56:03.004 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:56:03.004 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:56:03.004 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:56:03.004 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:56:03.004 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:56:03.004 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000012", 2024-08-18T15:56:03.004 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:56:03.004 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:56:03.005 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:56:03.005 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000084", 2024-08-18T15:56:03.005 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000043", 2024-08-18T15:56:03.005 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3", 2024-08-18T15:56:03.005 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:56:03.005 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:56:03.005 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.11", 2024-08-18T15:56:03.005 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:56:03.005 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:56:03.005 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:56:03.005 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:56:03.005 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:56:03.005 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:56:03.006 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:56:03.006 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:56:03.006 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000111", 2024-08-18T15:56:03.006 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:56:03.006 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000048", 2024-08-18T15:56:03.006 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:56:03.006 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000020", 2024-08-18T15:56:03.006 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:56:03.006 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:56:03.006 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:56:03.006 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:56:03.006 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:56:03.007 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000099", 2024-08-18T15:56:03.007 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000056", 2024-08-18T15:56:03.007 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:56:03.007 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:56:03.007 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.19", 2024-08-18T15:56:03.007 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:56:03.007 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.28", 2024-08-18T15:56:03.007 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:56:03.007 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:56:03.007 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:56:03.007 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:56:03.007 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:56:03.007 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000072", 2024-08-18T15:56:03.007 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:56:03.008 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:56:03.008 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000062", 2024-08-18T15:56:03.008 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:56:03.008 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:56:03.008 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:56:03.008 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000115", 2024-08-18T15:56:03.008 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:56:03.008 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.1", 2024-08-18T15:56:03.008 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2", 2024-08-18T15:56:03.008 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000033", 2024-08-18T15:56:03.008 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:56:03.008 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:56:03.008 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:56:03.009 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:56:03.009 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:56:03.009 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:56:03.009 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:56:03.009 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:56:03.009 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:56:03.009 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:56:03.009 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:56:03.009 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:56:03.009 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:56:03.009 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:56:03.009 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:56:03.009 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:56:03.010 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:56:03.010 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:56:03.010 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:56:03.010 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000025", 2024-08-18T15:56:03.010 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:56:03.010 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000123", 2024-08-18T15:56:03.010 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.31", 2024-08-18T15:56:03.010 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:56:03.010 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:56:03.010 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:56:03.010 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:56:03.010 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.18", 2024-08-18T15:56:03.011 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:56:03.011 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:56:03.011 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:56:03.011 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:56:03.011 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:56:03.011 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000108", 2024-08-18T15:56:03.011 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:56:03.011 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:56:03.011 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000094", 2024-08-18T15:56:03.011 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:56:03.011 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:56:03.011 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:56:03.012 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:56:03.012 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:56:03.012 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:56:03.012 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000060", 2024-08-18T15:56:03.012 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:56:03.012 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:56:03.012 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:56:03.012 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000109", 2024-08-18T15:56:03.012 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.15", 2024-08-18T15:56:03.012 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:56:03.012 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:56:03.012 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:56:03.013 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:56:03.013 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:56:03.013 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:56:03.013 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:56:03.013 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:56:03.013 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:56:03.013 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000013", 2024-08-18T15:56:03.013 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:56:03.013 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:56:03.013 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:56:03.013 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:56:03.013 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:56:03.013 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:56:03.013 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:56:03.014 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:56:03.014 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:56:03.014 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:56:03.014 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:56:03.014 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:56:03.014 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:56:03.014 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:56:03.014 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000051", 2024-08-18T15:56:03.014 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:56:03.015 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:56:03.015 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:56:03.015 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:56:03.015 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000106", 2024-08-18T15:56:03.015 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:56:03.015 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:56:03.015 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:56:03.015 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:56:03.015 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:56:03.016 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000017", 2024-08-18T15:56:03.016 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:56:03.016 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:56:03.016 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:56:03.016 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000023", 2024-08-18T15:56:03.016 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:56:03.016 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:56:03.017 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:56:03.017 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:56:03.017 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:56:03.017 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:56:03.017 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:56:03.018 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000107", 2024-08-18T15:56:03.018 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:56:03.018 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:56:03.018 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000046", 2024-08-18T15:56:03.018 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:56:03.018 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:56:03.018 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:56:03.018 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.26", 2024-08-18T15:56:03.018 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:56:03.018 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:56:03.018 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9", 2024-08-18T15:56:03.019 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.22", 2024-08-18T15:56:03.019 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:56:03.019 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:56:03.019 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:56:03.019 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:56:03.019 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:56:03.019 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:56:03.019 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:56:03.019 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2", 2024-08-18T15:56:03.019 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:56:03.019 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:56:03.019 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000067", 2024-08-18T15:56:03.019 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:56:03.020 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:56:03.020 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:56:03.020 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:56:03.020 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:56:03.020 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:56:03.020 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:56:03.020 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:56:03.020 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:56:03.020 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:56:03.020 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:56:03.021 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:56:03.021 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:56:03.021 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:56:03.021 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:56:03.021 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:56:03.021 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:56:03.021 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:56:03.021 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:56:03.021 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.9", 2024-08-18T15:56:03.021 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000086", 2024-08-18T15:56:03.022 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.5", 2024-08-18T15:56:03.022 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:56:03.022 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:56:03.022 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:56:03.022 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:56:03.022 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:56:03.022 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:56:03.022 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2.MFXJ583gCr1.0", 2024-08-18T15:56:03.023 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:56:03.023 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1.GPnO16liyZ+.0", 2024-08-18T15:56:03.023 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:56:03.023 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:56:03.023 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:56:03.023 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:56:03.023 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:56:03.023 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:56:03.024 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:56:03.024 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:56:03.024 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:56:03.024 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.20", 2024-08-18T15:56:03.024 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:56:03.024 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:56:03.024 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:56:03.024 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:56:03.025 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:56:03.025 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:56:03.025 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1.retry", 2024-08-18T15:56:03.025 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:56:03.025 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:56:03.025 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:56:03.025 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0.retry", 2024-08-18T15:56:03.025 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000050", 2024-08-18T15:56:03.026 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:56:03.026 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000102", 2024-08-18T15:56:03.026 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:56:03.026 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:56:03.026 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:56:03.026 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:56:03.026 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:56:03.027 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:56:03.027 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:56:03.027 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:56:03.027 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:56:03.027 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:56:03.027 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:56:03.027 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000124", 2024-08-18T15:56:03.028 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:56:03.028 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:56:03.028 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:56:03.028 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000001", 2024-08-18T15:56:03.028 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:56:03.029 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:56:03.029 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:56:03.029 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:56:03.029 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000126", 2024-08-18T15:56:03.029 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:56:03.029 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0.fZhctfLXnSt.0", 2024-08-18T15:56:03.029 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:56:03.029 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000055", 2024-08-18T15:56:03.030 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:56:03.030 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000061", 2024-08-18T15:56:03.030 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1", 2024-08-18T15:56:03.030 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:56:03.030 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:56:03.030 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:56:03.030 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000097", 2024-08-18T15:56:03.031 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:56:03.031 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:56:03.031 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:56:03.031 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:56:03.031 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:56:03.031 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:56:03.032 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:56:03.032 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:56:03.032 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000042", 2024-08-18T15:56:03.032 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:56:03.032 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:56:03.032 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:56:03.033 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1.retry", 2024-08-18T15:56:03.033 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:56:03.033 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:56:03.033 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:56:03.033 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:56:03.033 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:56:03.033 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:56:03.033 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:56:03.034 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000075", 2024-08-18T15:56:03.034 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:56:03.034 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000120", 2024-08-18T15:56:03.034 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000030", 2024-08-18T15:56:03.034 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:56:03.034 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:56:03.034 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:56:03.034 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:56:03.034 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:56:03.034 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:56:03.035 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9" 2024-08-18T15:56:03.035 INFO:teuthology.orchestra.run.smithi193.stdout:] 2024-08-18T15:56:03.035 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:56:08.036 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:56:08.037 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:56:08.037 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:56:08.090 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:56:08.090 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:56:08.154 INFO:teuthology.orchestra.run.smithi193.stdout:[ 2024-08-18T15:56:08.154 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:56:08.154 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:56:08.154 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.29", 2024-08-18T15:56:08.155 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000078", 2024-08-18T15:56:08.155 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.history", 2024-08-18T15:56:08.155 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:56:08.155 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:56:08.155 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:56:08.155 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:56:08.155 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.23", 2024-08-18T15:56:08.155 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000070", 2024-08-18T15:56:08.155 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000104", 2024-08-18T15:56:08.155 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:56:08.156 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2.retry", 2024-08-18T15:56:08.156 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:56:08.156 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:56:08.156 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:56:08.156 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:56:08.156 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000026", 2024-08-18T15:56:08.156 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:56:08.156 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:56:08.156 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:56:08.157 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:56:08.157 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:56:08.157 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:56:08.157 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.8", 2024-08-18T15:56:08.157 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:56:08.157 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:56:08.157 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:56:08.157 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:56:08.157 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000028", 2024-08-18T15:56:08.157 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:56:08.158 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:56:08.158 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:56:08.158 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:56:08.158 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:56:08.158 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:56:08.158 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:56:08.158 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2.retry", 2024-08-18T15:56:08.158 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:56:08.159 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.30", 2024-08-18T15:56:08.159 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:56:08.159 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:56:08.159 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:56:08.159 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:56:08.159 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000040", 2024-08-18T15:56:08.159 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:56:08.159 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.6", 2024-08-18T15:56:08.160 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:56:08.160 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:56:08.160 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:56:08.160 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:56:08.160 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:56:08.160 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:56:08.160 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000015", 2024-08-18T15:56:08.160 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:56:08.160 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:56:08.161 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000069", 2024-08-18T15:56:08.161 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:56:08.161 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:56:08.161 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:56:08.161 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000095", 2024-08-18T15:56:08.161 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0", 2024-08-18T15:56:08.161 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000003", 2024-08-18T15:56:08.161 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000047", 2024-08-18T15:56:08.162 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:56:08.162 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:56:08.162 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:56:08.162 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:56:08.162 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:56:08.162 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:56:08.162 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:56:08.162 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:56:08.162 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:56:08.163 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000035", 2024-08-18T15:56:08.163 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:56:08.163 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:56:08.163 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000037", 2024-08-18T15:56:08.163 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:56:08.163 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:56:08.163 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:56:08.163 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:56:08.163 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:56:08.164 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:56:08.164 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:56:08.164 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:56:08.164 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:56:08.164 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:56:08.164 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:56:08.164 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:56:08.164 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:56:08.164 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:56:08.164 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000024", 2024-08-18T15:56:08.164 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:56:08.164 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:56:08.164 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:56:08.165 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:56:08.165 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000000", 2024-08-18T15:56:08.165 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:56:08.165 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000031", 2024-08-18T15:56:08.165 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:56:08.165 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:56:08.165 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:56:08.165 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:56:08.165 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:56:08.165 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:56:08.165 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3", 2024-08-18T15:56:08.165 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:56:08.166 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:56:08.166 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:56:08.166 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:56:08.166 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:56:08.166 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:56:08.166 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:56:08.166 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:56:08.166 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:56:08.166 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:56:08.166 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:56:08.166 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:56:08.166 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:56:08.166 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:56:08.167 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:56:08.167 INFO:teuthology.orchestra.run.smithi193.stdout: "data_loggenerations_metadata", 2024-08-18T15:56:08.167 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2", 2024-08-18T15:56:08.167 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:56:08.167 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:56:08.167 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:56:08.167 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:56:08.167 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T15:56:08.167 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:56:08.167 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:56:08.167 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:56:08.167 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:56:08.168 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:56:08.168 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000076", 2024-08-18T15:56:08.168 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:56:08.168 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:56:08.168 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000113", 2024-08-18T15:56:08.168 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000005", 2024-08-18T15:56:08.168 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:56:08.168 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000011", 2024-08-18T15:56:08.168 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:56:08.168 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000058", 2024-08-18T15:56:08.168 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:56:08.168 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:56:08.168 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:56:08.168 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000052", 2024-08-18T15:56:08.169 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:56:08.169 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:56:08.169 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:56:08.169 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:56:08.169 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.21", 2024-08-18T15:56:08.169 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:56:08.169 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:56:08.169 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:56:08.169 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:56:08.169 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:56:08.169 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:56:08.169 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:56:08.170 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:56:08.170 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:56:08.170 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:56:08.170 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:56:08.170 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:56:08.170 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:56:08.170 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:56:08.170 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:56:08.170 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:56:08.170 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:56:08.170 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:56:08.170 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:56:08.171 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000088", 2024-08-18T15:56:08.171 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000080", 2024-08-18T15:56:08.171 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:56:08.171 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:56:08.171 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000090", 2024-08-18T15:56:08.171 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:56:08.171 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:56:08.171 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000110", 2024-08-18T15:56:08.171 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:56:08.171 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:56:08.171 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:56:08.171 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.0", 2024-08-18T15:56:08.171 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:56:08.172 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000096", 2024-08-18T15:56:08.172 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:56:08.172 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:56:08.172 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:56:08.172 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:56:08.172 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:56:08.172 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000087", 2024-08-18T15:56:08.172 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:56:08.172 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:56:08.172 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:56:08.172 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:56:08.172 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000008", 2024-08-18T15:56:08.172 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:56:08.172 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:56:08.173 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:56:08.173 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:56:08.173 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:56:08.173 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000006", 2024-08-18T15:56:08.173 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000029", 2024-08-18T15:56:08.173 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:56:08.173 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000089", 2024-08-18T15:56:08.173 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:56:08.173 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:56:08.173 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000022", 2024-08-18T15:56:08.173 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:56:08.173 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:56:08.173 INFO:teuthology.orchestra.run.smithi193.stdout: "bilog.trim", 2024-08-18T15:56:08.173 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000016", 2024-08-18T15:56:08.174 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:56:08.174 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:56:08.174 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:56:08.174 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:56:08.174 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:56:08.174 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000041", 2024-08-18T15:56:08.174 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:56:08.174 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:56:08.174 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.log.01050e30-179b-43be-bdbb-3bb23cf37ae8.3", 2024-08-18T15:56:08.174 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:56:08.174 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000018", 2024-08-18T15:56:08.174 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000092", 2024-08-18T15:56:08.175 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:56:08.175 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:56:08.175 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:56:08.175 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:56:08.175 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.27", 2024-08-18T15:56:08.175 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.4", 2024-08-18T15:56:08.175 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:56:08.175 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:56:08.175 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:56:08.175 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:56:08.175 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:56:08.175 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:56:08.175 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:56:08.175 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000014", 2024-08-18T15:56:08.176 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:56:08.176 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3.retry", 2024-08-18T15:56:08.176 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:56:08.176 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:56:08.176 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:56:08.176 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:56:08.176 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:56:08.176 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000112", 2024-08-18T15:56:08.176 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:56:08.176 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000007", 2024-08-18T15:56:08.176 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000021", 2024-08-18T15:56:08.176 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:56:08.177 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:56:08.177 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:56:08.177 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:56:08.177 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:56:08.177 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.16", 2024-08-18T15:56:08.177 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:56:08.177 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000064", 2024-08-18T15:56:08.177 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:56:08.177 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:56:08.177 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:56:08.177 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:56:08.177 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:56:08.177 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:56:08.177 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:56:08.178 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000071", 2024-08-18T15:56:08.178 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:56:08.178 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:56:08.178 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:56:08.178 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000074", 2024-08-18T15:56:08.178 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:56:08.178 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:56:08.178 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:56:08.178 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:56:08.178 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:56:08.178 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:56:08.178 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:56:08.178 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:56:08.179 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:56:08.179 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:56:08.179 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:56:08.179 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:56:08.179 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:56:08.179 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:56:08.179 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:56:08.179 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000081", 2024-08-18T15:56:08.179 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:56:08.179 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000009", 2024-08-18T15:56:08.179 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000121", 2024-08-18T15:56:08.179 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1", 2024-08-18T15:56:08.179 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:56:08.180 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:56:08.180 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:56:08.180 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:56:08.180 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:56:08.180 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:56:08.180 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.17", 2024-08-18T15:56:08.180 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000125", 2024-08-18T15:56:08.180 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:56:08.180 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000082", 2024-08-18T15:56:08.180 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:56:08.180 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:56:08.180 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:56:08.180 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:56:08.180 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:56:08.181 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:56:08.181 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:56:08.181 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:56:08.181 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:56:08.181 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:56:08.181 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:56:08.181 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000105", 2024-08-18T15:56:08.181 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000059", 2024-08-18T15:56:08.181 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:56:08.181 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:56:08.181 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:56:08.181 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:56:08.181 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:56:08.182 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:56:08.182 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:56:08.182 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:56:08.182 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:56:08.182 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:56:08.182 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:56:08.182 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000077", 2024-08-18T15:56:08.182 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000032", 2024-08-18T15:56:08.182 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:56:08.182 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:56:08.182 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:56:08.182 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:56:08.183 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:56:08.183 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:56:08.183 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.24", 2024-08-18T15:56:08.183 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:56:08.183 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:56:08.183 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:56:08.183 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:56:08.183 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3.retry", 2024-08-18T15:56:08.183 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:56:08.183 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:56:08.183 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:56:08.183 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:56:08.184 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:56:08.184 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:56:08.184 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000053", 2024-08-18T15:56:08.184 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000091", 2024-08-18T15:56:08.184 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:56:08.184 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:56:08.184 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:56:08.184 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000065", 2024-08-18T15:56:08.184 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:56:08.184 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:56:08.184 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:56:08.185 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:56:08.185 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000083", 2024-08-18T15:56:08.185 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000010", 2024-08-18T15:56:08.185 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:56:08.185 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:56:08.185 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:56:08.185 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:56:08.185 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:56:08.185 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:56:08.186 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:56:08.186 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:56:08.186 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:56:08.186 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:56:08.186 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:56:08.186 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.7", 2024-08-18T15:56:08.186 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.3", 2024-08-18T15:56:08.186 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:56:08.186 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:56:08.186 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:56:08.186 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:56:08.187 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:56:08.187 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:56:08.187 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:56:08.187 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:56:08.187 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.2", 2024-08-18T15:56:08.187 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:56:08.187 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:56:08.187 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:56:08.187 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:56:08.188 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:56:08.188 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:56:08.188 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:56:08.188 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000045", 2024-08-18T15:56:08.188 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000002", 2024-08-18T15:56:08.188 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:56:08.188 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3.2UbiLidwSTy.0", 2024-08-18T15:56:08.188 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:56:08.188 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.12", 2024-08-18T15:56:08.189 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:56:08.189 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:56:08.189 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:56:08.189 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:56:08.189 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:56:08.189 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:56:08.189 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000116", 2024-08-18T15:56:08.189 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.25", 2024-08-18T15:56:08.189 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:56:08.189 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:56:08.190 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:56:08.190 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:56:08.190 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000034", 2024-08-18T15:56:08.190 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:56:08.190 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:56:08.190 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000101", 2024-08-18T15:56:08.190 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:56:08.190 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:56:08.190 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.13", 2024-08-18T15:56:08.191 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:56:08.191 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:56:08.191 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.1", 2024-08-18T15:56:08.191 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0.retry", 2024-08-18T15:56:08.191 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:56:08.191 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:56:08.191 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000079", 2024-08-18T15:56:08.191 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000049", 2024-08-18T15:56:08.191 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:56:08.192 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0", 2024-08-18T15:56:08.192 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:56:08.192 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:56:08.192 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000117", 2024-08-18T15:56:08.192 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:56:08.192 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000044", 2024-08-18T15:56:08.192 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:56:08.192 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:56:08.192 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:56:08.193 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:56:08.193 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:56:08.193 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:56:08.193 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.14", 2024-08-18T15:56:08.193 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:56:08.193 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000066", 2024-08-18T15:56:08.193 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:56:08.193 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:56:08.193 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:56:08.194 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:56:08.194 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:56:08.194 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000068", 2024-08-18T15:56:08.194 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:56:08.194 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000085", 2024-08-18T15:56:08.194 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:56:08.194 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:56:08.194 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:56:08.195 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:56:08.195 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:56:08.195 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:56:08.195 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:56:08.195 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:56:08.195 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:56:08.195 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:56:08.195 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:56:08.195 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:56:08.196 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000073", 2024-08-18T15:56:08.196 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:56:08.196 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:56:08.196 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:56:08.196 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000038", 2024-08-18T15:56:08.196 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:56:08.196 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:56:08.196 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:56:08.196 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:56:08.196 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.10", 2024-08-18T15:56:08.196 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:56:08.196 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:56:08.197 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:56:08.197 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000118", 2024-08-18T15:56:08.197 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:56:08.197 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3", 2024-08-18T15:56:08.197 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:56:08.197 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000036", 2024-08-18T15:56:08.197 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:56:08.197 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:56:08.197 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:56:08.197 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:56:08.197 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:56:08.197 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000103", 2024-08-18T15:56:08.197 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.3", 2024-08-18T15:56:08.198 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:56:08.198 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000119", 2024-08-18T15:56:08.198 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000098", 2024-08-18T15:56:08.198 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:56:08.198 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:56:08.198 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:56:08.198 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:56:08.198 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:56:08.198 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:56:08.198 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:56:08.198 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:56:08.198 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.0", 2024-08-18T15:56:08.199 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:56:08.199 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:56:08.199 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:56:08.199 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:56:08.199 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:56:08.199 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:56:08.199 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:56:08.199 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:56:08.199 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000027", 2024-08-18T15:56:08.199 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:56:08.199 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status", 2024-08-18T15:56:08.199 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:56:08.199 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000019", 2024-08-18T15:56:08.200 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:56:08.200 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:56:08.200 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:56:08.200 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:56:08.200 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1", 2024-08-18T15:56:08.200 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:56:08.200 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:56:08.200 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:56:08.200 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:56:08.200 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000039", 2024-08-18T15:56:08.200 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:56:08.200 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:56:08.200 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:56:08.201 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:56:08.201 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:56:08.201 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:56:08.201 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:56:08.201 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:56:08.201 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:56:08.201 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:56:08.201 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:56:08.201 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000100", 2024-08-18T15:56:08.201 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:56:08.201 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:56:08.201 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:56:08.201 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:56:08.201 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:56:08.202 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000093", 2024-08-18T15:56:08.202 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:56:08.202 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000004", 2024-08-18T15:56:08.202 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000063", 2024-08-18T15:56:08.202 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000122", 2024-08-18T15:56:08.202 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:56:08.202 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:56:08.202 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:56:08.202 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000057", 2024-08-18T15:56:08.202 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:56:08.202 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:56:08.202 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:56:08.202 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:56:08.203 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000054", 2024-08-18T15:56:08.203 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:56:08.203 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000114", 2024-08-18T15:56:08.203 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:56:08.203 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:56:08.203 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:56:08.203 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.2", 2024-08-18T15:56:08.203 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:56:08.203 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:56:08.203 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0", 2024-08-18T15:56:08.203 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:56:08.203 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:56:08.203 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:56:08.204 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:56:08.204 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:56:08.204 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:56:08.204 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000012", 2024-08-18T15:56:08.204 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:56:08.204 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:56:08.204 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:56:08.204 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000084", 2024-08-18T15:56:08.204 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000043", 2024-08-18T15:56:08.204 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3", 2024-08-18T15:56:08.204 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:56:08.204 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:56:08.204 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.11", 2024-08-18T15:56:08.205 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:56:08.205 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:56:08.205 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:56:08.205 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:56:08.205 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:56:08.205 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:56:08.205 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:56:08.205 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:56:08.205 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000111", 2024-08-18T15:56:08.205 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:56:08.205 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000048", 2024-08-18T15:56:08.205 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:56:08.205 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000020", 2024-08-18T15:56:08.206 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:56:08.206 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:56:08.206 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:56:08.206 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:56:08.206 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:56:08.206 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000099", 2024-08-18T15:56:08.206 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000056", 2024-08-18T15:56:08.206 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:56:08.206 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:56:08.206 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.19", 2024-08-18T15:56:08.206 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:56:08.206 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.28", 2024-08-18T15:56:08.206 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:56:08.207 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:56:08.207 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:56:08.207 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:56:08.207 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:56:08.207 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000072", 2024-08-18T15:56:08.207 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:56:08.207 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:56:08.207 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000062", 2024-08-18T15:56:08.207 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:56:08.207 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:56:08.207 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:56:08.207 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000115", 2024-08-18T15:56:08.207 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:56:08.207 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.1", 2024-08-18T15:56:08.208 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2", 2024-08-18T15:56:08.208 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000033", 2024-08-18T15:56:08.208 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:56:08.208 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:56:08.208 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:56:08.208 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:56:08.208 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:56:08.208 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:56:08.208 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:56:08.208 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:56:08.208 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:56:08.208 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:56:08.208 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:56:08.209 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:56:08.209 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:56:08.209 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:56:08.209 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:56:08.209 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:56:08.209 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:56:08.209 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:56:08.209 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:56:08.209 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000025", 2024-08-18T15:56:08.209 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:56:08.209 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000123", 2024-08-18T15:56:08.209 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.31", 2024-08-18T15:56:08.209 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:56:08.209 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:56:08.210 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:56:08.210 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:56:08.210 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.18", 2024-08-18T15:56:08.210 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:56:08.210 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:56:08.210 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:56:08.210 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:56:08.210 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:56:08.210 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000108", 2024-08-18T15:56:08.210 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:56:08.210 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:56:08.210 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000094", 2024-08-18T15:56:08.210 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:56:08.211 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:56:08.211 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:56:08.211 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:56:08.211 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:56:08.211 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:56:08.211 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000060", 2024-08-18T15:56:08.211 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:56:08.211 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:56:08.211 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:56:08.211 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000109", 2024-08-18T15:56:08.211 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.15", 2024-08-18T15:56:08.211 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:56:08.212 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:56:08.212 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:56:08.212 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:56:08.212 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:56:08.212 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:56:08.212 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:56:08.212 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:56:08.212 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:56:08.212 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000013", 2024-08-18T15:56:08.212 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:56:08.212 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:56:08.212 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:56:08.212 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:56:08.212 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:56:08.213 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:56:08.213 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:56:08.213 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:56:08.213 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:56:08.213 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:56:08.213 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:56:08.213 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:56:08.213 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:56:08.213 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:56:08.213 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000051", 2024-08-18T15:56:08.213 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:56:08.213 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:56:08.213 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:56:08.214 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:56:08.214 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000106", 2024-08-18T15:56:08.214 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:56:08.214 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:56:08.214 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:56:08.214 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:56:08.214 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:56:08.214 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000017", 2024-08-18T15:56:08.214 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:56:08.214 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:56:08.215 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:56:08.215 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000023", 2024-08-18T15:56:08.215 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:56:08.215 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:56:08.215 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:56:08.215 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:56:08.215 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:56:08.215 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:56:08.215 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:56:08.215 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000107", 2024-08-18T15:56:08.215 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:56:08.216 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:56:08.216 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000046", 2024-08-18T15:56:08.216 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:56:08.216 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:56:08.216 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:56:08.216 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.26", 2024-08-18T15:56:08.216 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:56:08.216 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:56:08.216 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9", 2024-08-18T15:56:08.216 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.22", 2024-08-18T15:56:08.216 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:56:08.216 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:56:08.216 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:56:08.216 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:56:08.217 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:56:08.217 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:56:08.217 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:56:08.217 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2", 2024-08-18T15:56:08.217 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:56:08.217 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:56:08.217 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000067", 2024-08-18T15:56:08.217 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:56:08.217 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:56:08.217 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:56:08.217 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:56:08.217 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:56:08.217 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:56:08.218 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:56:08.218 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:56:08.218 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:56:08.218 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:56:08.218 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:56:08.218 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:56:08.218 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:56:08.218 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:56:08.218 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:56:08.218 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:56:08.218 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:56:08.218 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:56:08.218 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:56:08.219 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.9", 2024-08-18T15:56:08.219 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000086", 2024-08-18T15:56:08.219 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.5", 2024-08-18T15:56:08.219 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:56:08.219 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:56:08.219 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:56:08.219 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:56:08.219 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:56:08.219 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:56:08.219 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2.MFXJ583gCr1.0", 2024-08-18T15:56:08.219 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:56:08.219 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1.GPnO16liyZ+.0", 2024-08-18T15:56:08.219 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:56:08.219 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:56:08.220 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:56:08.220 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:56:08.220 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:56:08.220 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:56:08.220 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:56:08.220 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:56:08.220 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:56:08.220 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.20", 2024-08-18T15:56:08.220 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:56:08.220 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:56:08.220 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:56:08.220 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:56:08.220 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:56:08.221 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:56:08.221 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1.retry", 2024-08-18T15:56:08.221 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:56:08.221 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:56:08.221 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:56:08.221 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0.retry", 2024-08-18T15:56:08.221 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000050", 2024-08-18T15:56:08.221 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:56:08.221 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000102", 2024-08-18T15:56:08.221 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:56:08.222 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:56:08.222 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:56:08.222 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:56:08.222 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:56:08.222 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:56:08.222 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:56:08.222 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:56:08.222 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:56:08.222 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:56:08.223 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:56:08.223 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000124", 2024-08-18T15:56:08.223 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:56:08.223 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:56:08.223 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:56:08.223 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000001", 2024-08-18T15:56:08.223 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:56:08.223 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:56:08.223 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:56:08.223 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:56:08.224 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000126", 2024-08-18T15:56:08.224 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:56:08.224 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0.fZhctfLXnSt.0", 2024-08-18T15:56:08.224 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:56:08.224 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000055", 2024-08-18T15:56:08.224 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:56:08.224 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000061", 2024-08-18T15:56:08.224 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1", 2024-08-18T15:56:08.224 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:56:08.224 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:56:08.224 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:56:08.225 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000097", 2024-08-18T15:56:08.225 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:56:08.225 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:56:08.225 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:56:08.225 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:56:08.225 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:56:08.225 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:56:08.225 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:56:08.225 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:56:08.225 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000042", 2024-08-18T15:56:08.226 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:56:08.226 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:56:08.226 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:56:08.226 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1.retry", 2024-08-18T15:56:08.226 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:56:08.226 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:56:08.226 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:56:08.226 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:56:08.226 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:56:08.227 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:56:08.227 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:56:08.227 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000075", 2024-08-18T15:56:08.227 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:56:08.227 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000120", 2024-08-18T15:56:08.227 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000030", 2024-08-18T15:56:08.227 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:56:08.227 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:56:08.227 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:56:08.227 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:56:08.227 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:56:08.228 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:56:08.228 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9" 2024-08-18T15:56:08.228 INFO:teuthology.orchestra.run.smithi193.stdout:] 2024-08-18T15:56:08.228 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:56:13.230 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:56:13.230 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:56:13.230 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:56:13.275 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:56:13.275 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:56:13.332 INFO:teuthology.orchestra.run.smithi193.stdout:[ 2024-08-18T15:56:13.333 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:56:13.333 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:56:13.333 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.29", 2024-08-18T15:56:13.333 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000078", 2024-08-18T15:56:13.333 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.history", 2024-08-18T15:56:13.333 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:56:13.333 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:56:13.333 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:56:13.333 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:56:13.333 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.23", 2024-08-18T15:56:13.333 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000070", 2024-08-18T15:56:13.333 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000104", 2024-08-18T15:56:13.334 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:56:13.334 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2.retry", 2024-08-18T15:56:13.334 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:56:13.334 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:56:13.334 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:56:13.334 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:56:13.334 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000026", 2024-08-18T15:56:13.334 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:56:13.334 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:56:13.334 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:56:13.334 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:56:13.334 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:56:13.335 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:56:13.335 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.8", 2024-08-18T15:56:13.335 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:56:13.335 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:56:13.335 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:56:13.335 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:56:13.335 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000028", 2024-08-18T15:56:13.335 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:56:13.335 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:56:13.335 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:56:13.335 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:56:13.335 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:56:13.335 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:56:13.335 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:56:13.336 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2.retry", 2024-08-18T15:56:13.336 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:56:13.336 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.30", 2024-08-18T15:56:13.336 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:56:13.336 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:56:13.336 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:56:13.336 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:56:13.336 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000040", 2024-08-18T15:56:13.337 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:56:13.337 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.6", 2024-08-18T15:56:13.337 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:56:13.337 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:56:13.337 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:56:13.337 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:56:13.337 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:56:13.337 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:56:13.337 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000015", 2024-08-18T15:56:13.337 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:56:13.337 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:56:13.337 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000069", 2024-08-18T15:56:13.337 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:56:13.337 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:56:13.337 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:56:13.338 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000095", 2024-08-18T15:56:13.338 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0", 2024-08-18T15:56:13.338 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000003", 2024-08-18T15:56:13.338 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000047", 2024-08-18T15:56:13.338 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:56:13.338 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:56:13.338 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:56:13.338 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:56:13.338 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:56:13.338 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:56:13.338 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:56:13.338 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:56:13.338 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:56:13.339 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000035", 2024-08-18T15:56:13.339 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:56:13.339 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:56:13.339 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000037", 2024-08-18T15:56:13.339 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:56:13.339 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:56:13.339 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:56:13.339 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:56:13.339 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:56:13.339 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:56:13.340 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:56:13.340 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:56:13.340 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:56:13.340 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:56:13.340 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:56:13.340 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:56:13.340 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:56:13.340 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:56:13.340 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000024", 2024-08-18T15:56:13.340 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:56:13.340 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:56:13.340 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:56:13.340 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:56:13.341 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000000", 2024-08-18T15:56:13.341 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:56:13.341 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000031", 2024-08-18T15:56:13.341 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:56:13.341 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:56:13.341 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:56:13.341 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:56:13.341 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:56:13.341 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:56:13.341 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3", 2024-08-18T15:56:13.341 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:56:13.341 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:56:13.341 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:56:13.342 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:56:13.342 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:56:13.342 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:56:13.342 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:56:13.342 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:56:13.342 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:56:13.342 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:56:13.342 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:56:13.342 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:56:13.342 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:56:13.342 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:56:13.342 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:56:13.342 INFO:teuthology.orchestra.run.smithi193.stdout: "data_loggenerations_metadata", 2024-08-18T15:56:13.343 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2", 2024-08-18T15:56:13.343 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:56:13.343 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:56:13.343 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:56:13.343 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:56:13.343 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T15:56:13.343 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:56:13.343 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:56:13.343 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:56:13.343 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:56:13.343 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:56:13.343 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000076", 2024-08-18T15:56:13.343 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:56:13.343 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:56:13.344 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000113", 2024-08-18T15:56:13.344 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000005", 2024-08-18T15:56:13.344 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:56:13.344 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000011", 2024-08-18T15:56:13.344 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:56:13.344 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000058", 2024-08-18T15:56:13.344 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:56:13.344 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:56:13.344 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:56:13.344 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000052", 2024-08-18T15:56:13.344 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:56:13.344 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:56:13.344 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:56:13.345 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:56:13.345 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.21", 2024-08-18T15:56:13.345 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:56:13.345 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:56:13.345 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:56:13.345 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:56:13.345 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:56:13.345 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:56:13.345 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:56:13.345 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:56:13.345 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:56:13.345 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:56:13.345 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:56:13.346 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:56:13.346 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:56:13.346 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:56:13.346 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:56:13.346 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:56:13.346 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:56:13.346 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:56:13.346 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:56:13.346 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000088", 2024-08-18T15:56:13.346 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000080", 2024-08-18T15:56:13.346 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:56:13.347 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:56:13.347 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000090", 2024-08-18T15:56:13.347 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:56:13.347 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:56:13.347 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000110", 2024-08-18T15:56:13.347 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:56:13.348 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:56:13.348 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:56:13.348 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.0", 2024-08-18T15:56:13.348 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:56:13.348 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000096", 2024-08-18T15:56:13.348 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:56:13.348 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:56:13.348 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:56:13.348 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:56:13.348 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:56:13.348 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000087", 2024-08-18T15:56:13.348 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:56:13.348 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:56:13.349 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:56:13.349 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:56:13.349 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000008", 2024-08-18T15:56:13.349 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:56:13.349 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:56:13.349 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:56:13.349 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:56:13.349 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:56:13.349 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000006", 2024-08-18T15:56:13.349 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000029", 2024-08-18T15:56:13.349 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:56:13.349 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000089", 2024-08-18T15:56:13.349 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:56:13.350 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:56:13.350 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000022", 2024-08-18T15:56:13.350 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:56:13.350 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:56:13.350 INFO:teuthology.orchestra.run.smithi193.stdout: "bilog.trim", 2024-08-18T15:56:13.350 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000016", 2024-08-18T15:56:13.350 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:56:13.350 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:56:13.350 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:56:13.350 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:56:13.350 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:56:13.350 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000041", 2024-08-18T15:56:13.351 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:56:13.351 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:56:13.351 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.log.01050e30-179b-43be-bdbb-3bb23cf37ae8.3", 2024-08-18T15:56:13.351 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:56:13.351 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000018", 2024-08-18T15:56:13.351 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000092", 2024-08-18T15:56:13.351 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:56:13.351 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:56:13.351 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:56:13.351 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:56:13.351 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.27", 2024-08-18T15:56:13.351 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.4", 2024-08-18T15:56:13.351 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:56:13.351 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:56:13.352 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:56:13.352 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:56:13.352 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:56:13.352 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:56:13.352 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:56:13.352 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000014", 2024-08-18T15:56:13.352 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:56:13.352 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3.retry", 2024-08-18T15:56:13.352 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:56:13.352 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:56:13.352 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:56:13.352 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:56:13.352 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:56:13.352 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000112", 2024-08-18T15:56:13.353 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:56:13.353 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000007", 2024-08-18T15:56:13.353 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000021", 2024-08-18T15:56:13.353 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:56:13.353 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:56:13.353 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:56:13.353 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:56:13.353 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:56:13.353 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.16", 2024-08-18T15:56:13.353 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:56:13.354 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000064", 2024-08-18T15:56:13.354 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:56:13.354 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:56:13.354 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:56:13.354 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:56:13.354 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:56:13.354 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:56:13.354 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:56:13.354 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000071", 2024-08-18T15:56:13.354 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:56:13.354 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:56:13.354 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:56:13.354 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000074", 2024-08-18T15:56:13.355 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:56:13.355 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:56:13.355 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:56:13.355 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:56:13.355 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:56:13.355 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:56:13.355 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:56:13.355 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:56:13.355 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:56:13.355 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:56:13.355 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:56:13.355 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:56:13.355 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:56:13.355 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:56:13.356 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:56:13.356 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000081", 2024-08-18T15:56:13.356 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:56:13.356 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000009", 2024-08-18T15:56:13.356 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000121", 2024-08-18T15:56:13.356 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1", 2024-08-18T15:56:13.356 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:56:13.356 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:56:13.356 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:56:13.356 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:56:13.356 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:56:13.356 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:56:13.356 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.17", 2024-08-18T15:56:13.356 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000125", 2024-08-18T15:56:13.357 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:56:13.357 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000082", 2024-08-18T15:56:13.357 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:56:13.357 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:56:13.357 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:56:13.357 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:56:13.357 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:56:13.357 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:56:13.358 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:56:13.358 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:56:13.358 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:56:13.358 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:56:13.358 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:56:13.358 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000105", 2024-08-18T15:56:13.358 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000059", 2024-08-18T15:56:13.358 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:56:13.358 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:56:13.358 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:56:13.358 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:56:13.358 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:56:13.358 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:56:13.359 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:56:13.359 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:56:13.359 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:56:13.359 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:56:13.359 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:56:13.359 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000077", 2024-08-18T15:56:13.359 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000032", 2024-08-18T15:56:13.359 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:56:13.359 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:56:13.359 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:56:13.359 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:56:13.359 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:56:13.359 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:56:13.360 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.24", 2024-08-18T15:56:13.360 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:56:13.360 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:56:13.360 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:56:13.360 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:56:13.360 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3.retry", 2024-08-18T15:56:13.360 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:56:13.360 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:56:13.360 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:56:13.360 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:56:13.360 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:56:13.360 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:56:13.360 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000053", 2024-08-18T15:56:13.360 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000091", 2024-08-18T15:56:13.361 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:56:13.361 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:56:13.361 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:56:13.361 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000065", 2024-08-18T15:56:13.361 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:56:13.361 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:56:13.361 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:56:13.361 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:56:13.361 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000083", 2024-08-18T15:56:13.361 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000010", 2024-08-18T15:56:13.361 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:56:13.361 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:56:13.361 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:56:13.361 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:56:13.362 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:56:13.362 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:56:13.362 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:56:13.362 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:56:13.362 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:56:13.362 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:56:13.362 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:56:13.362 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.7", 2024-08-18T15:56:13.362 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.3", 2024-08-18T15:56:13.362 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:56:13.362 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:56:13.362 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:56:13.362 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:56:13.363 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:56:13.363 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:56:13.363 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:56:13.363 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:56:13.363 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.2", 2024-08-18T15:56:13.363 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:56:13.363 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:56:13.363 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:56:13.363 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:56:13.363 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:56:13.363 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:56:13.363 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:56:13.363 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000045", 2024-08-18T15:56:13.363 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000002", 2024-08-18T15:56:13.364 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:56:13.364 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3.2UbiLidwSTy.0", 2024-08-18T15:56:13.364 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:56:13.364 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.12", 2024-08-18T15:56:13.364 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:56:13.364 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:56:13.364 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:56:13.364 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:56:13.364 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:56:13.364 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:56:13.364 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000116", 2024-08-18T15:56:13.364 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.25", 2024-08-18T15:56:13.364 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:56:13.364 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:56:13.365 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:56:13.365 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:56:13.365 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000034", 2024-08-18T15:56:13.365 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:56:13.365 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:56:13.365 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000101", 2024-08-18T15:56:13.365 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:56:13.365 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:56:13.365 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.13", 2024-08-18T15:56:13.365 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:56:13.365 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:56:13.365 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.1", 2024-08-18T15:56:13.365 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0.retry", 2024-08-18T15:56:13.365 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:56:13.366 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:56:13.366 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000079", 2024-08-18T15:56:13.366 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000049", 2024-08-18T15:56:13.366 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:56:13.366 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0", 2024-08-18T15:56:13.366 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:56:13.366 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:56:13.366 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000117", 2024-08-18T15:56:13.366 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:56:13.366 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000044", 2024-08-18T15:56:13.366 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:56:13.366 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:56:13.366 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:56:13.367 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:56:13.367 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:56:13.367 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:56:13.367 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.14", 2024-08-18T15:56:13.367 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:56:13.367 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000066", 2024-08-18T15:56:13.367 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:56:13.367 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:56:13.368 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:56:13.368 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:56:13.368 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:56:13.368 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000068", 2024-08-18T15:56:13.368 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:56:13.368 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000085", 2024-08-18T15:56:13.368 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:56:13.368 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:56:13.368 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:56:13.368 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:56:13.368 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:56:13.368 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:56:13.368 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:56:13.369 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:56:13.369 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:56:13.369 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:56:13.369 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:56:13.369 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:56:13.369 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000073", 2024-08-18T15:56:13.369 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:56:13.369 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:56:13.369 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:56:13.369 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000038", 2024-08-18T15:56:13.369 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:56:13.369 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:56:13.369 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:56:13.369 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:56:13.370 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.10", 2024-08-18T15:56:13.370 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:56:13.370 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:56:13.370 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:56:13.370 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000118", 2024-08-18T15:56:13.370 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:56:13.370 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3", 2024-08-18T15:56:13.370 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:56:13.370 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000036", 2024-08-18T15:56:13.370 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:56:13.370 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:56:13.370 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:56:13.370 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:56:13.371 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:56:13.371 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000103", 2024-08-18T15:56:13.371 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.3", 2024-08-18T15:56:13.371 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:56:13.371 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000119", 2024-08-18T15:56:13.371 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000098", 2024-08-18T15:56:13.371 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:56:13.371 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:56:13.371 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:56:13.371 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:56:13.371 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:56:13.371 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:56:13.371 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:56:13.371 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:56:13.372 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.0", 2024-08-18T15:56:13.372 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:56:13.372 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:56:13.372 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:56:13.372 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:56:13.372 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:56:13.372 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:56:13.372 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:56:13.372 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:56:13.372 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000027", 2024-08-18T15:56:13.372 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:56:13.372 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status", 2024-08-18T15:56:13.372 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:56:13.372 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000019", 2024-08-18T15:56:13.373 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:56:13.373 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:56:13.373 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:56:13.373 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:56:13.373 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1", 2024-08-18T15:56:13.373 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:56:13.373 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:56:13.373 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:56:13.373 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:56:13.373 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000039", 2024-08-18T15:56:13.373 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:56:13.373 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:56:13.373 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:56:13.374 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:56:13.374 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:56:13.374 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:56:13.374 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:56:13.374 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:56:13.374 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:56:13.374 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:56:13.374 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:56:13.374 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000100", 2024-08-18T15:56:13.374 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:56:13.374 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:56:13.374 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:56:13.374 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:56:13.374 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:56:13.375 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000093", 2024-08-18T15:56:13.375 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:56:13.375 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000004", 2024-08-18T15:56:13.375 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000063", 2024-08-18T15:56:13.375 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000122", 2024-08-18T15:56:13.375 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:56:13.375 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:56:13.375 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:56:13.375 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000057", 2024-08-18T15:56:13.375 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:56:13.375 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:56:13.375 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:56:13.375 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:56:13.375 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000054", 2024-08-18T15:56:13.376 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:56:13.376 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000114", 2024-08-18T15:56:13.376 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:56:13.376 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:56:13.376 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:56:13.376 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.2", 2024-08-18T15:56:13.376 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:56:13.376 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:56:13.376 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0", 2024-08-18T15:56:13.376 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:56:13.376 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:56:13.376 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:56:13.376 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:56:13.377 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:56:13.377 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:56:13.377 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000012", 2024-08-18T15:56:13.377 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:56:13.377 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:56:13.377 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:56:13.377 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000084", 2024-08-18T15:56:13.377 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000043", 2024-08-18T15:56:13.377 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3", 2024-08-18T15:56:13.377 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:56:13.377 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:56:13.377 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.11", 2024-08-18T15:56:13.377 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:56:13.377 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:56:13.378 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:56:13.378 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:56:13.378 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:56:13.378 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:56:13.378 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:56:13.378 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:56:13.378 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000111", 2024-08-18T15:56:13.378 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:56:13.378 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000048", 2024-08-18T15:56:13.378 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:56:13.378 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000020", 2024-08-18T15:56:13.378 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:56:13.379 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:56:13.379 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:56:13.379 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:56:13.379 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:56:13.379 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000099", 2024-08-18T15:56:13.379 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000056", 2024-08-18T15:56:13.379 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:56:13.379 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:56:13.379 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.19", 2024-08-18T15:56:13.379 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:56:13.379 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.28", 2024-08-18T15:56:13.379 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:56:13.379 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:56:13.379 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:56:13.380 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:56:13.380 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:56:13.380 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000072", 2024-08-18T15:56:13.380 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:56:13.380 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:56:13.380 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000062", 2024-08-18T15:56:13.380 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:56:13.380 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:56:13.380 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:56:13.380 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000115", 2024-08-18T15:56:13.380 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:56:13.380 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.1", 2024-08-18T15:56:13.380 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2", 2024-08-18T15:56:13.380 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000033", 2024-08-18T15:56:13.381 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:56:13.381 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:56:13.381 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:56:13.381 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:56:13.381 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:56:13.381 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:56:13.381 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:56:13.381 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:56:13.381 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:56:13.381 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:56:13.381 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:56:13.381 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:56:13.381 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:56:13.382 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:56:13.382 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:56:13.382 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:56:13.382 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:56:13.382 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:56:13.382 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:56:13.382 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000025", 2024-08-18T15:56:13.382 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:56:13.382 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000123", 2024-08-18T15:56:13.382 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.31", 2024-08-18T15:56:13.382 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:56:13.382 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:56:13.382 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:56:13.382 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:56:13.383 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.18", 2024-08-18T15:56:13.383 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:56:13.383 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:56:13.383 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:56:13.383 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:56:13.383 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:56:13.383 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000108", 2024-08-18T15:56:13.383 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:56:13.383 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:56:13.383 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000094", 2024-08-18T15:56:13.383 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:56:13.383 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:56:13.383 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:56:13.384 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:56:13.384 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:56:13.384 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:56:13.384 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000060", 2024-08-18T15:56:13.384 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:56:13.384 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:56:13.384 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:56:13.384 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000109", 2024-08-18T15:56:13.384 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.15", 2024-08-18T15:56:13.384 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:56:13.384 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:56:13.384 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:56:13.384 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:56:13.384 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:56:13.385 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:56:13.385 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:56:13.385 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:56:13.385 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:56:13.385 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000013", 2024-08-18T15:56:13.385 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:56:13.385 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:56:13.385 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:56:13.385 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:56:13.385 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:56:13.385 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:56:13.385 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:56:13.385 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:56:13.386 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:56:13.386 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:56:13.386 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:56:13.386 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:56:13.386 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:56:13.386 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:56:13.386 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000051", 2024-08-18T15:56:13.386 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:56:13.386 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:56:13.386 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:56:13.386 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:56:13.386 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000106", 2024-08-18T15:56:13.386 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:56:13.386 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:56:13.387 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:56:13.387 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:56:13.387 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:56:13.387 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000017", 2024-08-18T15:56:13.387 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:56:13.387 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:56:13.387 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:56:13.387 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000023", 2024-08-18T15:56:13.387 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:56:13.388 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:56:13.388 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:56:13.388 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:56:13.388 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:56:13.388 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:56:13.388 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:56:13.388 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000107", 2024-08-18T15:56:13.388 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:56:13.388 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:56:13.388 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000046", 2024-08-18T15:56:13.388 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:56:13.388 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:56:13.388 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:56:13.388 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.26", 2024-08-18T15:56:13.389 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:56:13.389 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:56:13.389 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9", 2024-08-18T15:56:13.389 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.22", 2024-08-18T15:56:13.389 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:56:13.389 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:56:13.389 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:56:13.389 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:56:13.389 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:56:13.389 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:56:13.389 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:56:13.389 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2", 2024-08-18T15:56:13.389 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:56:13.390 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:56:13.390 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000067", 2024-08-18T15:56:13.390 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:56:13.390 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:56:13.390 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:56:13.390 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:56:13.390 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:56:13.390 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:56:13.390 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:56:13.390 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:56:13.390 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:56:13.390 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:56:13.390 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:56:13.391 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:56:13.391 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:56:13.391 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:56:13.391 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:56:13.391 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:56:13.391 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:56:13.391 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:56:13.391 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:56:13.391 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.9", 2024-08-18T15:56:13.391 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000086", 2024-08-18T15:56:13.391 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.5", 2024-08-18T15:56:13.391 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:56:13.391 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:56:13.391 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:56:13.392 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:56:13.392 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:56:13.392 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:56:13.392 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2.MFXJ583gCr1.0", 2024-08-18T15:56:13.392 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:56:13.392 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1.GPnO16liyZ+.0", 2024-08-18T15:56:13.392 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:56:13.392 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:56:13.392 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:56:13.392 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:56:13.392 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:56:13.392 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:56:13.392 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:56:13.392 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:56:13.393 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:56:13.393 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.20", 2024-08-18T15:56:13.393 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:56:13.393 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:56:13.393 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:56:13.393 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:56:13.393 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:56:13.393 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:56:13.393 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1.retry", 2024-08-18T15:56:13.393 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:56:13.393 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:56:13.393 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:56:13.393 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0.retry", 2024-08-18T15:56:13.394 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000050", 2024-08-18T15:56:13.394 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:56:13.394 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000102", 2024-08-18T15:56:13.394 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:56:13.394 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:56:13.394 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:56:13.394 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:56:13.394 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:56:13.394 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:56:13.394 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:56:13.394 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:56:13.394 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:56:13.394 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:56:13.395 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:56:13.395 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000124", 2024-08-18T15:56:13.395 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:56:13.395 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:56:13.395 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:56:13.395 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000001", 2024-08-18T15:56:13.395 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:56:13.395 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:56:13.395 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:56:13.395 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:56:13.395 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000126", 2024-08-18T15:56:13.395 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:56:13.395 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0.fZhctfLXnSt.0", 2024-08-18T15:56:13.395 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:56:13.396 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000055", 2024-08-18T15:56:13.396 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:56:13.396 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000061", 2024-08-18T15:56:13.396 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1", 2024-08-18T15:56:13.396 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:56:13.396 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:56:13.396 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:56:13.396 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000097", 2024-08-18T15:56:13.396 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:56:13.396 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:56:13.396 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:56:13.396 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:56:13.396 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:56:13.396 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:56:13.397 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:56:13.397 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:56:13.397 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000042", 2024-08-18T15:56:13.397 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:56:13.397 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:56:13.397 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:56:13.397 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1.retry", 2024-08-18T15:56:13.397 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:56:13.397 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:56:13.397 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:56:13.397 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:56:13.397 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:56:13.397 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:56:13.398 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:56:13.398 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000075", 2024-08-18T15:56:13.398 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:56:13.398 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000120", 2024-08-18T15:56:13.398 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000030", 2024-08-18T15:56:13.398 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:56:13.398 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:56:13.398 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:56:13.398 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:56:13.398 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:56:13.398 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:56:13.399 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9" 2024-08-18T15:56:13.399 INFO:teuthology.orchestra.run.smithi193.stdout:] 2024-08-18T15:56:13.399 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:56:18.400 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:56:18.401 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:56:18.401 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:56:18.453 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:56:18.453 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:56:18.518 INFO:teuthology.orchestra.run.smithi193.stdout:[ 2024-08-18T15:56:18.518 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:56:18.518 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:56:18.518 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.29", 2024-08-18T15:56:18.518 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000078", 2024-08-18T15:56:18.518 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.history", 2024-08-18T15:56:18.518 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:56:18.518 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:56:18.518 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:56:18.519 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:56:18.519 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.23", 2024-08-18T15:56:18.519 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000070", 2024-08-18T15:56:18.519 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000104", 2024-08-18T15:56:18.519 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:56:18.519 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2.retry", 2024-08-18T15:56:18.519 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:56:18.519 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:56:18.519 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:56:18.519 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:56:18.519 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000026", 2024-08-18T15:56:18.519 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:56:18.519 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:56:18.519 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:56:18.520 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:56:18.520 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:56:18.520 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:56:18.520 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.8", 2024-08-18T15:56:18.520 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:56:18.520 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:56:18.520 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:56:18.520 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:56:18.520 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000028", 2024-08-18T15:56:18.520 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:56:18.520 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:56:18.520 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:56:18.520 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:56:18.520 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:56:18.521 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:56:18.521 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:56:18.521 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2.retry", 2024-08-18T15:56:18.521 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:56:18.521 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.30", 2024-08-18T15:56:18.521 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:56:18.521 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:56:18.521 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:56:18.521 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:56:18.521 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000040", 2024-08-18T15:56:18.521 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:56:18.521 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.6", 2024-08-18T15:56:18.521 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:56:18.522 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:56:18.522 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:56:18.522 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:56:18.522 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:56:18.522 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:56:18.522 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000015", 2024-08-18T15:56:18.522 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:56:18.522 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:56:18.522 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000069", 2024-08-18T15:56:18.522 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:56:18.522 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:56:18.522 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:56:18.522 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000095", 2024-08-18T15:56:18.523 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0", 2024-08-18T15:56:18.523 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000003", 2024-08-18T15:56:18.523 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000047", 2024-08-18T15:56:18.523 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:56:18.523 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:56:18.523 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:56:18.523 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:56:18.523 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:56:18.523 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:56:18.523 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:56:18.523 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:56:18.523 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:56:18.523 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000035", 2024-08-18T15:56:18.524 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:56:18.524 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:56:18.524 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000037", 2024-08-18T15:56:18.524 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:56:18.524 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:56:18.524 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:56:18.524 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:56:18.524 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:56:18.524 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:56:18.524 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:56:18.524 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:56:18.524 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:56:18.525 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:56:18.525 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:56:18.525 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:56:18.525 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:56:18.525 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:56:18.525 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000024", 2024-08-18T15:56:18.525 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:56:18.525 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:56:18.525 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:56:18.525 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:56:18.525 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000000", 2024-08-18T15:56:18.525 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:56:18.525 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000031", 2024-08-18T15:56:18.526 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:56:18.526 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:56:18.526 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:56:18.526 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:56:18.526 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:56:18.526 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:56:18.526 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3", 2024-08-18T15:56:18.526 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:56:18.526 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:56:18.526 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:56:18.526 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:56:18.526 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:56:18.527 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:56:18.527 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:56:18.527 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:56:18.527 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:56:18.527 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:56:18.527 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:56:18.527 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:56:18.527 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:56:18.527 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:56:18.527 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:56:18.527 INFO:teuthology.orchestra.run.smithi193.stdout: "data_loggenerations_metadata", 2024-08-18T15:56:18.527 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2", 2024-08-18T15:56:18.527 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:56:18.528 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:56:18.528 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:56:18.528 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:56:18.528 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T15:56:18.528 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:56:18.528 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:56:18.528 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:56:18.528 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:56:18.528 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:56:18.528 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000076", 2024-08-18T15:56:18.528 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:56:18.528 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:56:18.529 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000113", 2024-08-18T15:56:18.529 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000005", 2024-08-18T15:56:18.529 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:56:18.529 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000011", 2024-08-18T15:56:18.529 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:56:18.529 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000058", 2024-08-18T15:56:18.529 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:56:18.529 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:56:18.529 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:56:18.529 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000052", 2024-08-18T15:56:18.529 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:56:18.529 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:56:18.529 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:56:18.529 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:56:18.530 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.21", 2024-08-18T15:56:18.530 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:56:18.530 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:56:18.530 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:56:18.531 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:56:18.531 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:56:18.531 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:56:18.531 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:56:18.531 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:56:18.531 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:56:18.531 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:56:18.531 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:56:18.531 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:56:18.531 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:56:18.531 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:56:18.532 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:56:18.532 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:56:18.532 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:56:18.532 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:56:18.532 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:56:18.532 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000088", 2024-08-18T15:56:18.532 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000080", 2024-08-18T15:56:18.532 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:56:18.532 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:56:18.532 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000090", 2024-08-18T15:56:18.532 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:56:18.532 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:56:18.532 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000110", 2024-08-18T15:56:18.533 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:56:18.533 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:56:18.533 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:56:18.533 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.0", 2024-08-18T15:56:18.533 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:56:18.533 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000096", 2024-08-18T15:56:18.533 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:56:18.533 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:56:18.533 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:56:18.533 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:56:18.533 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:56:18.533 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000087", 2024-08-18T15:56:18.533 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:56:18.533 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:56:18.534 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:56:18.534 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:56:18.534 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000008", 2024-08-18T15:56:18.534 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:56:18.534 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:56:18.534 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:56:18.534 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:56:18.534 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:56:18.534 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000006", 2024-08-18T15:56:18.534 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000029", 2024-08-18T15:56:18.534 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:56:18.534 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000089", 2024-08-18T15:56:18.534 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:56:18.535 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:56:18.535 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000022", 2024-08-18T15:56:18.535 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:56:18.535 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:56:18.535 INFO:teuthology.orchestra.run.smithi193.stdout: "bilog.trim", 2024-08-18T15:56:18.535 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000016", 2024-08-18T15:56:18.535 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:56:18.535 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:56:18.535 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:56:18.535 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:56:18.535 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:56:18.535 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000041", 2024-08-18T15:56:18.535 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:56:18.536 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:56:18.536 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.log.01050e30-179b-43be-bdbb-3bb23cf37ae8.3", 2024-08-18T15:56:18.536 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:56:18.536 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000018", 2024-08-18T15:56:18.536 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000092", 2024-08-18T15:56:18.536 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:56:18.536 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:56:18.536 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:56:18.536 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:56:18.536 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.27", 2024-08-18T15:56:18.536 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.4", 2024-08-18T15:56:18.536 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:56:18.536 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:56:18.536 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:56:18.536 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:56:18.537 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:56:18.537 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:56:18.537 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:56:18.537 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000014", 2024-08-18T15:56:18.537 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:56:18.537 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3.retry", 2024-08-18T15:56:18.537 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:56:18.537 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:56:18.537 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:56:18.537 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:56:18.537 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:56:18.537 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000112", 2024-08-18T15:56:18.538 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:56:18.538 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000007", 2024-08-18T15:56:18.538 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000021", 2024-08-18T15:56:18.538 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:56:18.538 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:56:18.538 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:56:18.538 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:56:18.538 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:56:18.538 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.16", 2024-08-18T15:56:18.538 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:56:18.538 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000064", 2024-08-18T15:56:18.538 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:56:18.539 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:56:18.539 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:56:18.539 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:56:18.539 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:56:18.539 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:56:18.539 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:56:18.539 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000071", 2024-08-18T15:56:18.539 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:56:18.539 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:56:18.539 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:56:18.539 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000074", 2024-08-18T15:56:18.539 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:56:18.539 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:56:18.539 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:56:18.540 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:56:18.540 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:56:18.540 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:56:18.540 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:56:18.540 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:56:18.540 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:56:18.540 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:56:18.540 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:56:18.540 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:56:18.540 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:56:18.540 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:56:18.540 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:56:18.540 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000081", 2024-08-18T15:56:18.540 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:56:18.540 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000009", 2024-08-18T15:56:18.541 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000121", 2024-08-18T15:56:18.541 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1", 2024-08-18T15:56:18.541 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:56:18.541 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:56:18.541 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:56:18.541 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:56:18.541 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:56:18.541 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:56:18.541 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.17", 2024-08-18T15:56:18.541 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000125", 2024-08-18T15:56:18.541 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:56:18.541 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000082", 2024-08-18T15:56:18.541 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:56:18.542 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:56:18.542 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:56:18.542 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:56:18.542 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:56:18.542 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:56:18.542 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:56:18.542 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:56:18.542 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:56:18.542 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:56:18.542 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:56:18.542 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000105", 2024-08-18T15:56:18.542 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000059", 2024-08-18T15:56:18.542 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:56:18.542 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:56:18.543 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:56:18.543 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:56:18.543 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:56:18.543 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:56:18.543 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:56:18.543 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:56:18.543 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:56:18.543 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:56:18.543 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:56:18.543 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000077", 2024-08-18T15:56:18.543 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000032", 2024-08-18T15:56:18.543 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:56:18.543 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:56:18.543 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:56:18.543 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:56:18.544 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:56:18.544 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:56:18.544 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.24", 2024-08-18T15:56:18.544 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:56:18.544 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:56:18.544 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:56:18.544 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:56:18.544 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3.retry", 2024-08-18T15:56:18.544 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:56:18.544 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:56:18.544 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:56:18.544 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:56:18.544 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:56:18.545 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:56:18.545 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000053", 2024-08-18T15:56:18.545 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000091", 2024-08-18T15:56:18.545 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:56:18.545 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:56:18.545 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:56:18.545 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000065", 2024-08-18T15:56:18.545 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:56:18.545 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:56:18.545 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:56:18.545 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:56:18.545 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000083", 2024-08-18T15:56:18.545 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000010", 2024-08-18T15:56:18.545 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:56:18.546 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:56:18.546 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:56:18.546 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:56:18.546 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:56:18.546 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:56:18.546 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:56:18.546 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:56:18.546 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:56:18.546 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:56:18.546 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:56:18.546 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.7", 2024-08-18T15:56:18.546 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.3", 2024-08-18T15:56:18.547 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:56:18.547 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:56:18.547 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:56:18.547 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:56:18.547 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:56:18.547 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:56:18.547 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:56:18.547 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:56:18.547 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.2", 2024-08-18T15:56:18.547 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:56:18.547 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:56:18.547 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:56:18.547 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:56:18.547 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:56:18.547 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:56:18.548 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:56:18.548 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000045", 2024-08-18T15:56:18.548 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000002", 2024-08-18T15:56:18.548 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:56:18.548 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3.2UbiLidwSTy.0", 2024-08-18T15:56:18.548 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:56:18.548 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.12", 2024-08-18T15:56:18.548 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:56:18.548 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:56:18.548 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:56:18.548 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:56:18.548 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:56:18.548 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:56:18.549 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000116", 2024-08-18T15:56:18.549 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.25", 2024-08-18T15:56:18.549 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:56:18.549 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:56:18.549 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:56:18.549 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:56:18.549 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000034", 2024-08-18T15:56:18.549 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:56:18.549 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:56:18.549 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000101", 2024-08-18T15:56:18.549 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:56:18.549 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:56:18.549 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.13", 2024-08-18T15:56:18.549 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:56:18.550 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:56:18.550 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.1", 2024-08-18T15:56:18.550 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0.retry", 2024-08-18T15:56:18.550 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:56:18.550 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:56:18.550 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000079", 2024-08-18T15:56:18.550 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000049", 2024-08-18T15:56:18.550 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:56:18.550 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0", 2024-08-18T15:56:18.550 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:56:18.550 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:56:18.550 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000117", 2024-08-18T15:56:18.550 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:56:18.550 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000044", 2024-08-18T15:56:18.551 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:56:18.551 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:56:18.551 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:56:18.551 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:56:18.551 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:56:18.551 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:56:18.551 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.14", 2024-08-18T15:56:18.551 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:56:18.551 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000066", 2024-08-18T15:56:18.551 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:56:18.551 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:56:18.551 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:56:18.551 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:56:18.552 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:56:18.552 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000068", 2024-08-18T15:56:18.552 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:56:18.552 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000085", 2024-08-18T15:56:18.552 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:56:18.552 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:56:18.552 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:56:18.552 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:56:18.552 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:56:18.552 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:56:18.552 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:56:18.553 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:56:18.553 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:56:18.553 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:56:18.553 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:56:18.553 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:56:18.553 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000073", 2024-08-18T15:56:18.553 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:56:18.553 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:56:18.553 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:56:18.553 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000038", 2024-08-18T15:56:18.553 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:56:18.553 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:56:18.553 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:56:18.553 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:56:18.554 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.10", 2024-08-18T15:56:18.554 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:56:18.554 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:56:18.554 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:56:18.554 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000118", 2024-08-18T15:56:18.554 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:56:18.554 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3", 2024-08-18T15:56:18.554 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:56:18.554 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000036", 2024-08-18T15:56:18.554 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:56:18.554 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:56:18.554 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:56:18.555 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:56:18.555 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:56:18.555 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000103", 2024-08-18T15:56:18.555 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.3", 2024-08-18T15:56:18.555 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:56:18.555 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000119", 2024-08-18T15:56:18.555 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000098", 2024-08-18T15:56:18.555 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:56:18.555 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:56:18.555 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:56:18.555 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:56:18.555 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:56:18.555 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:56:18.555 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:56:18.555 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:56:18.556 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.0", 2024-08-18T15:56:18.556 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:56:18.556 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:56:18.556 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:56:18.556 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:56:18.556 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:56:18.556 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:56:18.556 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:56:18.556 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:56:18.556 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000027", 2024-08-18T15:56:18.556 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:56:18.556 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status", 2024-08-18T15:56:18.557 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:56:18.557 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000019", 2024-08-18T15:56:18.557 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:56:18.557 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:56:18.557 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:56:18.557 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:56:18.557 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1", 2024-08-18T15:56:18.557 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:56:18.557 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:56:18.557 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:56:18.557 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:56:18.557 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000039", 2024-08-18T15:56:18.557 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:56:18.558 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:56:18.558 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:56:18.558 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:56:18.558 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:56:18.558 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:56:18.558 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:56:18.558 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:56:18.558 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:56:18.558 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:56:18.558 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:56:18.558 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000100", 2024-08-18T15:56:18.558 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:56:18.559 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:56:18.559 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:56:18.559 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:56:18.559 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:56:18.559 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000093", 2024-08-18T15:56:18.559 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:56:18.559 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000004", 2024-08-18T15:56:18.559 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000063", 2024-08-18T15:56:18.559 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000122", 2024-08-18T15:56:18.559 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:56:18.559 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:56:18.559 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:56:18.559 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000057", 2024-08-18T15:56:18.560 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:56:18.560 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:56:18.560 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:56:18.560 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:56:18.560 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000054", 2024-08-18T15:56:18.560 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:56:18.560 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000114", 2024-08-18T15:56:18.560 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:56:18.560 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:56:18.560 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:56:18.560 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.2", 2024-08-18T15:56:18.560 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:56:18.560 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:56:18.561 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0", 2024-08-18T15:56:18.561 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:56:18.561 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:56:18.561 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:56:18.561 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:56:18.561 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:56:18.561 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:56:18.561 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000012", 2024-08-18T15:56:18.561 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:56:18.561 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:56:18.561 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:56:18.561 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000084", 2024-08-18T15:56:18.561 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000043", 2024-08-18T15:56:18.561 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3", 2024-08-18T15:56:18.562 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:56:18.562 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:56:18.562 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.11", 2024-08-18T15:56:18.562 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:56:18.562 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:56:18.562 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:56:18.562 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:56:18.562 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:56:18.562 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:56:18.562 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:56:18.562 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:56:18.562 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000111", 2024-08-18T15:56:18.563 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:56:18.563 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000048", 2024-08-18T15:56:18.563 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:56:18.563 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000020", 2024-08-18T15:56:18.563 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:56:18.563 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:56:18.563 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:56:18.563 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:56:18.563 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:56:18.563 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000099", 2024-08-18T15:56:18.563 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000056", 2024-08-18T15:56:18.563 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:56:18.563 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:56:18.564 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.19", 2024-08-18T15:56:18.564 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:56:18.564 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.28", 2024-08-18T15:56:18.564 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:56:18.564 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:56:18.564 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:56:18.564 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:56:18.564 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:56:18.564 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000072", 2024-08-18T15:56:18.564 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:56:18.564 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:56:18.564 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000062", 2024-08-18T15:56:18.564 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:56:18.565 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:56:18.565 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:56:18.565 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000115", 2024-08-18T15:56:18.565 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:56:18.565 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.1", 2024-08-18T15:56:18.565 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2", 2024-08-18T15:56:18.565 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000033", 2024-08-18T15:56:18.565 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:56:18.565 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:56:18.565 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:56:18.565 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:56:18.565 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:56:18.565 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:56:18.566 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:56:18.566 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:56:18.566 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:56:18.566 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:56:18.566 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:56:18.566 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:56:18.566 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:56:18.566 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:56:18.566 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:56:18.566 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:56:18.566 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:56:18.566 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:56:18.566 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:56:18.567 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000025", 2024-08-18T15:56:18.567 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:56:18.567 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000123", 2024-08-18T15:56:18.567 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.31", 2024-08-18T15:56:18.567 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:56:18.567 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:56:18.567 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:56:18.567 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:56:18.567 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.18", 2024-08-18T15:56:18.567 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:56:18.567 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:56:18.567 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:56:18.567 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:56:18.567 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:56:18.568 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000108", 2024-08-18T15:56:18.568 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:56:18.568 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:56:18.568 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000094", 2024-08-18T15:56:18.568 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:56:18.568 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:56:18.568 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:56:18.568 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:56:18.568 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:56:18.568 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:56:18.569 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000060", 2024-08-18T15:56:18.569 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:56:18.569 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:56:18.569 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:56:18.569 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000109", 2024-08-18T15:56:18.569 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.15", 2024-08-18T15:56:18.569 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:56:18.569 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:56:18.569 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:56:18.569 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:56:18.569 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:56:18.569 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:56:18.569 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:56:18.570 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:56:18.570 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:56:18.570 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000013", 2024-08-18T15:56:18.570 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:56:18.570 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:56:18.570 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:56:18.570 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:56:18.570 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:56:18.570 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:56:18.570 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:56:18.570 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:56:18.570 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:56:18.570 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:56:18.570 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:56:18.571 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:56:18.571 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:56:18.571 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:56:18.571 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000051", 2024-08-18T15:56:18.571 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:56:18.571 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:56:18.571 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:56:18.571 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:56:18.571 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000106", 2024-08-18T15:56:18.571 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:56:18.571 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:56:18.571 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:56:18.572 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:56:18.572 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:56:18.572 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000017", 2024-08-18T15:56:18.572 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:56:18.572 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:56:18.572 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:56:18.572 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000023", 2024-08-18T15:56:18.572 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:56:18.572 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:56:18.572 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:56:18.572 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:56:18.572 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:56:18.572 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:56:18.572 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:56:18.573 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000107", 2024-08-18T15:56:18.573 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:56:18.573 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:56:18.573 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000046", 2024-08-18T15:56:18.573 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:56:18.573 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:56:18.573 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:56:18.573 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.26", 2024-08-18T15:56:18.573 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:56:18.573 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:56:18.573 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9", 2024-08-18T15:56:18.573 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.22", 2024-08-18T15:56:18.573 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:56:18.573 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:56:18.573 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:56:18.574 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:56:18.574 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:56:18.574 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:56:18.574 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:56:18.574 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2", 2024-08-18T15:56:18.574 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:56:18.574 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:56:18.574 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000067", 2024-08-18T15:56:18.574 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:56:18.574 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:56:18.574 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:56:18.574 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:56:18.575 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:56:18.575 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:56:18.575 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:56:18.575 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:56:18.575 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:56:18.575 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:56:18.575 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:56:18.575 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:56:18.575 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:56:18.575 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:56:18.575 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:56:18.575 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:56:18.575 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:56:18.575 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:56:18.576 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:56:18.576 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.9", 2024-08-18T15:56:18.576 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000086", 2024-08-18T15:56:18.576 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.5", 2024-08-18T15:56:18.576 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:56:18.576 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:56:18.576 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:56:18.576 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:56:18.576 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:56:18.576 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:56:18.576 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2.MFXJ583gCr1.0", 2024-08-18T15:56:18.576 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:56:18.576 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1.GPnO16liyZ+.0", 2024-08-18T15:56:18.576 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:56:18.577 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:56:18.577 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:56:18.577 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:56:18.577 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:56:18.577 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:56:18.577 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:56:18.577 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:56:18.577 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:56:18.577 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.20", 2024-08-18T15:56:18.577 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:56:18.577 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:56:18.577 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:56:18.577 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:56:18.578 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:56:18.578 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:56:18.578 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1.retry", 2024-08-18T15:56:18.578 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:56:18.578 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:56:18.578 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:56:18.578 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0.retry", 2024-08-18T15:56:18.578 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000050", 2024-08-18T15:56:18.578 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:56:18.578 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000102", 2024-08-18T15:56:18.578 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:56:18.578 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:56:18.579 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:56:18.579 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:56:18.579 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:56:18.579 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:56:18.579 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:56:18.579 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:56:18.579 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:56:18.579 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:56:18.579 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:56:18.579 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000124", 2024-08-18T15:56:18.579 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:56:18.579 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:56:18.579 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:56:18.579 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000001", 2024-08-18T15:56:18.580 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:56:18.580 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:56:18.580 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:56:18.580 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:56:18.580 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000126", 2024-08-18T15:56:18.580 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:56:18.580 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0.fZhctfLXnSt.0", 2024-08-18T15:56:18.580 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:56:18.580 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000055", 2024-08-18T15:56:18.580 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:56:18.580 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000061", 2024-08-18T15:56:18.580 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1", 2024-08-18T15:56:18.580 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:56:18.580 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:56:18.581 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:56:18.581 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000097", 2024-08-18T15:56:18.581 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:56:18.581 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:56:18.581 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:56:18.581 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:56:18.581 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:56:18.581 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:56:18.581 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:56:18.581 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:56:18.581 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000042", 2024-08-18T15:56:18.581 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:56:18.581 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:56:18.581 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:56:18.582 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1.retry", 2024-08-18T15:56:18.582 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:56:18.582 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:56:18.582 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:56:18.582 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:56:18.582 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:56:18.582 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:56:18.582 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:56:18.582 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000075", 2024-08-18T15:56:18.582 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:56:18.582 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000120", 2024-08-18T15:56:18.582 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000030", 2024-08-18T15:56:18.583 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:56:18.583 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:56:18.583 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:56:18.583 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:56:18.583 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:56:18.583 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:56:18.583 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9" 2024-08-18T15:56:18.583 INFO:teuthology.orchestra.run.smithi193.stdout:] 2024-08-18T15:56:18.583 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:56:23.585 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:56:23.585 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:56:23.585 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:56:23.640 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:56:23.640 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:56:23.704 INFO:teuthology.orchestra.run.smithi193.stdout:[ 2024-08-18T15:56:23.704 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:56:23.705 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:56:23.705 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.29", 2024-08-18T15:56:23.705 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000078", 2024-08-18T15:56:23.705 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.history", 2024-08-18T15:56:23.705 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:56:23.705 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:56:23.705 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:56:23.705 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:56:23.705 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.23", 2024-08-18T15:56:23.705 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000070", 2024-08-18T15:56:23.705 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000104", 2024-08-18T15:56:23.705 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:56:23.706 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2.retry", 2024-08-18T15:56:23.706 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:56:23.706 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:56:23.706 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:56:23.706 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:56:23.706 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000026", 2024-08-18T15:56:23.706 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:56:23.706 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:56:23.706 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:56:23.706 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:56:23.706 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:56:23.706 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:56:23.707 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.8", 2024-08-18T15:56:23.707 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:56:23.707 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:56:23.707 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:56:23.707 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:56:23.707 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000028", 2024-08-18T15:56:23.707 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:56:23.707 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:56:23.707 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:56:23.707 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:56:23.707 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:56:23.707 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:56:23.707 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:56:23.708 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2.retry", 2024-08-18T15:56:23.708 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:56:23.708 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.30", 2024-08-18T15:56:23.708 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:56:23.708 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:56:23.708 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:56:23.708 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:56:23.708 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000040", 2024-08-18T15:56:23.708 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:56:23.708 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.6", 2024-08-18T15:56:23.708 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:56:23.708 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:56:23.708 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:56:23.708 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:56:23.709 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:56:23.709 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:56:23.709 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000015", 2024-08-18T15:56:23.709 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:56:23.709 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:56:23.709 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000069", 2024-08-18T15:56:23.709 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:56:23.709 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:56:23.709 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:56:23.709 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000095", 2024-08-18T15:56:23.709 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0", 2024-08-18T15:56:23.709 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000003", 2024-08-18T15:56:23.709 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000047", 2024-08-18T15:56:23.709 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:56:23.710 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:56:23.710 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:56:23.710 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:56:23.710 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:56:23.710 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:56:23.710 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:56:23.710 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:56:23.710 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:56:23.710 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000035", 2024-08-18T15:56:23.710 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:56:23.710 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:56:23.710 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000037", 2024-08-18T15:56:23.710 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:56:23.711 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:56:23.711 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:56:23.711 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:56:23.711 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:56:23.711 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:56:23.711 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:56:23.711 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:56:23.711 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:56:23.711 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:56:23.711 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:56:23.711 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:56:23.711 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:56:23.712 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:56:23.712 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000024", 2024-08-18T15:56:23.712 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:56:23.712 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:56:23.712 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:56:23.712 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:56:23.712 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000000", 2024-08-18T15:56:23.712 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:56:23.712 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000031", 2024-08-18T15:56:23.712 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:56:23.712 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:56:23.712 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:56:23.712 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:56:23.712 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:56:23.713 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:56:23.713 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3", 2024-08-18T15:56:23.713 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:56:23.713 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:56:23.713 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:56:23.713 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:56:23.713 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:56:23.713 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:56:23.713 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:56:23.713 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:56:23.713 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:56:23.713 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:56:23.713 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:56:23.713 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:56:23.714 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:56:23.714 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:56:23.714 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:56:23.714 INFO:teuthology.orchestra.run.smithi193.stdout: "data_loggenerations_metadata", 2024-08-18T15:56:23.714 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2", 2024-08-18T15:56:23.714 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:56:23.714 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:56:23.714 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:56:23.714 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:56:23.714 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T15:56:23.714 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:56:23.714 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:56:23.714 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:56:23.715 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:56:23.715 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:56:23.715 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000076", 2024-08-18T15:56:23.715 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:56:23.715 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:56:23.715 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000113", 2024-08-18T15:56:23.715 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000005", 2024-08-18T15:56:23.715 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:56:23.715 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000011", 2024-08-18T15:56:23.715 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:56:23.715 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000058", 2024-08-18T15:56:23.715 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:56:23.715 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:56:23.715 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:56:23.716 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000052", 2024-08-18T15:56:23.716 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:56:23.716 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:56:23.716 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:56:23.716 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:56:23.716 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.21", 2024-08-18T15:56:23.716 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:56:23.716 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:56:23.716 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:56:23.716 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:56:23.716 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:56:23.716 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:56:23.716 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:56:23.717 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:56:23.717 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:56:23.717 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:56:23.717 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:56:23.717 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:56:23.717 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:56:23.717 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:56:23.717 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:56:23.717 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:56:23.717 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:56:23.717 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:56:23.717 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:56:23.717 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000088", 2024-08-18T15:56:23.717 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000080", 2024-08-18T15:56:23.718 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:56:23.718 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:56:23.718 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000090", 2024-08-18T15:56:23.718 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:56:23.718 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:56:23.718 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000110", 2024-08-18T15:56:23.718 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:56:23.718 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:56:23.718 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:56:23.718 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.0", 2024-08-18T15:56:23.718 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:56:23.718 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000096", 2024-08-18T15:56:23.718 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:56:23.719 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:56:23.719 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:56:23.719 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:56:23.719 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:56:23.719 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000087", 2024-08-18T15:56:23.719 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:56:23.719 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:56:23.719 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:56:23.719 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:56:23.719 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000008", 2024-08-18T15:56:23.719 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:56:23.719 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:56:23.720 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:56:23.720 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:56:23.720 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:56:23.720 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000006", 2024-08-18T15:56:23.720 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000029", 2024-08-18T15:56:23.720 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:56:23.720 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000089", 2024-08-18T15:56:23.720 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:56:23.720 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:56:23.720 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000022", 2024-08-18T15:56:23.720 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:56:23.720 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:56:23.720 INFO:teuthology.orchestra.run.smithi193.stdout: "bilog.trim", 2024-08-18T15:56:23.721 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000016", 2024-08-18T15:56:23.721 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:56:23.721 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:56:23.721 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:56:23.721 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:56:23.721 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:56:23.721 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000041", 2024-08-18T15:56:23.721 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:56:23.721 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:56:23.721 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.log.01050e30-179b-43be-bdbb-3bb23cf37ae8.3", 2024-08-18T15:56:23.721 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:56:23.721 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000018", 2024-08-18T15:56:23.721 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000092", 2024-08-18T15:56:23.721 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:56:23.722 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:56:23.722 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:56:23.722 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:56:23.722 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.27", 2024-08-18T15:56:23.722 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.4", 2024-08-18T15:56:23.722 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:56:23.722 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:56:23.722 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:56:23.722 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:56:23.722 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:56:23.722 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:56:23.722 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:56:23.723 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000014", 2024-08-18T15:56:23.723 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:56:23.723 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3.retry", 2024-08-18T15:56:23.723 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:56:23.723 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:56:23.723 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:56:23.723 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:56:23.723 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:56:23.723 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000112", 2024-08-18T15:56:23.723 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:56:23.723 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000007", 2024-08-18T15:56:23.723 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000021", 2024-08-18T15:56:23.723 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:56:23.724 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:56:23.724 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:56:23.724 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:56:23.724 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:56:23.724 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.16", 2024-08-18T15:56:23.724 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:56:23.724 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000064", 2024-08-18T15:56:23.724 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:56:23.724 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:56:23.724 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:56:23.724 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:56:23.724 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:56:23.724 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:56:23.724 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:56:23.725 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000071", 2024-08-18T15:56:23.725 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:56:23.725 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:56:23.725 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:56:23.725 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000074", 2024-08-18T15:56:23.725 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:56:23.725 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:56:23.725 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:56:23.725 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:56:23.725 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:56:23.725 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:56:23.725 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:56:23.725 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:56:23.725 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:56:23.726 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:56:23.726 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:56:23.726 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:56:23.726 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:56:23.726 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:56:23.726 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:56:23.726 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000081", 2024-08-18T15:56:23.726 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:56:23.726 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000009", 2024-08-18T15:56:23.726 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000121", 2024-08-18T15:56:23.726 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1", 2024-08-18T15:56:23.726 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:56:23.726 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:56:23.727 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:56:23.727 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:56:23.727 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:56:23.727 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:56:23.727 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.17", 2024-08-18T15:56:23.727 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000125", 2024-08-18T15:56:23.727 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:56:23.727 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000082", 2024-08-18T15:56:23.727 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:56:23.727 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:56:23.727 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:56:23.727 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:56:23.727 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:56:23.727 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:56:23.728 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:56:23.728 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:56:23.728 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:56:23.728 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:56:23.728 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:56:23.728 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000105", 2024-08-18T15:56:23.728 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000059", 2024-08-18T15:56:23.728 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:56:23.728 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:56:23.728 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:56:23.728 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:56:23.728 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:56:23.728 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:56:23.728 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:56:23.728 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:56:23.729 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:56:23.729 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:56:23.729 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:56:23.729 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000077", 2024-08-18T15:56:23.729 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000032", 2024-08-18T15:56:23.729 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:56:23.729 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:56:23.729 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:56:23.729 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:56:23.729 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:56:23.729 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:56:23.729 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.24", 2024-08-18T15:56:23.729 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:56:23.729 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:56:23.730 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:56:23.730 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:56:23.730 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3.retry", 2024-08-18T15:56:23.730 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:56:23.730 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:56:23.730 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:56:23.730 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:56:23.730 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:56:23.730 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:56:23.730 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000053", 2024-08-18T15:56:23.730 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000091", 2024-08-18T15:56:23.730 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:56:23.730 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:56:23.731 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:56:23.731 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000065", 2024-08-18T15:56:23.731 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:56:23.731 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:56:23.731 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:56:23.731 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:56:23.731 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000083", 2024-08-18T15:56:23.731 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000010", 2024-08-18T15:56:23.731 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:56:23.731 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:56:23.731 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:56:23.731 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:56:23.731 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:56:23.732 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:56:23.732 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:56:23.732 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:56:23.732 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:56:23.732 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:56:23.732 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:56:23.732 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.7", 2024-08-18T15:56:23.732 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.3", 2024-08-18T15:56:23.732 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:56:23.732 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:56:23.732 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:56:23.732 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:56:23.732 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:56:23.732 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:56:23.733 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:56:23.733 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:56:23.733 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.2", 2024-08-18T15:56:23.733 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:56:23.733 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:56:23.733 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:56:23.733 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:56:23.733 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:56:23.733 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:56:23.733 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:56:23.733 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000045", 2024-08-18T15:56:23.733 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000002", 2024-08-18T15:56:23.733 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:56:23.734 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3.2UbiLidwSTy.0", 2024-08-18T15:56:23.734 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:56:23.734 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.12", 2024-08-18T15:56:23.734 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:56:23.734 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:56:23.734 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:56:23.734 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:56:23.734 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:56:23.734 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:56:23.734 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000116", 2024-08-18T15:56:23.734 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.25", 2024-08-18T15:56:23.734 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:56:23.734 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:56:23.735 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:56:23.735 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:56:23.735 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000034", 2024-08-18T15:56:23.735 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:56:23.735 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:56:23.735 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000101", 2024-08-18T15:56:23.735 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:56:23.735 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:56:23.735 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.13", 2024-08-18T15:56:23.735 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:56:23.735 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:56:23.735 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.1", 2024-08-18T15:56:23.735 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0.retry", 2024-08-18T15:56:23.735 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:56:23.736 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:56:23.736 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000079", 2024-08-18T15:56:23.736 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000049", 2024-08-18T15:56:23.736 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:56:23.736 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0", 2024-08-18T15:56:23.736 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:56:23.736 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:56:23.736 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000117", 2024-08-18T15:56:23.736 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:56:23.736 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000044", 2024-08-18T15:56:23.736 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:56:23.736 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:56:23.736 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:56:23.737 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:56:23.737 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:56:23.737 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:56:23.737 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.14", 2024-08-18T15:56:23.737 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:56:23.737 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000066", 2024-08-18T15:56:23.737 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:56:23.737 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:56:23.737 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:56:23.737 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:56:23.737 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:56:23.737 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000068", 2024-08-18T15:56:23.737 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:56:23.738 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000085", 2024-08-18T15:56:23.738 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:56:23.738 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:56:23.738 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:56:23.738 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:56:23.738 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:56:23.738 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:56:23.738 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:56:23.738 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:56:23.738 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:56:23.738 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:56:23.738 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:56:23.738 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:56:23.739 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000073", 2024-08-18T15:56:23.739 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:56:23.739 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:56:23.739 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:56:23.739 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000038", 2024-08-18T15:56:23.739 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:56:23.739 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:56:23.739 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:56:23.739 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:56:23.739 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.10", 2024-08-18T15:56:23.739 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:56:23.739 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:56:23.739 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:56:23.739 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000118", 2024-08-18T15:56:23.740 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:56:23.740 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3", 2024-08-18T15:56:23.740 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:56:23.740 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000036", 2024-08-18T15:56:23.740 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:56:23.740 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:56:23.740 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:56:23.740 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:56:23.740 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:56:23.740 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000103", 2024-08-18T15:56:23.740 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.3", 2024-08-18T15:56:23.740 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:56:23.740 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000119", 2024-08-18T15:56:23.741 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000098", 2024-08-18T15:56:23.741 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:56:23.741 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:56:23.741 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:56:23.741 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:56:23.741 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:56:23.741 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:56:23.741 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:56:23.741 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:56:23.741 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.0", 2024-08-18T15:56:23.741 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:56:23.741 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:56:23.742 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:56:23.742 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:56:23.742 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:56:23.742 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:56:23.742 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:56:23.742 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:56:23.742 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000027", 2024-08-18T15:56:23.742 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:56:23.742 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status", 2024-08-18T15:56:23.742 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:56:23.742 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000019", 2024-08-18T15:56:23.742 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:56:23.743 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:56:23.743 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:56:23.743 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:56:23.743 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1", 2024-08-18T15:56:23.743 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:56:23.743 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:56:23.743 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:56:23.743 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:56:23.743 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000039", 2024-08-18T15:56:23.743 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:56:23.743 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:56:23.743 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:56:23.743 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:56:23.744 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:56:23.744 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:56:23.744 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:56:23.744 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:56:23.744 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:56:23.744 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:56:23.744 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:56:23.744 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000100", 2024-08-18T15:56:23.744 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:56:23.744 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:56:23.744 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:56:23.744 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:56:23.744 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:56:23.745 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000093", 2024-08-18T15:56:23.745 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:56:23.745 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000004", 2024-08-18T15:56:23.745 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000063", 2024-08-18T15:56:23.745 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000122", 2024-08-18T15:56:23.745 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:56:23.745 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:56:23.745 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:56:23.745 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000057", 2024-08-18T15:56:23.745 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:56:23.745 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:56:23.745 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:56:23.745 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:56:23.745 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000054", 2024-08-18T15:56:23.746 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:56:23.746 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000114", 2024-08-18T15:56:23.746 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:56:23.746 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:56:23.746 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:56:23.746 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.2", 2024-08-18T15:56:23.746 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:56:23.746 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:56:23.746 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0", 2024-08-18T15:56:23.746 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:56:23.746 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:56:23.746 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:56:23.746 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:56:23.747 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:56:23.747 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:56:23.747 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000012", 2024-08-18T15:56:23.747 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:56:23.747 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:56:23.747 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:56:23.747 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000084", 2024-08-18T15:56:23.747 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000043", 2024-08-18T15:56:23.747 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3", 2024-08-18T15:56:23.747 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:56:23.747 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:56:23.747 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.11", 2024-08-18T15:56:23.747 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:56:23.747 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:56:23.748 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:56:23.748 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:56:23.748 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:56:23.748 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:56:23.748 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:56:23.748 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:56:23.748 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000111", 2024-08-18T15:56:23.748 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:56:23.748 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000048", 2024-08-18T15:56:23.748 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:56:23.748 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000020", 2024-08-18T15:56:23.748 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:56:23.748 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:56:23.749 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:56:23.749 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:56:23.749 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:56:23.749 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000099", 2024-08-18T15:56:23.749 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000056", 2024-08-18T15:56:23.749 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:56:23.749 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:56:23.749 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.19", 2024-08-18T15:56:23.749 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:56:23.749 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.28", 2024-08-18T15:56:23.749 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:56:23.749 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:56:23.749 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:56:23.750 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:56:23.750 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:56:23.750 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000072", 2024-08-18T15:56:23.750 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:56:23.750 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:56:23.750 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000062", 2024-08-18T15:56:23.750 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:56:23.750 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:56:23.750 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:56:23.750 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000115", 2024-08-18T15:56:23.750 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:56:23.750 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.1", 2024-08-18T15:56:23.751 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2", 2024-08-18T15:56:23.751 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000033", 2024-08-18T15:56:23.751 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:56:23.751 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:56:23.751 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:56:23.751 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:56:23.751 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:56:23.751 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:56:23.751 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:56:23.751 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:56:23.751 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:56:23.751 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:56:23.751 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:56:23.751 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:56:23.752 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:56:23.752 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:56:23.752 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:56:23.752 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:56:23.752 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:56:23.752 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:56:23.752 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:56:23.752 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000025", 2024-08-18T15:56:23.752 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:56:23.752 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000123", 2024-08-18T15:56:23.752 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.31", 2024-08-18T15:56:23.752 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:56:23.752 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:56:23.753 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:56:23.753 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:56:23.753 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.18", 2024-08-18T15:56:23.753 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:56:23.753 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:56:23.753 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:56:23.753 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:56:23.753 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:56:23.753 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000108", 2024-08-18T15:56:23.753 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:56:23.753 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:56:23.753 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000094", 2024-08-18T15:56:23.753 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:56:23.754 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:56:23.754 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:56:23.754 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:56:23.754 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:56:23.754 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:56:23.754 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000060", 2024-08-18T15:56:23.754 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:56:23.754 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:56:23.754 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:56:23.754 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000109", 2024-08-18T15:56:23.754 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.15", 2024-08-18T15:56:23.754 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:56:23.755 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:56:23.755 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:56:23.755 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:56:23.755 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:56:23.755 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:56:23.755 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:56:23.755 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:56:23.755 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:56:23.755 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000013", 2024-08-18T15:56:23.755 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:56:23.755 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:56:23.755 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:56:23.755 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:56:23.756 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:56:23.756 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:56:23.756 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:56:23.756 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:56:23.756 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:56:23.756 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:56:23.756 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:56:23.756 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:56:23.756 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:56:23.756 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:56:23.756 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000051", 2024-08-18T15:56:23.756 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:56:23.756 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:56:23.757 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:56:23.757 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:56:23.757 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000106", 2024-08-18T15:56:23.757 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:56:23.757 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:56:23.757 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:56:23.757 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:56:23.757 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:56:23.757 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000017", 2024-08-18T15:56:23.757 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:56:23.757 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:56:23.757 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:56:23.757 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000023", 2024-08-18T15:56:23.758 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:56:23.758 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:56:23.758 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:56:23.758 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:56:23.758 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:56:23.758 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:56:23.758 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:56:23.758 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000107", 2024-08-18T15:56:23.758 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:56:23.758 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:56:23.758 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000046", 2024-08-18T15:56:23.759 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:56:23.759 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:56:23.759 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:56:23.759 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.26", 2024-08-18T15:56:23.759 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:56:23.759 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:56:23.759 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9", 2024-08-18T15:56:23.759 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.22", 2024-08-18T15:56:23.759 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:56:23.759 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:56:23.759 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:56:23.759 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:56:23.759 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:56:23.760 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:56:23.760 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:56:23.760 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2", 2024-08-18T15:56:23.760 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:56:23.760 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:56:23.760 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000067", 2024-08-18T15:56:23.760 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:56:23.760 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:56:23.760 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:56:23.760 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:56:23.760 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:56:23.760 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:56:23.760 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:56:23.760 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:56:23.761 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:56:23.761 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:56:23.761 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:56:23.761 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:56:23.761 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:56:23.761 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:56:23.761 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:56:23.761 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:56:23.761 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:56:23.761 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:56:23.761 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:56:23.761 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.9", 2024-08-18T15:56:23.761 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000086", 2024-08-18T15:56:23.761 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.5", 2024-08-18T15:56:23.762 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:56:23.762 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:56:23.762 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:56:23.762 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:56:23.762 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:56:23.762 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:56:23.762 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2.MFXJ583gCr1.0", 2024-08-18T15:56:23.762 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:56:23.762 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1.GPnO16liyZ+.0", 2024-08-18T15:56:23.762 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:56:23.762 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:56:23.762 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:56:23.763 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:56:23.763 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:56:23.763 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:56:23.763 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:56:23.763 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:56:23.763 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:56:23.763 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.20", 2024-08-18T15:56:23.763 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:56:23.763 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:56:23.763 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:56:23.763 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:56:23.763 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:56:23.763 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:56:23.764 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1.retry", 2024-08-18T15:56:23.764 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:56:23.764 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:56:23.764 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:56:23.764 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0.retry", 2024-08-18T15:56:23.764 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000050", 2024-08-18T15:56:23.764 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:56:23.764 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000102", 2024-08-18T15:56:23.764 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:56:23.764 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:56:23.764 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:56:23.764 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:56:23.765 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:56:23.765 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:56:23.765 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:56:23.765 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:56:23.765 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:56:23.765 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:56:23.765 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:56:23.765 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000124", 2024-08-18T15:56:23.765 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:56:23.765 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:56:23.765 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:56:23.765 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000001", 2024-08-18T15:56:23.765 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:56:23.765 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:56:23.766 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:56:23.766 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:56:23.766 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000126", 2024-08-18T15:56:23.766 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:56:23.766 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0.fZhctfLXnSt.0", 2024-08-18T15:56:23.766 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:56:23.766 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000055", 2024-08-18T15:56:23.766 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:56:23.766 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000061", 2024-08-18T15:56:23.766 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1", 2024-08-18T15:56:23.766 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:56:23.766 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:56:23.767 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:56:23.767 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000097", 2024-08-18T15:56:23.767 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:56:23.767 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:56:23.767 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:56:23.767 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:56:23.767 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:56:23.767 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:56:23.767 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:56:23.767 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:56:23.767 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000042", 2024-08-18T15:56:23.767 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:56:23.768 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:56:23.768 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:56:23.768 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1.retry", 2024-08-18T15:56:23.768 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:56:23.768 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:56:23.768 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:56:23.768 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:56:23.768 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:56:23.768 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:56:23.768 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:56:23.768 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000075", 2024-08-18T15:56:23.769 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:56:23.769 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000120", 2024-08-18T15:56:23.769 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000030", 2024-08-18T15:56:23.769 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:56:23.769 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:56:23.769 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:56:23.769 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:56:23.769 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:56:23.769 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:56:23.769 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9" 2024-08-18T15:56:23.769 INFO:teuthology.orchestra.run.smithi193.stdout:] 2024-08-18T15:56:23.769 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:56:28.772 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:56:28.772 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:56:28.772 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:56:28.824 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:56:28.825 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:56:28.889 INFO:teuthology.orchestra.run.smithi193.stdout:[ 2024-08-18T15:56:28.889 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:56:28.889 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:56:28.889 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.29", 2024-08-18T15:56:28.889 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000078", 2024-08-18T15:56:28.889 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.history", 2024-08-18T15:56:28.889 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:56:28.889 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:56:28.889 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:56:28.889 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:56:28.890 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.23", 2024-08-18T15:56:28.890 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000070", 2024-08-18T15:56:28.890 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000104", 2024-08-18T15:56:28.890 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:56:28.890 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2.retry", 2024-08-18T15:56:28.890 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:56:28.890 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:56:28.890 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:56:28.890 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:56:28.890 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000026", 2024-08-18T15:56:28.890 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:56:28.890 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:56:28.890 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:56:28.890 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:56:28.891 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:56:28.891 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:56:28.891 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.8", 2024-08-18T15:56:28.891 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:56:28.891 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:56:28.891 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:56:28.891 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:56:28.891 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000028", 2024-08-18T15:56:28.891 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:56:28.891 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:56:28.891 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:56:28.891 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:56:28.891 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:56:28.892 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:56:28.892 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:56:28.892 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2.retry", 2024-08-18T15:56:28.892 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:56:28.892 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.30", 2024-08-18T15:56:28.892 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:56:28.892 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:56:28.892 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:56:28.892 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:56:28.892 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000040", 2024-08-18T15:56:28.892 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:56:28.893 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.6", 2024-08-18T15:56:28.893 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:56:28.893 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:56:28.893 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:56:28.893 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:56:28.893 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:56:28.893 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:56:28.894 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000015", 2024-08-18T15:56:28.894 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:56:28.894 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:56:28.894 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000069", 2024-08-18T15:56:28.894 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:56:28.894 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:56:28.894 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:56:28.894 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000095", 2024-08-18T15:56:28.894 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0", 2024-08-18T15:56:28.894 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000003", 2024-08-18T15:56:28.894 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000047", 2024-08-18T15:56:28.894 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:56:28.894 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:56:28.895 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:56:28.895 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:56:28.895 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:56:28.895 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:56:28.895 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:56:28.895 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:56:28.895 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:56:28.895 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000035", 2024-08-18T15:56:28.895 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:56:28.895 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:56:28.895 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000037", 2024-08-18T15:56:28.895 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:56:28.895 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:56:28.896 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:56:28.896 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:56:28.896 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:56:28.896 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:56:28.896 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:56:28.896 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:56:28.896 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:56:28.896 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:56:28.896 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:56:28.896 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:56:28.896 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:56:28.896 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:56:28.896 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000024", 2024-08-18T15:56:28.897 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:56:28.897 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:56:28.897 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:56:28.897 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:56:28.897 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000000", 2024-08-18T15:56:28.897 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:56:28.897 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000031", 2024-08-18T15:56:28.897 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:56:28.897 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:56:28.897 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:56:28.897 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:56:28.897 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:56:28.897 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:56:28.898 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3", 2024-08-18T15:56:28.898 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:56:28.898 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:56:28.898 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:56:28.898 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:56:28.898 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:56:28.898 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:56:28.898 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:56:28.898 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:56:28.898 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:56:28.898 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:56:28.898 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:56:28.899 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:56:28.899 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:56:28.899 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:56:28.899 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:56:28.899 INFO:teuthology.orchestra.run.smithi193.stdout: "data_loggenerations_metadata", 2024-08-18T15:56:28.899 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2", 2024-08-18T15:56:28.899 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:56:28.899 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:56:28.899 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:56:28.899 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:56:28.899 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T15:56:28.899 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:56:28.899 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:56:28.900 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:56:28.900 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:56:28.900 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:56:28.900 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000076", 2024-08-18T15:56:28.900 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:56:28.900 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:56:28.900 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000113", 2024-08-18T15:56:28.900 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000005", 2024-08-18T15:56:28.900 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:56:28.900 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000011", 2024-08-18T15:56:28.900 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:56:28.900 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000058", 2024-08-18T15:56:28.900 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:56:28.900 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:56:28.901 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:56:28.901 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000052", 2024-08-18T15:56:28.901 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:56:28.901 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:56:28.901 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:56:28.901 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:56:28.901 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.21", 2024-08-18T15:56:28.901 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:56:28.901 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:56:28.901 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:56:28.901 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:56:28.901 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:56:28.901 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:56:28.902 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:56:28.902 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:56:28.902 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:56:28.902 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:56:28.902 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:56:28.902 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:56:28.902 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:56:28.902 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:56:28.902 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:56:28.902 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:56:28.902 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:56:28.902 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:56:28.902 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:56:28.903 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000088", 2024-08-18T15:56:28.903 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000080", 2024-08-18T15:56:28.903 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:56:28.903 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:56:28.903 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000090", 2024-08-18T15:56:28.903 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:56:28.903 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:56:28.903 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000110", 2024-08-18T15:56:28.903 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:56:28.903 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:56:28.903 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:56:28.904 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.0", 2024-08-18T15:56:28.904 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:56:28.904 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000096", 2024-08-18T15:56:28.904 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:56:28.904 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:56:28.904 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:56:28.904 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:56:28.904 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:56:28.904 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000087", 2024-08-18T15:56:28.904 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:56:28.904 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:56:28.904 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:56:28.904 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:56:28.904 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000008", 2024-08-18T15:56:28.905 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:56:28.905 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:56:28.905 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:56:28.905 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:56:28.905 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:56:28.905 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000006", 2024-08-18T15:56:28.905 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000029", 2024-08-18T15:56:28.905 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:56:28.905 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000089", 2024-08-18T15:56:28.905 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:56:28.905 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:56:28.905 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000022", 2024-08-18T15:56:28.905 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:56:28.906 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:56:28.906 INFO:teuthology.orchestra.run.smithi193.stdout: "bilog.trim", 2024-08-18T15:56:28.906 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000016", 2024-08-18T15:56:28.906 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:56:28.906 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:56:28.906 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:56:28.906 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:56:28.906 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:56:28.906 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000041", 2024-08-18T15:56:28.906 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:56:28.906 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:56:28.906 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.log.01050e30-179b-43be-bdbb-3bb23cf37ae8.3", 2024-08-18T15:56:28.906 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:56:28.907 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000018", 2024-08-18T15:56:28.907 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000092", 2024-08-18T15:56:28.907 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:56:28.907 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:56:28.907 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:56:28.907 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:56:28.907 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.27", 2024-08-18T15:56:28.907 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.4", 2024-08-18T15:56:28.907 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:56:28.907 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:56:28.907 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:56:28.907 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:56:28.908 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:56:28.908 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:56:28.908 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:56:28.908 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000014", 2024-08-18T15:56:28.908 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:56:28.908 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3.retry", 2024-08-18T15:56:28.908 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:56:28.908 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:56:28.908 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:56:28.908 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:56:28.908 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:56:28.908 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000112", 2024-08-18T15:56:28.909 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:56:28.909 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000007", 2024-08-18T15:56:28.909 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000021", 2024-08-18T15:56:28.909 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:56:28.909 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:56:28.909 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:56:28.909 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:56:28.909 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:56:28.909 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.16", 2024-08-18T15:56:28.909 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:56:28.909 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000064", 2024-08-18T15:56:28.909 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:56:28.909 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:56:28.910 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:56:28.910 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:56:28.910 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:56:28.910 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:56:28.910 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:56:28.910 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000071", 2024-08-18T15:56:28.910 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:56:28.910 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:56:28.910 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:56:28.910 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000074", 2024-08-18T15:56:28.910 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:56:28.910 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:56:28.911 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:56:28.911 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:56:28.911 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:56:28.911 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:56:28.911 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:56:28.911 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:56:28.911 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:56:28.911 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:56:28.911 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:56:28.911 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:56:28.911 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:56:28.911 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:56:28.912 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:56:28.912 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000081", 2024-08-18T15:56:28.912 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:56:28.912 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000009", 2024-08-18T15:56:28.912 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000121", 2024-08-18T15:56:28.912 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1", 2024-08-18T15:56:28.912 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:56:28.912 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:56:28.912 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:56:28.912 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:56:28.912 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:56:28.912 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:56:28.912 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.17", 2024-08-18T15:56:28.913 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000125", 2024-08-18T15:56:28.913 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:56:28.913 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000082", 2024-08-18T15:56:28.913 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:56:28.913 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:56:28.913 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:56:28.913 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:56:28.913 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:56:28.913 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:56:28.913 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:56:28.913 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:56:28.913 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:56:28.913 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:56:28.913 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:56:28.914 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000105", 2024-08-18T15:56:28.914 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000059", 2024-08-18T15:56:28.914 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:56:28.914 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:56:28.914 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:56:28.914 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:56:28.914 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:56:28.914 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:56:28.914 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:56:28.914 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:56:28.914 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:56:28.914 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:56:28.914 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:56:28.915 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000077", 2024-08-18T15:56:28.915 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000032", 2024-08-18T15:56:28.915 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:56:28.915 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:56:28.915 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:56:28.915 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:56:28.915 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:56:28.915 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:56:28.915 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.24", 2024-08-18T15:56:28.915 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:56:28.915 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:56:28.915 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:56:28.915 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:56:28.916 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3.retry", 2024-08-18T15:56:28.916 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:56:28.916 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:56:28.916 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:56:28.916 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:56:28.916 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:56:28.916 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:56:28.916 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000053", 2024-08-18T15:56:28.916 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000091", 2024-08-18T15:56:28.916 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:56:28.916 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:56:28.916 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:56:28.916 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000065", 2024-08-18T15:56:28.916 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:56:28.917 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:56:28.917 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:56:28.917 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:56:28.917 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000083", 2024-08-18T15:56:28.917 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000010", 2024-08-18T15:56:28.917 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:56:28.917 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:56:28.917 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:56:28.917 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:56:28.917 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:56:28.917 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:56:28.917 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:56:28.917 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:56:28.918 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:56:28.918 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:56:28.918 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:56:28.918 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.7", 2024-08-18T15:56:28.918 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.3", 2024-08-18T15:56:28.918 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:56:28.918 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:56:28.918 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:56:28.918 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:56:28.918 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:56:28.918 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:56:28.918 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:56:28.918 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:56:28.919 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.2", 2024-08-18T15:56:28.919 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:56:28.919 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:56:28.919 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:56:28.919 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:56:28.919 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:56:28.919 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:56:28.919 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:56:28.919 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000045", 2024-08-18T15:56:28.919 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000002", 2024-08-18T15:56:28.919 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:56:28.919 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3.2UbiLidwSTy.0", 2024-08-18T15:56:28.919 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:56:28.920 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.12", 2024-08-18T15:56:28.920 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:56:28.920 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:56:28.920 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:56:28.920 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:56:28.920 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:56:28.920 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:56:28.920 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000116", 2024-08-18T15:56:28.920 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.25", 2024-08-18T15:56:28.920 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:56:28.920 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:56:28.920 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:56:28.920 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:56:28.920 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000034", 2024-08-18T15:56:28.921 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:56:28.921 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:56:28.921 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000101", 2024-08-18T15:56:28.921 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:56:28.921 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:56:28.921 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.13", 2024-08-18T15:56:28.921 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:56:28.921 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:56:28.921 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.1", 2024-08-18T15:56:28.921 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0.retry", 2024-08-18T15:56:28.921 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:56:28.921 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:56:28.922 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000079", 2024-08-18T15:56:28.922 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000049", 2024-08-18T15:56:28.922 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:56:28.922 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0", 2024-08-18T15:56:28.922 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:56:28.922 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:56:28.922 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000117", 2024-08-18T15:56:28.922 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:56:28.922 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000044", 2024-08-18T15:56:28.922 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:56:28.922 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:56:28.922 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:56:28.922 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:56:28.923 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:56:28.923 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:56:28.923 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.14", 2024-08-18T15:56:28.923 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:56:28.923 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000066", 2024-08-18T15:56:28.923 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:56:28.923 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:56:28.923 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:56:28.923 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:56:28.923 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:56:28.923 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000068", 2024-08-18T15:56:28.923 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:56:28.923 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000085", 2024-08-18T15:56:28.924 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:56:28.924 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:56:28.924 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:56:28.924 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:56:28.924 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:56:28.924 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:56:28.924 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:56:28.924 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:56:28.924 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:56:28.924 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:56:28.924 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:56:28.924 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:56:28.924 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000073", 2024-08-18T15:56:28.924 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:56:28.925 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:56:28.925 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:56:28.925 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000038", 2024-08-18T15:56:28.925 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:56:28.925 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:56:28.925 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:56:28.925 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:56:28.925 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.10", 2024-08-18T15:56:28.925 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:56:28.925 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:56:28.925 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:56:28.925 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000118", 2024-08-18T15:56:28.925 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:56:28.926 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3", 2024-08-18T15:56:28.926 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:56:28.926 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000036", 2024-08-18T15:56:28.926 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:56:28.926 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:56:28.926 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:56:28.926 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:56:28.926 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:56:28.926 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000103", 2024-08-18T15:56:28.926 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.3", 2024-08-18T15:56:28.926 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:56:28.926 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000119", 2024-08-18T15:56:28.927 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000098", 2024-08-18T15:56:28.927 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:56:28.927 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:56:28.927 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:56:28.927 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:56:28.927 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:56:28.927 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:56:28.927 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:56:28.927 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:56:28.927 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.0", 2024-08-18T15:56:28.927 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:56:28.927 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:56:28.927 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:56:28.928 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:56:28.928 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:56:28.928 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:56:28.928 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:56:28.928 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:56:28.928 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000027", 2024-08-18T15:56:28.928 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:56:28.928 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status", 2024-08-18T15:56:28.928 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:56:28.928 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000019", 2024-08-18T15:56:28.928 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:56:28.928 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:56:28.928 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:56:28.929 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:56:28.929 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1", 2024-08-18T15:56:28.929 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:56:28.929 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:56:28.929 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:56:28.929 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:56:28.929 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000039", 2024-08-18T15:56:28.929 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:56:28.929 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:56:28.929 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:56:28.929 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:56:28.929 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:56:28.929 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:56:28.930 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:56:28.930 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:56:28.930 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:56:28.930 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:56:28.930 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:56:28.930 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000100", 2024-08-18T15:56:28.930 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:56:28.930 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:56:28.930 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:56:28.930 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:56:28.930 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:56:28.930 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000093", 2024-08-18T15:56:28.930 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:56:28.931 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000004", 2024-08-18T15:56:28.931 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000063", 2024-08-18T15:56:28.931 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000122", 2024-08-18T15:56:28.931 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:56:28.931 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:56:28.931 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:56:28.931 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000057", 2024-08-18T15:56:28.931 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:56:28.931 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:56:28.931 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:56:28.931 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:56:28.931 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000054", 2024-08-18T15:56:28.931 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:56:28.932 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000114", 2024-08-18T15:56:28.932 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:56:28.932 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:56:28.932 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:56:28.932 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.2", 2024-08-18T15:56:28.932 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:56:28.932 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:56:28.932 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0", 2024-08-18T15:56:28.932 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:56:28.932 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:56:28.932 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:56:28.932 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:56:28.932 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:56:28.932 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:56:28.933 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000012", 2024-08-18T15:56:28.933 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:56:28.933 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:56:28.933 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:56:28.933 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000084", 2024-08-18T15:56:28.933 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000043", 2024-08-18T15:56:28.933 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3", 2024-08-18T15:56:28.933 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:56:28.933 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:56:28.933 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.11", 2024-08-18T15:56:28.933 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:56:28.933 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:56:28.933 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:56:28.934 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:56:28.934 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:56:28.934 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:56:28.934 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:56:28.934 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:56:28.934 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000111", 2024-08-18T15:56:28.934 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:56:28.934 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000048", 2024-08-18T15:56:28.934 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:56:28.934 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000020", 2024-08-18T15:56:28.934 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:56:28.934 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:56:28.935 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:56:28.935 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:56:28.935 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:56:28.935 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000099", 2024-08-18T15:56:28.935 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000056", 2024-08-18T15:56:28.935 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:56:28.935 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:56:28.935 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.19", 2024-08-18T15:56:28.935 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:56:28.935 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.28", 2024-08-18T15:56:28.935 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:56:28.935 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:56:28.935 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:56:28.936 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:56:28.936 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:56:28.936 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000072", 2024-08-18T15:56:28.936 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:56:28.936 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:56:28.936 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000062", 2024-08-18T15:56:28.936 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:56:28.936 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:56:28.936 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:56:28.936 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000115", 2024-08-18T15:56:28.936 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:56:28.936 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.1", 2024-08-18T15:56:28.936 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2", 2024-08-18T15:56:28.936 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000033", 2024-08-18T15:56:28.937 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:56:28.937 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:56:28.937 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:56:28.937 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:56:28.937 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:56:28.937 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:56:28.937 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:56:28.937 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:56:28.937 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:56:28.937 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:56:28.937 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:56:28.937 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:56:28.937 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:56:28.938 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:56:28.938 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:56:28.938 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:56:28.938 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:56:28.938 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:56:28.938 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:56:28.938 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000025", 2024-08-18T15:56:28.938 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:56:28.938 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000123", 2024-08-18T15:56:28.938 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.31", 2024-08-18T15:56:28.938 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:56:28.938 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:56:28.938 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:56:28.939 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:56:28.939 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.18", 2024-08-18T15:56:28.939 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:56:28.939 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:56:28.939 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:56:28.939 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:56:28.939 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:56:28.939 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000108", 2024-08-18T15:56:28.939 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:56:28.939 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:56:28.939 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000094", 2024-08-18T15:56:28.939 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:56:28.939 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:56:28.940 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:56:28.940 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:56:28.940 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:56:28.940 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:56:28.940 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000060", 2024-08-18T15:56:28.940 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:56:28.940 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:56:28.940 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:56:28.940 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000109", 2024-08-18T15:56:28.940 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.15", 2024-08-18T15:56:28.940 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:56:28.940 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:56:28.940 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:56:28.941 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:56:28.941 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:56:28.941 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:56:28.941 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:56:28.941 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:56:28.941 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:56:28.941 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000013", 2024-08-18T15:56:28.941 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:56:28.941 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:56:28.941 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:56:28.941 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:56:28.941 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:56:28.941 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:56:28.942 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:56:28.942 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:56:28.942 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:56:28.942 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:56:28.942 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:56:28.942 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:56:28.942 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:56:28.942 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:56:28.942 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000051", 2024-08-18T15:56:28.942 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:56:28.942 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:56:28.942 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:56:28.942 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:56:28.942 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000106", 2024-08-18T15:56:28.943 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:56:28.943 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:56:28.943 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:56:28.943 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:56:28.943 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:56:28.943 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000017", 2024-08-18T15:56:28.943 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:56:28.943 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:56:28.943 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:56:28.943 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000023", 2024-08-18T15:56:28.943 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:56:28.943 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:56:28.943 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:56:28.944 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:56:28.944 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:56:28.944 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:56:28.944 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:56:28.944 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000107", 2024-08-18T15:56:28.944 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:56:28.944 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:56:28.944 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000046", 2024-08-18T15:56:28.944 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:56:28.944 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:56:28.944 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:56:28.944 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.26", 2024-08-18T15:56:28.944 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:56:28.944 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:56:28.945 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9", 2024-08-18T15:56:28.945 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.22", 2024-08-18T15:56:28.945 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:56:28.945 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:56:28.945 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:56:28.945 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:56:28.945 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:56:28.945 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:56:28.945 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:56:28.945 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2", 2024-08-18T15:56:28.945 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:56:28.945 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:56:28.945 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000067", 2024-08-18T15:56:28.945 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:56:28.946 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:56:28.946 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:56:28.946 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:56:28.946 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:56:28.946 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:56:28.946 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:56:28.946 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:56:28.946 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:56:28.946 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:56:28.946 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:56:28.946 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:56:28.946 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:56:28.947 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:56:28.947 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:56:28.947 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:56:28.947 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:56:28.947 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:56:28.947 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:56:28.947 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.9", 2024-08-18T15:56:28.947 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000086", 2024-08-18T15:56:28.947 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.5", 2024-08-18T15:56:28.947 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:56:28.947 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:56:28.947 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:56:28.948 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:56:28.948 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:56:28.948 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:56:28.948 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2.MFXJ583gCr1.0", 2024-08-18T15:56:28.948 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:56:28.948 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1.GPnO16liyZ+.0", 2024-08-18T15:56:28.948 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:56:28.948 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:56:28.948 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:56:28.948 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:56:28.949 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:56:28.949 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:56:28.949 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:56:28.949 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:56:28.949 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:56:28.949 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.20", 2024-08-18T15:56:28.949 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:56:28.949 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:56:28.949 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:56:28.949 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:56:28.949 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:56:28.949 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:56:28.950 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1.retry", 2024-08-18T15:56:28.950 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:56:28.950 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:56:28.950 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:56:28.950 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0.retry", 2024-08-18T15:56:28.950 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000050", 2024-08-18T15:56:28.950 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:56:28.950 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000102", 2024-08-18T15:56:28.950 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:56:28.950 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:56:28.950 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:56:28.950 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:56:28.951 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:56:28.951 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:56:28.951 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:56:28.951 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:56:28.951 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:56:28.951 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:56:28.951 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:56:28.951 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000124", 2024-08-18T15:56:28.951 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:56:28.951 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:56:28.951 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:56:28.951 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000001", 2024-08-18T15:56:28.952 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:56:28.952 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:56:28.952 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:56:28.952 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:56:28.952 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000126", 2024-08-18T15:56:28.952 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:56:28.952 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0.fZhctfLXnSt.0", 2024-08-18T15:56:28.952 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:56:28.952 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000055", 2024-08-18T15:56:28.952 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:56:28.952 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000061", 2024-08-18T15:56:28.952 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1", 2024-08-18T15:56:28.952 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:56:28.953 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:56:28.953 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:56:28.953 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000097", 2024-08-18T15:56:28.953 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:56:28.953 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:56:28.953 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:56:28.953 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:56:28.953 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:56:28.953 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:56:28.953 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:56:28.953 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:56:28.953 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000042", 2024-08-18T15:56:28.953 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:56:28.954 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:56:28.954 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:56:28.954 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1.retry", 2024-08-18T15:56:28.954 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:56:28.954 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:56:28.954 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:56:28.954 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:56:28.954 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:56:28.955 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:56:28.955 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:56:28.955 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000075", 2024-08-18T15:56:28.955 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:56:28.955 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000120", 2024-08-18T15:56:28.955 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000030", 2024-08-18T15:56:28.955 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:56:28.955 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:56:28.955 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:56:28.955 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:56:28.955 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:56:28.955 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:56:28.956 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9" 2024-08-18T15:56:28.956 INFO:teuthology.orchestra.run.smithi193.stdout:] 2024-08-18T15:56:28.956 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:56:33.957 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:56:33.958 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:56:33.958 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:56:34.012 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:56:34.012 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:56:34.077 INFO:teuthology.orchestra.run.smithi193.stdout:[ 2024-08-18T15:56:34.077 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:56:34.078 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:56:34.078 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.29", 2024-08-18T15:56:34.078 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000078", 2024-08-18T15:56:34.078 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.history", 2024-08-18T15:56:34.078 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:56:34.078 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:56:34.078 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:56:34.078 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:56:34.078 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.23", 2024-08-18T15:56:34.078 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000070", 2024-08-18T15:56:34.079 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000104", 2024-08-18T15:56:34.079 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:56:34.079 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2.retry", 2024-08-18T15:56:34.079 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:56:34.079 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:56:34.079 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:56:34.079 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:56:34.079 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000026", 2024-08-18T15:56:34.079 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:56:34.079 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:56:34.080 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:56:34.080 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:56:34.080 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:56:34.080 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:56:34.080 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.8", 2024-08-18T15:56:34.080 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:56:34.080 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:56:34.080 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:56:34.080 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:56:34.080 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000028", 2024-08-18T15:56:34.080 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:56:34.080 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:56:34.080 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:56:34.080 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:56:34.081 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:56:34.081 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:56:34.081 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:56:34.081 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2.retry", 2024-08-18T15:56:34.081 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:56:34.081 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.30", 2024-08-18T15:56:34.081 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:56:34.081 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:56:34.081 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:56:34.081 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:56:34.081 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000040", 2024-08-18T15:56:34.081 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:56:34.081 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.6", 2024-08-18T15:56:34.081 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:56:34.082 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:56:34.082 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:56:34.082 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:56:34.082 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:56:34.082 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:56:34.082 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000015", 2024-08-18T15:56:34.082 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:56:34.082 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:56:34.082 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000069", 2024-08-18T15:56:34.082 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:56:34.082 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:56:34.082 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:56:34.082 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000095", 2024-08-18T15:56:34.083 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0", 2024-08-18T15:56:34.083 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000003", 2024-08-18T15:56:34.083 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000047", 2024-08-18T15:56:34.083 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:56:34.083 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:56:34.083 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:56:34.083 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:56:34.083 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:56:34.083 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:56:34.083 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:56:34.083 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:56:34.083 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:56:34.083 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000035", 2024-08-18T15:56:34.084 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:56:34.084 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:56:34.084 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000037", 2024-08-18T15:56:34.084 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:56:34.084 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:56:34.084 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:56:34.084 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:56:34.084 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:56:34.084 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:56:34.084 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:56:34.084 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:56:34.084 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:56:34.085 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:56:34.085 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:56:34.085 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:56:34.085 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:56:34.085 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:56:34.085 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000024", 2024-08-18T15:56:34.085 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:56:34.085 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:56:34.085 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:56:34.085 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:56:34.085 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000000", 2024-08-18T15:56:34.085 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:56:34.085 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000031", 2024-08-18T15:56:34.086 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:56:34.086 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:56:34.086 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:56:34.086 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:56:34.086 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:56:34.086 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:56:34.086 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3", 2024-08-18T15:56:34.086 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:56:34.086 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:56:34.086 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:56:34.086 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:56:34.086 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:56:34.086 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:56:34.087 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:56:34.087 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:56:34.087 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:56:34.087 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:56:34.087 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:56:34.087 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:56:34.087 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:56:34.087 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:56:34.087 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:56:34.087 INFO:teuthology.orchestra.run.smithi193.stdout: "data_loggenerations_metadata", 2024-08-18T15:56:34.087 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2", 2024-08-18T15:56:34.087 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:56:34.087 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:56:34.088 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:56:34.088 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:56:34.088 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T15:56:34.088 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:56:34.088 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:56:34.088 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:56:34.088 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:56:34.088 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:56:34.088 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000076", 2024-08-18T15:56:34.088 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:56:34.088 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:56:34.088 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000113", 2024-08-18T15:56:34.088 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000005", 2024-08-18T15:56:34.089 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:56:34.089 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000011", 2024-08-18T15:56:34.089 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:56:34.089 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000058", 2024-08-18T15:56:34.089 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:56:34.089 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:56:34.089 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:56:34.089 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000052", 2024-08-18T15:56:34.089 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:56:34.089 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:56:34.089 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:56:34.089 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:56:34.089 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.21", 2024-08-18T15:56:34.090 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:56:34.090 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:56:34.090 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:56:34.090 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:56:34.090 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:56:34.090 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:56:34.090 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:56:34.090 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:56:34.090 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:56:34.090 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:56:34.090 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:56:34.090 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:56:34.091 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:56:34.091 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:56:34.091 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:56:34.091 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:56:34.091 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:56:34.091 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:56:34.091 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:56:34.091 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000088", 2024-08-18T15:56:34.091 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000080", 2024-08-18T15:56:34.091 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:56:34.091 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:56:34.091 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000090", 2024-08-18T15:56:34.091 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:56:34.092 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:56:34.092 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000110", 2024-08-18T15:56:34.092 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:56:34.092 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:56:34.092 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:56:34.092 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.0", 2024-08-18T15:56:34.092 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:56:34.092 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000096", 2024-08-18T15:56:34.092 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:56:34.092 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:56:34.092 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:56:34.092 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:56:34.092 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:56:34.092 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000087", 2024-08-18T15:56:34.093 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:56:34.093 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:56:34.093 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:56:34.093 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:56:34.093 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000008", 2024-08-18T15:56:34.093 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:56:34.093 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:56:34.093 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:56:34.093 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:56:34.093 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:56:34.093 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000006", 2024-08-18T15:56:34.093 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000029", 2024-08-18T15:56:34.093 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:56:34.094 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000089", 2024-08-18T15:56:34.094 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:56:34.094 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:56:34.094 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000022", 2024-08-18T15:56:34.094 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:56:34.094 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:56:34.094 INFO:teuthology.orchestra.run.smithi193.stdout: "bilog.trim", 2024-08-18T15:56:34.094 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000016", 2024-08-18T15:56:34.094 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:56:34.094 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:56:34.094 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:56:34.094 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:56:34.095 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:56:34.095 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000041", 2024-08-18T15:56:34.095 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:56:34.095 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:56:34.095 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.log.01050e30-179b-43be-bdbb-3bb23cf37ae8.3", 2024-08-18T15:56:34.095 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:56:34.095 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000018", 2024-08-18T15:56:34.095 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000092", 2024-08-18T15:56:34.095 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:56:34.095 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:56:34.095 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:56:34.095 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:56:34.095 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.27", 2024-08-18T15:56:34.095 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.4", 2024-08-18T15:56:34.096 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:56:34.096 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:56:34.096 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:56:34.096 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:56:34.096 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:56:34.096 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:56:34.096 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:56:34.096 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000014", 2024-08-18T15:56:34.096 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:56:34.096 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3.retry", 2024-08-18T15:56:34.096 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:56:34.096 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:56:34.096 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:56:34.096 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:56:34.097 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:56:34.097 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000112", 2024-08-18T15:56:34.097 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:56:34.097 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000007", 2024-08-18T15:56:34.097 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000021", 2024-08-18T15:56:34.097 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:56:34.097 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:56:34.097 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:56:34.097 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:56:34.097 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:56:34.097 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.16", 2024-08-18T15:56:34.097 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:56:34.097 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000064", 2024-08-18T15:56:34.098 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:56:34.098 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:56:34.098 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:56:34.098 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:56:34.098 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:56:34.098 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:56:34.098 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:56:34.098 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000071", 2024-08-18T15:56:34.098 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:56:34.098 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:56:34.098 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:56:34.098 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000074", 2024-08-18T15:56:34.099 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:56:34.099 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:56:34.099 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:56:34.099 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:56:34.099 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:56:34.099 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:56:34.099 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:56:34.099 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:56:34.099 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:56:34.099 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:56:34.099 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:56:34.099 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:56:34.099 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:56:34.100 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:56:34.100 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:56:34.100 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000081", 2024-08-18T15:56:34.100 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:56:34.100 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000009", 2024-08-18T15:56:34.100 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000121", 2024-08-18T15:56:34.100 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1", 2024-08-18T15:56:34.100 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:56:34.100 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:56:34.100 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:56:34.100 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:56:34.100 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:56:34.100 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:56:34.100 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.17", 2024-08-18T15:56:34.101 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000125", 2024-08-18T15:56:34.101 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:56:34.101 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000082", 2024-08-18T15:56:34.101 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:56:34.101 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:56:34.101 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:56:34.101 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:56:34.101 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:56:34.101 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:56:34.101 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:56:34.101 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:56:34.101 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:56:34.101 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:56:34.101 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:56:34.102 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000105", 2024-08-18T15:56:34.102 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000059", 2024-08-18T15:56:34.102 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:56:34.102 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:56:34.102 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:56:34.102 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:56:34.102 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:56:34.102 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:56:34.102 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:56:34.102 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:56:34.103 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:56:34.103 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:56:34.103 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:56:34.103 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000077", 2024-08-18T15:56:34.103 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000032", 2024-08-18T15:56:34.103 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:56:34.103 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:56:34.103 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:56:34.103 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:56:34.103 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:56:34.103 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:56:34.104 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.24", 2024-08-18T15:56:34.104 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:56:34.104 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:56:34.104 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:56:34.104 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:56:34.104 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3.retry", 2024-08-18T15:56:34.104 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:56:34.104 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:56:34.104 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:56:34.104 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:56:34.104 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:56:34.104 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:56:34.104 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000053", 2024-08-18T15:56:34.105 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000091", 2024-08-18T15:56:34.105 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:56:34.105 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:56:34.105 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:56:34.105 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000065", 2024-08-18T15:56:34.105 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:56:34.105 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:56:34.105 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:56:34.105 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:56:34.105 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000083", 2024-08-18T15:56:34.106 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000010", 2024-08-18T15:56:34.106 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:56:34.106 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:56:34.106 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:56:34.106 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:56:34.106 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:56:34.106 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:56:34.106 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:56:34.106 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:56:34.106 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:56:34.106 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:56:34.106 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:56:34.106 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.7", 2024-08-18T15:56:34.107 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.3", 2024-08-18T15:56:34.107 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:56:34.107 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:56:34.107 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:56:34.107 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:56:34.107 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:56:34.107 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:56:34.107 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:56:34.107 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:56:34.107 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.2", 2024-08-18T15:56:34.107 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:56:34.107 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:56:34.107 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:56:34.108 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:56:34.108 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:56:34.108 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:56:34.108 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:56:34.108 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000045", 2024-08-18T15:56:34.108 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000002", 2024-08-18T15:56:34.108 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:56:34.108 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3.2UbiLidwSTy.0", 2024-08-18T15:56:34.108 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:56:34.108 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.12", 2024-08-18T15:56:34.108 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:56:34.108 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:56:34.108 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:56:34.108 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:56:34.109 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:56:34.109 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:56:34.109 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000116", 2024-08-18T15:56:34.109 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.25", 2024-08-18T15:56:34.109 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:56:34.109 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:56:34.109 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:56:34.109 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:56:34.109 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000034", 2024-08-18T15:56:34.109 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:56:34.109 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:56:34.109 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000101", 2024-08-18T15:56:34.110 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:56:34.110 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:56:34.110 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.13", 2024-08-18T15:56:34.110 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:56:34.110 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:56:34.110 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.1", 2024-08-18T15:56:34.110 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0.retry", 2024-08-18T15:56:34.110 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:56:34.110 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:56:34.110 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000079", 2024-08-18T15:56:34.110 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000049", 2024-08-18T15:56:34.110 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:56:34.110 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0", 2024-08-18T15:56:34.111 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:56:34.111 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:56:34.111 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000117", 2024-08-18T15:56:34.111 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:56:34.111 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000044", 2024-08-18T15:56:34.111 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:56:34.111 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:56:34.111 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:56:34.111 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:56:34.111 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:56:34.111 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:56:34.111 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.14", 2024-08-18T15:56:34.111 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:56:34.111 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000066", 2024-08-18T15:56:34.112 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:56:34.112 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:56:34.112 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:56:34.112 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:56:34.112 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:56:34.112 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000068", 2024-08-18T15:56:34.112 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:56:34.112 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000085", 2024-08-18T15:56:34.112 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:56:34.112 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:56:34.112 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:56:34.112 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:56:34.112 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:56:34.113 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:56:34.113 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:56:34.113 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:56:34.113 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:56:34.113 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:56:34.113 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:56:34.113 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:56:34.113 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000073", 2024-08-18T15:56:34.113 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:56:34.113 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:56:34.113 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:56:34.113 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000038", 2024-08-18T15:56:34.113 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:56:34.113 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:56:34.114 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:56:34.114 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:56:34.114 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.10", 2024-08-18T15:56:34.114 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:56:34.114 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:56:34.114 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:56:34.114 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000118", 2024-08-18T15:56:34.114 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:56:34.114 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3", 2024-08-18T15:56:34.114 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:56:34.114 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000036", 2024-08-18T15:56:34.114 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:56:34.117 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:56:34.118 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:56:34.118 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:56:34.118 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:56:34.118 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000103", 2024-08-18T15:56:34.118 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.3", 2024-08-18T15:56:34.118 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:56:34.118 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000119", 2024-08-18T15:56:34.118 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000098", 2024-08-18T15:56:34.118 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:56:34.118 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:56:34.118 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:56:34.118 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:56:34.118 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:56:34.119 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:56:34.119 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:56:34.119 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:56:34.119 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.0", 2024-08-18T15:56:34.119 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:56:34.119 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:56:34.119 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:56:34.119 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:56:34.119 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:56:34.119 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:56:34.119 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:56:34.119 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:56:34.120 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000027", 2024-08-18T15:56:34.120 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:56:34.120 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status", 2024-08-18T15:56:34.120 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:56:34.120 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000019", 2024-08-18T15:56:34.120 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:56:34.120 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:56:34.120 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:56:34.120 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:56:34.120 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1", 2024-08-18T15:56:34.120 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:56:34.120 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:56:34.120 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:56:34.121 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:56:34.121 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000039", 2024-08-18T15:56:34.121 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:56:34.121 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:56:34.121 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:56:34.121 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:56:34.121 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:56:34.121 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:56:34.121 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:56:34.121 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:56:34.121 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:56:34.121 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:56:34.121 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:56:34.121 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000100", 2024-08-18T15:56:34.122 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:56:34.122 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:56:34.122 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:56:34.122 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:56:34.122 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:56:34.122 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000093", 2024-08-18T15:56:34.122 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:56:34.122 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000004", 2024-08-18T15:56:34.122 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000063", 2024-08-18T15:56:34.122 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000122", 2024-08-18T15:56:34.122 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:56:34.122 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:56:34.122 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:56:34.123 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000057", 2024-08-18T15:56:34.123 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:56:34.123 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:56:34.123 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:56:34.123 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:56:34.123 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000054", 2024-08-18T15:56:34.123 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:56:34.123 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000114", 2024-08-18T15:56:34.123 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:56:34.123 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:56:34.123 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:56:34.123 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.2", 2024-08-18T15:56:34.123 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:56:34.124 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:56:34.124 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0", 2024-08-18T15:56:34.124 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:56:34.124 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:56:34.124 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:56:34.124 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:56:34.124 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:56:34.124 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:56:34.124 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000012", 2024-08-18T15:56:34.124 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:56:34.124 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:56:34.124 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:56:34.124 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000084", 2024-08-18T15:56:34.124 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000043", 2024-08-18T15:56:34.125 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3", 2024-08-18T15:56:34.125 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:56:34.125 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:56:34.125 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.11", 2024-08-18T15:56:34.125 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:56:34.125 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:56:34.125 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:56:34.125 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:56:34.125 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:56:34.125 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:56:34.125 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:56:34.125 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:56:34.125 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000111", 2024-08-18T15:56:34.126 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:56:34.126 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000048", 2024-08-18T15:56:34.126 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:56:34.126 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000020", 2024-08-18T15:56:34.126 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:56:34.126 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:56:34.126 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:56:34.126 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:56:34.126 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:56:34.126 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000099", 2024-08-18T15:56:34.126 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000056", 2024-08-18T15:56:34.126 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:56:34.126 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:56:34.127 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.19", 2024-08-18T15:56:34.127 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:56:34.127 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.28", 2024-08-18T15:56:34.127 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:56:34.127 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:56:34.127 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:56:34.127 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:56:34.127 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:56:34.127 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000072", 2024-08-18T15:56:34.127 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:56:34.127 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:56:34.127 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000062", 2024-08-18T15:56:34.127 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:56:34.128 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:56:34.128 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:56:34.128 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000115", 2024-08-18T15:56:34.128 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:56:34.128 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.1", 2024-08-18T15:56:34.128 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2", 2024-08-18T15:56:34.128 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000033", 2024-08-18T15:56:34.128 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:56:34.128 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:56:34.128 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:56:34.128 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:56:34.128 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:56:34.128 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:56:34.129 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:56:34.129 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:56:34.129 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:56:34.129 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:56:34.129 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:56:34.129 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:56:34.129 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:56:34.129 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:56:34.129 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:56:34.129 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:56:34.129 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:56:34.129 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:56:34.129 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:56:34.129 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000025", 2024-08-18T15:56:34.130 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:56:34.130 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000123", 2024-08-18T15:56:34.130 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.31", 2024-08-18T15:56:34.130 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:56:34.130 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:56:34.130 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:56:34.130 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:56:34.130 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.18", 2024-08-18T15:56:34.130 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:56:34.130 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:56:34.130 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:56:34.130 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:56:34.130 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:56:34.131 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000108", 2024-08-18T15:56:34.131 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:56:34.131 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:56:34.131 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000094", 2024-08-18T15:56:34.131 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:56:34.131 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:56:34.131 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:56:34.131 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:56:34.131 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:56:34.132 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:56:34.132 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000060", 2024-08-18T15:56:34.132 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:56:34.132 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:56:34.132 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:56:34.132 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000109", 2024-08-18T15:56:34.132 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.15", 2024-08-18T15:56:34.132 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:56:34.132 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:56:34.132 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:56:34.132 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:56:34.132 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:56:34.132 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:56:34.133 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:56:34.133 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:56:34.133 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:56:34.133 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000013", 2024-08-18T15:56:34.133 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:56:34.133 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:56:34.134 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:56:34.134 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:56:34.134 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:56:34.134 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:56:34.134 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:56:34.134 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:56:34.134 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:56:34.134 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:56:34.134 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:56:34.134 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:56:34.135 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:56:34.135 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:56:34.135 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000051", 2024-08-18T15:56:34.135 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:56:34.135 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:56:34.135 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:56:34.135 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:56:34.135 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000106", 2024-08-18T15:56:34.135 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:56:34.135 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:56:34.135 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:56:34.135 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:56:34.135 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:56:34.136 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000017", 2024-08-18T15:56:34.136 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:56:34.136 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:56:34.136 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:56:34.136 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000023", 2024-08-18T15:56:34.136 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:56:34.136 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:56:34.136 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:56:34.136 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:56:34.136 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:56:34.136 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:56:34.136 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:56:34.136 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000107", 2024-08-18T15:56:34.137 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:56:34.137 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:56:34.137 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000046", 2024-08-18T15:56:34.137 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:56:34.137 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:56:34.137 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:56:34.137 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.26", 2024-08-18T15:56:34.137 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:56:34.137 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:56:34.138 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9", 2024-08-18T15:56:34.138 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.22", 2024-08-18T15:56:34.138 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:56:34.138 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:56:34.138 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:56:34.138 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:56:34.138 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:56:34.138 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:56:34.138 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:56:34.138 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2", 2024-08-18T15:56:34.138 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:56:34.138 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:56:34.138 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000067", 2024-08-18T15:56:34.139 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:56:34.139 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:56:34.139 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:56:34.139 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:56:34.139 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:56:34.139 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:56:34.139 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:56:34.139 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:56:34.139 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:56:34.139 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:56:34.139 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:56:34.139 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:56:34.139 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:56:34.140 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:56:34.140 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:56:34.140 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:56:34.140 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:56:34.140 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:56:34.140 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:56:34.140 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.9", 2024-08-18T15:56:34.140 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000086", 2024-08-18T15:56:34.140 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.5", 2024-08-18T15:56:34.140 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:56:34.140 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:56:34.140 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:56:34.140 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:56:34.141 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:56:34.141 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:56:34.141 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2.MFXJ583gCr1.0", 2024-08-18T15:56:34.141 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:56:34.141 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1.GPnO16liyZ+.0", 2024-08-18T15:56:34.141 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:56:34.141 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:56:34.141 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:56:34.141 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:56:34.141 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:56:34.141 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:56:34.141 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:56:34.141 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:56:34.141 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:56:34.142 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.20", 2024-08-18T15:56:34.142 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:56:34.142 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:56:34.142 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:56:34.142 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:56:34.142 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:56:34.142 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:56:34.142 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1.retry", 2024-08-18T15:56:34.142 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:56:34.142 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:56:34.142 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:56:34.142 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0.retry", 2024-08-18T15:56:34.143 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000050", 2024-08-18T15:56:34.143 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:56:34.143 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000102", 2024-08-18T15:56:34.143 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:56:34.143 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:56:34.143 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:56:34.143 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:56:34.143 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:56:34.143 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:56:34.143 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:56:34.143 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:56:34.143 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:56:34.143 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:56:34.144 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:56:34.144 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000124", 2024-08-18T15:56:34.144 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:56:34.144 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:56:34.144 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:56:34.144 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000001", 2024-08-18T15:56:34.144 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:56:34.144 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:56:34.144 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:56:34.144 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:56:34.144 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000126", 2024-08-18T15:56:34.145 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:56:34.145 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0.fZhctfLXnSt.0", 2024-08-18T15:56:34.145 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:56:34.145 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000055", 2024-08-18T15:56:34.145 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:56:34.145 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000061", 2024-08-18T15:56:34.145 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1", 2024-08-18T15:56:34.145 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:56:34.145 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:56:34.145 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:56:34.145 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000097", 2024-08-18T15:56:34.145 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:56:34.146 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:56:34.146 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:56:34.146 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:56:34.146 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:56:34.146 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:56:34.146 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:56:34.146 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:56:34.146 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000042", 2024-08-18T15:56:34.146 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:56:34.146 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:56:34.146 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:56:34.146 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1.retry", 2024-08-18T15:56:34.147 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:56:34.147 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:56:34.147 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:56:34.147 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:56:34.147 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:56:34.147 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:56:34.147 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:56:34.147 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000075", 2024-08-18T15:56:34.147 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:56:34.147 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000120", 2024-08-18T15:56:34.147 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000030", 2024-08-18T15:56:34.147 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:56:34.147 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:56:34.148 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:56:34.148 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:56:34.148 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:56:34.148 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:56:34.148 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9" 2024-08-18T15:56:34.148 INFO:teuthology.orchestra.run.smithi193.stdout:] 2024-08-18T15:56:34.148 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:56:39.150 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:56:39.150 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:56:39.151 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:56:39.202 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:56:39.203 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:56:39.266 INFO:teuthology.orchestra.run.smithi193.stdout:[ 2024-08-18T15:56:39.266 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:56:39.266 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:56:39.266 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.29", 2024-08-18T15:56:39.266 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000078", 2024-08-18T15:56:39.266 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.history", 2024-08-18T15:56:39.266 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:56:39.266 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:56:39.266 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:56:39.266 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:56:39.267 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.23", 2024-08-18T15:56:39.267 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000070", 2024-08-18T15:56:39.267 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000104", 2024-08-18T15:56:39.267 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:56:39.267 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2.retry", 2024-08-18T15:56:39.267 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:56:39.267 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:56:39.267 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:56:39.267 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:56:39.267 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000026", 2024-08-18T15:56:39.267 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:56:39.267 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:56:39.267 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:56:39.268 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:56:39.268 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:56:39.268 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:56:39.268 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.8", 2024-08-18T15:56:39.268 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:56:39.268 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:56:39.268 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:56:39.268 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:56:39.268 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000028", 2024-08-18T15:56:39.268 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:56:39.268 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:56:39.268 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:56:39.268 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:56:39.269 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:56:39.269 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:56:39.269 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:56:39.269 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2.retry", 2024-08-18T15:56:39.269 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:56:39.269 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.30", 2024-08-18T15:56:39.269 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:56:39.269 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:56:39.269 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:56:39.269 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:56:39.269 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000040", 2024-08-18T15:56:39.269 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:56:39.269 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.6", 2024-08-18T15:56:39.269 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:56:39.270 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:56:39.270 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:56:39.270 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:56:39.270 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:56:39.270 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:56:39.270 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000015", 2024-08-18T15:56:39.270 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:56:39.270 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:56:39.270 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000069", 2024-08-18T15:56:39.270 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:56:39.270 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:56:39.270 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:56:39.270 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000095", 2024-08-18T15:56:39.270 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0", 2024-08-18T15:56:39.271 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000003", 2024-08-18T15:56:39.271 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000047", 2024-08-18T15:56:39.271 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:56:39.271 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:56:39.271 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:56:39.271 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:56:39.271 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:56:39.271 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:56:39.271 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:56:39.271 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:56:39.271 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:56:39.271 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000035", 2024-08-18T15:56:39.271 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:56:39.272 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:56:39.272 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000037", 2024-08-18T15:56:39.272 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:56:39.351 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:56:39.351 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:56:39.351 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:56:39.352 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:56:39.352 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:56:39.352 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:56:39.352 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:56:39.352 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:56:39.352 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:56:39.352 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:56:39.352 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:56:39.352 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:56:39.352 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:56:39.353 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000024", 2024-08-18T15:56:39.353 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:56:39.353 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:56:39.353 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:56:39.353 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:56:39.353 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000000", 2024-08-18T15:56:39.353 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:56:39.353 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000031", 2024-08-18T15:56:39.353 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:56:39.353 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:56:39.353 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:56:39.354 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:56:39.354 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:56:39.354 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:56:39.354 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3", 2024-08-18T15:56:39.354 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:56:39.354 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:56:39.354 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:56:39.355 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:56:39.355 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:56:39.355 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:56:39.355 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:56:39.355 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:56:39.355 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:56:39.355 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:56:39.355 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:56:39.355 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:56:39.356 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:56:39.356 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:56:39.356 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:56:39.356 INFO:teuthology.orchestra.run.smithi193.stdout: "data_loggenerations_metadata", 2024-08-18T15:56:39.356 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2", 2024-08-18T15:56:39.356 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:56:39.356 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:56:39.356 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:56:39.356 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:56:39.356 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T15:56:39.356 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:56:39.356 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:56:39.356 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:56:39.357 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:56:39.357 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:56:39.357 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000076", 2024-08-18T15:56:39.357 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:56:39.357 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:56:39.357 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000113", 2024-08-18T15:56:39.357 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000005", 2024-08-18T15:56:39.357 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:56:39.357 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000011", 2024-08-18T15:56:39.357 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:56:39.357 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000058", 2024-08-18T15:56:39.358 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:56:39.358 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:56:39.358 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:56:39.358 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000052", 2024-08-18T15:56:39.358 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:56:39.358 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:56:39.358 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:56:39.358 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:56:39.358 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.21", 2024-08-18T15:56:39.358 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:56:39.359 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:56:39.359 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:56:39.359 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:56:39.359 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:56:39.359 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:56:39.359 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:56:39.359 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:56:39.359 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:56:39.360 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:56:39.360 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:56:39.360 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:56:39.360 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:56:39.360 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:56:39.360 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:56:39.360 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:56:39.360 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:56:39.361 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:56:39.361 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:56:39.361 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000088", 2024-08-18T15:56:39.361 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000080", 2024-08-18T15:56:39.361 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:56:39.361 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:56:39.361 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000090", 2024-08-18T15:56:39.361 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:56:39.361 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:56:39.361 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000110", 2024-08-18T15:56:39.361 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:56:39.362 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:56:39.362 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:56:39.362 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.0", 2024-08-18T15:56:39.362 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:56:39.362 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000096", 2024-08-18T15:56:39.362 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:56:39.363 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:56:39.363 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:56:39.363 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:56:39.363 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:56:39.363 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000087", 2024-08-18T15:56:39.363 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:56:39.364 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:56:39.364 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:56:39.364 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:56:39.364 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000008", 2024-08-18T15:56:39.364 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:56:39.364 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:56:39.364 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:56:39.364 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:56:39.364 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:56:39.365 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000006", 2024-08-18T15:56:39.365 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000029", 2024-08-18T15:56:39.365 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:56:39.365 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000089", 2024-08-18T15:56:39.365 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:56:39.365 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:56:39.365 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000022", 2024-08-18T15:56:39.365 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:56:39.365 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:56:39.365 INFO:teuthology.orchestra.run.smithi193.stdout: "bilog.trim", 2024-08-18T15:56:39.365 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000016", 2024-08-18T15:56:39.365 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:56:39.366 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:56:39.366 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:56:39.366 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:56:39.366 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:56:39.366 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000041", 2024-08-18T15:56:39.366 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:56:39.366 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:56:39.366 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.log.01050e30-179b-43be-bdbb-3bb23cf37ae8.3", 2024-08-18T15:56:39.366 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:56:39.367 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000018", 2024-08-18T15:56:39.367 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000092", 2024-08-18T15:56:39.368 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:56:39.368 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:56:39.368 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:56:39.368 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:56:39.368 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.27", 2024-08-18T15:56:39.368 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.4", 2024-08-18T15:56:39.368 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:56:39.368 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:56:39.368 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:56:39.368 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:56:39.368 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:56:39.368 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:56:39.369 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:56:39.369 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000014", 2024-08-18T15:56:39.369 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:56:39.369 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3.retry", 2024-08-18T15:56:39.369 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:56:39.369 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:56:39.369 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:56:39.369 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:56:39.369 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:56:39.369 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000112", 2024-08-18T15:56:39.369 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:56:39.369 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000007", 2024-08-18T15:56:39.369 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000021", 2024-08-18T15:56:39.370 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:56:39.370 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:56:39.370 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:56:39.370 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:56:39.370 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:56:39.370 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.16", 2024-08-18T15:56:39.370 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:56:39.370 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000064", 2024-08-18T15:56:39.370 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:56:39.370 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:56:39.371 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:56:39.371 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:56:39.371 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:56:39.371 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:56:39.371 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:56:39.371 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000071", 2024-08-18T15:56:39.371 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:56:39.371 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:56:39.371 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:56:39.371 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000074", 2024-08-18T15:56:39.371 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:56:39.371 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:56:39.372 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:56:39.372 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:56:39.372 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:56:39.372 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:56:39.372 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:56:39.372 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:56:39.372 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:56:39.372 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:56:39.372 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:56:39.372 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:56:39.372 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:56:39.372 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:56:39.372 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:56:39.372 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000081", 2024-08-18T15:56:39.373 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:56:39.373 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000009", 2024-08-18T15:56:39.373 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000121", 2024-08-18T15:56:39.373 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1", 2024-08-18T15:56:39.373 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:56:39.373 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:56:39.373 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:56:39.373 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:56:39.373 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:56:39.373 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:56:39.373 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.17", 2024-08-18T15:56:39.373 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000125", 2024-08-18T15:56:39.373 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:56:39.374 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000082", 2024-08-18T15:56:39.374 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:56:39.374 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:56:39.374 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:56:39.374 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:56:39.374 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:56:39.374 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:56:39.374 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:56:39.374 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:56:39.374 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:56:39.374 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:56:39.374 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:56:39.375 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000105", 2024-08-18T15:56:39.375 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000059", 2024-08-18T15:56:39.375 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:56:39.375 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:56:39.375 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:56:39.375 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:56:39.375 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:56:39.375 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:56:39.375 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:56:39.375 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:56:39.375 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:56:39.375 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:56:39.375 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:56:39.376 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000077", 2024-08-18T15:56:39.376 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000032", 2024-08-18T15:56:39.376 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:56:39.376 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:56:39.376 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:56:39.376 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:56:39.376 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:56:39.376 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:56:39.377 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.24", 2024-08-18T15:56:39.377 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:56:39.377 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:56:39.377 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:56:39.377 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:56:39.377 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3.retry", 2024-08-18T15:56:39.377 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:56:39.377 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:56:39.378 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:56:39.378 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:56:39.378 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:56:39.378 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:56:39.378 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000053", 2024-08-18T15:56:39.378 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000091", 2024-08-18T15:56:39.378 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:56:39.378 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:56:39.378 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:56:39.378 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000065", 2024-08-18T15:56:39.378 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:56:39.378 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:56:39.378 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:56:39.378 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:56:39.379 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000083", 2024-08-18T15:56:39.379 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000010", 2024-08-18T15:56:39.379 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:56:39.379 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:56:39.379 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:56:39.379 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:56:39.379 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:56:39.379 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:56:39.379 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:56:39.379 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:56:39.379 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:56:39.379 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:56:39.379 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:56:39.380 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.7", 2024-08-18T15:56:39.380 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.3", 2024-08-18T15:56:39.380 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:56:39.380 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:56:39.380 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:56:39.380 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:56:39.380 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:56:39.380 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:56:39.380 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:56:39.380 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:56:39.380 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.2", 2024-08-18T15:56:39.380 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:56:39.380 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:56:39.380 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:56:39.380 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:56:39.381 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:56:39.381 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:56:39.381 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:56:39.381 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000045", 2024-08-18T15:56:39.381 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000002", 2024-08-18T15:56:39.381 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:56:39.381 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3.2UbiLidwSTy.0", 2024-08-18T15:56:39.381 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:56:39.381 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.12", 2024-08-18T15:56:39.381 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:56:39.381 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:56:39.381 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:56:39.381 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:56:39.381 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:56:39.382 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:56:39.382 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000116", 2024-08-18T15:56:39.382 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.25", 2024-08-18T15:56:39.382 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:56:39.382 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:56:39.382 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:56:39.382 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:56:39.382 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000034", 2024-08-18T15:56:39.382 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:56:39.382 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:56:39.382 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000101", 2024-08-18T15:56:39.382 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:56:39.382 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:56:39.383 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.13", 2024-08-18T15:56:39.383 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:56:39.383 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:56:39.383 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.1", 2024-08-18T15:56:39.383 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0.retry", 2024-08-18T15:56:39.383 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:56:39.383 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:56:39.383 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000079", 2024-08-18T15:56:39.383 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000049", 2024-08-18T15:56:39.383 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:56:39.383 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0", 2024-08-18T15:56:39.383 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:56:39.383 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:56:39.384 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000117", 2024-08-18T15:56:39.384 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:56:39.384 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000044", 2024-08-18T15:56:39.384 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:56:39.384 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:56:39.384 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:56:39.384 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:56:39.384 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:56:39.384 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:56:39.384 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.14", 2024-08-18T15:56:39.384 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:56:39.384 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000066", 2024-08-18T15:56:39.384 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:56:39.384 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:56:39.384 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:56:39.385 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:56:39.385 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:56:39.385 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000068", 2024-08-18T15:56:39.385 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:56:39.385 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000085", 2024-08-18T15:56:39.385 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:56:39.385 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:56:39.385 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:56:39.385 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:56:39.385 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:56:39.385 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:56:39.385 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:56:39.385 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:56:39.386 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:56:39.386 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:56:39.386 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:56:39.386 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:56:39.386 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000073", 2024-08-18T15:56:39.386 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:56:39.386 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:56:39.386 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:56:39.386 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000038", 2024-08-18T15:56:39.386 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:56:39.387 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:56:39.387 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:56:39.387 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:56:39.387 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.10", 2024-08-18T15:56:39.387 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:56:39.387 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:56:39.387 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:56:39.387 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000118", 2024-08-18T15:56:39.387 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:56:39.387 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3", 2024-08-18T15:56:39.387 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:56:39.387 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000036", 2024-08-18T15:56:39.388 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:56:39.388 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:56:39.388 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:56:39.388 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:56:39.388 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:56:39.388 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000103", 2024-08-18T15:56:39.388 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.3", 2024-08-18T15:56:39.388 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:56:39.388 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000119", 2024-08-18T15:56:39.388 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000098", 2024-08-18T15:56:39.388 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:56:39.388 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:56:39.389 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:56:39.389 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:56:39.389 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:56:39.389 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:56:39.389 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:56:39.389 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:56:39.389 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.0", 2024-08-18T15:56:39.389 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:56:39.389 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:56:39.389 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:56:39.389 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:56:39.389 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:56:39.389 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:56:39.389 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:56:39.390 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:56:39.390 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000027", 2024-08-18T15:56:39.390 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:56:39.390 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status", 2024-08-18T15:56:39.390 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:56:39.390 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000019", 2024-08-18T15:56:39.390 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:56:39.390 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:56:39.390 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:56:39.390 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:56:39.390 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1", 2024-08-18T15:56:39.390 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:56:39.390 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:56:39.391 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:56:39.391 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:56:39.391 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000039", 2024-08-18T15:56:39.391 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:56:39.391 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:56:39.391 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:56:39.391 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:56:39.391 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:56:39.391 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:56:39.391 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:56:39.391 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:56:39.391 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:56:39.391 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:56:39.391 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:56:39.392 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000100", 2024-08-18T15:56:39.392 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:56:39.392 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:56:39.392 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:56:39.392 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:56:39.392 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:56:39.392 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000093", 2024-08-18T15:56:39.392 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:56:39.392 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000004", 2024-08-18T15:56:39.392 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000063", 2024-08-18T15:56:39.392 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000122", 2024-08-18T15:56:39.392 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:56:39.392 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:56:39.392 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:56:39.393 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000057", 2024-08-18T15:56:39.393 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:56:39.393 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:56:39.393 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:56:39.393 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:56:39.393 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000054", 2024-08-18T15:56:39.393 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:56:39.393 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000114", 2024-08-18T15:56:39.393 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:56:39.393 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:56:39.393 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:56:39.393 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.2", 2024-08-18T15:56:39.393 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:56:39.394 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:56:39.394 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0", 2024-08-18T15:56:39.394 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:56:39.394 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:56:39.394 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:56:39.394 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:56:39.394 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:56:39.394 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:56:39.394 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000012", 2024-08-18T15:56:39.394 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:56:39.394 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:56:39.394 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:56:39.394 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000084", 2024-08-18T15:56:39.394 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000043", 2024-08-18T15:56:39.395 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3", 2024-08-18T15:56:39.395 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:56:39.395 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:56:39.395 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.11", 2024-08-18T15:56:39.395 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:56:39.395 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:56:39.395 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:56:39.395 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:56:39.395 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:56:39.395 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:56:39.395 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:56:39.395 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:56:39.396 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000111", 2024-08-18T15:56:39.396 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:56:39.396 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000048", 2024-08-18T15:56:39.396 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:56:39.396 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000020", 2024-08-18T15:56:39.396 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:56:39.396 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:56:39.396 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:56:39.396 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:56:39.396 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:56:39.396 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000099", 2024-08-18T15:56:39.396 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000056", 2024-08-18T15:56:39.396 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:56:39.396 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:56:39.397 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.19", 2024-08-18T15:56:39.397 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:56:39.397 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.28", 2024-08-18T15:56:39.397 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:56:39.397 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:56:39.397 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:56:39.397 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:56:39.397 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:56:39.397 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000072", 2024-08-18T15:56:39.397 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:56:39.397 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:56:39.397 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000062", 2024-08-18T15:56:39.397 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:56:39.398 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:56:39.398 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:56:39.398 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000115", 2024-08-18T15:56:39.398 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:56:39.398 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.1", 2024-08-18T15:56:39.398 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2", 2024-08-18T15:56:39.398 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000033", 2024-08-18T15:56:39.398 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:56:39.398 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:56:39.398 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:56:39.398 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:56:39.398 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:56:39.398 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:56:39.399 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:56:39.399 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:56:39.399 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:56:39.399 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:56:39.399 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:56:39.399 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:56:39.399 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:56:39.399 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:56:39.399 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:56:39.399 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:56:39.399 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:56:39.399 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:56:39.399 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:56:39.399 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000025", 2024-08-18T15:56:39.400 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:56:39.400 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000123", 2024-08-18T15:56:39.400 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.31", 2024-08-18T15:56:39.400 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:56:39.400 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:56:39.400 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:56:39.400 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:56:39.400 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.18", 2024-08-18T15:56:39.400 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:56:39.400 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:56:39.400 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:56:39.400 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:56:39.400 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:56:39.400 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000108", 2024-08-18T15:56:39.401 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:56:39.401 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:56:39.401 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000094", 2024-08-18T15:56:39.401 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:56:39.401 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:56:39.401 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:56:39.401 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:56:39.401 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:56:39.401 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:56:39.401 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000060", 2024-08-18T15:56:39.401 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:56:39.401 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:56:39.401 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:56:39.402 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000109", 2024-08-18T15:56:39.402 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.15", 2024-08-18T15:56:39.402 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:56:39.402 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:56:39.402 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:56:39.402 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:56:39.402 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:56:39.402 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:56:39.402 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:56:39.402 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:56:39.402 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:56:39.402 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000013", 2024-08-18T15:56:39.402 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:56:39.403 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:56:39.403 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:56:39.403 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:56:39.403 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:56:39.403 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:56:39.403 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:56:39.403 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:56:39.403 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:56:39.403 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:56:39.403 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:56:39.403 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:56:39.403 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:56:39.403 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:56:39.403 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000051", 2024-08-18T15:56:39.404 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:56:39.404 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:56:39.404 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:56:39.404 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:56:39.404 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000106", 2024-08-18T15:56:39.404 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:56:39.404 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:56:39.404 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:56:39.404 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:56:39.404 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:56:39.404 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000017", 2024-08-18T15:56:39.404 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:56:39.404 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:56:39.404 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:56:39.405 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000023", 2024-08-18T15:56:39.405 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:56:39.405 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:56:39.405 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:56:39.405 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:56:39.405 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:56:39.405 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:56:39.405 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:56:39.405 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000107", 2024-08-18T15:56:39.405 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:56:39.405 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:56:39.405 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000046", 2024-08-18T15:56:39.405 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:56:39.405 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:56:39.406 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:56:39.406 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.26", 2024-08-18T15:56:39.406 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:56:39.406 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:56:39.406 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9", 2024-08-18T15:56:39.406 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.22", 2024-08-18T15:56:39.406 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:56:39.406 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:56:39.406 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:56:39.406 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:56:39.406 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:56:39.406 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:56:39.406 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:56:39.407 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2", 2024-08-18T15:56:39.407 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:56:39.407 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:56:39.407 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000067", 2024-08-18T15:56:39.407 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:56:39.407 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:56:39.407 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:56:39.407 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:56:39.407 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:56:39.407 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:56:39.407 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:56:39.407 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:56:39.407 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:56:39.407 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:56:39.408 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:56:39.408 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:56:39.408 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:56:39.408 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:56:39.408 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:56:39.408 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:56:39.408 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:56:39.408 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:56:39.408 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:56:39.408 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.9", 2024-08-18T15:56:39.408 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000086", 2024-08-18T15:56:39.408 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.5", 2024-08-18T15:56:39.408 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:56:39.409 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:56:39.409 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:56:39.409 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:56:39.409 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:56:39.409 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:56:39.409 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2.MFXJ583gCr1.0", 2024-08-18T15:56:39.409 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:56:39.409 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1.GPnO16liyZ+.0", 2024-08-18T15:56:39.409 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:56:39.409 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:56:39.409 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:56:39.409 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:56:39.409 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:56:39.409 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:56:39.410 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:56:39.410 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:56:39.410 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:56:39.410 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.20", 2024-08-18T15:56:39.410 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:56:39.410 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:56:39.410 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:56:39.410 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:56:39.410 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:56:39.410 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:56:39.410 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1.retry", 2024-08-18T15:56:39.410 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:56:39.410 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:56:39.411 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:56:39.411 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0.retry", 2024-08-18T15:56:39.411 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000050", 2024-08-18T15:56:39.411 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:56:39.411 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000102", 2024-08-18T15:56:39.411 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:56:39.411 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:56:39.411 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:56:39.411 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:56:39.411 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:56:39.411 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:56:39.411 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:56:39.411 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:56:39.411 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:56:39.412 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:56:39.412 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:56:39.412 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000124", 2024-08-18T15:56:39.412 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:56:39.412 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:56:39.412 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:56:39.412 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000001", 2024-08-18T15:56:39.412 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:56:39.412 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:56:39.412 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:56:39.412 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:56:39.412 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000126", 2024-08-18T15:56:39.412 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:56:39.412 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0.fZhctfLXnSt.0", 2024-08-18T15:56:39.413 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:56:39.413 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000055", 2024-08-18T15:56:39.413 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:56:39.413 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000061", 2024-08-18T15:56:39.413 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1", 2024-08-18T15:56:39.413 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:56:39.413 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:56:39.413 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:56:39.413 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000097", 2024-08-18T15:56:39.413 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:56:39.413 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:56:39.413 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:56:39.413 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:56:39.414 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:56:39.414 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:56:39.414 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:56:39.414 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:56:39.414 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000042", 2024-08-18T15:56:39.414 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:56:39.414 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:56:39.414 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:56:39.414 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1.retry", 2024-08-18T15:56:39.414 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:56:39.414 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:56:39.414 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:56:39.414 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:56:39.415 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:56:39.415 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:56:39.415 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:56:39.415 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000075", 2024-08-18T15:56:39.415 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:56:39.415 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000120", 2024-08-18T15:56:39.415 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000030", 2024-08-18T15:56:39.415 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:56:39.415 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:56:39.415 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:56:39.415 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:56:39.415 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:56:39.415 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:56:39.415 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9" 2024-08-18T15:56:39.416 INFO:teuthology.orchestra.run.smithi193.stdout:] 2024-08-18T15:56:39.416 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:56:44.417 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:56:44.418 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:56:44.418 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:56:44.471 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:56:44.471 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:56:44.535 INFO:teuthology.orchestra.run.smithi193.stdout:[ 2024-08-18T15:56:44.535 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:56:44.535 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:56:44.535 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.29", 2024-08-18T15:56:44.536 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000078", 2024-08-18T15:56:44.536 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.history", 2024-08-18T15:56:44.536 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:56:44.536 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:56:44.536 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:56:44.536 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:56:44.536 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.23", 2024-08-18T15:56:44.536 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000070", 2024-08-18T15:56:44.536 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000104", 2024-08-18T15:56:44.536 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:56:44.536 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2.retry", 2024-08-18T15:56:44.536 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:56:44.536 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:56:44.536 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:56:44.537 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:56:44.537 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000026", 2024-08-18T15:56:44.537 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:56:44.537 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:56:44.537 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:56:44.537 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:56:44.537 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:56:44.537 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:56:44.537 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.8", 2024-08-18T15:56:44.537 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:56:44.537 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:56:44.537 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:56:44.537 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:56:44.538 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000028", 2024-08-18T15:56:44.538 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:56:44.538 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:56:44.538 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:56:44.538 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:56:44.538 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:56:44.538 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:56:44.538 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:56:44.538 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2.retry", 2024-08-18T15:56:44.538 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:56:44.538 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.30", 2024-08-18T15:56:44.538 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:56:44.538 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:56:44.538 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:56:44.539 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:56:44.539 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000040", 2024-08-18T15:56:44.539 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:56:44.539 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.6", 2024-08-18T15:56:44.539 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:56:44.539 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:56:44.539 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:56:44.539 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:56:44.539 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:56:44.539 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:56:44.539 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000015", 2024-08-18T15:56:44.539 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:56:44.539 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:56:44.539 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000069", 2024-08-18T15:56:44.540 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:56:44.540 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:56:44.540 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:56:44.540 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000095", 2024-08-18T15:56:44.540 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0", 2024-08-18T15:56:44.540 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000003", 2024-08-18T15:56:44.540 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000047", 2024-08-18T15:56:44.540 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:56:44.540 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:56:44.540 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:56:44.540 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:56:44.540 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:56:44.540 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:56:44.540 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:56:44.541 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:56:44.541 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:56:44.541 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000035", 2024-08-18T15:56:44.541 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:56:44.541 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:56:44.541 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000037", 2024-08-18T15:56:44.541 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:56:44.541 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:56:44.541 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:56:44.541 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:56:44.542 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:56:44.542 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:56:44.542 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:56:44.542 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:56:44.542 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:56:44.542 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:56:44.542 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:56:44.542 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:56:44.542 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:56:44.542 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:56:44.542 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000024", 2024-08-18T15:56:44.542 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:56:44.543 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:56:44.543 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:56:44.543 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:56:44.543 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000000", 2024-08-18T15:56:44.543 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:56:44.543 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000031", 2024-08-18T15:56:44.543 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:56:44.543 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:56:44.543 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:56:44.543 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:56:44.543 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:56:44.543 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:56:44.543 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3", 2024-08-18T15:56:44.544 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:56:44.544 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:56:44.544 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:56:44.544 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:56:44.544 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:56:44.544 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:56:44.544 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:56:44.544 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:56:44.544 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:56:44.544 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:56:44.544 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:56:44.544 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:56:44.544 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:56:44.544 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:56:44.545 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:56:44.545 INFO:teuthology.orchestra.run.smithi193.stdout: "data_loggenerations_metadata", 2024-08-18T15:56:44.545 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2", 2024-08-18T15:56:44.545 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:56:44.545 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:56:44.545 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:56:44.545 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:56:44.545 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T15:56:44.545 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:56:44.545 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:56:44.545 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:56:44.545 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:56:44.545 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:56:44.546 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000076", 2024-08-18T15:56:44.546 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:56:44.546 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:56:44.546 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000113", 2024-08-18T15:56:44.546 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000005", 2024-08-18T15:56:44.546 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:56:44.546 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000011", 2024-08-18T15:56:44.546 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:56:44.546 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000058", 2024-08-18T15:56:44.546 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:56:44.546 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:56:44.546 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:56:44.546 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000052", 2024-08-18T15:56:44.547 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:56:44.547 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:56:44.547 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:56:44.547 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:56:44.547 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.21", 2024-08-18T15:56:44.547 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:56:44.547 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:56:44.547 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:56:44.547 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:56:44.547 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:56:44.547 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:56:44.547 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:56:44.547 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:56:44.548 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:56:44.548 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:56:44.548 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:56:44.548 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:56:44.548 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:56:44.548 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:56:44.548 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:56:44.548 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:56:44.548 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:56:44.548 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:56:44.548 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:56:44.548 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000088", 2024-08-18T15:56:44.549 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000080", 2024-08-18T15:56:44.549 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:56:44.549 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:56:44.549 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000090", 2024-08-18T15:56:44.549 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:56:44.549 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:56:44.549 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000110", 2024-08-18T15:56:44.549 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:56:44.549 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:56:44.549 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:56:44.549 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.0", 2024-08-18T15:56:44.549 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:56:44.549 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000096", 2024-08-18T15:56:44.549 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:56:44.550 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:56:44.550 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:56:44.550 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:56:44.550 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:56:44.550 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000087", 2024-08-18T15:56:44.550 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:56:44.550 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:56:44.550 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:56:44.550 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:56:44.550 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000008", 2024-08-18T15:56:44.550 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:56:44.550 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:56:44.550 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:56:44.551 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:56:44.551 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:56:44.551 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000006", 2024-08-18T15:56:44.551 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000029", 2024-08-18T15:56:44.551 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:56:44.551 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000089", 2024-08-18T15:56:44.551 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:56:44.551 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:56:44.551 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000022", 2024-08-18T15:56:44.551 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:56:44.551 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:56:44.551 INFO:teuthology.orchestra.run.smithi193.stdout: "bilog.trim", 2024-08-18T15:56:44.552 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000016", 2024-08-18T15:56:44.552 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:56:44.552 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:56:44.552 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:56:44.552 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:56:44.552 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:56:44.552 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000041", 2024-08-18T15:56:44.552 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:56:44.552 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:56:44.552 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.log.01050e30-179b-43be-bdbb-3bb23cf37ae8.3", 2024-08-18T15:56:44.552 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:56:44.552 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000018", 2024-08-18T15:56:44.552 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000092", 2024-08-18T15:56:44.553 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:56:44.553 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:56:44.553 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:56:44.553 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:56:44.553 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.27", 2024-08-18T15:56:44.553 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.4", 2024-08-18T15:56:44.553 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:56:44.553 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:56:44.553 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:56:44.553 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:56:44.553 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:56:44.553 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:56:44.553 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:56:44.554 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000014", 2024-08-18T15:56:44.554 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:56:44.554 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3.retry", 2024-08-18T15:56:44.554 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:56:44.554 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:56:44.554 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:56:44.554 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:56:44.554 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:56:44.554 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000112", 2024-08-18T15:56:44.554 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:56:44.554 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000007", 2024-08-18T15:56:44.554 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000021", 2024-08-18T15:56:44.554 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:56:44.555 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:56:44.555 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:56:44.555 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:56:44.555 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:56:44.555 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.16", 2024-08-18T15:56:44.555 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:56:44.555 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000064", 2024-08-18T15:56:44.555 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:56:44.555 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:56:44.555 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:56:44.555 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:56:44.555 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:56:44.555 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:56:44.556 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:56:44.556 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000071", 2024-08-18T15:56:44.556 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:56:44.556 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:56:44.556 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:56:44.556 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000074", 2024-08-18T15:56:44.556 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:56:44.556 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:56:44.556 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:56:44.556 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:56:44.556 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:56:44.556 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:56:44.556 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:56:44.556 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:56:44.557 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:56:44.557 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:56:44.557 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:56:44.557 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:56:44.557 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:56:44.557 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:56:44.557 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:56:44.557 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000081", 2024-08-18T15:56:44.557 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:56:44.557 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000009", 2024-08-18T15:56:44.557 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000121", 2024-08-18T15:56:44.557 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1", 2024-08-18T15:56:44.558 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:56:44.558 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:56:44.558 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:56:44.558 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:56:44.558 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:56:44.558 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:56:44.558 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.17", 2024-08-18T15:56:44.558 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000125", 2024-08-18T15:56:44.558 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:56:44.558 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000082", 2024-08-18T15:56:44.559 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:56:44.559 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:56:44.559 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:56:44.559 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:56:44.559 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:56:44.559 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:56:44.559 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:56:44.559 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:56:44.559 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:56:44.559 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:56:44.559 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:56:44.559 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000105", 2024-08-18T15:56:44.560 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000059", 2024-08-18T15:56:44.560 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:56:44.560 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:56:44.560 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:56:44.560 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:56:44.560 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:56:44.560 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:56:44.560 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:56:44.560 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:56:44.560 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:56:44.560 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:56:44.560 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:56:44.560 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000077", 2024-08-18T15:56:44.561 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000032", 2024-08-18T15:56:44.561 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:56:44.561 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:56:44.561 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:56:44.561 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:56:44.561 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:56:44.561 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:56:44.561 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.24", 2024-08-18T15:56:44.561 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:56:44.561 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:56:44.561 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:56:44.561 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:56:44.561 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3.retry", 2024-08-18T15:56:44.562 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:56:44.562 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:56:44.562 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:56:44.562 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:56:44.562 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:56:44.562 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:56:44.562 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000053", 2024-08-18T15:56:44.562 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000091", 2024-08-18T15:56:44.562 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:56:44.562 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:56:44.562 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:56:44.562 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000065", 2024-08-18T15:56:44.562 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:56:44.563 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:56:44.563 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:56:44.563 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:56:44.563 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000083", 2024-08-18T15:56:44.563 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000010", 2024-08-18T15:56:44.563 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:56:44.563 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:56:44.563 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:56:44.563 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:56:44.563 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:56:44.564 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:56:44.564 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:56:44.564 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:56:44.564 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:56:44.564 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:56:44.564 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:56:44.564 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.7", 2024-08-18T15:56:44.564 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.3", 2024-08-18T15:56:44.564 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:56:44.564 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:56:44.564 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:56:44.564 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:56:44.564 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:56:44.565 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:56:44.565 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:56:44.565 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:56:44.565 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.2", 2024-08-18T15:56:44.565 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:56:44.565 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:56:44.565 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:56:44.565 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:56:44.565 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:56:44.565 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:56:44.565 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:56:44.565 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000045", 2024-08-18T15:56:44.566 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000002", 2024-08-18T15:56:44.566 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:56:44.566 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3.2UbiLidwSTy.0", 2024-08-18T15:56:44.566 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:56:44.566 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.12", 2024-08-18T15:56:44.566 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:56:44.566 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:56:44.566 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:56:44.566 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:56:44.566 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:56:44.566 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:56:44.566 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000116", 2024-08-18T15:56:44.566 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.25", 2024-08-18T15:56:44.566 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:56:44.567 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:56:44.567 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:56:44.567 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:56:44.567 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000034", 2024-08-18T15:56:44.567 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:56:44.567 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:56:44.567 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000101", 2024-08-18T15:56:44.567 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:56:44.567 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:56:44.567 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.13", 2024-08-18T15:56:44.567 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:56:44.567 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:56:44.568 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.1", 2024-08-18T15:56:44.568 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0.retry", 2024-08-18T15:56:44.568 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:56:44.568 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:56:44.568 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000079", 2024-08-18T15:56:44.568 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000049", 2024-08-18T15:56:44.568 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:56:44.568 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0", 2024-08-18T15:56:44.568 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:56:44.568 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:56:44.568 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000117", 2024-08-18T15:56:44.568 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:56:44.568 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000044", 2024-08-18T15:56:44.569 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:56:44.569 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:56:44.569 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:56:44.569 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:56:44.569 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:56:44.569 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:56:44.569 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.14", 2024-08-18T15:56:44.569 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:56:44.569 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000066", 2024-08-18T15:56:44.569 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:56:44.569 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:56:44.569 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:56:44.569 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:56:44.570 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:56:44.570 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000068", 2024-08-18T15:56:44.570 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:56:44.570 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000085", 2024-08-18T15:56:44.570 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:56:44.570 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:56:44.570 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:56:44.570 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:56:44.570 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:56:44.570 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:56:44.574 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:56:44.574 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:56:44.574 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:56:44.574 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:56:44.574 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:56:44.574 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:56:44.574 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000073", 2024-08-18T15:56:44.574 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:56:44.574 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:56:44.574 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:56:44.574 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000038", 2024-08-18T15:56:44.575 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:56:44.575 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:56:44.575 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:56:44.575 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:56:44.575 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.10", 2024-08-18T15:56:44.575 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:56:44.575 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:56:44.575 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:56:44.575 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000118", 2024-08-18T15:56:44.575 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:56:44.575 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3", 2024-08-18T15:56:44.575 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:56:44.575 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000036", 2024-08-18T15:56:44.575 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:56:44.576 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:56:44.576 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:56:44.576 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:56:44.576 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:56:44.576 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000103", 2024-08-18T15:56:44.576 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.3", 2024-08-18T15:56:44.576 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:56:44.576 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000119", 2024-08-18T15:56:44.576 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000098", 2024-08-18T15:56:44.576 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:56:44.576 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:56:44.576 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:56:44.576 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:56:44.577 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:56:44.577 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:56:44.577 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:56:44.577 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:56:44.577 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.0", 2024-08-18T15:56:44.577 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:56:44.577 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:56:44.577 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:56:44.577 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:56:44.577 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:56:44.577 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:56:44.577 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:56:44.577 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:56:44.577 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000027", 2024-08-18T15:56:44.578 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:56:44.578 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status", 2024-08-18T15:56:44.578 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:56:44.578 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000019", 2024-08-18T15:56:44.578 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:56:44.578 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:56:44.578 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:56:44.578 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:56:44.578 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1", 2024-08-18T15:56:44.578 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:56:44.578 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:56:44.578 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:56:44.578 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:56:44.579 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000039", 2024-08-18T15:56:44.579 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:56:44.579 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:56:44.579 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:56:44.579 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:56:44.579 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:56:44.579 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:56:44.579 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:56:44.579 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:56:44.579 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:56:44.579 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:56:44.579 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:56:44.579 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000100", 2024-08-18T15:56:44.579 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:56:44.580 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:56:44.580 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:56:44.580 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:56:44.580 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:56:44.580 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000093", 2024-08-18T15:56:44.580 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:56:44.580 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000004", 2024-08-18T15:56:44.580 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000063", 2024-08-18T15:56:44.580 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000122", 2024-08-18T15:56:44.580 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:56:44.580 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:56:44.580 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:56:44.580 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000057", 2024-08-18T15:56:44.580 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:56:44.581 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:56:44.581 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:56:44.581 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:56:44.581 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000054", 2024-08-18T15:56:44.581 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:56:44.581 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000114", 2024-08-18T15:56:44.581 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:56:44.581 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:56:44.581 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:56:44.581 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.2", 2024-08-18T15:56:44.581 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:56:44.581 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:56:44.581 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0", 2024-08-18T15:56:44.581 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:56:44.582 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:56:44.582 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:56:44.582 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:56:44.582 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:56:44.582 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:56:44.582 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000012", 2024-08-18T15:56:44.582 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:56:44.582 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:56:44.582 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:56:44.582 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000084", 2024-08-18T15:56:44.582 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000043", 2024-08-18T15:56:44.582 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3", 2024-08-18T15:56:44.582 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:56:44.583 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:56:44.583 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.11", 2024-08-18T15:56:44.583 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:56:44.583 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:56:44.583 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:56:44.583 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:56:44.583 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:56:44.583 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:56:44.583 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:56:44.583 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:56:44.583 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000111", 2024-08-18T15:56:44.583 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:56:44.583 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000048", 2024-08-18T15:56:44.584 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:56:44.584 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000020", 2024-08-18T15:56:44.584 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:56:44.584 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:56:44.584 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:56:44.584 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:56:44.584 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:56:44.584 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000099", 2024-08-18T15:56:44.584 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000056", 2024-08-18T15:56:44.584 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:56:44.584 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:56:44.584 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.19", 2024-08-18T15:56:44.584 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:56:44.584 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.28", 2024-08-18T15:56:44.585 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:56:44.585 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:56:44.585 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:56:44.585 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:56:44.585 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:56:44.585 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000072", 2024-08-18T15:56:44.585 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:56:44.585 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:56:44.585 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000062", 2024-08-18T15:56:44.586 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:56:44.586 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:56:44.586 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:56:44.586 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000115", 2024-08-18T15:56:44.586 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:56:44.586 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.1", 2024-08-18T15:56:44.586 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2", 2024-08-18T15:56:44.586 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000033", 2024-08-18T15:56:44.586 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:56:44.586 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:56:44.586 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:56:44.586 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:56:44.586 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:56:44.587 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:56:44.587 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:56:44.587 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:56:44.587 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:56:44.587 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:56:44.587 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:56:44.587 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:56:44.587 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:56:44.587 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:56:44.587 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:56:44.587 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:56:44.587 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:56:44.587 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:56:44.587 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:56:44.588 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000025", 2024-08-18T15:56:44.588 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:56:44.588 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000123", 2024-08-18T15:56:44.588 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.31", 2024-08-18T15:56:44.588 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:56:44.588 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:56:44.588 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:56:44.588 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:56:44.588 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.18", 2024-08-18T15:56:44.588 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:56:44.588 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:56:44.588 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:56:44.588 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:56:44.588 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:56:44.589 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000108", 2024-08-18T15:56:44.589 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:56:44.589 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:56:44.589 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000094", 2024-08-18T15:56:44.589 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:56:44.589 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:56:44.589 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:56:44.589 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:56:44.589 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:56:44.589 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:56:44.589 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000060", 2024-08-18T15:56:44.589 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:56:44.589 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:56:44.589 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:56:44.590 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000109", 2024-08-18T15:56:44.590 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.15", 2024-08-18T15:56:44.590 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:56:44.590 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:56:44.590 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:56:44.590 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:56:44.590 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:56:44.590 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:56:44.590 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:56:44.590 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:56:44.590 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:56:44.590 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000013", 2024-08-18T15:56:44.590 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:56:44.591 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:56:44.591 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:56:44.591 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:56:44.591 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:56:44.591 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:56:44.591 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:56:44.591 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:56:44.591 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:56:44.591 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:56:44.591 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:56:44.591 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:56:44.591 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:56:44.591 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:56:44.591 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000051", 2024-08-18T15:56:44.592 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:56:44.592 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:56:44.592 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:56:44.592 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:56:44.592 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000106", 2024-08-18T15:56:44.592 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:56:44.592 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:56:44.592 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:56:44.592 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:56:44.592 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:56:44.592 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000017", 2024-08-18T15:56:44.592 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:56:44.592 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:56:44.592 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:56:44.593 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000023", 2024-08-18T15:56:44.593 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:56:44.593 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:56:44.593 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:56:44.593 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:56:44.593 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:56:44.593 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:56:44.593 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:56:44.593 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000107", 2024-08-18T15:56:44.593 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:56:44.593 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:56:44.593 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000046", 2024-08-18T15:56:44.593 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:56:44.593 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:56:44.594 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:56:44.594 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.26", 2024-08-18T15:56:44.594 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:56:44.594 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:56:44.594 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9", 2024-08-18T15:56:44.594 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.22", 2024-08-18T15:56:44.594 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:56:44.594 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:56:44.594 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:56:44.594 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:56:44.594 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:56:44.594 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:56:44.594 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:56:44.595 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2", 2024-08-18T15:56:44.595 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:56:44.595 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:56:44.595 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000067", 2024-08-18T15:56:44.595 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:56:44.595 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:56:44.595 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:56:44.595 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:56:44.595 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:56:44.595 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:56:44.595 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:56:44.595 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:56:44.595 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:56:44.596 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:56:44.596 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:56:44.596 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:56:44.596 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:56:44.596 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:56:44.596 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:56:44.596 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:56:44.596 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:56:44.596 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:56:44.596 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:56:44.596 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.9", 2024-08-18T15:56:44.596 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000086", 2024-08-18T15:56:44.596 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.5", 2024-08-18T15:56:44.596 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:56:44.597 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:56:44.597 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:56:44.597 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:56:44.597 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:56:44.597 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:56:44.597 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2.MFXJ583gCr1.0", 2024-08-18T15:56:44.597 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:56:44.597 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1.GPnO16liyZ+.0", 2024-08-18T15:56:44.597 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:56:44.597 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:56:44.597 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:56:44.597 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:56:44.597 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:56:44.597 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:56:44.598 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:56:44.598 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:56:44.598 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:56:44.598 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.20", 2024-08-18T15:56:44.598 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:56:44.598 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:56:44.598 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:56:44.598 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:56:44.598 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:56:44.598 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:56:44.598 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1.retry", 2024-08-18T15:56:44.598 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:56:44.598 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:56:44.599 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:56:44.599 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0.retry", 2024-08-18T15:56:44.599 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000050", 2024-08-18T15:56:44.599 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:56:44.599 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000102", 2024-08-18T15:56:44.599 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:56:44.599 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:56:44.599 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:56:44.599 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:56:44.599 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:56:44.599 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:56:44.599 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:56:44.599 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:56:44.599 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:56:44.599 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:56:44.600 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:56:44.600 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000124", 2024-08-18T15:56:44.600 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:56:44.600 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:56:44.600 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:56:44.600 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000001", 2024-08-18T15:56:44.600 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:56:44.600 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:56:44.600 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:56:44.600 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:56:44.600 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000126", 2024-08-18T15:56:44.600 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:56:44.600 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0.fZhctfLXnSt.0", 2024-08-18T15:56:44.601 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:56:44.601 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000055", 2024-08-18T15:56:44.601 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:56:44.601 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000061", 2024-08-18T15:56:44.601 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1", 2024-08-18T15:56:44.601 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:56:44.601 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:56:44.601 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:56:44.601 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000097", 2024-08-18T15:56:44.601 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:56:44.601 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:56:44.601 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:56:44.601 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:56:44.602 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:56:44.602 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:56:44.602 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:56:44.602 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:56:44.602 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000042", 2024-08-18T15:56:44.602 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:56:44.602 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:56:44.602 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:56:44.602 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1.retry", 2024-08-18T15:56:44.602 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:56:44.602 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:56:44.602 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:56:44.602 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:56:44.603 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:56:44.603 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:56:44.603 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:56:44.603 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000075", 2024-08-18T15:56:44.603 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:56:44.603 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000120", 2024-08-18T15:56:44.603 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000030", 2024-08-18T15:56:44.603 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:56:44.603 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:56:44.603 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:56:44.603 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:56:44.603 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:56:44.603 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:56:44.604 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9" 2024-08-18T15:56:44.604 INFO:teuthology.orchestra.run.smithi193.stdout:] 2024-08-18T15:56:44.604 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:56:49.606 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:56:49.606 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:56:49.606 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:56:49.660 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:56:49.660 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:56:49.723 INFO:teuthology.orchestra.run.smithi193.stdout:[ 2024-08-18T15:56:49.723 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:56:49.723 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:56:49.723 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.29", 2024-08-18T15:56:49.723 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000078", 2024-08-18T15:56:49.723 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.history", 2024-08-18T15:56:49.723 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:56:49.723 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:56:49.723 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:56:49.723 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:56:49.724 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.23", 2024-08-18T15:56:49.724 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000070", 2024-08-18T15:56:49.724 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000104", 2024-08-18T15:56:49.724 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:56:49.724 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2.retry", 2024-08-18T15:56:49.724 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:56:49.724 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:56:49.724 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:56:49.724 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:56:49.724 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000026", 2024-08-18T15:56:49.724 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:56:49.724 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:56:49.724 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:56:49.724 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:56:49.725 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:56:49.725 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:56:49.725 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.8", 2024-08-18T15:56:49.725 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:56:49.725 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:56:49.725 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:56:49.725 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:56:49.725 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000028", 2024-08-18T15:56:49.725 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:56:49.726 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:56:49.726 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:56:49.726 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:56:49.726 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:56:49.726 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:56:49.726 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:56:49.726 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2.retry", 2024-08-18T15:56:49.726 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:56:49.726 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.30", 2024-08-18T15:56:49.726 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:56:49.726 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:56:49.727 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:56:49.727 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:56:49.727 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000040", 2024-08-18T15:56:49.727 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:56:49.727 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.6", 2024-08-18T15:56:49.727 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:56:49.727 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:56:49.727 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:56:49.727 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:56:49.727 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:56:49.727 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:56:49.727 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000015", 2024-08-18T15:56:49.727 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:56:49.728 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:56:49.728 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000069", 2024-08-18T15:56:49.728 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:56:49.728 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:56:49.728 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:56:49.728 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000095", 2024-08-18T15:56:49.728 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0", 2024-08-18T15:56:49.728 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000003", 2024-08-18T15:56:49.728 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000047", 2024-08-18T15:56:49.728 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:56:49.728 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:56:49.728 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:56:49.728 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:56:49.729 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:56:49.729 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:56:49.729 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:56:49.729 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:56:49.729 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:56:49.729 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000035", 2024-08-18T15:56:49.729 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:56:49.729 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:56:49.729 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000037", 2024-08-18T15:56:49.729 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:56:49.729 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:56:49.729 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:56:49.729 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:56:49.730 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:56:49.730 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:56:49.730 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:56:49.730 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:56:49.730 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:56:49.730 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:56:49.730 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:56:49.730 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:56:49.730 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:56:49.730 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:56:49.730 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000024", 2024-08-18T15:56:49.730 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:56:49.730 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:56:49.731 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:56:49.731 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:56:49.731 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000000", 2024-08-18T15:56:49.731 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:56:49.731 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000031", 2024-08-18T15:56:49.731 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:56:49.731 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:56:49.731 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:56:49.731 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:56:49.732 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:56:49.732 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:56:49.732 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3", 2024-08-18T15:56:49.732 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:56:49.732 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:56:49.732 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:56:49.732 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:56:49.732 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:56:49.732 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:56:49.732 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:56:49.732 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:56:49.732 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:56:49.732 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:56:49.732 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:56:49.733 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:56:49.733 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:56:49.733 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:56:49.733 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:56:49.733 INFO:teuthology.orchestra.run.smithi193.stdout: "data_loggenerations_metadata", 2024-08-18T15:56:49.733 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2", 2024-08-18T15:56:49.733 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:56:49.733 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:56:49.733 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:56:49.733 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:56:49.733 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T15:56:49.733 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:56:49.734 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:56:49.734 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:56:49.734 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:56:49.734 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:56:49.734 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000076", 2024-08-18T15:56:49.734 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:56:49.734 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:56:49.734 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000113", 2024-08-18T15:56:49.734 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000005", 2024-08-18T15:56:49.734 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:56:49.734 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000011", 2024-08-18T15:56:49.734 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:56:49.734 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000058", 2024-08-18T15:56:49.734 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:56:49.735 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:56:49.735 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:56:49.735 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000052", 2024-08-18T15:56:49.735 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:56:49.735 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:56:49.735 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:56:49.735 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:56:49.735 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.21", 2024-08-18T15:56:49.735 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:56:49.735 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:56:49.735 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:56:49.735 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:56:49.735 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:56:49.736 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:56:49.736 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:56:49.736 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:56:49.736 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:56:49.736 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:56:49.736 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:56:49.736 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:56:49.736 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:56:49.736 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:56:49.737 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:56:49.737 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:56:49.737 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:56:49.737 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:56:49.737 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:56:49.737 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000088", 2024-08-18T15:56:49.737 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000080", 2024-08-18T15:56:49.737 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:56:49.737 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:56:49.737 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000090", 2024-08-18T15:56:49.737 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:56:49.737 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:56:49.738 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000110", 2024-08-18T15:56:49.738 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:56:49.738 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:56:49.738 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:56:49.738 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.0", 2024-08-18T15:56:49.738 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:56:49.738 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000096", 2024-08-18T15:56:49.738 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:56:49.738 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:56:49.738 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:56:49.738 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:56:49.738 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:56:49.738 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000087", 2024-08-18T15:56:49.739 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:56:49.739 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:56:49.739 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:56:49.739 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:56:49.739 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000008", 2024-08-18T15:56:49.739 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:56:49.739 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:56:49.739 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:56:49.739 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:56:49.739 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:56:49.739 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000006", 2024-08-18T15:56:49.739 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000029", 2024-08-18T15:56:49.740 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:56:49.740 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000089", 2024-08-18T15:56:49.740 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:56:49.740 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:56:49.740 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000022", 2024-08-18T15:56:49.740 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:56:49.740 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:56:49.740 INFO:teuthology.orchestra.run.smithi193.stdout: "bilog.trim", 2024-08-18T15:56:49.740 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000016", 2024-08-18T15:56:49.740 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:56:49.740 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:56:49.740 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:56:49.740 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:56:49.741 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:56:49.741 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000041", 2024-08-18T15:56:49.741 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:56:49.741 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:56:49.741 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.log.01050e30-179b-43be-bdbb-3bb23cf37ae8.3", 2024-08-18T15:56:49.741 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:56:49.741 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000018", 2024-08-18T15:56:49.741 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000092", 2024-08-18T15:56:49.741 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:56:49.741 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:56:49.741 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:56:49.741 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:56:49.741 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.27", 2024-08-18T15:56:49.742 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.4", 2024-08-18T15:56:49.742 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:56:49.742 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:56:49.742 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:56:49.742 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:56:49.742 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:56:49.742 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:56:49.742 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:56:49.742 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000014", 2024-08-18T15:56:49.742 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:56:49.742 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3.retry", 2024-08-18T15:56:49.742 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:56:49.742 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:56:49.743 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:56:49.743 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:56:49.743 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:56:49.743 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000112", 2024-08-18T15:56:49.743 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:56:49.743 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000007", 2024-08-18T15:56:49.743 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000021", 2024-08-18T15:56:49.743 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:56:49.743 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:56:49.743 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:56:49.743 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:56:49.743 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:56:49.743 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.16", 2024-08-18T15:56:49.744 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:56:49.744 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000064", 2024-08-18T15:56:49.744 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:56:49.744 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:56:49.744 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:56:49.744 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:56:49.744 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:56:49.744 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:56:49.744 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:56:49.744 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000071", 2024-08-18T15:56:49.744 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:56:49.744 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:56:49.744 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:56:49.745 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000074", 2024-08-18T15:56:49.745 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:56:49.745 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:56:49.745 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:56:49.745 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:56:49.745 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:56:49.745 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:56:49.745 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:56:49.745 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:56:49.745 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:56:49.745 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:56:49.745 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:56:49.745 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:56:49.745 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:56:49.745 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:56:49.746 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:56:49.746 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000081", 2024-08-18T15:56:49.746 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:56:49.746 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000009", 2024-08-18T15:56:49.746 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000121", 2024-08-18T15:56:49.746 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1", 2024-08-18T15:56:49.746 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:56:49.746 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:56:49.746 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:56:49.746 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:56:49.746 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:56:49.746 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:56:49.747 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.17", 2024-08-18T15:56:49.747 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000125", 2024-08-18T15:56:49.747 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:56:49.747 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000082", 2024-08-18T15:56:49.747 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:56:49.747 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:56:49.747 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:56:49.747 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:56:49.747 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:56:49.747 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:56:49.747 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:56:49.747 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:56:49.747 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:56:49.747 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:56:49.748 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:56:49.748 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000105", 2024-08-18T15:56:49.748 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000059", 2024-08-18T15:56:49.748 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:56:49.748 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:56:49.748 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:56:49.748 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:56:49.748 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:56:49.748 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:56:49.748 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:56:49.748 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:56:49.748 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:56:49.748 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:56:49.748 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:56:49.749 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000077", 2024-08-18T15:56:49.749 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000032", 2024-08-18T15:56:49.749 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:56:49.749 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:56:49.749 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:56:49.749 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:56:49.749 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:56:49.749 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:56:49.749 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.24", 2024-08-18T15:56:49.749 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:56:49.749 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:56:49.750 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:56:49.750 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:56:49.750 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3.retry", 2024-08-18T15:56:49.750 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:56:49.750 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:56:49.750 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:56:49.750 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:56:49.750 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:56:49.750 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:56:49.750 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000053", 2024-08-18T15:56:49.750 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000091", 2024-08-18T15:56:49.750 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:56:49.750 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:56:49.751 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:56:49.751 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000065", 2024-08-18T15:56:49.751 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:56:49.751 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:56:49.751 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:56:49.751 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:56:49.751 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000083", 2024-08-18T15:56:49.751 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000010", 2024-08-18T15:56:49.751 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:56:49.751 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:56:49.751 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:56:49.751 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:56:49.751 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:56:49.752 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:56:49.752 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:56:49.752 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:56:49.752 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:56:49.752 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:56:49.752 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:56:49.752 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.7", 2024-08-18T15:56:49.752 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.3", 2024-08-18T15:56:49.752 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:56:49.752 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:56:49.752 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:56:49.752 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:56:49.752 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:56:49.752 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:56:49.753 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:56:49.753 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:56:49.753 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.2", 2024-08-18T15:56:49.753 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:56:49.753 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:56:49.753 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:56:49.753 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:56:49.753 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:56:49.753 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:56:49.753 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:56:49.753 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000045", 2024-08-18T15:56:49.753 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000002", 2024-08-18T15:56:49.753 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:56:49.754 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3.2UbiLidwSTy.0", 2024-08-18T15:56:49.754 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:56:49.754 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.12", 2024-08-18T15:56:49.754 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:56:49.754 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:56:49.754 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:56:49.754 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:56:49.754 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:56:49.754 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:56:49.754 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000116", 2024-08-18T15:56:49.754 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.25", 2024-08-18T15:56:49.754 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:56:49.754 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:56:49.755 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:56:49.755 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:56:49.755 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000034", 2024-08-18T15:56:49.755 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:56:49.755 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:56:49.755 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000101", 2024-08-18T15:56:49.755 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:56:49.755 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:56:49.755 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.13", 2024-08-18T15:56:49.755 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:56:49.755 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:56:49.755 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.1", 2024-08-18T15:56:49.755 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0.retry", 2024-08-18T15:56:49.756 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:56:49.756 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:56:49.756 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000079", 2024-08-18T15:56:49.756 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000049", 2024-08-18T15:56:49.756 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:56:49.756 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0", 2024-08-18T15:56:49.756 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:56:49.756 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:56:49.756 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000117", 2024-08-18T15:56:49.756 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:56:49.756 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000044", 2024-08-18T15:56:49.756 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:56:49.756 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:56:49.757 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:56:49.757 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:56:49.757 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:56:49.757 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:56:49.757 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.14", 2024-08-18T15:56:49.757 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:56:49.757 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000066", 2024-08-18T15:56:49.757 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:56:49.757 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:56:49.757 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:56:49.757 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:56:49.757 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:56:49.757 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000068", 2024-08-18T15:56:49.758 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:56:49.758 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000085", 2024-08-18T15:56:49.758 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:56:49.758 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:56:49.758 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:56:49.758 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:56:49.758 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:56:49.758 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:56:49.758 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:56:49.758 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:56:49.758 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:56:49.758 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:56:49.758 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:56:49.759 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:56:49.759 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000073", 2024-08-18T15:56:49.759 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:56:49.759 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:56:49.759 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:56:49.759 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000038", 2024-08-18T15:56:49.759 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:56:49.759 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:56:49.759 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:56:49.759 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:56:49.759 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.10", 2024-08-18T15:56:49.759 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:56:49.760 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:56:49.760 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:56:49.760 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000118", 2024-08-18T15:56:49.760 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:56:49.760 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3", 2024-08-18T15:56:49.760 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:56:49.760 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000036", 2024-08-18T15:56:49.760 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:56:49.760 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:56:49.760 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:56:49.760 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:56:49.760 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:56:49.760 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000103", 2024-08-18T15:56:49.761 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.3", 2024-08-18T15:56:49.761 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:56:49.761 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000119", 2024-08-18T15:56:49.761 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000098", 2024-08-18T15:56:49.761 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:56:49.761 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:56:49.761 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:56:49.761 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:56:49.761 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:56:49.761 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:56:49.761 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:56:49.761 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:56:49.761 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.0", 2024-08-18T15:56:49.762 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:56:49.762 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:56:49.762 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:56:49.762 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:56:49.762 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:56:49.762 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:56:49.762 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:56:49.762 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:56:49.762 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000027", 2024-08-18T15:56:49.762 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:56:49.762 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status", 2024-08-18T15:56:49.762 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:56:49.763 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000019", 2024-08-18T15:56:49.763 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:56:49.763 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:56:49.763 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:56:49.763 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:56:49.763 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1", 2024-08-18T15:56:49.763 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:56:49.763 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:56:49.763 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:56:49.763 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:56:49.763 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000039", 2024-08-18T15:56:49.763 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:56:49.763 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:56:49.764 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:56:49.764 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:56:49.764 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:56:49.764 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:56:49.764 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:56:49.764 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:56:49.764 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:56:49.764 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:56:49.764 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:56:49.764 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000100", 2024-08-18T15:56:49.764 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:56:49.764 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:56:49.765 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:56:49.765 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:56:49.765 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:56:49.765 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000093", 2024-08-18T15:56:49.765 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:56:49.765 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000004", 2024-08-18T15:56:49.765 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000063", 2024-08-18T15:56:49.765 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000122", 2024-08-18T15:56:49.765 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:56:49.765 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:56:49.765 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:56:49.765 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000057", 2024-08-18T15:56:49.765 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:56:49.765 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:56:49.766 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:56:49.766 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:56:49.766 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000054", 2024-08-18T15:56:49.766 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:56:49.766 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000114", 2024-08-18T15:56:49.766 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:56:49.766 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:56:49.766 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:56:49.766 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.2", 2024-08-18T15:56:49.766 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:56:49.766 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:56:49.766 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0", 2024-08-18T15:56:49.767 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:56:49.767 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:56:49.767 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:56:49.767 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:56:49.767 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:56:49.767 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:56:49.767 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000012", 2024-08-18T15:56:49.767 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:56:49.767 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:56:49.767 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:56:49.767 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000084", 2024-08-18T15:56:49.767 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000043", 2024-08-18T15:56:49.767 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3", 2024-08-18T15:56:49.768 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:56:49.768 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:56:49.768 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.11", 2024-08-18T15:56:49.768 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:56:49.768 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:56:49.768 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:56:49.768 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:56:49.768 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:56:49.768 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:56:49.768 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:56:49.768 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:56:49.768 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000111", 2024-08-18T15:56:49.769 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:56:49.769 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000048", 2024-08-18T15:56:49.769 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:56:49.769 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000020", 2024-08-18T15:56:49.769 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:56:49.769 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:56:49.769 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:56:49.769 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:56:49.769 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:56:49.769 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000099", 2024-08-18T15:56:49.769 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000056", 2024-08-18T15:56:49.769 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:56:49.769 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:56:49.770 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.19", 2024-08-18T15:56:49.770 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:56:49.770 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.28", 2024-08-18T15:56:49.770 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:56:49.770 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:56:49.770 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:56:49.770 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:56:49.770 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:56:49.770 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000072", 2024-08-18T15:56:49.770 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:56:49.770 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:56:49.770 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000062", 2024-08-18T15:56:49.771 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:56:49.771 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:56:49.771 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:56:49.771 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000115", 2024-08-18T15:56:49.771 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:56:49.771 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.1", 2024-08-18T15:56:49.771 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2", 2024-08-18T15:56:49.771 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000033", 2024-08-18T15:56:49.771 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:56:49.771 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:56:49.771 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:56:49.771 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:56:49.771 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:56:49.771 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:56:49.772 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:56:49.772 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:56:49.772 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:56:49.772 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:56:49.772 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:56:49.772 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:56:49.772 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:56:49.772 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:56:49.772 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:56:49.772 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:56:49.772 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:56:49.772 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:56:49.772 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:56:49.773 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000025", 2024-08-18T15:56:49.773 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:56:49.773 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000123", 2024-08-18T15:56:49.773 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.31", 2024-08-18T15:56:49.773 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:56:49.773 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:56:49.773 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:56:49.773 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:56:49.773 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.18", 2024-08-18T15:56:49.773 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:56:49.773 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:56:49.773 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:56:49.773 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:56:49.774 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:56:49.774 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000108", 2024-08-18T15:56:49.774 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:56:49.774 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:56:49.774 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000094", 2024-08-18T15:56:49.774 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:56:49.774 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:56:49.774 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:56:49.774 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:56:49.774 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:56:49.774 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:56:49.774 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000060", 2024-08-18T15:56:49.774 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:56:49.775 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:56:49.775 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:56:49.775 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000109", 2024-08-18T15:56:49.775 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.15", 2024-08-18T15:56:49.775 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:56:49.775 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:56:49.775 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:56:49.775 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:56:49.775 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:56:49.775 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:56:49.775 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:56:49.775 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:56:49.775 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:56:49.776 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000013", 2024-08-18T15:56:49.776 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:56:49.776 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:56:49.776 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:56:49.776 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:56:49.776 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:56:49.776 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:56:49.776 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:56:49.776 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:56:49.776 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:56:49.776 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:56:49.776 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:56:49.777 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:56:49.777 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:56:49.777 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:56:49.777 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000051", 2024-08-18T15:56:49.777 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:56:49.777 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:56:49.777 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:56:49.777 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:56:49.777 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000106", 2024-08-18T15:56:49.777 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:56:49.777 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:56:49.777 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:56:49.777 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:56:49.777 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:56:49.778 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000017", 2024-08-18T15:56:49.778 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:56:49.778 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:56:49.778 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:56:49.778 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000023", 2024-08-18T15:56:49.778 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:56:49.778 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:56:49.778 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:56:49.778 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:56:49.778 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:56:49.778 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:56:49.778 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:56:49.779 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000107", 2024-08-18T15:56:49.779 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:56:49.779 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:56:49.779 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000046", 2024-08-18T15:56:49.779 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:56:49.779 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:56:49.779 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:56:49.779 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.26", 2024-08-18T15:56:49.779 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:56:49.779 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:56:49.779 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9", 2024-08-18T15:56:49.779 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.22", 2024-08-18T15:56:49.779 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:56:49.779 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:56:49.779 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:56:49.780 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:56:49.780 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:56:49.780 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:56:49.780 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:56:49.780 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2", 2024-08-18T15:56:49.780 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:56:49.780 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:56:49.780 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000067", 2024-08-18T15:56:49.780 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:56:49.780 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:56:49.780 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:56:49.780 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:56:49.780 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:56:49.780 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:56:49.781 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:56:49.781 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:56:49.781 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:56:49.781 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:56:49.781 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:56:49.781 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:56:49.781 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:56:49.781 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:56:49.781 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:56:49.781 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:56:49.781 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:56:49.781 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:56:49.781 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:56:49.781 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.9", 2024-08-18T15:56:49.782 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000086", 2024-08-18T15:56:49.782 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.5", 2024-08-18T15:56:49.782 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:56:49.782 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:56:49.782 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:56:49.782 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:56:49.782 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:56:49.782 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:56:49.782 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2.MFXJ583gCr1.0", 2024-08-18T15:56:49.782 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:56:49.782 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1.GPnO16liyZ+.0", 2024-08-18T15:56:49.782 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:56:49.783 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:56:49.783 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:56:49.783 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:56:49.783 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:56:49.783 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:56:49.783 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:56:49.783 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:56:49.783 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:56:49.783 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.20", 2024-08-18T15:56:49.783 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:56:49.783 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:56:49.783 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:56:49.783 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:56:49.783 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:56:49.783 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:56:49.784 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1.retry", 2024-08-18T15:56:49.784 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:56:49.784 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:56:49.784 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:56:49.784 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0.retry", 2024-08-18T15:56:49.784 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000050", 2024-08-18T15:56:49.784 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:56:49.784 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000102", 2024-08-18T15:56:49.784 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:56:49.784 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:56:49.784 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:56:49.784 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:56:49.784 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:56:49.784 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:56:49.785 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:56:49.785 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:56:49.785 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:56:49.785 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:56:49.785 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:56:49.785 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000124", 2024-08-18T15:56:49.785 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:56:49.785 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:56:49.785 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:56:49.785 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000001", 2024-08-18T15:56:49.785 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:56:49.785 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:56:49.785 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:56:49.786 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:56:49.786 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000126", 2024-08-18T15:56:49.786 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:56:49.786 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0.fZhctfLXnSt.0", 2024-08-18T15:56:49.786 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:56:49.786 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000055", 2024-08-18T15:56:49.786 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:56:49.786 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000061", 2024-08-18T15:56:49.786 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1", 2024-08-18T15:56:49.786 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:56:49.786 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:56:49.786 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:56:49.787 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000097", 2024-08-18T15:56:49.787 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:56:49.787 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:56:49.787 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:56:49.787 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:56:49.787 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:56:49.787 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:56:49.787 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:56:49.787 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:56:49.787 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000042", 2024-08-18T15:56:49.787 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:56:49.787 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:56:49.787 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:56:49.787 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1.retry", 2024-08-18T15:56:49.788 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:56:49.788 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:56:49.788 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:56:49.788 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:56:49.788 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:56:49.788 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:56:49.788 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:56:49.788 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000075", 2024-08-18T15:56:49.788 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:56:49.788 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000120", 2024-08-18T15:56:49.788 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000030", 2024-08-18T15:56:49.788 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:56:49.788 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:56:49.789 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:56:49.789 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:56:49.789 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:56:49.789 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:56:49.789 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9" 2024-08-18T15:56:49.789 INFO:teuthology.orchestra.run.smithi193.stdout:] 2024-08-18T15:56:49.789 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:56:54.791 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:56:54.791 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:56:54.791 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:56:54.843 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:56:54.843 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:56:54.906 INFO:teuthology.orchestra.run.smithi193.stdout:[ 2024-08-18T15:56:54.906 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:56:54.906 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:56:54.907 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.29", 2024-08-18T15:56:54.907 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000078", 2024-08-18T15:56:54.907 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.history", 2024-08-18T15:56:54.907 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:56:54.907 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:56:54.907 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:56:54.907 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:56:54.907 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.23", 2024-08-18T15:56:54.907 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000070", 2024-08-18T15:56:54.907 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000104", 2024-08-18T15:56:54.907 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:56:54.907 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2.retry", 2024-08-18T15:56:54.908 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:56:54.908 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:56:54.908 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:56:54.908 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:56:54.908 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000026", 2024-08-18T15:56:54.908 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:56:54.908 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:56:54.908 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:56:54.908 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:56:54.908 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:56:54.908 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:56:54.908 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.8", 2024-08-18T15:56:54.908 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:56:54.909 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:56:54.909 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:56:54.909 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:56:54.909 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000028", 2024-08-18T15:56:54.909 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:56:54.909 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:56:54.909 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:56:54.909 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:56:54.909 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:56:54.909 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:56:54.909 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:56:54.909 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2.retry", 2024-08-18T15:56:54.909 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:56:54.910 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.30", 2024-08-18T15:56:54.910 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:56:54.910 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:56:54.910 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:56:54.910 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:56:54.910 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000040", 2024-08-18T15:56:54.910 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:56:54.910 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.6", 2024-08-18T15:56:54.910 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:56:54.910 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:56:54.910 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:56:54.910 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:56:54.910 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:56:54.911 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:56:54.911 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000015", 2024-08-18T15:56:54.911 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:56:54.911 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:56:54.911 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000069", 2024-08-18T15:56:54.911 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:56:54.911 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:56:54.911 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:56:54.911 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000095", 2024-08-18T15:56:54.911 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0", 2024-08-18T15:56:54.911 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000003", 2024-08-18T15:56:54.911 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000047", 2024-08-18T15:56:54.911 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:56:54.912 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:56:54.912 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:56:54.912 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:56:54.912 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:56:54.912 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:56:54.912 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:56:54.912 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:56:54.912 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:56:54.912 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000035", 2024-08-18T15:56:54.912 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:56:54.912 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:56:54.912 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000037", 2024-08-18T15:56:54.912 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:56:54.913 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:56:54.913 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:56:54.913 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:56:54.913 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:56:54.913 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:56:54.913 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:56:54.913 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:56:54.913 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:56:54.913 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:56:54.913 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:56:54.913 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:56:54.914 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:56:54.914 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:56:54.914 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000024", 2024-08-18T15:56:54.914 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:56:54.914 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:56:54.914 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:56:54.914 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:56:54.914 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000000", 2024-08-18T15:56:54.914 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:56:54.914 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000031", 2024-08-18T15:56:54.914 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:56:54.914 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:56:54.915 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:56:54.915 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:56:54.915 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:56:54.915 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:56:54.915 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3", 2024-08-18T15:56:54.915 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:56:54.915 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:56:54.915 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:56:54.915 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:56:54.915 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:56:54.915 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:56:54.916 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:56:54.916 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:56:54.916 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:56:54.916 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:56:54.916 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:56:54.916 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:56:54.916 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:56:54.916 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:56:54.916 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:56:54.916 INFO:teuthology.orchestra.run.smithi193.stdout: "data_loggenerations_metadata", 2024-08-18T15:56:54.916 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2", 2024-08-18T15:56:54.916 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:56:54.916 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:56:54.917 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:56:54.917 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:56:54.917 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T15:56:54.917 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:56:54.917 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:56:54.917 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:56:54.917 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:56:54.917 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:56:54.917 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000076", 2024-08-18T15:56:54.917 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:56:54.917 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:56:54.917 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000113", 2024-08-18T15:56:54.917 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000005", 2024-08-18T15:56:54.918 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:56:54.918 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000011", 2024-08-18T15:56:54.918 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:56:54.918 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000058", 2024-08-18T15:56:54.918 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:56:54.918 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:56:54.918 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:56:54.918 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000052", 2024-08-18T15:56:54.918 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:56:54.918 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:56:54.918 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:56:54.918 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:56:54.919 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.21", 2024-08-18T15:56:54.919 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:56:54.919 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:56:54.919 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:56:54.919 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:56:54.919 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:56:54.919 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:56:54.919 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:56:54.919 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:56:54.919 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:56:54.919 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:56:54.919 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:56:54.919 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:56:54.920 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:56:54.920 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:56:54.920 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:56:54.920 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:56:54.920 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:56:54.920 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:56:54.920 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:56:54.920 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000088", 2024-08-18T15:56:54.920 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000080", 2024-08-18T15:56:54.920 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:56:54.920 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:56:54.920 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000090", 2024-08-18T15:56:54.920 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:56:54.921 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:56:54.921 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000110", 2024-08-18T15:56:54.921 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:56:54.921 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:56:54.921 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:56:54.921 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.0", 2024-08-18T15:56:54.921 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:56:54.921 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000096", 2024-08-18T15:56:54.921 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:56:54.921 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:56:54.921 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:56:54.921 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:56:54.921 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:56:54.922 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000087", 2024-08-18T15:56:54.922 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:56:54.922 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:56:54.922 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:56:54.922 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:56:54.922 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000008", 2024-08-18T15:56:54.922 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:56:54.922 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:56:54.922 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:56:54.922 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:56:54.922 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:56:54.922 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000006", 2024-08-18T15:56:54.922 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000029", 2024-08-18T15:56:54.923 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:56:54.923 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000089", 2024-08-18T15:56:54.923 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:56:54.923 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:56:54.923 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000022", 2024-08-18T15:56:54.923 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:56:54.923 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:56:54.923 INFO:teuthology.orchestra.run.smithi193.stdout: "bilog.trim", 2024-08-18T15:56:54.923 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000016", 2024-08-18T15:56:54.923 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:56:54.923 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:56:54.923 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:56:54.924 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:56:54.924 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:56:54.924 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000041", 2024-08-18T15:56:54.924 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:56:54.924 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:56:54.924 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.log.01050e30-179b-43be-bdbb-3bb23cf37ae8.3", 2024-08-18T15:56:54.924 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:56:54.924 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000018", 2024-08-18T15:56:54.924 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000092", 2024-08-18T15:56:54.924 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:56:54.924 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:56:54.924 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:56:54.924 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:56:54.924 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.27", 2024-08-18T15:56:54.925 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.4", 2024-08-18T15:56:54.925 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:56:54.925 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:56:54.925 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:56:54.925 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:56:54.925 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:56:54.925 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:56:54.925 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:56:54.925 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000014", 2024-08-18T15:56:54.925 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:56:54.925 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3.retry", 2024-08-18T15:56:54.925 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:56:54.925 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:56:54.925 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:56:54.926 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:56:54.926 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:56:54.926 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000112", 2024-08-18T15:56:54.926 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:56:54.926 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000007", 2024-08-18T15:56:54.926 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000021", 2024-08-18T15:56:54.926 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:56:54.926 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:56:54.926 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:56:54.926 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:56:54.926 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:56:54.927 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.16", 2024-08-18T15:56:54.927 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:56:54.927 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000064", 2024-08-18T15:56:54.927 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:56:54.928 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:56:54.928 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:56:54.928 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:56:54.928 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:56:54.928 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:56:54.928 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:56:54.928 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000071", 2024-08-18T15:56:54.928 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:56:54.928 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:56:54.928 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:56:54.928 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000074", 2024-08-18T15:56:54.928 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:56:54.929 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:56:54.929 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:56:54.929 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:56:54.929 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:56:54.929 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:56:54.929 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:56:54.929 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:56:54.929 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:56:54.929 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:56:54.929 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:56:54.929 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:56:54.929 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:56:54.930 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:56:54.930 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:56:54.930 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000081", 2024-08-18T15:56:54.930 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:56:54.930 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000009", 2024-08-18T15:56:54.930 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000121", 2024-08-18T15:56:54.930 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1", 2024-08-18T15:56:54.930 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:56:54.930 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:56:54.930 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:56:54.930 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:56:54.930 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:56:54.931 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:56:54.931 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.17", 2024-08-18T15:56:54.931 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000125", 2024-08-18T15:56:54.931 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:56:54.931 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000082", 2024-08-18T15:56:54.931 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:56:54.931 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:56:54.931 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:56:54.931 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:56:54.931 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:56:54.931 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:56:54.931 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:56:54.931 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:56:54.931 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:56:54.932 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:56:54.932 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:56:54.932 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000105", 2024-08-18T15:56:54.932 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000059", 2024-08-18T15:56:54.932 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:56:54.932 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:56:54.932 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:56:54.932 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:56:54.932 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:56:54.932 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:56:54.932 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:56:54.932 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:56:54.932 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:56:54.932 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:56:54.933 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:56:54.933 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000077", 2024-08-18T15:56:54.933 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000032", 2024-08-18T15:56:54.933 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:56:54.933 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:56:54.933 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:56:54.933 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:56:54.933 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:56:54.933 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:56:54.933 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.24", 2024-08-18T15:56:54.934 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:56:54.934 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:56:54.934 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:56:54.934 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:56:54.934 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3.retry", 2024-08-18T15:56:54.934 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:56:54.934 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:56:54.934 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:56:54.934 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:56:54.934 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:56:54.934 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:56:54.934 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000053", 2024-08-18T15:56:54.934 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000091", 2024-08-18T15:56:54.935 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:56:54.935 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:56:54.935 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:56:54.935 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000065", 2024-08-18T15:56:54.935 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:56:54.935 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:56:54.935 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:56:54.935 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:56:54.935 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000083", 2024-08-18T15:56:54.935 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000010", 2024-08-18T15:56:54.935 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:56:54.935 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:56:54.935 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:56:54.935 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:56:54.936 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:56:54.936 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:56:54.936 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:56:54.936 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:56:54.936 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:56:54.936 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:56:54.936 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:56:54.936 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.7", 2024-08-18T15:56:54.936 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.3", 2024-08-18T15:56:54.936 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:56:54.936 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:56:54.936 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:56:54.936 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:56:54.936 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:56:54.936 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:56:54.937 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:56:54.937 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:56:54.937 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.2", 2024-08-18T15:56:54.937 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:56:54.937 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:56:54.937 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:56:54.937 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:56:54.937 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:56:54.937 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:56:54.937 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:56:54.937 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000045", 2024-08-18T15:56:54.937 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000002", 2024-08-18T15:56:54.937 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:56:54.937 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3.2UbiLidwSTy.0", 2024-08-18T15:56:54.937 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:56:54.938 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.12", 2024-08-18T15:56:54.938 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:56:54.938 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:56:54.938 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:56:54.938 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:56:54.938 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:56:54.938 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:56:54.938 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000116", 2024-08-18T15:56:54.938 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.25", 2024-08-18T15:56:54.938 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:56:54.938 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:56:54.938 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:56:54.938 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:56:54.939 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000034", 2024-08-18T15:56:54.939 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:56:54.939 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:56:54.939 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000101", 2024-08-18T15:56:54.939 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:56:54.939 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:56:54.939 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.13", 2024-08-18T15:56:54.939 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:56:54.939 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:56:54.939 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.1", 2024-08-18T15:56:54.939 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0.retry", 2024-08-18T15:56:54.939 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:56:54.939 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:56:54.940 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000079", 2024-08-18T15:56:54.940 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000049", 2024-08-18T15:56:54.940 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:56:54.940 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0", 2024-08-18T15:56:54.940 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:56:54.940 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:56:54.940 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000117", 2024-08-18T15:56:54.940 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:56:54.940 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000044", 2024-08-18T15:56:54.940 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:56:54.940 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:56:54.940 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:56:54.940 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:56:54.941 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:56:54.941 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:56:54.941 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.14", 2024-08-18T15:56:54.941 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:56:54.941 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000066", 2024-08-18T15:56:54.941 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:56:54.941 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:56:54.941 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:56:54.941 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:56:54.941 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:56:54.941 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000068", 2024-08-18T15:56:54.941 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:56:54.941 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000085", 2024-08-18T15:56:54.942 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:56:54.942 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:56:54.942 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:56:54.942 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:56:54.942 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:56:54.942 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:56:54.942 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:56:54.942 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:56:54.942 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:56:54.942 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:56:54.942 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:56:54.942 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:56:54.943 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000073", 2024-08-18T15:56:54.943 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:56:54.943 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:56:54.943 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:56:54.943 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000038", 2024-08-18T15:56:54.943 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:56:54.943 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:56:54.943 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:56:54.943 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:56:54.943 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.10", 2024-08-18T15:56:54.943 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:56:54.943 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:56:54.943 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:56:54.944 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000118", 2024-08-18T15:56:54.944 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:56:54.944 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3", 2024-08-18T15:56:54.944 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:56:54.944 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000036", 2024-08-18T15:56:54.944 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:56:54.944 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:56:54.944 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:56:54.944 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:56:54.944 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:56:54.944 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000103", 2024-08-18T15:56:54.944 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.3", 2024-08-18T15:56:54.944 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:56:54.945 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000119", 2024-08-18T15:56:54.945 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000098", 2024-08-18T15:56:54.945 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:56:54.945 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:56:54.945 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:56:54.945 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:56:54.945 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:56:54.945 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:56:54.945 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:56:54.945 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:56:54.945 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.0", 2024-08-18T15:56:54.945 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:56:54.945 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:56:54.946 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:56:54.946 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:56:54.946 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:56:54.946 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:56:54.946 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:56:54.946 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:56:54.946 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000027", 2024-08-18T15:56:54.946 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:56:54.946 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status", 2024-08-18T15:56:54.946 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:56:54.946 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000019", 2024-08-18T15:56:54.946 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:56:54.947 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:56:54.947 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:56:54.947 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:56:54.947 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1", 2024-08-18T15:56:54.947 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:56:54.947 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:56:54.947 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:56:54.947 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:56:54.947 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000039", 2024-08-18T15:56:54.947 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:56:54.947 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:56:54.947 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:56:54.947 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:56:54.948 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:56:54.948 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:56:54.948 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:56:54.948 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:56:54.948 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:56:54.948 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:56:54.948 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:56:54.948 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000100", 2024-08-18T15:56:54.948 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:56:54.948 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:56:54.948 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:56:54.948 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:56:54.948 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:56:54.949 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000093", 2024-08-18T15:56:54.949 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:56:54.949 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000004", 2024-08-18T15:56:54.949 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000063", 2024-08-18T15:56:54.949 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000122", 2024-08-18T15:56:54.949 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:56:54.949 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:56:54.949 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:56:54.949 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000057", 2024-08-18T15:56:54.949 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:56:54.949 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:56:54.949 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:56:54.949 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:56:54.949 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000054", 2024-08-18T15:56:54.950 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:56:54.950 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000114", 2024-08-18T15:56:54.950 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:56:54.950 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:56:54.950 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:56:54.950 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.2", 2024-08-18T15:56:54.950 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:56:54.950 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:56:54.950 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0", 2024-08-18T15:56:54.950 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:56:54.950 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:56:54.950 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:56:54.951 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:56:54.951 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:56:54.951 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:56:54.951 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000012", 2024-08-18T15:56:54.951 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:56:54.951 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:56:54.951 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:56:54.951 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000084", 2024-08-18T15:56:54.951 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000043", 2024-08-18T15:56:54.951 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3", 2024-08-18T15:56:54.951 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:56:54.951 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:56:54.951 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.11", 2024-08-18T15:56:54.951 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:56:54.952 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:56:54.952 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:56:54.952 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:56:54.952 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:56:54.952 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:56:54.952 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:56:54.952 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:56:54.952 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000111", 2024-08-18T15:56:54.952 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:56:54.952 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000048", 2024-08-18T15:56:54.952 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:56:54.952 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000020", 2024-08-18T15:56:54.953 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:56:54.953 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:56:54.953 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:56:54.953 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:56:54.953 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:56:54.953 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000099", 2024-08-18T15:56:54.953 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000056", 2024-08-18T15:56:54.953 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:56:54.953 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:56:54.953 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.19", 2024-08-18T15:56:54.953 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:56:54.953 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.28", 2024-08-18T15:56:54.953 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:56:54.953 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:56:54.954 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:56:54.954 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:56:54.954 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:56:54.954 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000072", 2024-08-18T15:56:54.954 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:56:54.954 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:56:54.954 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000062", 2024-08-18T15:56:54.954 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:56:54.954 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:56:54.954 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:56:54.954 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000115", 2024-08-18T15:56:54.954 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:56:54.955 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.1", 2024-08-18T15:56:54.955 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2", 2024-08-18T15:56:54.955 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000033", 2024-08-18T15:56:54.955 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:56:54.955 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:56:54.955 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:56:54.955 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:56:54.955 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:56:54.955 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:56:54.955 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:56:54.955 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:56:54.955 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:56:54.955 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:56:54.956 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:56:54.956 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:56:54.956 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:56:54.956 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:56:54.956 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:56:54.956 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:56:54.956 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:56:54.956 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:56:54.956 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:56:54.956 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000025", 2024-08-18T15:56:54.956 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:56:54.956 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000123", 2024-08-18T15:56:54.956 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.31", 2024-08-18T15:56:54.957 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:56:54.957 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:56:54.957 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:56:54.957 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:56:54.957 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.18", 2024-08-18T15:56:54.957 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:56:54.957 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:56:54.957 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:56:54.957 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:56:54.957 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:56:54.957 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000108", 2024-08-18T15:56:54.957 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:56:54.957 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:56:54.958 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000094", 2024-08-18T15:56:54.958 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:56:54.958 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:56:54.958 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:56:54.958 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:56:54.958 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:56:54.958 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:56:54.958 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000060", 2024-08-18T15:56:54.958 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:56:54.958 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:56:54.958 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:56:54.958 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000109", 2024-08-18T15:56:54.959 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.15", 2024-08-18T15:56:54.959 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:56:54.959 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:56:54.959 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:56:54.959 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:56:54.959 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:56:54.959 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:56:54.959 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:56:54.959 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:56:54.959 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:56:54.959 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000013", 2024-08-18T15:56:54.959 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:56:54.960 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:56:54.960 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:56:54.960 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:56:54.960 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:56:54.960 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:56:54.960 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:56:54.960 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:56:54.960 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:56:54.960 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:56:54.960 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:56:54.960 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:56:54.960 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:56:54.960 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:56:54.961 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000051", 2024-08-18T15:56:54.961 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:56:54.961 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:56:54.961 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:56:54.961 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:56:54.961 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000106", 2024-08-18T15:56:54.961 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:56:54.961 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:56:54.961 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:56:54.961 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:56:54.961 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:56:54.961 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000017", 2024-08-18T15:56:54.961 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:56:54.962 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:56:54.962 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:56:54.962 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000023", 2024-08-18T15:56:54.962 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:56:54.962 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:56:54.962 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:56:54.962 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:56:54.963 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:56:54.963 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:56:54.963 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:56:54.963 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000107", 2024-08-18T15:56:54.963 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:56:54.963 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:56:54.963 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000046", 2024-08-18T15:56:54.963 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:56:54.963 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:56:54.963 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:56:54.963 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.26", 2024-08-18T15:56:54.963 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:56:54.963 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:56:54.964 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9", 2024-08-18T15:56:54.964 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.22", 2024-08-18T15:56:54.964 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:56:54.964 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:56:54.964 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:56:54.964 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:56:54.964 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:56:54.964 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:56:54.964 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:56:54.964 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2", 2024-08-18T15:56:54.964 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:56:54.964 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:56:54.965 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000067", 2024-08-18T15:56:54.965 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:56:54.965 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:56:54.965 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:56:54.965 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:56:54.965 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:56:54.965 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:56:54.965 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:56:54.965 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:56:54.965 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:56:54.965 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:56:54.965 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:56:54.965 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:56:54.966 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:56:54.966 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:56:54.966 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:56:54.966 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:56:54.966 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:56:54.966 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:56:54.966 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:56:54.966 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.9", 2024-08-18T15:56:54.966 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000086", 2024-08-18T15:56:54.966 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.5", 2024-08-18T15:56:54.966 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:56:54.966 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:56:54.967 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:56:54.967 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:56:54.967 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:56:54.967 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:56:54.967 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2.MFXJ583gCr1.0", 2024-08-18T15:56:54.967 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:56:54.967 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1.GPnO16liyZ+.0", 2024-08-18T15:56:54.967 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:56:54.967 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:56:54.967 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:56:54.967 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:56:54.967 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:56:54.967 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:56:54.968 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:56:54.968 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:56:54.968 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:56:54.968 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.20", 2024-08-18T15:56:54.968 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:56:54.968 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:56:54.968 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:56:54.968 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:56:54.968 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:56:54.968 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:56:54.968 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1.retry", 2024-08-18T15:56:54.968 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:56:54.968 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:56:54.968 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:56:54.968 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0.retry", 2024-08-18T15:56:54.969 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000050", 2024-08-18T15:56:54.969 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:56:54.969 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000102", 2024-08-18T15:56:54.969 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:56:54.969 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:56:54.969 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:56:54.969 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:56:54.969 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:56:54.969 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:56:54.969 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:56:54.969 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:56:54.969 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:56:54.969 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:56:54.970 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:56:54.970 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000124", 2024-08-18T15:56:54.970 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:56:54.970 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:56:54.970 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:56:54.970 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000001", 2024-08-18T15:56:54.970 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:56:54.970 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:56:54.970 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:56:54.970 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:56:54.970 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000126", 2024-08-18T15:56:54.970 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:56:54.971 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0.fZhctfLXnSt.0", 2024-08-18T15:56:54.971 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:56:54.971 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000055", 2024-08-18T15:56:54.971 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:56:54.971 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000061", 2024-08-18T15:56:54.971 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1", 2024-08-18T15:56:54.971 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:56:54.971 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:56:54.971 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:56:54.971 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000097", 2024-08-18T15:56:54.971 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:56:54.971 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:56:54.971 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:56:54.971 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:56:54.972 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:56:54.972 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:56:54.972 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:56:54.972 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:56:54.972 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000042", 2024-08-18T15:56:54.972 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:56:54.972 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:56:54.972 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:56:54.972 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1.retry", 2024-08-18T15:56:54.972 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:56:54.972 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:56:54.972 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:56:54.972 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:56:54.972 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:56:54.973 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:56:54.973 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:56:54.973 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000075", 2024-08-18T15:56:54.973 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:56:54.973 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000120", 2024-08-18T15:56:54.973 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000030", 2024-08-18T15:56:54.973 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:56:54.973 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:56:54.973 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:56:54.973 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:56:54.973 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:56:54.973 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:56:54.973 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9" 2024-08-18T15:56:54.973 INFO:teuthology.orchestra.run.smithi193.stdout:] 2024-08-18T15:56:54.973 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:56:59.976 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:56:59.976 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:56:59.976 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:57:00.028 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:57:00.029 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:57:00.092 INFO:teuthology.orchestra.run.smithi193.stdout:[ 2024-08-18T15:57:00.092 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:57:00.093 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:57:00.093 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.29", 2024-08-18T15:57:00.093 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000078", 2024-08-18T15:57:00.093 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.history", 2024-08-18T15:57:00.093 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:57:00.093 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:57:00.093 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:57:00.093 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:57:00.094 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.23", 2024-08-18T15:57:00.094 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000070", 2024-08-18T15:57:00.094 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000104", 2024-08-18T15:57:00.094 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:57:00.094 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2.retry", 2024-08-18T15:57:00.094 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:57:00.094 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:57:00.095 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:57:00.095 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:57:00.095 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000026", 2024-08-18T15:57:00.095 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:57:00.095 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:57:00.095 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:57:00.095 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:57:00.095 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:57:00.096 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:57:00.096 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.8", 2024-08-18T15:57:00.096 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:57:00.096 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:57:00.096 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:57:00.096 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:57:00.096 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000028", 2024-08-18T15:57:00.097 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:57:00.097 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:57:00.097 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:57:00.097 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:57:00.097 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:57:00.097 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:57:00.097 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:57:00.097 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2.retry", 2024-08-18T15:57:00.098 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:57:00.098 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.30", 2024-08-18T15:57:00.098 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:57:00.098 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:57:00.098 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:57:00.098 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:57:00.098 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000040", 2024-08-18T15:57:00.099 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:57:00.099 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.6", 2024-08-18T15:57:00.099 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:57:00.099 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:57:00.099 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:57:00.099 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:57:00.099 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:57:00.099 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:57:00.100 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000015", 2024-08-18T15:57:00.100 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:57:00.100 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:57:00.100 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000069", 2024-08-18T15:57:00.100 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:57:00.100 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:57:00.100 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:57:00.100 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000095", 2024-08-18T15:57:00.101 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0", 2024-08-18T15:57:00.101 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000003", 2024-08-18T15:57:00.101 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000047", 2024-08-18T15:57:00.101 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:57:00.101 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:57:00.101 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:57:00.101 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:57:00.102 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:57:00.102 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:57:00.102 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:57:00.102 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:57:00.102 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:57:00.102 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000035", 2024-08-18T15:57:00.102 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:57:00.102 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:57:00.103 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000037", 2024-08-18T15:57:00.103 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:57:00.103 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:57:00.103 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:57:00.103 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:57:00.103 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:57:00.103 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:57:00.103 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:57:00.104 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:57:00.104 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:57:00.104 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:57:00.104 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:57:00.104 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:57:00.104 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:57:00.104 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:57:00.104 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000024", 2024-08-18T15:57:00.104 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:57:00.105 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:57:00.105 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:57:00.105 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:57:00.105 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000000", 2024-08-18T15:57:00.105 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:57:00.105 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000031", 2024-08-18T15:57:00.105 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:57:00.105 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:57:00.106 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:57:00.106 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:57:00.106 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:57:00.106 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:57:00.106 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3", 2024-08-18T15:57:00.106 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:57:00.106 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:57:00.107 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:57:00.107 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:57:00.107 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:57:00.107 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:57:00.107 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:57:00.107 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:57:00.107 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:57:00.107 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:57:00.107 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:57:00.107 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:57:00.108 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:57:00.108 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:57:00.108 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:57:00.108 INFO:teuthology.orchestra.run.smithi193.stdout: "data_loggenerations_metadata", 2024-08-18T15:57:00.108 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2", 2024-08-18T15:57:00.108 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:57:00.108 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:57:00.108 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:57:00.108 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:57:00.108 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T15:57:00.108 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:57:00.108 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:57:00.108 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:57:00.108 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:57:00.109 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:57:00.109 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000076", 2024-08-18T15:57:00.109 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:57:00.109 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:57:00.109 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000113", 2024-08-18T15:57:00.109 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000005", 2024-08-18T15:57:00.109 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:57:00.109 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000011", 2024-08-18T15:57:00.109 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:57:00.109 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000058", 2024-08-18T15:57:00.109 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:57:00.109 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:57:00.109 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:57:00.109 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000052", 2024-08-18T15:57:00.110 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:57:00.110 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:57:00.110 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:57:00.110 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:57:00.110 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.21", 2024-08-18T15:57:00.110 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:57:00.110 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:57:00.110 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:57:00.110 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:57:00.110 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:57:00.110 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:57:00.111 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:57:00.111 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:57:00.111 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:57:00.111 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:57:00.111 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:57:00.111 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:57:00.111 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:57:00.111 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:57:00.111 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:57:00.111 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:57:00.111 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:57:00.111 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:57:00.112 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:57:00.112 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000088", 2024-08-18T15:57:00.112 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000080", 2024-08-18T15:57:00.112 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:57:00.112 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:57:00.112 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000090", 2024-08-18T15:57:00.112 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:57:00.112 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:57:00.112 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000110", 2024-08-18T15:57:00.112 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:57:00.112 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:57:00.113 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:57:00.113 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.0", 2024-08-18T15:57:00.113 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:57:00.113 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000096", 2024-08-18T15:57:00.113 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:57:00.113 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:57:00.113 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:57:00.113 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:57:00.113 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:57:00.113 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000087", 2024-08-18T15:57:00.113 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:57:00.113 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:57:00.113 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:57:00.114 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:57:00.114 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000008", 2024-08-18T15:57:00.114 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:57:00.114 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:57:00.114 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:57:00.114 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:57:00.114 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:57:00.114 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000006", 2024-08-18T15:57:00.115 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000029", 2024-08-18T15:57:00.115 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:57:00.115 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000089", 2024-08-18T15:57:00.115 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:57:00.115 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:57:00.115 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000022", 2024-08-18T15:57:00.116 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:57:00.116 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:57:00.116 INFO:teuthology.orchestra.run.smithi193.stdout: "bilog.trim", 2024-08-18T15:57:00.116 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000016", 2024-08-18T15:57:00.116 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:57:00.116 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:57:00.117 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:57:00.117 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:57:00.117 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:57:00.117 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000041", 2024-08-18T15:57:00.117 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:57:00.117 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:57:00.117 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.log.01050e30-179b-43be-bdbb-3bb23cf37ae8.3", 2024-08-18T15:57:00.118 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:57:00.118 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000018", 2024-08-18T15:57:00.118 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000092", 2024-08-18T15:57:00.118 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:57:00.118 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:57:00.118 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:57:00.118 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:57:00.119 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.27", 2024-08-18T15:57:00.119 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.4", 2024-08-18T15:57:00.119 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:57:00.119 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:57:00.119 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:57:00.119 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:57:00.119 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:57:00.120 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:57:00.120 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:57:00.120 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000014", 2024-08-18T15:57:00.120 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:57:00.120 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3.retry", 2024-08-18T15:57:00.120 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:57:00.120 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:57:00.120 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:57:00.120 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:57:00.121 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:57:00.121 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000112", 2024-08-18T15:57:00.121 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:57:00.121 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000007", 2024-08-18T15:57:00.121 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000021", 2024-08-18T15:57:00.121 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:57:00.121 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:57:00.122 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:57:00.122 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:57:00.122 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:57:00.122 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.16", 2024-08-18T15:57:00.122 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:57:00.122 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000064", 2024-08-18T15:57:00.123 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:57:00.123 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:57:00.123 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:57:00.123 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:57:00.123 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:57:00.123 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:57:00.123 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:57:00.123 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000071", 2024-08-18T15:57:00.123 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:57:00.123 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:57:00.124 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:57:00.124 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000074", 2024-08-18T15:57:00.124 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:57:00.124 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:57:00.124 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:57:00.124 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:57:00.124 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:57:00.124 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:57:00.124 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:57:00.124 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:57:00.124 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:57:00.124 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:57:00.124 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:57:00.124 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:57:00.125 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:57:00.125 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:57:00.125 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:57:00.125 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000081", 2024-08-18T15:57:00.125 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:57:00.125 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000009", 2024-08-18T15:57:00.125 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000121", 2024-08-18T15:57:00.125 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1", 2024-08-18T15:57:00.125 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:57:00.125 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:57:00.125 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:57:00.125 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:57:00.126 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:57:00.126 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:57:00.126 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.17", 2024-08-18T15:57:00.126 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000125", 2024-08-18T15:57:00.126 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:57:00.126 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000082", 2024-08-18T15:57:00.126 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:57:00.126 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:57:00.126 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:57:00.126 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:57:00.126 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:57:00.126 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:57:00.126 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:57:00.127 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:57:00.127 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:57:00.127 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:57:00.127 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:57:00.127 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000105", 2024-08-18T15:57:00.127 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000059", 2024-08-18T15:57:00.127 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:57:00.127 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:57:00.127 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:57:00.127 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:57:00.127 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:57:00.127 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:57:00.127 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:57:00.128 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:57:00.128 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:57:00.128 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:57:00.128 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:57:00.128 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000077", 2024-08-18T15:57:00.128 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000032", 2024-08-18T15:57:00.128 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:57:00.128 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:57:00.128 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:57:00.128 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:57:00.128 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:57:00.128 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:57:00.128 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.24", 2024-08-18T15:57:00.129 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:57:00.129 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:57:00.129 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:57:00.129 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:57:00.129 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3.retry", 2024-08-18T15:57:00.129 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:57:00.129 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:57:00.129 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:57:00.129 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:57:00.129 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:57:00.129 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:57:00.129 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000053", 2024-08-18T15:57:00.129 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000091", 2024-08-18T15:57:00.129 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:57:00.130 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:57:00.130 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:57:00.130 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000065", 2024-08-18T15:57:00.130 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:57:00.130 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:57:00.130 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:57:00.130 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:57:00.130 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000083", 2024-08-18T15:57:00.130 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000010", 2024-08-18T15:57:00.130 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:57:00.130 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:57:00.130 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:57:00.130 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:57:00.131 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:57:00.131 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:57:00.131 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:57:00.131 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:57:00.131 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:57:00.131 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:57:00.131 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:57:00.131 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.7", 2024-08-18T15:57:00.131 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.3", 2024-08-18T15:57:00.131 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:57:00.131 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:57:00.131 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:57:00.131 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:57:00.131 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:57:00.132 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:57:00.132 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:57:00.132 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:57:00.132 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.2", 2024-08-18T15:57:00.132 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:57:00.132 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:57:00.132 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:57:00.132 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:57:00.132 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:57:00.132 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:57:00.132 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:57:00.133 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000045", 2024-08-18T15:57:00.133 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000002", 2024-08-18T15:57:00.133 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:57:00.133 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3.2UbiLidwSTy.0", 2024-08-18T15:57:00.133 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:57:00.133 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.12", 2024-08-18T15:57:00.133 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:57:00.133 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:57:00.133 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:57:00.133 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:57:00.133 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:57:00.133 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:57:00.134 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000116", 2024-08-18T15:57:00.134 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.25", 2024-08-18T15:57:00.134 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:57:00.134 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:57:00.134 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:57:00.134 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:57:00.134 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000034", 2024-08-18T15:57:00.134 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:57:00.134 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:57:00.134 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000101", 2024-08-18T15:57:00.134 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:57:00.135 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:57:00.135 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.13", 2024-08-18T15:57:00.135 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:57:00.135 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:57:00.135 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.1", 2024-08-18T15:57:00.135 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0.retry", 2024-08-18T15:57:00.135 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:57:00.135 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:57:00.135 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000079", 2024-08-18T15:57:00.135 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000049", 2024-08-18T15:57:00.135 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:57:00.135 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0", 2024-08-18T15:57:00.135 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:57:00.136 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:57:00.136 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000117", 2024-08-18T15:57:00.136 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:57:00.136 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000044", 2024-08-18T15:57:00.136 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:57:00.136 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:57:00.136 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:57:00.136 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:57:00.136 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:57:00.136 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:57:00.136 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.14", 2024-08-18T15:57:00.136 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:57:00.136 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000066", 2024-08-18T15:57:00.137 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:57:00.137 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:57:00.137 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:57:00.137 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:57:00.137 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:57:00.137 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000068", 2024-08-18T15:57:00.137 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:57:00.137 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000085", 2024-08-18T15:57:00.137 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:57:00.137 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:57:00.137 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:57:00.137 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:57:00.138 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:57:00.138 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:57:00.138 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:57:00.138 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:57:00.138 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:57:00.138 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:57:00.138 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:57:00.138 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:57:00.138 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000073", 2024-08-18T15:57:00.138 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:57:00.138 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:57:00.139 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:57:00.139 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000038", 2024-08-18T15:57:00.139 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:57:00.139 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:57:00.139 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:57:00.139 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:57:00.139 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.10", 2024-08-18T15:57:00.139 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:57:00.139 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:57:00.139 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:57:00.139 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000118", 2024-08-18T15:57:00.139 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:57:00.139 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3", 2024-08-18T15:57:00.140 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:57:00.140 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000036", 2024-08-18T15:57:00.140 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:57:00.140 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:57:00.140 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:57:00.140 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:57:00.140 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:57:00.140 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000103", 2024-08-18T15:57:00.140 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.3", 2024-08-18T15:57:00.140 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:57:00.140 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000119", 2024-08-18T15:57:00.140 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000098", 2024-08-18T15:57:00.141 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:57:00.141 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:57:00.141 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:57:00.141 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:57:00.141 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:57:00.141 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:57:00.141 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:57:00.141 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:57:00.141 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.0", 2024-08-18T15:57:00.141 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:57:00.141 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:57:00.141 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:57:00.141 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:57:00.141 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:57:00.142 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:57:00.142 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:57:00.142 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:57:00.142 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000027", 2024-08-18T15:57:00.142 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:57:00.142 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status", 2024-08-18T15:57:00.142 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:57:00.142 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000019", 2024-08-18T15:57:00.142 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:57:00.142 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:57:00.142 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:57:00.142 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:57:00.143 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1", 2024-08-18T15:57:00.143 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:57:00.143 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:57:00.143 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:57:00.143 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:57:00.143 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000039", 2024-08-18T15:57:00.143 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:57:00.143 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:57:00.143 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:57:00.143 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:57:00.143 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:57:00.144 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:57:00.144 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:57:00.144 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:57:00.144 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:57:00.144 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:57:00.144 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:57:00.144 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000100", 2024-08-18T15:57:00.144 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:57:00.144 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:57:00.144 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:57:00.144 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:57:00.144 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:57:00.145 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000093", 2024-08-18T15:57:00.145 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:57:00.145 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000004", 2024-08-18T15:57:00.145 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000063", 2024-08-18T15:57:00.145 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000122", 2024-08-18T15:57:00.145 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:57:00.145 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:57:00.145 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:57:00.145 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000057", 2024-08-18T15:57:00.145 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:57:00.145 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:57:00.145 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:57:00.145 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:57:00.146 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000054", 2024-08-18T15:57:00.146 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:57:00.146 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000114", 2024-08-18T15:57:00.146 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:57:00.146 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:57:00.146 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:57:00.146 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.2", 2024-08-18T15:57:00.146 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:57:00.146 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:57:00.147 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0", 2024-08-18T15:57:00.147 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:57:00.147 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:57:00.147 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:57:00.147 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:57:00.147 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:57:00.147 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:57:00.147 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000012", 2024-08-18T15:57:00.147 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:57:00.147 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:57:00.148 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:57:00.148 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000084", 2024-08-18T15:57:00.148 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000043", 2024-08-18T15:57:00.148 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3", 2024-08-18T15:57:00.148 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:57:00.148 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:57:00.148 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.11", 2024-08-18T15:57:00.148 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:57:00.148 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:57:00.148 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:57:00.149 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:57:00.149 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:57:00.149 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:57:00.149 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:57:00.149 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:57:00.149 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000111", 2024-08-18T15:57:00.149 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:57:00.149 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000048", 2024-08-18T15:57:00.149 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:57:00.149 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000020", 2024-08-18T15:57:00.149 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:57:00.150 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:57:00.150 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:57:00.150 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:57:00.150 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:57:00.150 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000099", 2024-08-18T15:57:00.150 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000056", 2024-08-18T15:57:00.150 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:57:00.150 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:57:00.150 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.19", 2024-08-18T15:57:00.150 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:57:00.150 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.28", 2024-08-18T15:57:00.150 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:57:00.150 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:57:00.151 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:57:00.151 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:57:00.151 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:57:00.151 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000072", 2024-08-18T15:57:00.151 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:57:00.151 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:57:00.151 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000062", 2024-08-18T15:57:00.151 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:57:00.151 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:57:00.151 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:57:00.151 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000115", 2024-08-18T15:57:00.151 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:57:00.151 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.1", 2024-08-18T15:57:00.152 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2", 2024-08-18T15:57:00.152 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000033", 2024-08-18T15:57:00.152 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:57:00.152 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:57:00.152 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:57:00.152 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:57:00.152 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:57:00.152 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:57:00.152 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:57:00.152 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:57:00.152 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:57:00.152 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:57:00.152 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:57:00.152 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:57:00.153 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:57:00.153 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:57:00.153 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:57:00.153 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:57:00.153 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:57:00.153 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:57:00.153 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:57:00.153 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000025", 2024-08-18T15:57:00.153 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:57:00.153 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000123", 2024-08-18T15:57:00.153 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.31", 2024-08-18T15:57:00.154 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:57:00.154 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:57:00.154 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:57:00.154 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:57:00.154 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.18", 2024-08-18T15:57:00.154 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:57:00.154 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:57:00.155 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:57:00.155 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:57:00.155 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:57:00.155 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000108", 2024-08-18T15:57:00.155 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:57:00.155 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:57:00.156 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000094", 2024-08-18T15:57:00.156 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:57:00.156 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:57:00.156 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:57:00.156 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:57:00.156 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:57:00.156 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:57:00.156 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000060", 2024-08-18T15:57:00.157 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:57:00.157 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:57:00.157 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:57:00.157 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000109", 2024-08-18T15:57:00.157 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.15", 2024-08-18T15:57:00.157 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:57:00.157 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:57:00.157 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:57:00.157 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:57:00.158 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:57:00.158 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:57:00.158 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:57:00.158 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:57:00.158 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:57:00.158 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000013", 2024-08-18T15:57:00.158 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:57:00.158 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:57:00.158 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:57:00.158 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:57:00.158 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:57:00.159 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:57:00.159 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:57:00.159 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:57:00.159 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:57:00.159 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:57:00.159 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:57:00.159 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:57:00.159 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:57:00.159 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:57:00.160 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000051", 2024-08-18T15:57:00.160 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:57:00.160 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:57:00.160 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:57:00.160 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:57:00.160 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000106", 2024-08-18T15:57:00.160 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:57:00.160 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:57:00.160 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:57:00.160 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:57:00.160 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:57:00.160 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000017", 2024-08-18T15:57:00.160 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:57:00.161 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:57:00.161 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:57:00.161 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000023", 2024-08-18T15:57:00.161 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:57:00.161 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:57:00.161 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:57:00.161 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:57:00.161 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:57:00.161 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:57:00.161 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:57:00.161 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000107", 2024-08-18T15:57:00.161 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:57:00.161 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:57:00.161 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000046", 2024-08-18T15:57:00.162 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:57:00.162 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:57:00.162 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:57:00.162 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.26", 2024-08-18T15:57:00.162 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:57:00.162 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:57:00.162 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9", 2024-08-18T15:57:00.162 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.22", 2024-08-18T15:57:00.162 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:57:00.162 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:57:00.162 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:57:00.162 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:57:00.162 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:57:00.163 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:57:00.163 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:57:00.163 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2", 2024-08-18T15:57:00.163 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:57:00.163 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:57:00.163 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000067", 2024-08-18T15:57:00.163 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:57:00.163 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:57:00.163 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:57:00.163 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:57:00.164 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:57:00.164 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:57:00.164 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:57:00.164 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:57:00.164 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:57:00.164 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:57:00.164 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:57:00.164 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:57:00.164 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:57:00.164 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:57:00.164 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:57:00.164 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:57:00.164 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:57:00.165 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:57:00.165 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:57:00.165 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.9", 2024-08-18T15:57:00.165 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000086", 2024-08-18T15:57:00.165 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.5", 2024-08-18T15:57:00.165 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:57:00.165 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:57:00.165 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:57:00.165 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:57:00.165 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:57:00.165 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:57:00.165 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2.MFXJ583gCr1.0", 2024-08-18T15:57:00.165 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:57:00.166 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1.GPnO16liyZ+.0", 2024-08-18T15:57:00.166 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:57:00.166 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:57:00.166 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:57:00.166 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:57:00.166 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:57:00.166 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:57:00.166 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:57:00.166 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:57:00.166 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:57:00.166 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.20", 2024-08-18T15:57:00.166 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:57:00.167 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:57:00.167 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:57:00.167 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:57:00.167 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:57:00.167 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:57:00.167 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1.retry", 2024-08-18T15:57:00.167 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:57:00.167 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:57:00.167 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:57:00.167 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0.retry", 2024-08-18T15:57:00.167 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000050", 2024-08-18T15:57:00.167 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:57:00.168 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000102", 2024-08-18T15:57:00.168 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:57:00.168 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:57:00.168 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:57:00.168 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:57:00.168 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:57:00.168 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:57:00.168 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:57:00.168 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:57:00.168 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:57:00.168 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:57:00.168 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:57:00.168 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000124", 2024-08-18T15:57:00.169 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:57:00.169 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:57:00.169 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:57:00.169 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000001", 2024-08-18T15:57:00.169 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:57:00.169 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:57:00.169 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:57:00.169 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:57:00.169 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000126", 2024-08-18T15:57:00.169 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:57:00.169 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0.fZhctfLXnSt.0", 2024-08-18T15:57:00.170 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:57:00.170 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000055", 2024-08-18T15:57:00.170 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:57:00.170 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000061", 2024-08-18T15:57:00.170 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1", 2024-08-18T15:57:00.171 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:57:00.171 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:57:00.171 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:57:00.171 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000097", 2024-08-18T15:57:00.171 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:57:00.171 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:57:00.172 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:57:00.172 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:57:00.172 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:57:00.172 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:57:00.172 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:57:00.172 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:57:00.172 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000042", 2024-08-18T15:57:00.172 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:57:00.173 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:57:00.173 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:57:00.173 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1.retry", 2024-08-18T15:57:00.173 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:57:00.173 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:57:00.173 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:57:00.173 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:57:00.173 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:57:00.174 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:57:00.174 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:57:00.174 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000075", 2024-08-18T15:57:00.174 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:57:00.174 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000120", 2024-08-18T15:57:00.174 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000030", 2024-08-18T15:57:00.174 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:57:00.174 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:57:00.174 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:57:00.175 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:57:00.175 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:57:00.175 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:57:00.175 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9" 2024-08-18T15:57:00.175 INFO:teuthology.orchestra.run.smithi193.stdout:] 2024-08-18T15:57:00.175 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:57:05.177 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:57:05.177 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:57:05.178 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:57:05.230 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:57:05.230 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:57:05.293 INFO:teuthology.orchestra.run.smithi193.stdout:[ 2024-08-18T15:57:05.293 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:57:05.293 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:57:05.293 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.29", 2024-08-18T15:57:05.293 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000078", 2024-08-18T15:57:05.293 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.history", 2024-08-18T15:57:05.293 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:57:05.293 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:57:05.293 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:57:05.294 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:57:05.294 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.23", 2024-08-18T15:57:05.294 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000070", 2024-08-18T15:57:05.294 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000104", 2024-08-18T15:57:05.294 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:57:05.294 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2.retry", 2024-08-18T15:57:05.294 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:57:05.294 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:57:05.294 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:57:05.294 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:57:05.294 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000026", 2024-08-18T15:57:05.295 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:57:05.295 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:57:05.295 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:57:05.295 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:57:05.295 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:57:05.295 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:57:05.295 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.8", 2024-08-18T15:57:05.295 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:57:05.295 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:57:05.295 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:57:05.295 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:57:05.295 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000028", 2024-08-18T15:57:05.296 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:57:05.296 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:57:05.296 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:57:05.296 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:57:05.296 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:57:05.296 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:57:05.296 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:57:05.296 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2.retry", 2024-08-18T15:57:05.296 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:57:05.296 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.30", 2024-08-18T15:57:05.296 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:57:05.296 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:57:05.296 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:57:05.297 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:57:05.297 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000040", 2024-08-18T15:57:05.297 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:57:05.297 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.6", 2024-08-18T15:57:05.297 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:57:05.297 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:57:05.297 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:57:05.297 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:57:05.297 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:57:05.297 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:57:05.297 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000015", 2024-08-18T15:57:05.297 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:57:05.297 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:57:05.298 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000069", 2024-08-18T15:57:05.298 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:57:05.298 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:57:05.298 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:57:05.298 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000095", 2024-08-18T15:57:05.298 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0", 2024-08-18T15:57:05.298 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000003", 2024-08-18T15:57:05.298 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000047", 2024-08-18T15:57:05.298 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:57:05.298 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:57:05.298 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:57:05.298 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:57:05.299 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:57:05.299 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:57:05.299 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:57:05.299 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:57:05.299 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:57:05.299 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000035", 2024-08-18T15:57:05.299 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:57:05.299 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:57:05.299 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000037", 2024-08-18T15:57:05.299 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:57:05.299 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:57:05.299 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:57:05.299 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:57:05.299 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:57:05.300 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:57:05.300 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:57:05.300 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:57:05.300 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:57:05.300 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:57:05.300 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:57:05.300 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:57:05.300 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:57:05.300 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:57:05.300 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000024", 2024-08-18T15:57:05.300 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:57:05.300 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:57:05.300 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:57:05.301 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:57:05.301 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000000", 2024-08-18T15:57:05.301 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:57:05.301 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000031", 2024-08-18T15:57:05.301 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:57:05.301 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:57:05.301 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:57:05.301 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:57:05.301 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:57:05.301 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:57:05.301 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3", 2024-08-18T15:57:05.301 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:57:05.301 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:57:05.302 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:57:05.302 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:57:05.302 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:57:05.302 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:57:05.302 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:57:05.302 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:57:05.302 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:57:05.302 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:57:05.302 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:57:05.302 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:57:05.302 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:57:05.302 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:57:05.302 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:57:05.303 INFO:teuthology.orchestra.run.smithi193.stdout: "data_loggenerations_metadata", 2024-08-18T15:57:05.303 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2", 2024-08-18T15:57:05.303 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:57:05.303 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:57:05.303 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:57:05.303 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:57:05.303 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T15:57:05.303 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:57:05.303 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:57:05.303 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:57:05.303 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:57:05.303 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:57:05.303 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000076", 2024-08-18T15:57:05.304 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:57:05.304 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:57:05.304 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000113", 2024-08-18T15:57:05.304 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000005", 2024-08-18T15:57:05.304 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:57:05.304 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000011", 2024-08-18T15:57:05.304 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:57:05.304 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000058", 2024-08-18T15:57:05.304 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:57:05.304 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:57:05.304 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:57:05.304 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000052", 2024-08-18T15:57:05.304 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:57:05.305 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:57:05.305 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:57:05.305 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:57:05.305 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.21", 2024-08-18T15:57:05.305 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:57:05.305 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:57:05.305 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:57:05.305 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:57:05.305 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:57:05.305 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:57:05.305 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:57:05.305 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:57:05.305 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:57:05.306 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:57:05.306 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:57:05.306 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:57:05.306 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:57:05.306 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:57:05.306 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:57:05.306 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:57:05.306 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:57:05.306 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:57:05.306 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:57:05.306 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000088", 2024-08-18T15:57:05.306 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000080", 2024-08-18T15:57:05.307 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:57:05.307 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:57:05.307 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000090", 2024-08-18T15:57:05.307 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:57:05.307 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:57:05.307 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000110", 2024-08-18T15:57:05.307 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:57:05.307 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:57:05.307 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:57:05.307 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.0", 2024-08-18T15:57:05.307 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:57:05.307 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000096", 2024-08-18T15:57:05.307 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:57:05.308 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:57:05.308 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:57:05.308 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:57:05.308 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:57:05.308 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000087", 2024-08-18T15:57:05.308 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:57:05.308 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:57:05.308 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:57:05.308 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:57:05.308 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000008", 2024-08-18T15:57:05.308 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:57:05.308 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:57:05.309 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:57:05.309 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:57:05.309 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:57:05.309 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000006", 2024-08-18T15:57:05.309 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000029", 2024-08-18T15:57:05.309 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:57:05.309 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000089", 2024-08-18T15:57:05.309 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:57:05.309 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:57:05.309 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000022", 2024-08-18T15:57:05.309 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:57:05.309 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:57:05.309 INFO:teuthology.orchestra.run.smithi193.stdout: "bilog.trim", 2024-08-18T15:57:05.309 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000016", 2024-08-18T15:57:05.310 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:57:05.310 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:57:05.310 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:57:05.310 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:57:05.310 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:57:05.310 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000041", 2024-08-18T15:57:05.310 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:57:05.310 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:57:05.310 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.log.01050e30-179b-43be-bdbb-3bb23cf37ae8.3", 2024-08-18T15:57:05.310 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:57:05.310 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000018", 2024-08-18T15:57:05.310 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000092", 2024-08-18T15:57:05.310 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:57:05.311 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:57:05.311 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:57:05.311 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:57:05.311 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.27", 2024-08-18T15:57:05.311 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.4", 2024-08-18T15:57:05.311 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:57:05.311 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:57:05.311 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:57:05.311 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:57:05.311 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:57:05.311 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:57:05.311 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:57:05.311 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000014", 2024-08-18T15:57:05.312 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:57:05.312 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3.retry", 2024-08-18T15:57:05.312 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:57:05.312 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:57:05.312 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:57:05.312 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:57:05.312 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:57:05.312 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000112", 2024-08-18T15:57:05.312 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:57:05.312 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000007", 2024-08-18T15:57:05.312 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000021", 2024-08-18T15:57:05.312 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:57:05.313 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:57:05.313 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:57:05.313 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:57:05.313 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:57:05.313 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.16", 2024-08-18T15:57:05.313 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:57:05.313 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000064", 2024-08-18T15:57:05.313 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:57:05.313 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:57:05.313 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:57:05.313 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:57:05.313 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:57:05.313 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:57:05.314 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:57:05.314 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000071", 2024-08-18T15:57:05.314 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:57:05.314 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:57:05.314 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:57:05.314 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000074", 2024-08-18T15:57:05.314 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:57:05.314 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:57:05.314 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:57:05.314 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:57:05.314 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:57:05.314 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:57:05.315 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:57:05.315 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:57:05.315 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:57:05.315 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:57:05.315 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:57:05.315 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:57:05.315 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:57:05.315 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:57:05.315 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:57:05.315 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000081", 2024-08-18T15:57:05.315 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:57:05.315 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000009", 2024-08-18T15:57:05.315 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000121", 2024-08-18T15:57:05.315 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1", 2024-08-18T15:57:05.316 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:57:05.316 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:57:05.316 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:57:05.316 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:57:05.316 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:57:05.316 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:57:05.316 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.17", 2024-08-18T15:57:05.316 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000125", 2024-08-18T15:57:05.316 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:57:05.316 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000082", 2024-08-18T15:57:05.316 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:57:05.316 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:57:05.316 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:57:05.317 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:57:05.317 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:57:05.317 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:57:05.317 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:57:05.317 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:57:05.317 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:57:05.317 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:57:05.317 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:57:05.317 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000105", 2024-08-18T15:57:05.317 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000059", 2024-08-18T15:57:05.317 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:57:05.317 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:57:05.317 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:57:05.318 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:57:05.318 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:57:05.318 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:57:05.318 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:57:05.318 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:57:05.318 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:57:05.318 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:57:05.318 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:57:05.318 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000077", 2024-08-18T15:57:05.318 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000032", 2024-08-18T15:57:05.318 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:57:05.318 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:57:05.319 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:57:05.319 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:57:05.319 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:57:05.319 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:57:05.319 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.24", 2024-08-18T15:57:05.319 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:57:05.319 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:57:05.319 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:57:05.319 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:57:05.319 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3.retry", 2024-08-18T15:57:05.319 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:57:05.319 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:57:05.320 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:57:05.320 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:57:05.320 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:57:05.320 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:57:05.320 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000053", 2024-08-18T15:57:05.320 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000091", 2024-08-18T15:57:05.320 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:57:05.320 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:57:05.320 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:57:05.320 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000065", 2024-08-18T15:57:05.320 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:57:05.320 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:57:05.320 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:57:05.321 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:57:05.321 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000083", 2024-08-18T15:57:05.321 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000010", 2024-08-18T15:57:05.321 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:57:05.321 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:57:05.321 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:57:05.321 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:57:05.321 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:57:05.321 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:57:05.321 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:57:05.321 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:57:05.321 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:57:05.321 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:57:05.322 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:57:05.322 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.7", 2024-08-18T15:57:05.322 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.3", 2024-08-18T15:57:05.322 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:57:05.322 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:57:05.322 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:57:05.322 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:57:05.322 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:57:05.322 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:57:05.322 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:57:05.322 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:57:05.322 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.2", 2024-08-18T15:57:05.322 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:57:05.323 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:57:05.323 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:57:05.323 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:57:05.323 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:57:05.323 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:57:05.323 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:57:05.323 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000045", 2024-08-18T15:57:05.323 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000002", 2024-08-18T15:57:05.323 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:57:05.323 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3.2UbiLidwSTy.0", 2024-08-18T15:57:05.323 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:57:05.323 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.12", 2024-08-18T15:57:05.323 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:57:05.324 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:57:05.324 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:57:05.324 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:57:05.324 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:57:05.324 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:57:05.324 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000116", 2024-08-18T15:57:05.324 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.25", 2024-08-18T15:57:05.324 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:57:05.324 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:57:05.324 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:57:05.324 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:57:05.324 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000034", 2024-08-18T15:57:05.324 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:57:05.325 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:57:05.325 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000101", 2024-08-18T15:57:05.325 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:57:05.325 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:57:05.325 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.13", 2024-08-18T15:57:05.325 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:57:05.325 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:57:05.325 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.1", 2024-08-18T15:57:05.325 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0.retry", 2024-08-18T15:57:05.325 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:57:05.325 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:57:05.325 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000079", 2024-08-18T15:57:05.326 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000049", 2024-08-18T15:57:05.326 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:57:05.326 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0", 2024-08-18T15:57:05.326 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:57:05.326 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:57:05.326 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000117", 2024-08-18T15:57:05.326 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:57:05.326 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000044", 2024-08-18T15:57:05.326 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:57:05.326 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:57:05.326 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:57:05.326 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:57:05.326 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:57:05.327 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:57:05.327 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.14", 2024-08-18T15:57:05.327 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:57:05.327 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000066", 2024-08-18T15:57:05.327 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:57:05.327 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:57:05.327 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:57:05.327 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:57:05.327 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:57:05.327 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000068", 2024-08-18T15:57:05.327 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:57:05.327 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000085", 2024-08-18T15:57:05.328 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:57:05.328 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:57:05.328 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:57:05.328 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:57:05.328 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:57:05.328 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:57:05.328 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:57:05.328 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:57:05.328 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:57:05.328 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:57:05.328 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:57:05.328 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:57:05.328 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000073", 2024-08-18T15:57:05.329 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:57:05.329 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:57:05.329 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:57:05.329 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000038", 2024-08-18T15:57:05.329 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:57:05.329 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:57:05.329 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:57:05.329 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:57:05.329 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.10", 2024-08-18T15:57:05.329 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:57:05.329 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:57:05.329 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:57:05.329 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000118", 2024-08-18T15:57:05.330 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:57:05.330 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3", 2024-08-18T15:57:05.330 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:57:05.330 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000036", 2024-08-18T15:57:05.330 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:57:05.330 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:57:05.330 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:57:05.330 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:57:05.330 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:57:05.330 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000103", 2024-08-18T15:57:05.330 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.3", 2024-08-18T15:57:05.330 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:57:05.330 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000119", 2024-08-18T15:57:05.331 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000098", 2024-08-18T15:57:05.331 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:57:05.331 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:57:05.331 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:57:05.331 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:57:05.331 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:57:05.331 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:57:05.331 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:57:05.331 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:57:05.332 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.0", 2024-08-18T15:57:05.332 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:57:05.332 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:57:05.332 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:57:05.332 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:57:05.332 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:57:05.332 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:57:05.332 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:57:05.332 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:57:05.332 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000027", 2024-08-18T15:57:05.332 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:57:05.332 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status", 2024-08-18T15:57:05.332 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:57:05.333 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000019", 2024-08-18T15:57:05.333 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:57:05.333 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:57:05.333 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:57:05.333 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:57:05.333 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1", 2024-08-18T15:57:05.333 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:57:05.333 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:57:05.333 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:57:05.333 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:57:05.333 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000039", 2024-08-18T15:57:05.333 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:57:05.334 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:57:05.334 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:57:05.334 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:57:05.334 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:57:05.334 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:57:05.334 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:57:05.334 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:57:05.334 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:57:05.334 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:57:05.334 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:57:05.334 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000100", 2024-08-18T15:57:05.334 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:57:05.334 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:57:05.335 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:57:05.335 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:57:05.335 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:57:05.335 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000093", 2024-08-18T15:57:05.335 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:57:05.335 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000004", 2024-08-18T15:57:05.335 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000063", 2024-08-18T15:57:05.335 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000122", 2024-08-18T15:57:05.335 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:57:05.335 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:57:05.335 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:57:05.335 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000057", 2024-08-18T15:57:05.335 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:57:05.336 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:57:05.336 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:57:05.336 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:57:05.336 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000054", 2024-08-18T15:57:05.336 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:57:05.336 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000114", 2024-08-18T15:57:05.336 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:57:05.336 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:57:05.336 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:57:05.336 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.2", 2024-08-18T15:57:05.336 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:57:05.336 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:57:05.336 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0", 2024-08-18T15:57:05.337 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:57:05.337 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:57:05.337 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:57:05.337 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:57:05.337 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:57:05.337 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:57:05.337 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000012", 2024-08-18T15:57:05.337 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:57:05.337 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:57:05.337 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:57:05.337 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000084", 2024-08-18T15:57:05.337 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000043", 2024-08-18T15:57:05.338 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3", 2024-08-18T15:57:05.338 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:57:05.338 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:57:05.338 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.11", 2024-08-18T15:57:05.338 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:57:05.338 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:57:05.338 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:57:05.338 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:57:05.338 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:57:05.338 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:57:05.338 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:57:05.339 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:57:05.339 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000111", 2024-08-18T15:57:05.339 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:57:05.339 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000048", 2024-08-18T15:57:05.339 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:57:05.339 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000020", 2024-08-18T15:57:05.339 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:57:05.339 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:57:05.339 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:57:05.339 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:57:05.339 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:57:05.339 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000099", 2024-08-18T15:57:05.339 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000056", 2024-08-18T15:57:05.340 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:57:05.340 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:57:05.340 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.19", 2024-08-18T15:57:05.340 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:57:05.340 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.28", 2024-08-18T15:57:05.340 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:57:05.340 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:57:05.340 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:57:05.340 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:57:05.340 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:57:05.340 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000072", 2024-08-18T15:57:05.340 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:57:05.340 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:57:05.341 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000062", 2024-08-18T15:57:05.341 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:57:05.341 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:57:05.341 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:57:05.341 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000115", 2024-08-18T15:57:05.341 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:57:05.341 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.1", 2024-08-18T15:57:05.341 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2", 2024-08-18T15:57:05.341 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000033", 2024-08-18T15:57:05.341 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:57:05.341 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:57:05.341 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:57:05.341 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:57:05.342 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:57:05.342 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:57:05.342 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:57:05.342 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:57:05.342 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:57:05.342 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:57:05.342 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:57:05.342 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:57:05.342 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:57:05.342 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:57:05.342 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:57:05.342 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:57:05.342 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:57:05.343 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:57:05.343 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:57:05.343 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000025", 2024-08-18T15:57:05.343 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:57:05.343 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000123", 2024-08-18T15:57:05.343 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.31", 2024-08-18T15:57:05.343 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:57:05.343 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:57:05.343 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:57:05.343 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:57:05.343 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.18", 2024-08-18T15:57:05.343 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:57:05.343 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:57:05.344 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:57:05.344 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:57:05.344 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:57:05.344 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000108", 2024-08-18T15:57:05.344 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:57:05.344 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:57:05.344 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000094", 2024-08-18T15:57:05.344 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:57:05.344 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:57:05.344 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:57:05.344 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:57:05.344 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:57:05.345 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:57:05.345 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000060", 2024-08-18T15:57:05.345 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:57:05.345 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:57:05.345 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:57:05.345 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000109", 2024-08-18T15:57:05.345 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.15", 2024-08-18T15:57:05.345 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:57:05.345 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:57:05.345 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:57:05.345 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:57:05.345 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:57:05.345 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:57:05.345 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:57:05.346 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:57:05.346 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:57:05.346 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000013", 2024-08-18T15:57:05.346 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:57:05.346 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:57:05.346 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:57:05.346 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:57:05.346 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:57:05.346 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:57:05.346 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:57:05.346 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:57:05.346 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:57:05.347 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:57:05.347 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:57:05.347 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:57:05.347 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:57:05.347 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:57:05.347 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000051", 2024-08-18T15:57:05.347 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:57:05.347 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:57:05.347 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:57:05.347 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:57:05.347 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000106", 2024-08-18T15:57:05.347 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:57:05.347 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:57:05.348 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:57:05.348 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:57:05.348 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:57:05.348 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000017", 2024-08-18T15:57:05.348 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:57:05.348 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:57:05.348 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:57:05.348 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000023", 2024-08-18T15:57:05.348 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:57:05.348 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:57:05.348 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:57:05.348 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:57:05.348 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:57:05.349 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:57:05.349 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:57:05.349 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000107", 2024-08-18T15:57:05.349 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:57:05.349 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:57:05.349 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000046", 2024-08-18T15:57:05.349 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:57:05.349 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:57:05.349 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:57:05.349 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.26", 2024-08-18T15:57:05.349 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:57:05.349 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:57:05.349 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9", 2024-08-18T15:57:05.350 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.22", 2024-08-18T15:57:05.350 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:57:05.350 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:57:05.350 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:57:05.350 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:57:05.350 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:57:05.350 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:57:05.350 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:57:05.350 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2", 2024-08-18T15:57:05.350 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:57:05.350 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:57:05.350 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000067", 2024-08-18T15:57:05.350 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:57:05.351 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:57:05.351 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:57:05.351 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:57:05.351 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:57:05.351 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:57:05.351 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:57:05.351 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:57:05.351 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:57:05.351 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:57:05.351 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:57:05.351 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:57:05.351 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:57:05.351 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:57:05.352 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:57:05.352 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:57:05.352 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:57:05.352 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:57:05.352 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:57:05.352 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.9", 2024-08-18T15:57:05.352 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000086", 2024-08-18T15:57:05.352 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.5", 2024-08-18T15:57:05.352 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:57:05.352 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:57:05.352 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:57:05.352 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:57:05.352 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:57:05.353 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:57:05.353 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2.MFXJ583gCr1.0", 2024-08-18T15:57:05.353 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:57:05.353 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1.GPnO16liyZ+.0", 2024-08-18T15:57:05.353 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:57:05.353 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:57:05.353 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:57:05.353 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:57:05.353 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:57:05.353 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:57:05.353 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:57:05.353 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:57:05.353 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:57:05.354 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.20", 2024-08-18T15:57:05.354 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:57:05.354 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:57:05.354 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:57:05.354 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:57:05.354 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:57:05.354 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:57:05.354 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1.retry", 2024-08-18T15:57:05.354 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:57:05.354 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:57:05.354 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:57:05.354 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0.retry", 2024-08-18T15:57:05.354 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000050", 2024-08-18T15:57:05.355 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:57:05.355 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000102", 2024-08-18T15:57:05.355 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:57:05.355 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:57:05.355 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:57:05.355 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:57:05.355 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:57:05.355 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:57:05.355 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:57:05.355 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:57:05.355 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:57:05.355 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:57:05.355 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:57:05.355 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000124", 2024-08-18T15:57:05.356 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:57:05.356 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:57:05.356 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:57:05.356 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000001", 2024-08-18T15:57:05.356 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:57:05.356 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:57:05.356 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:57:05.356 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:57:05.356 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000126", 2024-08-18T15:57:05.356 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:57:05.356 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0.fZhctfLXnSt.0", 2024-08-18T15:57:05.356 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:57:05.356 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000055", 2024-08-18T15:57:05.357 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:57:05.357 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000061", 2024-08-18T15:57:05.357 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1", 2024-08-18T15:57:05.357 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:57:05.357 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:57:05.357 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:57:05.357 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000097", 2024-08-18T15:57:05.357 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:57:05.357 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:57:05.357 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:57:05.357 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:57:05.357 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:57:05.357 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:57:05.358 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:57:05.358 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:57:05.358 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000042", 2024-08-18T15:57:05.358 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:57:05.358 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:57:05.358 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:57:05.358 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1.retry", 2024-08-18T15:57:05.358 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:57:05.358 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:57:05.358 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:57:05.358 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:57:05.358 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:57:05.358 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:57:05.359 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:57:05.359 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000075", 2024-08-18T15:57:05.359 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:57:05.359 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000120", 2024-08-18T15:57:05.359 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000030", 2024-08-18T15:57:05.359 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:57:05.359 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:57:05.359 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:57:05.359 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:57:05.359 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:57:05.359 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:57:05.359 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9" 2024-08-18T15:57:05.360 INFO:teuthology.orchestra.run.smithi193.stdout:] 2024-08-18T15:57:05.360 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:57:10.361 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:57:10.361 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:57:10.362 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:57:10.415 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:57:10.415 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:57:10.478 INFO:teuthology.orchestra.run.smithi193.stdout:[ 2024-08-18T15:57:10.478 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:57:10.479 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:57:10.479 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.29", 2024-08-18T15:57:10.479 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000078", 2024-08-18T15:57:10.479 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.history", 2024-08-18T15:57:10.479 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:57:10.479 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:57:10.479 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:57:10.479 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:57:10.479 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.23", 2024-08-18T15:57:10.479 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000070", 2024-08-18T15:57:10.479 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000104", 2024-08-18T15:57:10.479 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:57:10.480 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2.retry", 2024-08-18T15:57:10.480 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:57:10.480 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:57:10.480 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:57:10.480 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:57:10.480 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000026", 2024-08-18T15:57:10.480 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:57:10.480 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:57:10.480 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:57:10.480 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:57:10.480 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:57:10.480 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:57:10.481 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.8", 2024-08-18T15:57:10.481 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:57:10.481 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:57:10.481 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:57:10.481 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:57:10.481 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000028", 2024-08-18T15:57:10.481 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:57:10.481 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:57:10.481 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:57:10.481 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:57:10.481 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:57:10.481 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:57:10.482 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:57:10.482 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2.retry", 2024-08-18T15:57:10.482 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:57:10.482 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.30", 2024-08-18T15:57:10.482 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:57:10.482 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:57:10.482 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:57:10.482 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:57:10.482 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000040", 2024-08-18T15:57:10.482 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:57:10.482 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.6", 2024-08-18T15:57:10.482 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:57:10.483 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:57:10.483 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:57:10.483 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:57:10.483 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:57:10.483 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:57:10.483 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000015", 2024-08-18T15:57:10.483 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:57:10.483 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:57:10.483 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000069", 2024-08-18T15:57:10.483 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:57:10.483 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:57:10.483 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:57:10.484 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000095", 2024-08-18T15:57:10.484 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0", 2024-08-18T15:57:10.484 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000003", 2024-08-18T15:57:10.484 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000047", 2024-08-18T15:57:10.484 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:57:10.484 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:57:10.484 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:57:10.484 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:57:10.484 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:57:10.484 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:57:10.484 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:57:10.484 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:57:10.484 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:57:10.485 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000035", 2024-08-18T15:57:10.485 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:57:10.485 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:57:10.485 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000037", 2024-08-18T15:57:10.485 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:57:10.485 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:57:10.485 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:57:10.485 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:57:10.485 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:57:10.485 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:57:10.485 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:57:10.485 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:57:10.485 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:57:10.486 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:57:10.486 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:57:10.486 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:57:10.486 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:57:10.486 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:57:10.486 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000024", 2024-08-18T15:57:10.486 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:57:10.486 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:57:10.486 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:57:10.486 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:57:10.486 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000000", 2024-08-18T15:57:10.486 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:57:10.487 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000031", 2024-08-18T15:57:10.487 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:57:10.487 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:57:10.487 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:57:10.487 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:57:10.487 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:57:10.487 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:57:10.487 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3", 2024-08-18T15:57:10.487 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:57:10.487 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:57:10.487 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:57:10.487 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:57:10.487 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:57:10.488 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:57:10.488 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:57:10.488 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:57:10.488 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:57:10.488 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:57:10.488 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:57:10.488 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:57:10.488 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:57:10.488 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:57:10.488 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:57:10.488 INFO:teuthology.orchestra.run.smithi193.stdout: "data_loggenerations_metadata", 2024-08-18T15:57:10.488 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2", 2024-08-18T15:57:10.488 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:57:10.489 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:57:10.489 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:57:10.489 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:57:10.489 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T15:57:10.489 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:57:10.489 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:57:10.489 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:57:10.489 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:57:10.489 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:57:10.489 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000076", 2024-08-18T15:57:10.489 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:57:10.489 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:57:10.489 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000113", 2024-08-18T15:57:10.490 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000005", 2024-08-18T15:57:10.490 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:57:10.490 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000011", 2024-08-18T15:57:10.490 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:57:10.490 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000058", 2024-08-18T15:57:10.490 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:57:10.490 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:57:10.490 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:57:10.490 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000052", 2024-08-18T15:57:10.490 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:57:10.490 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:57:10.490 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:57:10.491 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:57:10.491 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.21", 2024-08-18T15:57:10.491 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:57:10.491 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:57:10.491 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:57:10.491 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:57:10.491 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:57:10.491 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:57:10.491 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:57:10.491 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:57:10.491 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:57:10.491 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:57:10.491 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:57:10.492 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:57:10.492 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:57:10.492 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:57:10.492 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:57:10.492 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:57:10.492 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:57:10.492 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:57:10.492 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:57:10.492 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000088", 2024-08-18T15:57:10.492 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000080", 2024-08-18T15:57:10.492 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:57:10.492 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:57:10.492 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000090", 2024-08-18T15:57:10.493 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:57:10.493 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:57:10.493 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000110", 2024-08-18T15:57:10.493 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:57:10.493 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:57:10.493 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:57:10.493 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.0", 2024-08-18T15:57:10.493 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:57:10.493 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000096", 2024-08-18T15:57:10.493 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:57:10.493 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:57:10.493 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:57:10.494 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:57:10.494 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:57:10.494 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000087", 2024-08-18T15:57:10.494 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:57:10.494 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:57:10.494 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:57:10.494 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:57:10.494 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000008", 2024-08-18T15:57:10.494 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:57:10.494 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:57:10.494 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:57:10.494 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:57:10.495 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:57:10.495 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000006", 2024-08-18T15:57:10.495 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000029", 2024-08-18T15:57:10.495 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:57:10.495 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000089", 2024-08-18T15:57:10.495 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:57:10.495 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:57:10.495 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000022", 2024-08-18T15:57:10.495 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:57:10.495 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:57:10.495 INFO:teuthology.orchestra.run.smithi193.stdout: "bilog.trim", 2024-08-18T15:57:10.495 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000016", 2024-08-18T15:57:10.495 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:57:10.496 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:57:10.496 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:57:10.496 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:57:10.496 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:57:10.496 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000041", 2024-08-18T15:57:10.496 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:57:10.496 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:57:10.496 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.log.01050e30-179b-43be-bdbb-3bb23cf37ae8.3", 2024-08-18T15:57:10.496 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:57:10.496 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000018", 2024-08-18T15:57:10.496 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000092", 2024-08-18T15:57:10.496 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:57:10.496 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:57:10.496 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:57:10.497 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:57:10.497 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.27", 2024-08-18T15:57:10.497 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.4", 2024-08-18T15:57:10.497 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:57:10.497 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:57:10.497 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:57:10.497 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:57:10.497 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:57:10.497 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:57:10.497 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:57:10.497 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000014", 2024-08-18T15:57:10.497 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:57:10.497 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3.retry", 2024-08-18T15:57:10.498 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:57:10.498 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:57:10.498 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:57:10.498 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:57:10.498 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:57:10.498 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000112", 2024-08-18T15:57:10.498 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:57:10.498 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000007", 2024-08-18T15:57:10.498 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000021", 2024-08-18T15:57:10.498 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:57:10.498 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:57:10.498 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:57:10.499 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:57:10.499 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:57:10.499 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.16", 2024-08-18T15:57:10.499 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:57:10.499 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000064", 2024-08-18T15:57:10.499 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:57:10.499 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:57:10.499 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:57:10.499 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:57:10.499 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:57:10.499 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:57:10.499 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:57:10.499 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000071", 2024-08-18T15:57:10.500 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:57:10.500 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:57:10.500 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:57:10.500 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000074", 2024-08-18T15:57:10.500 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:57:10.500 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:57:10.500 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:57:10.500 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:57:10.500 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:57:10.500 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:57:10.500 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:57:10.500 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:57:10.500 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:57:10.501 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:57:10.501 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:57:10.501 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:57:10.501 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:57:10.501 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:57:10.501 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:57:10.501 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000081", 2024-08-18T15:57:10.501 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:57:10.501 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000009", 2024-08-18T15:57:10.501 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000121", 2024-08-18T15:57:10.501 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1", 2024-08-18T15:57:10.501 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:57:10.501 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:57:10.502 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:57:10.502 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:57:10.502 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:57:10.502 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:57:10.502 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.17", 2024-08-18T15:57:10.502 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000125", 2024-08-18T15:57:10.502 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:57:10.502 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000082", 2024-08-18T15:57:10.502 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:57:10.502 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:57:10.502 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:57:10.502 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:57:10.503 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:57:10.503 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:57:10.503 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:57:10.503 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:57:10.503 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:57:10.503 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:57:10.503 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:57:10.503 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000105", 2024-08-18T15:57:10.503 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000059", 2024-08-18T15:57:10.503 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:57:10.503 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:57:10.503 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:57:10.503 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:57:10.504 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:57:10.504 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:57:10.504 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:57:10.504 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:57:10.504 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:57:10.504 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:57:10.504 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:57:10.504 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000077", 2024-08-18T15:57:10.504 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000032", 2024-08-18T15:57:10.504 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:57:10.504 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:57:10.504 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:57:10.504 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:57:10.505 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:57:10.505 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:57:10.505 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.24", 2024-08-18T15:57:10.505 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:57:10.505 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:57:10.505 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:57:10.505 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:57:10.505 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3.retry", 2024-08-18T15:57:10.505 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:57:10.505 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:57:10.505 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:57:10.505 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:57:10.505 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:57:10.506 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:57:10.506 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000053", 2024-08-18T15:57:10.506 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000091", 2024-08-18T15:57:10.506 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:57:10.506 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:57:10.506 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:57:10.506 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000065", 2024-08-18T15:57:10.506 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:57:10.506 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:57:10.506 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:57:10.506 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:57:10.506 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000083", 2024-08-18T15:57:10.506 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000010", 2024-08-18T15:57:10.507 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:57:10.507 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:57:10.507 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:57:10.507 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:57:10.507 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:57:10.507 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:57:10.507 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:57:10.507 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:57:10.507 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:57:10.507 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:57:10.507 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:57:10.507 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.7", 2024-08-18T15:57:10.508 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.3", 2024-08-18T15:57:10.508 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:57:10.508 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:57:10.508 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:57:10.508 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:57:10.508 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:57:10.508 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:57:10.508 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:57:10.508 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:57:10.508 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.2", 2024-08-18T15:57:10.508 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:57:10.508 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:57:10.508 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:57:10.508 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:57:10.509 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:57:10.509 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:57:10.509 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:57:10.509 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000045", 2024-08-18T15:57:10.509 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000002", 2024-08-18T15:57:10.509 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:57:10.509 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3.2UbiLidwSTy.0", 2024-08-18T15:57:10.509 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:57:10.509 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.12", 2024-08-18T15:57:10.509 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:57:10.509 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:57:10.509 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:57:10.510 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:57:10.510 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:57:10.510 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:57:10.510 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000116", 2024-08-18T15:57:10.510 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.25", 2024-08-18T15:57:10.510 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:57:10.510 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:57:10.510 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:57:10.510 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:57:10.510 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000034", 2024-08-18T15:57:10.510 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:57:10.510 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:57:10.511 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000101", 2024-08-18T15:57:10.511 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:57:10.511 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:57:10.511 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.13", 2024-08-18T15:57:10.511 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:57:10.511 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:57:10.511 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.1", 2024-08-18T15:57:10.511 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0.retry", 2024-08-18T15:57:10.511 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:57:10.511 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:57:10.511 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000079", 2024-08-18T15:57:10.511 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000049", 2024-08-18T15:57:10.511 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:57:10.512 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0", 2024-08-18T15:57:10.512 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:57:10.512 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:57:10.512 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000117", 2024-08-18T15:57:10.512 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:57:10.512 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000044", 2024-08-18T15:57:10.512 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:57:10.512 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:57:10.512 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:57:10.512 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:57:10.512 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:57:10.512 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:57:10.512 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.14", 2024-08-18T15:57:10.512 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:57:10.513 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000066", 2024-08-18T15:57:10.513 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:57:10.513 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:57:10.513 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:57:10.513 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:57:10.513 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:57:10.513 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000068", 2024-08-18T15:57:10.513 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:57:10.513 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000085", 2024-08-18T15:57:10.513 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:57:10.513 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:57:10.513 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:57:10.513 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:57:10.513 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:57:10.514 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:57:10.514 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:57:10.514 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:57:10.514 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:57:10.514 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:57:10.514 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:57:10.514 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:57:10.514 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000073", 2024-08-18T15:57:10.514 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:57:10.514 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:57:10.514 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:57:10.514 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000038", 2024-08-18T15:57:10.515 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:57:10.515 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:57:10.515 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:57:10.515 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:57:10.515 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.10", 2024-08-18T15:57:10.515 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:57:10.515 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:57:10.515 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:57:10.515 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000118", 2024-08-18T15:57:10.515 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:57:10.515 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3", 2024-08-18T15:57:10.515 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:57:10.515 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000036", 2024-08-18T15:57:10.515 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:57:10.516 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:57:10.516 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:57:10.516 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:57:10.516 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:57:10.516 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000103", 2024-08-18T15:57:10.516 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.3", 2024-08-18T15:57:10.516 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:57:10.516 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000119", 2024-08-18T15:57:10.516 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000098", 2024-08-18T15:57:10.516 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:57:10.516 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:57:10.516 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:57:10.517 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:57:10.517 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:57:10.517 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:57:10.517 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:57:10.517 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:57:10.517 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.0", 2024-08-18T15:57:10.517 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:57:10.517 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:57:10.517 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:57:10.517 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:57:10.517 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:57:10.517 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:57:10.517 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:57:10.517 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:57:10.518 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000027", 2024-08-18T15:57:10.518 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:57:10.518 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status", 2024-08-18T15:57:10.518 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:57:10.518 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000019", 2024-08-18T15:57:10.518 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:57:10.518 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:57:10.518 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:57:10.518 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:57:10.518 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1", 2024-08-18T15:57:10.518 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:57:10.518 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:57:10.519 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:57:10.519 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:57:10.519 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000039", 2024-08-18T15:57:10.519 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:57:10.519 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:57:10.519 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:57:10.519 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:57:10.519 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:57:10.519 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:57:10.519 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:57:10.519 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:57:10.519 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:57:10.519 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:57:10.520 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:57:10.520 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000100", 2024-08-18T15:57:10.520 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:57:10.520 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:57:10.520 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:57:10.520 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:57:10.520 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:57:10.520 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000093", 2024-08-18T15:57:10.520 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:57:10.520 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000004", 2024-08-18T15:57:10.520 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000063", 2024-08-18T15:57:10.520 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000122", 2024-08-18T15:57:10.520 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:57:10.521 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:57:10.521 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:57:10.521 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000057", 2024-08-18T15:57:10.521 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:57:10.521 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:57:10.521 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:57:10.521 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:57:10.521 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000054", 2024-08-18T15:57:10.521 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:57:10.521 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000114", 2024-08-18T15:57:10.521 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:57:10.521 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:57:10.521 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:57:10.522 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.2", 2024-08-18T15:57:10.522 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:57:10.522 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:57:10.522 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0", 2024-08-18T15:57:10.522 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:57:10.522 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:57:10.522 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:57:10.522 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:57:10.522 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:57:10.522 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:57:10.522 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000012", 2024-08-18T15:57:10.522 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:57:10.522 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:57:10.523 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:57:10.523 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000084", 2024-08-18T15:57:10.523 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000043", 2024-08-18T15:57:10.523 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3", 2024-08-18T15:57:10.523 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:57:10.523 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:57:10.523 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.11", 2024-08-18T15:57:10.523 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:57:10.523 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:57:10.523 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:57:10.523 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:57:10.523 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:57:10.523 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:57:10.523 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:57:10.524 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:57:10.524 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000111", 2024-08-18T15:57:10.524 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:57:10.524 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000048", 2024-08-18T15:57:10.524 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:57:10.524 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000020", 2024-08-18T15:57:10.524 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:57:10.524 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:57:10.524 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:57:10.524 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:57:10.524 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:57:10.524 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000099", 2024-08-18T15:57:10.525 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000056", 2024-08-18T15:57:10.525 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:57:10.525 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:57:10.525 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.19", 2024-08-18T15:57:10.525 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:57:10.525 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.28", 2024-08-18T15:57:10.525 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:57:10.525 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:57:10.525 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:57:10.525 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:57:10.525 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:57:10.525 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000072", 2024-08-18T15:57:10.525 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:57:10.526 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:57:10.526 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000062", 2024-08-18T15:57:10.526 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:57:10.526 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:57:10.526 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:57:10.526 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000115", 2024-08-18T15:57:10.526 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:57:10.526 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.1", 2024-08-18T15:57:10.526 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2", 2024-08-18T15:57:10.526 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000033", 2024-08-18T15:57:10.529 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:57:10.529 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:57:10.529 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:57:10.529 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:57:10.529 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:57:10.530 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:57:10.530 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:57:10.530 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:57:10.530 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:57:10.530 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:57:10.530 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:57:10.530 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:57:10.530 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:57:10.530 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:57:10.530 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:57:10.530 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:57:10.530 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:57:10.530 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:57:10.531 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:57:10.531 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000025", 2024-08-18T15:57:10.531 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:57:10.531 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000123", 2024-08-18T15:57:10.531 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.31", 2024-08-18T15:57:10.531 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:57:10.531 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:57:10.531 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:57:10.531 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:57:10.531 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.18", 2024-08-18T15:57:10.531 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:57:10.531 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:57:10.531 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:57:10.532 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:57:10.532 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:57:10.532 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000108", 2024-08-18T15:57:10.532 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:57:10.532 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:57:10.532 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000094", 2024-08-18T15:57:10.532 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:57:10.532 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:57:10.532 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:57:10.532 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:57:10.532 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:57:10.532 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:57:10.532 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000060", 2024-08-18T15:57:10.533 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:57:10.533 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:57:10.533 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:57:10.533 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000109", 2024-08-18T15:57:10.533 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.15", 2024-08-18T15:57:10.533 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:57:10.533 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:57:10.533 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:57:10.533 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:57:10.533 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:57:10.533 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:57:10.533 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:57:10.534 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:57:10.534 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:57:10.534 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000013", 2024-08-18T15:57:10.534 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:57:10.534 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:57:10.534 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:57:10.534 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:57:10.534 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:57:10.534 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:57:10.534 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:57:10.534 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:57:10.534 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:57:10.534 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:57:10.535 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:57:10.535 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:57:10.535 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:57:10.535 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:57:10.535 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000051", 2024-08-18T15:57:10.535 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:57:10.535 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:57:10.535 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:57:10.535 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:57:10.535 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000106", 2024-08-18T15:57:10.535 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:57:10.535 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:57:10.535 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:57:10.536 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:57:10.536 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:57:10.536 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000017", 2024-08-18T15:57:10.536 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:57:10.536 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:57:10.536 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:57:10.536 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000023", 2024-08-18T15:57:10.536 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:57:10.536 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:57:10.536 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:57:10.536 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:57:10.536 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:57:10.536 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:57:10.537 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:57:10.537 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000107", 2024-08-18T15:57:10.537 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:57:10.537 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:57:10.537 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000046", 2024-08-18T15:57:10.537 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:57:10.537 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:57:10.537 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:57:10.537 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.26", 2024-08-18T15:57:10.537 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:57:10.537 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:57:10.537 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9", 2024-08-18T15:57:10.538 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.22", 2024-08-18T15:57:10.538 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:57:10.538 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:57:10.538 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:57:10.538 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:57:10.538 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:57:10.538 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:57:10.538 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:57:10.538 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2", 2024-08-18T15:57:10.538 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:57:10.538 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:57:10.538 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000067", 2024-08-18T15:57:10.538 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:57:10.539 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:57:10.539 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:57:10.539 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:57:10.539 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:57:10.539 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:57:10.539 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:57:10.539 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:57:10.539 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:57:10.539 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:57:10.539 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:57:10.539 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:57:10.539 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:57:10.540 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:57:10.540 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:57:10.540 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:57:10.540 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:57:10.540 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:57:10.540 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:57:10.540 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.9", 2024-08-18T15:57:10.540 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000086", 2024-08-18T15:57:10.540 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.5", 2024-08-18T15:57:10.540 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:57:10.540 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:57:10.540 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:57:10.540 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:57:10.541 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:57:10.541 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:57:10.541 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2.MFXJ583gCr1.0", 2024-08-18T15:57:10.541 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:57:10.541 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1.GPnO16liyZ+.0", 2024-08-18T15:57:10.541 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:57:10.541 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:57:10.541 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:57:10.541 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:57:10.541 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:57:10.541 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:57:10.541 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:57:10.541 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:57:10.541 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:57:10.542 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.20", 2024-08-18T15:57:10.542 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:57:10.542 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:57:10.542 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:57:10.542 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:57:10.542 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:57:10.542 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:57:10.542 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1.retry", 2024-08-18T15:57:10.542 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:57:10.542 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:57:10.542 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:57:10.542 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0.retry", 2024-08-18T15:57:10.542 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000050", 2024-08-18T15:57:10.543 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:57:10.543 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000102", 2024-08-18T15:57:10.543 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:57:10.543 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:57:10.543 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:57:10.543 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:57:10.543 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:57:10.543 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:57:10.543 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:57:10.543 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:57:10.543 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:57:10.543 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:57:10.543 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:57:10.544 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000124", 2024-08-18T15:57:10.544 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:57:10.544 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:57:10.544 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:57:10.544 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000001", 2024-08-18T15:57:10.544 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:57:10.544 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:57:10.544 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:57:10.544 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:57:10.544 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000126", 2024-08-18T15:57:10.544 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:57:10.544 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0.fZhctfLXnSt.0", 2024-08-18T15:57:10.544 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:57:10.545 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000055", 2024-08-18T15:57:10.545 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:57:10.545 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000061", 2024-08-18T15:57:10.545 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1", 2024-08-18T15:57:10.545 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:57:10.545 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:57:10.545 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:57:10.545 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000097", 2024-08-18T15:57:10.545 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:57:10.545 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:57:10.545 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:57:10.545 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:57:10.545 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:57:10.546 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:57:10.546 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:57:10.546 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:57:10.546 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000042", 2024-08-18T15:57:10.546 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:57:10.546 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:57:10.546 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:57:10.546 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1.retry", 2024-08-18T15:57:10.546 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:57:10.546 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:57:10.546 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:57:10.546 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:57:10.547 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:57:10.547 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:57:10.547 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:57:10.547 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000075", 2024-08-18T15:57:10.547 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:57:10.547 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000120", 2024-08-18T15:57:10.547 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000030", 2024-08-18T15:57:10.547 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:57:10.547 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:57:10.547 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:57:10.547 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:57:10.547 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:57:10.547 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:57:10.548 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9" 2024-08-18T15:57:10.548 INFO:teuthology.orchestra.run.smithi193.stdout:] 2024-08-18T15:57:10.548 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:57:15.550 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:57:15.550 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:57:15.550 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:57:15.602 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:57:15.603 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:57:15.667 INFO:teuthology.orchestra.run.smithi193.stdout:[ 2024-08-18T15:57:15.667 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:57:15.667 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:57:15.667 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.29", 2024-08-18T15:57:15.667 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000078", 2024-08-18T15:57:15.667 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.history", 2024-08-18T15:57:15.667 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:57:15.667 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:57:15.668 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:57:15.668 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:57:15.668 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.23", 2024-08-18T15:57:15.668 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000070", 2024-08-18T15:57:15.668 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000104", 2024-08-18T15:57:15.668 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:57:15.668 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2.retry", 2024-08-18T15:57:15.668 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:57:15.668 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:57:15.668 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:57:15.668 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:57:15.668 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000026", 2024-08-18T15:57:15.668 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:57:15.669 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:57:15.669 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:57:15.669 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:57:15.669 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:57:15.669 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:57:15.669 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.8", 2024-08-18T15:57:15.669 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:57:15.669 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:57:15.669 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:57:15.669 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:57:15.669 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000028", 2024-08-18T15:57:15.669 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:57:15.669 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:57:15.670 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:57:15.670 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:57:15.670 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:57:15.670 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:57:15.670 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:57:15.670 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2.retry", 2024-08-18T15:57:15.670 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:57:15.670 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.30", 2024-08-18T15:57:15.670 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:57:15.670 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:57:15.670 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:57:15.670 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:57:15.670 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000040", 2024-08-18T15:57:15.671 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:57:15.671 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.6", 2024-08-18T15:57:15.671 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:57:15.671 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:57:15.671 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:57:15.671 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:57:15.671 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:57:15.671 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:57:15.671 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000015", 2024-08-18T15:57:15.671 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:57:15.671 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:57:15.671 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000069", 2024-08-18T15:57:15.671 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:57:15.671 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:57:15.672 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:57:15.672 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000095", 2024-08-18T15:57:15.672 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0", 2024-08-18T15:57:15.672 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000003", 2024-08-18T15:57:15.672 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000047", 2024-08-18T15:57:15.672 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:57:15.672 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:57:15.672 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:57:15.672 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:57:15.672 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:57:15.672 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:57:15.672 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:57:15.672 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:57:15.672 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:57:15.673 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000035", 2024-08-18T15:57:15.673 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:57:15.673 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:57:15.673 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000037", 2024-08-18T15:57:15.673 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:57:15.673 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:57:15.673 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:57:15.673 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:57:15.673 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:57:15.673 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:57:15.673 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:57:15.673 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:57:15.674 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:57:15.674 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:57:15.674 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:57:15.674 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:57:15.674 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:57:15.674 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:57:15.674 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000024", 2024-08-18T15:57:15.674 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:57:15.674 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:57:15.674 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:57:15.674 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:57:15.674 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000000", 2024-08-18T15:57:15.674 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:57:15.675 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000031", 2024-08-18T15:57:15.675 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:57:15.675 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:57:15.675 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:57:15.675 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:57:15.675 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:57:15.675 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:57:15.675 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3", 2024-08-18T15:57:15.675 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:57:15.675 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:57:15.675 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:57:15.675 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:57:15.675 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:57:15.676 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:57:15.676 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:57:15.676 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:57:15.676 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:57:15.676 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:57:15.676 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:57:15.676 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:57:15.676 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:57:15.676 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:57:15.676 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:57:15.676 INFO:teuthology.orchestra.run.smithi193.stdout: "data_loggenerations_metadata", 2024-08-18T15:57:15.676 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2", 2024-08-18T15:57:15.676 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:57:15.677 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:57:15.677 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:57:15.677 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:57:15.677 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T15:57:15.677 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:57:15.677 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:57:15.677 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:57:15.677 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:57:15.677 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:57:15.677 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000076", 2024-08-18T15:57:15.677 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:57:15.677 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:57:15.677 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000113", 2024-08-18T15:57:15.678 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000005", 2024-08-18T15:57:15.678 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:57:15.678 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000011", 2024-08-18T15:57:15.678 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:57:15.678 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000058", 2024-08-18T15:57:15.678 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:57:15.678 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:57:15.678 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:57:15.678 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000052", 2024-08-18T15:57:15.678 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:57:15.678 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:57:15.678 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:57:15.678 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:57:15.679 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.21", 2024-08-18T15:57:15.679 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:57:15.679 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:57:15.679 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:57:15.679 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:57:15.679 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:57:15.679 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:57:15.679 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:57:15.679 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:57:15.679 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:57:15.679 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:57:15.679 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:57:15.679 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:57:15.680 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:57:15.680 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:57:15.680 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:57:15.680 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:57:15.680 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:57:15.680 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:57:15.680 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:57:15.680 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000088", 2024-08-18T15:57:15.680 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000080", 2024-08-18T15:57:15.680 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:57:15.680 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:57:15.680 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000090", 2024-08-18T15:57:15.680 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:57:15.681 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:57:15.681 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000110", 2024-08-18T15:57:15.681 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:57:15.681 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:57:15.681 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:57:15.681 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.0", 2024-08-18T15:57:15.681 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:57:15.681 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000096", 2024-08-18T15:57:15.681 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:57:15.681 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:57:15.681 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:57:15.681 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:57:15.681 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:57:15.682 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000087", 2024-08-18T15:57:15.682 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:57:15.682 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:57:15.682 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:57:15.682 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:57:15.682 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000008", 2024-08-18T15:57:15.682 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:57:15.682 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:57:15.682 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:57:15.682 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:57:15.682 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:57:15.682 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000006", 2024-08-18T15:57:15.683 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000029", 2024-08-18T15:57:15.683 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:57:15.683 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000089", 2024-08-18T15:57:15.683 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:57:15.683 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:57:15.683 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000022", 2024-08-18T15:57:15.683 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:57:15.683 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:57:15.683 INFO:teuthology.orchestra.run.smithi193.stdout: "bilog.trim", 2024-08-18T15:57:15.683 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000016", 2024-08-18T15:57:15.683 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:57:15.683 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:57:15.683 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:57:15.684 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:57:15.684 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:57:15.684 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000041", 2024-08-18T15:57:15.684 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:57:15.684 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:57:15.684 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.log.01050e30-179b-43be-bdbb-3bb23cf37ae8.3", 2024-08-18T15:57:15.684 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:57:15.684 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000018", 2024-08-18T15:57:15.684 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000092", 2024-08-18T15:57:15.684 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:57:15.684 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:57:15.684 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:57:15.684 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:57:15.685 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.27", 2024-08-18T15:57:15.685 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.4", 2024-08-18T15:57:15.685 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:57:15.685 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:57:15.685 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:57:15.685 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:57:15.685 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:57:15.685 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:57:15.685 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:57:15.685 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000014", 2024-08-18T15:57:15.685 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:57:15.685 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3.retry", 2024-08-18T15:57:15.685 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:57:15.685 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:57:15.686 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:57:15.686 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:57:15.686 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:57:15.686 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000112", 2024-08-18T15:57:15.686 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:57:15.686 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000007", 2024-08-18T15:57:15.686 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000021", 2024-08-18T15:57:15.686 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:57:15.686 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:57:15.686 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:57:15.686 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:57:15.687 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:57:15.687 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.16", 2024-08-18T15:57:15.687 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:57:15.687 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000064", 2024-08-18T15:57:15.687 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:57:15.687 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:57:15.687 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:57:15.687 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:57:15.687 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:57:15.687 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:57:15.687 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:57:15.687 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000071", 2024-08-18T15:57:15.687 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:57:15.688 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:57:15.688 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:57:15.688 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000074", 2024-08-18T15:57:15.688 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:57:15.688 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:57:15.688 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:57:15.688 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:57:15.688 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:57:15.688 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:57:15.688 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:57:15.688 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:57:15.688 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:57:15.689 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:57:15.689 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:57:15.689 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:57:15.689 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:57:15.689 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:57:15.689 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:57:15.689 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000081", 2024-08-18T15:57:15.689 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:57:15.689 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000009", 2024-08-18T15:57:15.689 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000121", 2024-08-18T15:57:15.689 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1", 2024-08-18T15:57:15.689 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:57:15.689 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:57:15.690 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:57:15.690 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:57:15.690 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:57:15.690 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:57:15.690 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.17", 2024-08-18T15:57:15.690 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000125", 2024-08-18T15:57:15.690 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:57:15.690 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000082", 2024-08-18T15:57:15.690 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:57:15.690 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:57:15.690 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:57:15.691 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:57:15.691 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:57:15.691 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:57:15.691 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:57:15.691 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:57:15.691 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:57:15.691 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:57:15.691 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:57:15.691 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000105", 2024-08-18T15:57:15.691 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000059", 2024-08-18T15:57:15.691 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:57:15.691 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:57:15.691 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:57:15.692 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:57:15.692 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:57:15.692 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:57:15.692 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:57:15.692 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:57:15.692 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:57:15.692 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:57:15.692 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:57:15.692 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000077", 2024-08-18T15:57:15.692 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000032", 2024-08-18T15:57:15.692 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:57:15.692 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:57:15.692 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:57:15.693 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:57:15.693 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:57:15.693 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:57:15.693 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.24", 2024-08-18T15:57:15.693 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:57:15.693 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:57:15.693 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:57:15.693 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:57:15.693 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3.retry", 2024-08-18T15:57:15.693 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:57:15.693 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:57:15.693 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:57:15.693 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:57:15.694 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:57:15.694 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:57:15.694 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000053", 2024-08-18T15:57:15.694 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000091", 2024-08-18T15:57:15.694 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:57:15.694 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:57:15.694 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:57:15.694 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000065", 2024-08-18T15:57:15.694 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:57:15.694 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:57:15.694 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:57:15.694 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:57:15.695 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000083", 2024-08-18T15:57:15.695 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000010", 2024-08-18T15:57:15.695 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:57:15.695 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:57:15.695 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:57:15.695 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:57:15.695 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:57:15.695 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:57:15.695 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:57:15.695 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:57:15.695 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:57:15.695 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:57:15.695 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:57:15.696 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.7", 2024-08-18T15:57:15.696 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.3", 2024-08-18T15:57:15.696 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:57:15.696 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:57:15.696 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:57:15.696 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:57:15.696 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:57:15.696 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:57:15.696 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:57:15.696 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:57:15.696 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.2", 2024-08-18T15:57:15.696 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:57:15.696 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:57:15.697 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:57:15.697 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:57:15.697 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:57:15.697 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:57:15.697 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:57:15.697 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000045", 2024-08-18T15:57:15.697 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000002", 2024-08-18T15:57:15.697 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:57:15.697 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3.2UbiLidwSTy.0", 2024-08-18T15:57:15.697 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:57:15.697 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.12", 2024-08-18T15:57:15.697 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:57:15.697 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:57:15.698 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:57:15.698 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:57:15.698 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:57:15.698 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:57:15.698 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000116", 2024-08-18T15:57:15.698 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.25", 2024-08-18T15:57:15.698 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:57:15.698 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:57:15.698 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:57:15.698 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:57:15.698 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000034", 2024-08-18T15:57:15.699 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:57:15.699 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:57:15.699 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000101", 2024-08-18T15:57:15.699 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:57:15.699 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:57:15.699 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.13", 2024-08-18T15:57:15.699 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:57:15.699 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:57:15.699 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.1", 2024-08-18T15:57:15.699 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0.retry", 2024-08-18T15:57:15.699 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:57:15.699 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:57:15.699 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000079", 2024-08-18T15:57:15.700 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000049", 2024-08-18T15:57:15.700 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:57:15.700 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0", 2024-08-18T15:57:15.700 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:57:15.700 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:57:15.700 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000117", 2024-08-18T15:57:15.700 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:57:15.700 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000044", 2024-08-18T15:57:15.700 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:57:15.700 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:57:15.700 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:57:15.700 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:57:15.700 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:57:15.701 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:57:15.701 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.14", 2024-08-18T15:57:15.701 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:57:15.701 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000066", 2024-08-18T15:57:15.701 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:57:15.701 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:57:15.701 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:57:15.701 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:57:15.701 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:57:15.701 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000068", 2024-08-18T15:57:15.701 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:57:15.702 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000085", 2024-08-18T15:57:15.702 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:57:15.702 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:57:15.702 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:57:15.702 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:57:15.702 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:57:15.702 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:57:15.702 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:57:15.702 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:57:15.702 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:57:15.702 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:57:15.703 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:57:15.703 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:57:15.703 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000073", 2024-08-18T15:57:15.703 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:57:15.703 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:57:15.703 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:57:15.703 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000038", 2024-08-18T15:57:15.703 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:57:15.703 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:57:15.703 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:57:15.703 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:57:15.703 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.10", 2024-08-18T15:57:15.704 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:57:15.704 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:57:15.704 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:57:15.704 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000118", 2024-08-18T15:57:15.704 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:57:15.704 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3", 2024-08-18T15:57:15.704 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:57:15.704 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000036", 2024-08-18T15:57:15.704 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:57:15.704 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:57:15.705 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:57:15.705 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:57:15.705 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:57:15.705 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000103", 2024-08-18T15:57:15.705 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.3", 2024-08-18T15:57:15.705 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:57:15.705 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000119", 2024-08-18T15:57:15.705 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000098", 2024-08-18T15:57:15.705 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:57:15.705 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:57:15.705 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:57:15.705 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:57:15.706 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:57:15.706 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:57:15.706 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:57:15.706 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:57:15.706 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.0", 2024-08-18T15:57:15.706 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:57:15.706 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:57:15.706 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:57:15.706 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:57:15.706 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:57:15.706 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:57:15.706 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:57:15.707 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:57:15.707 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000027", 2024-08-18T15:57:15.707 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:57:15.707 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status", 2024-08-18T15:57:15.707 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:57:15.707 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000019", 2024-08-18T15:57:15.707 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:57:15.707 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:57:15.707 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:57:15.708 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:57:15.708 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1", 2024-08-18T15:57:15.708 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:57:15.708 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:57:15.708 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:57:15.708 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:57:15.708 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000039", 2024-08-18T15:57:15.708 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:57:15.708 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:57:15.708 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:57:15.708 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:57:15.708 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:57:15.708 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:57:15.709 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:57:15.709 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:57:15.709 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:57:15.709 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:57:15.709 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:57:15.709 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000100", 2024-08-18T15:57:15.709 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:57:15.709 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:57:15.709 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:57:15.709 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:57:15.709 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:57:15.709 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000093", 2024-08-18T15:57:15.709 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:57:15.709 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000004", 2024-08-18T15:57:15.710 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000063", 2024-08-18T15:57:15.710 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000122", 2024-08-18T15:57:15.710 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:57:15.710 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:57:15.710 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:57:15.710 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000057", 2024-08-18T15:57:15.710 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:57:15.710 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:57:15.710 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:57:15.710 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:57:15.710 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000054", 2024-08-18T15:57:15.710 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:57:15.710 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000114", 2024-08-18T15:57:15.711 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:57:15.711 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:57:15.711 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:57:15.711 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.2", 2024-08-18T15:57:15.711 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:57:15.711 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:57:15.711 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0", 2024-08-18T15:57:15.711 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:57:15.711 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:57:15.712 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:57:15.712 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:57:15.712 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:57:15.712 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:57:15.713 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000012", 2024-08-18T15:57:15.714 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:57:15.714 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:57:15.714 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:57:15.714 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000084", 2024-08-18T15:57:15.714 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000043", 2024-08-18T15:57:15.714 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3", 2024-08-18T15:57:15.714 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:57:15.714 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:57:15.714 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.11", 2024-08-18T15:57:15.714 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:57:15.714 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:57:15.715 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:57:15.715 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:57:15.715 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:57:15.715 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:57:15.715 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:57:15.715 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:57:15.715 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000111", 2024-08-18T15:57:15.715 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:57:15.715 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000048", 2024-08-18T15:57:15.715 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:57:15.715 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000020", 2024-08-18T15:57:15.716 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:57:15.716 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:57:15.716 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:57:15.716 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:57:15.716 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:57:15.716 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000099", 2024-08-18T15:57:15.716 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000056", 2024-08-18T15:57:15.716 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:57:15.716 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:57:15.716 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.19", 2024-08-18T15:57:15.716 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:57:15.716 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.28", 2024-08-18T15:57:15.716 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:57:15.716 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:57:15.716 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:57:15.717 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:57:15.717 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:57:15.717 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000072", 2024-08-18T15:57:15.717 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:57:15.717 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:57:15.717 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000062", 2024-08-18T15:57:15.717 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:57:15.717 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:57:15.717 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:57:15.717 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000115", 2024-08-18T15:57:15.717 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:57:15.717 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.1", 2024-08-18T15:57:15.717 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2", 2024-08-18T15:57:15.717 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000033", 2024-08-18T15:57:15.718 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:57:15.718 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:57:15.718 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:57:15.718 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:57:15.718 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:57:15.718 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:57:15.718 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:57:15.718 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:57:15.718 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:57:15.718 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:57:15.718 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:57:15.718 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:57:15.719 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:57:15.719 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:57:15.719 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:57:15.719 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:57:15.719 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:57:15.719 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:57:15.719 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:57:15.719 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000025", 2024-08-18T15:57:15.719 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:57:15.719 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000123", 2024-08-18T15:57:15.719 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.31", 2024-08-18T15:57:15.719 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:57:15.719 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:57:15.720 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:57:15.720 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:57:15.720 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.18", 2024-08-18T15:57:15.720 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:57:15.720 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:57:15.720 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:57:15.720 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:57:15.720 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:57:15.720 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000108", 2024-08-18T15:57:15.720 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:57:15.720 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:57:15.720 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000094", 2024-08-18T15:57:15.720 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:57:15.720 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:57:15.720 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:57:15.721 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:57:15.721 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:57:15.721 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:57:15.721 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000060", 2024-08-18T15:57:15.721 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:57:15.721 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:57:15.721 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:57:15.721 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000109", 2024-08-18T15:57:15.721 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.15", 2024-08-18T15:57:15.721 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:57:15.721 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:57:15.721 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:57:15.721 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:57:15.721 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:57:15.722 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:57:15.722 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:57:15.722 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:57:15.722 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:57:15.722 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000013", 2024-08-18T15:57:15.722 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:57:15.722 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:57:15.722 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:57:15.722 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:57:15.722 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:57:15.722 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:57:15.722 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:57:15.722 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:57:15.723 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:57:15.723 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:57:15.723 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:57:15.723 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:57:15.723 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:57:15.723 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:57:15.723 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000051", 2024-08-18T15:57:15.723 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:57:15.723 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:57:15.723 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:57:15.723 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:57:15.723 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000106", 2024-08-18T15:57:15.724 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:57:15.724 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:57:15.724 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:57:15.724 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:57:15.724 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:57:15.724 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000017", 2024-08-18T15:57:15.724 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:57:15.724 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:57:15.724 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:57:15.724 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000023", 2024-08-18T15:57:15.724 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:57:15.725 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:57:15.725 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:57:15.725 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:57:15.725 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:57:15.725 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:57:15.725 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:57:15.725 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000107", 2024-08-18T15:57:15.725 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:57:15.725 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:57:15.725 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000046", 2024-08-18T15:57:15.725 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:57:15.726 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:57:15.726 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:57:15.726 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.26", 2024-08-18T15:57:15.726 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:57:15.726 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:57:15.726 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9", 2024-08-18T15:57:15.726 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.22", 2024-08-18T15:57:15.726 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:57:15.726 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:57:15.726 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:57:15.726 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:57:15.727 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:57:15.727 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:57:15.727 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:57:15.727 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2", 2024-08-18T15:57:15.727 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:57:15.727 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:57:15.727 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000067", 2024-08-18T15:57:15.727 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:57:15.727 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:57:15.727 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:57:15.727 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:57:15.727 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:57:15.728 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:57:15.728 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:57:15.728 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:57:15.728 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:57:15.728 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:57:15.728 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:57:15.728 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:57:15.728 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:57:15.728 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:57:15.728 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:57:15.728 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:57:15.729 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:57:15.729 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:57:15.729 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:57:15.729 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.9", 2024-08-18T15:57:15.729 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000086", 2024-08-18T15:57:15.729 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.5", 2024-08-18T15:57:15.729 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:57:15.729 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:57:15.729 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:57:15.729 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:57:15.729 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:57:15.729 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:57:15.730 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2.MFXJ583gCr1.0", 2024-08-18T15:57:15.730 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:57:15.730 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1.GPnO16liyZ+.0", 2024-08-18T15:57:15.730 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:57:15.730 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:57:15.730 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:57:15.730 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:57:15.730 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:57:15.730 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:57:15.730 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:57:15.731 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:57:15.731 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:57:15.731 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.20", 2024-08-18T15:57:15.731 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:57:15.731 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:57:15.731 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:57:15.731 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:57:15.731 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:57:15.731 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:57:15.731 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1.retry", 2024-08-18T15:57:15.731 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:57:15.731 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:57:15.731 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:57:15.732 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0.retry", 2024-08-18T15:57:15.732 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000050", 2024-08-18T15:57:15.732 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:57:15.732 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000102", 2024-08-18T15:57:15.732 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:57:15.732 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:57:15.732 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:57:15.732 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:57:15.732 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:57:15.732 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:57:15.732 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:57:15.732 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:57:15.732 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:57:15.733 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:57:15.733 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:57:15.733 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000124", 2024-08-18T15:57:15.733 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:57:15.733 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:57:15.733 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:57:15.733 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000001", 2024-08-18T15:57:15.733 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:57:15.733 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:57:15.733 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:57:15.733 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:57:15.733 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000126", 2024-08-18T15:57:15.733 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:57:15.734 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0.fZhctfLXnSt.0", 2024-08-18T15:57:15.734 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:57:15.734 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000055", 2024-08-18T15:57:15.734 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:57:15.734 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000061", 2024-08-18T15:57:15.734 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1", 2024-08-18T15:57:15.734 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:57:15.734 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:57:15.734 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:57:15.734 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000097", 2024-08-18T15:57:15.734 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:57:15.734 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:57:15.734 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:57:15.735 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:57:15.735 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:57:15.735 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:57:15.735 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:57:15.735 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:57:15.735 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000042", 2024-08-18T15:57:15.735 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:57:15.735 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:57:15.735 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:57:15.735 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1.retry", 2024-08-18T15:57:15.735 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:57:15.735 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:57:15.735 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:57:15.736 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:57:15.736 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:57:15.736 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:57:15.736 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:57:15.736 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000075", 2024-08-18T15:57:15.736 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:57:15.736 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000120", 2024-08-18T15:57:15.736 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000030", 2024-08-18T15:57:15.736 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:57:15.736 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:57:15.736 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:57:15.736 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:57:15.736 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:57:15.737 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:57:15.737 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9" 2024-08-18T15:57:15.737 INFO:teuthology.orchestra.run.smithi193.stdout:] 2024-08-18T15:57:15.737 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:57:20.739 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:57:20.739 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:57:20.740 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:57:20.792 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:57:20.792 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:57:20.856 INFO:teuthology.orchestra.run.smithi193.stdout:[ 2024-08-18T15:57:20.856 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:57:20.856 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:57:20.856 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.29", 2024-08-18T15:57:20.857 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000078", 2024-08-18T15:57:20.857 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.history", 2024-08-18T15:57:20.857 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:57:20.857 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:57:20.857 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:57:20.857 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:57:20.857 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.23", 2024-08-18T15:57:20.857 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000070", 2024-08-18T15:57:20.857 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000104", 2024-08-18T15:57:20.857 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:57:20.857 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2.retry", 2024-08-18T15:57:20.857 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:57:20.857 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:57:20.858 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:57:20.858 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:57:20.858 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000026", 2024-08-18T15:57:20.858 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:57:20.858 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:57:20.858 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:57:20.858 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:57:20.858 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:57:20.858 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:57:20.858 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.8", 2024-08-18T15:57:20.858 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:57:20.858 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:57:20.859 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:57:20.859 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:57:20.859 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000028", 2024-08-18T15:57:20.859 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:57:20.859 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:57:20.859 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:57:20.859 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:57:20.859 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:57:20.859 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:57:20.859 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:57:20.859 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2.retry", 2024-08-18T15:57:20.859 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:57:20.859 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.30", 2024-08-18T15:57:20.860 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:57:20.860 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:57:20.860 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:57:20.860 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:57:20.860 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000040", 2024-08-18T15:57:20.860 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:57:20.860 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.6", 2024-08-18T15:57:20.860 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:57:20.860 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:57:20.860 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:57:20.860 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:57:20.861 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:57:20.861 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:57:20.861 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000015", 2024-08-18T15:57:20.861 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:57:20.861 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:57:20.861 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000069", 2024-08-18T15:57:20.861 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:57:20.861 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:57:20.861 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:57:20.861 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000095", 2024-08-18T15:57:20.861 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0", 2024-08-18T15:57:20.861 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000003", 2024-08-18T15:57:20.861 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000047", 2024-08-18T15:57:20.862 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:57:20.862 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:57:20.862 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:57:20.862 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:57:20.862 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:57:20.862 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:57:20.862 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:57:20.862 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:57:20.862 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:57:20.862 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000035", 2024-08-18T15:57:20.862 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:57:20.862 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:57:20.863 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000037", 2024-08-18T15:57:20.863 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:57:20.863 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:57:20.863 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:57:20.863 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:57:20.863 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:57:20.863 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:57:20.863 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:57:20.863 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:57:20.863 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:57:20.863 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:57:20.863 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:57:20.864 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:57:20.864 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:57:20.864 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:57:20.864 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000024", 2024-08-18T15:57:20.864 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:57:20.864 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:57:20.864 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:57:20.864 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:57:20.864 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000000", 2024-08-18T15:57:20.864 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:57:20.864 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000031", 2024-08-18T15:57:20.865 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:57:20.865 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:57:20.865 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:57:20.865 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:57:20.865 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:57:20.865 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:57:20.865 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3", 2024-08-18T15:57:20.865 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:57:20.865 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:57:20.865 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:57:20.865 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:57:20.865 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:57:20.865 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:57:20.866 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:57:20.866 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:57:20.866 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:57:20.866 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:57:20.866 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:57:20.866 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:57:20.866 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:57:20.866 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:57:20.866 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:57:20.866 INFO:teuthology.orchestra.run.smithi193.stdout: "data_loggenerations_metadata", 2024-08-18T15:57:20.866 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2", 2024-08-18T15:57:20.867 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:57:20.867 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:57:20.867 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:57:20.867 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:57:20.867 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T15:57:20.867 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:57:20.867 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:57:20.867 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:57:20.867 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:57:20.867 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:57:20.867 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000076", 2024-08-18T15:57:20.867 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:57:20.867 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:57:20.868 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000113", 2024-08-18T15:57:20.868 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000005", 2024-08-18T15:57:20.868 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:57:20.868 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000011", 2024-08-18T15:57:20.868 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:57:20.868 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000058", 2024-08-18T15:57:20.868 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:57:20.868 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:57:20.868 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:57:20.868 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000052", 2024-08-18T15:57:20.869 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:57:20.869 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:57:20.869 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:57:20.869 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:57:20.869 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.21", 2024-08-18T15:57:20.869 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:57:20.869 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:57:20.869 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:57:20.869 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:57:20.869 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:57:20.869 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:57:20.869 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:57:20.869 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:57:20.870 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:57:20.870 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:57:20.870 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:57:20.870 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:57:20.870 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:57:20.870 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:57:20.870 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:57:20.871 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:57:20.871 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:57:20.871 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:57:20.871 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:57:20.871 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000088", 2024-08-18T15:57:20.871 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000080", 2024-08-18T15:57:20.871 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:57:20.871 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:57:20.871 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000090", 2024-08-18T15:57:20.871 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:57:20.871 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:57:20.872 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000110", 2024-08-18T15:57:20.872 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:57:20.872 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:57:20.872 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:57:20.872 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.0", 2024-08-18T15:57:20.872 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:57:20.872 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000096", 2024-08-18T15:57:20.873 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:57:20.873 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:57:20.873 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:57:20.873 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:57:20.873 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:57:20.873 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000087", 2024-08-18T15:57:20.873 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:57:20.873 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:57:20.873 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:57:20.873 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:57:20.873 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000008", 2024-08-18T15:57:20.873 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:57:20.874 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:57:20.874 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:57:20.874 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:57:20.874 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:57:20.874 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000006", 2024-08-18T15:57:20.874 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000029", 2024-08-18T15:57:20.874 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:57:20.874 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000089", 2024-08-18T15:57:20.874 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:57:20.874 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:57:20.874 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000022", 2024-08-18T15:57:20.874 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:57:20.874 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:57:20.875 INFO:teuthology.orchestra.run.smithi193.stdout: "bilog.trim", 2024-08-18T15:57:20.875 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000016", 2024-08-18T15:57:20.875 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:57:20.875 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:57:20.875 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:57:20.875 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:57:20.875 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:57:20.875 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000041", 2024-08-18T15:57:20.875 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:57:20.876 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:57:20.876 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.log.01050e30-179b-43be-bdbb-3bb23cf37ae8.3", 2024-08-18T15:57:20.876 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:57:20.876 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000018", 2024-08-18T15:57:20.876 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000092", 2024-08-18T15:57:20.876 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:57:20.876 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:57:20.876 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:57:20.876 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:57:20.876 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.27", 2024-08-18T15:57:20.876 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.4", 2024-08-18T15:57:20.876 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:57:20.877 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:57:20.877 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:57:20.877 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:57:20.877 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:57:20.877 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:57:20.877 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:57:20.878 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000014", 2024-08-18T15:57:20.878 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:57:20.878 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3.retry", 2024-08-18T15:57:20.878 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:57:20.878 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:57:20.879 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:57:20.879 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:57:20.879 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:57:20.879 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000112", 2024-08-18T15:57:20.879 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:57:20.879 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000007", 2024-08-18T15:57:20.879 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000021", 2024-08-18T15:57:20.880 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:57:20.880 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:57:20.880 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:57:20.880 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:57:20.880 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:57:20.880 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.16", 2024-08-18T15:57:20.880 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:57:20.880 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000064", 2024-08-18T15:57:20.880 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:57:20.880 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:57:20.880 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:57:20.880 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:57:20.880 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:57:20.881 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:57:20.881 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:57:20.881 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000071", 2024-08-18T15:57:20.881 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:57:20.881 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:57:20.881 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:57:20.881 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000074", 2024-08-18T15:57:20.881 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:57:20.881 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:57:20.881 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:57:20.881 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:57:20.881 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:57:20.881 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:57:20.882 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:57:20.882 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:57:20.882 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:57:20.882 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:57:20.882 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:57:20.882 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:57:20.883 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:57:20.883 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:57:20.883 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:57:20.883 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000081", 2024-08-18T15:57:20.883 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:57:20.883 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000009", 2024-08-18T15:57:20.884 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000121", 2024-08-18T15:57:20.884 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1", 2024-08-18T15:57:20.884 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:57:20.884 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:57:20.884 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:57:20.884 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:57:20.884 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:57:20.885 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:57:20.885 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.17", 2024-08-18T15:57:20.885 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000125", 2024-08-18T15:57:20.885 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:57:20.885 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000082", 2024-08-18T15:57:20.885 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:57:20.885 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:57:20.885 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:57:20.885 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:57:20.885 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:57:20.885 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:57:20.885 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:57:20.885 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:57:20.886 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:57:20.886 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:57:20.886 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:57:20.886 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000105", 2024-08-18T15:57:20.886 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000059", 2024-08-18T15:57:20.886 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:57:20.886 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:57:20.886 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:57:20.887 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:57:20.887 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:57:20.887 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:57:20.887 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:57:20.887 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:57:20.888 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:57:20.888 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:57:20.888 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:57:20.888 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000077", 2024-08-18T15:57:20.888 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000032", 2024-08-18T15:57:20.888 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:57:20.888 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:57:20.889 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:57:20.889 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:57:20.889 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:57:20.889 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:57:20.889 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.24", 2024-08-18T15:57:20.889 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:57:20.889 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:57:20.889 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:57:20.890 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:57:20.890 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3.retry", 2024-08-18T15:57:20.890 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:57:20.890 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:57:20.890 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:57:20.890 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:57:20.890 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:57:20.890 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:57:20.891 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000053", 2024-08-18T15:57:20.891 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000091", 2024-08-18T15:57:20.891 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:57:20.891 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:57:20.891 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:57:20.891 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000065", 2024-08-18T15:57:20.891 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:57:20.891 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:57:20.892 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:57:20.892 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:57:20.892 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000083", 2024-08-18T15:57:20.892 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000010", 2024-08-18T15:57:20.892 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:57:20.892 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:57:20.892 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:57:20.892 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:57:20.893 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:57:20.893 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:57:20.893 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:57:20.893 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:57:20.893 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:57:20.893 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:57:20.893 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:57:20.893 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.7", 2024-08-18T15:57:20.893 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.3", 2024-08-18T15:57:20.894 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:57:20.894 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:57:20.894 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:57:20.894 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:57:20.894 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:57:20.894 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:57:20.894 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:57:20.894 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:57:20.895 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.2", 2024-08-18T15:57:20.895 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:57:20.895 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:57:20.895 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:57:20.895 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:57:20.895 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:57:20.895 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:57:20.896 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:57:20.896 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000045", 2024-08-18T15:57:20.896 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000002", 2024-08-18T15:57:20.896 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:57:20.896 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3.2UbiLidwSTy.0", 2024-08-18T15:57:20.896 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:57:20.896 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.12", 2024-08-18T15:57:20.896 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:57:20.897 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:57:20.897 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:57:20.897 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:57:20.897 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:57:20.897 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:57:20.897 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000116", 2024-08-18T15:57:20.897 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.25", 2024-08-18T15:57:20.897 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:57:20.898 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:57:20.898 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:57:20.898 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:57:20.898 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000034", 2024-08-18T15:57:20.898 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:57:20.898 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:57:20.899 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000101", 2024-08-18T15:57:20.899 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:57:20.899 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:57:20.899 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.13", 2024-08-18T15:57:20.899 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:57:20.899 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:57:20.899 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.1", 2024-08-18T15:57:20.899 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0.retry", 2024-08-18T15:57:20.900 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:57:20.900 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:57:20.900 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000079", 2024-08-18T15:57:20.900 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000049", 2024-08-18T15:57:20.900 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:57:20.900 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0", 2024-08-18T15:57:20.900 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:57:20.900 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:57:20.901 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000117", 2024-08-18T15:57:20.901 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:57:20.901 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000044", 2024-08-18T15:57:20.901 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:57:20.901 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:57:20.901 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:57:20.901 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:57:20.902 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:57:20.902 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:57:20.902 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.14", 2024-08-18T15:57:20.902 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:57:20.902 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000066", 2024-08-18T15:57:20.902 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:57:20.902 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:57:20.902 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:57:20.902 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:57:20.902 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:57:20.902 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000068", 2024-08-18T15:57:20.903 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:57:20.903 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000085", 2024-08-18T15:57:20.903 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:57:20.903 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:57:20.903 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:57:20.903 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:57:20.903 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:57:20.903 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:57:20.903 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:57:20.903 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:57:20.903 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:57:20.903 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:57:20.903 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:57:20.904 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:57:20.904 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000073", 2024-08-18T15:57:20.904 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:57:20.904 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:57:20.904 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:57:20.904 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000038", 2024-08-18T15:57:20.904 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:57:20.904 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:57:20.904 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:57:20.904 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:57:20.904 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.10", 2024-08-18T15:57:20.904 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:57:20.904 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:57:20.905 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:57:20.905 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000118", 2024-08-18T15:57:20.905 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:57:20.905 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3", 2024-08-18T15:57:20.905 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:57:20.905 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000036", 2024-08-18T15:57:20.905 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:57:20.905 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:57:20.905 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:57:20.905 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:57:20.905 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:57:20.905 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000103", 2024-08-18T15:57:20.905 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.3", 2024-08-18T15:57:20.905 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:57:20.906 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000119", 2024-08-18T15:57:20.906 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000098", 2024-08-18T15:57:20.906 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:57:20.906 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:57:20.906 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:57:20.906 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:57:20.906 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:57:20.906 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:57:20.906 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:57:20.906 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:57:20.906 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.0", 2024-08-18T15:57:20.907 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:57:20.907 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:57:20.907 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:57:20.907 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:57:20.907 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:57:20.907 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:57:20.907 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:57:20.907 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:57:20.907 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000027", 2024-08-18T15:57:20.908 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:57:20.908 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status", 2024-08-18T15:57:20.908 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:57:20.908 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000019", 2024-08-18T15:57:20.908 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:57:20.908 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:57:20.909 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:57:20.909 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:57:20.909 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1", 2024-08-18T15:57:20.909 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:57:20.909 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:57:20.909 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:57:20.909 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:57:20.909 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000039", 2024-08-18T15:57:20.909 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:57:20.909 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:57:20.909 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:57:20.909 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:57:20.909 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:57:20.910 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:57:20.910 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:57:20.910 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:57:20.910 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:57:20.910 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:57:20.910 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:57:20.910 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000100", 2024-08-18T15:57:20.910 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:57:20.910 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:57:20.910 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:57:20.910 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:57:20.911 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:57:20.911 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000093", 2024-08-18T15:57:20.911 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:57:20.911 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000004", 2024-08-18T15:57:20.911 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000063", 2024-08-18T15:57:20.911 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000122", 2024-08-18T15:57:20.911 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:57:20.911 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:57:20.911 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:57:20.911 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000057", 2024-08-18T15:57:20.911 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:57:20.911 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:57:20.911 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:57:20.912 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:57:20.912 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000054", 2024-08-18T15:57:20.912 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:57:20.912 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000114", 2024-08-18T15:57:20.912 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:57:20.912 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:57:20.912 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:57:20.912 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.2", 2024-08-18T15:57:20.912 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:57:20.912 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:57:20.913 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0", 2024-08-18T15:57:20.913 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:57:20.913 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:57:20.913 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:57:20.913 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:57:20.913 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:57:20.913 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:57:20.914 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000012", 2024-08-18T15:57:20.914 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:57:20.914 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:57:20.914 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:57:20.914 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000084", 2024-08-18T15:57:20.914 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000043", 2024-08-18T15:57:20.914 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3", 2024-08-18T15:57:20.914 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:57:20.914 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:57:20.914 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.11", 2024-08-18T15:57:20.914 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:57:20.914 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:57:20.915 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:57:20.915 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:57:20.915 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:57:20.915 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:57:20.915 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:57:20.915 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:57:20.915 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000111", 2024-08-18T15:57:20.915 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:57:20.915 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000048", 2024-08-18T15:57:20.915 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:57:20.915 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000020", 2024-08-18T15:57:20.915 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:57:20.915 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:57:20.916 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:57:20.916 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:57:20.916 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:57:20.916 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000099", 2024-08-18T15:57:20.916 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000056", 2024-08-18T15:57:20.916 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:57:20.916 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:57:20.916 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.19", 2024-08-18T15:57:20.916 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:57:20.916 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.28", 2024-08-18T15:57:20.916 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:57:20.916 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:57:20.916 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:57:20.917 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:57:20.917 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:57:20.917 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000072", 2024-08-18T15:57:20.917 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:57:20.917 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:57:20.918 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000062", 2024-08-18T15:57:20.918 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:57:20.918 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:57:20.918 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:57:20.918 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000115", 2024-08-18T15:57:20.918 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:57:20.919 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.1", 2024-08-18T15:57:20.919 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2", 2024-08-18T15:57:20.919 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000033", 2024-08-18T15:57:20.919 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:57:20.919 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:57:20.919 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:57:20.920 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:57:20.920 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:57:20.920 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:57:20.920 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:57:20.920 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:57:20.920 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:57:20.920 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:57:20.920 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:57:20.920 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:57:20.921 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:57:20.921 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:57:20.921 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:57:20.921 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:57:20.921 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:57:20.921 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:57:20.921 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:57:20.921 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000025", 2024-08-18T15:57:20.921 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:57:20.921 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000123", 2024-08-18T15:57:20.921 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.31", 2024-08-18T15:57:20.921 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:57:20.921 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:57:20.921 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:57:20.922 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:57:20.922 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.18", 2024-08-18T15:57:20.922 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:57:20.922 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:57:20.922 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:57:20.922 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:57:20.922 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:57:20.922 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000108", 2024-08-18T15:57:20.922 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:57:20.923 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:57:20.923 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000094", 2024-08-18T15:57:20.923 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:57:20.923 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:57:20.923 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:57:20.923 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:57:20.923 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:57:20.923 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:57:20.923 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000060", 2024-08-18T15:57:20.924 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:57:20.924 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:57:20.924 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:57:20.924 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000109", 2024-08-18T15:57:20.924 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.15", 2024-08-18T15:57:20.925 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:57:20.925 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:57:20.925 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:57:20.925 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:57:20.925 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:57:20.925 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:57:20.925 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:57:20.925 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:57:20.925 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:57:20.925 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000013", 2024-08-18T15:57:20.925 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:57:20.926 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:57:20.926 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:57:20.926 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:57:20.926 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:57:20.926 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:57:20.926 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:57:20.926 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:57:20.926 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:57:20.926 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:57:20.927 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:57:20.927 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:57:20.927 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:57:20.927 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:57:20.927 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000051", 2024-08-18T15:57:20.927 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:57:20.927 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:57:20.928 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:57:20.928 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:57:20.928 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000106", 2024-08-18T15:57:20.928 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:57:20.928 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:57:20.928 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:57:20.928 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:57:20.928 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:57:20.928 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000017", 2024-08-18T15:57:20.928 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:57:20.928 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:57:20.928 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:57:20.929 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000023", 2024-08-18T15:57:20.929 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:57:20.929 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:57:20.929 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:57:20.929 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:57:20.929 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:57:20.929 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:57:20.929 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:57:20.930 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000107", 2024-08-18T15:57:20.930 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:57:20.930 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:57:20.930 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000046", 2024-08-18T15:57:20.930 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:57:20.930 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:57:20.930 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:57:20.930 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.26", 2024-08-18T15:57:20.930 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:57:20.930 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:57:20.930 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9", 2024-08-18T15:57:20.930 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.22", 2024-08-18T15:57:20.931 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:57:20.931 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:57:20.931 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:57:20.931 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:57:20.931 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:57:20.931 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:57:20.931 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:57:20.931 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2", 2024-08-18T15:57:20.931 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:57:20.931 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:57:20.931 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000067", 2024-08-18T15:57:20.931 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:57:20.931 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:57:20.932 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:57:20.932 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:57:20.932 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:57:20.932 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:57:20.932 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:57:20.932 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:57:20.932 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:57:20.933 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:57:20.933 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:57:20.933 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:57:20.933 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:57:20.933 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:57:20.933 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:57:20.933 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:57:20.933 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:57:20.933 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:57:20.933 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:57:20.933 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.9", 2024-08-18T15:57:20.933 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000086", 2024-08-18T15:57:20.934 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.5", 2024-08-18T15:57:20.934 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:57:20.934 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:57:20.934 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:57:20.934 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:57:20.934 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:57:20.934 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:57:20.934 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2.MFXJ583gCr1.0", 2024-08-18T15:57:20.934 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:57:20.934 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1.GPnO16liyZ+.0", 2024-08-18T15:57:20.934 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:57:20.934 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:57:20.934 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:57:20.935 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:57:20.935 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:57:20.935 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:57:20.935 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:57:20.935 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:57:20.935 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:57:20.935 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.20", 2024-08-18T15:57:20.935 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:57:20.935 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:57:20.935 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:57:20.935 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:57:20.935 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:57:20.935 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:57:20.936 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1.retry", 2024-08-18T15:57:20.936 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:57:20.936 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:57:20.936 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:57:20.936 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0.retry", 2024-08-18T15:57:20.936 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000050", 2024-08-18T15:57:20.936 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:57:20.936 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000102", 2024-08-18T15:57:20.936 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:57:20.936 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:57:20.936 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:57:20.936 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:57:20.936 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:57:20.937 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:57:20.937 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:57:20.937 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:57:20.937 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:57:20.937 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:57:20.937 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:57:20.937 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000124", 2024-08-18T15:57:20.937 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:57:20.937 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:57:20.937 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:57:20.937 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000001", 2024-08-18T15:57:20.937 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:57:20.937 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:57:20.937 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:57:20.938 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:57:20.938 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000126", 2024-08-18T15:57:20.938 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:57:20.938 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0.fZhctfLXnSt.0", 2024-08-18T15:57:20.938 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:57:20.938 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000055", 2024-08-18T15:57:20.938 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:57:20.938 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000061", 2024-08-18T15:57:20.938 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1", 2024-08-18T15:57:20.938 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:57:20.938 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:57:20.938 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:57:20.939 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000097", 2024-08-18T15:57:20.939 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:57:20.939 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:57:20.939 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:57:20.939 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:57:20.939 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:57:20.939 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:57:20.939 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:57:20.939 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:57:20.939 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000042", 2024-08-18T15:57:20.939 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:57:20.939 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:57:20.939 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:57:20.940 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1.retry", 2024-08-18T15:57:20.940 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:57:20.940 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:57:20.940 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:57:20.940 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:57:20.940 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:57:20.940 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:57:20.940 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:57:20.940 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000075", 2024-08-18T15:57:20.940 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:57:20.940 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000120", 2024-08-18T15:57:20.940 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000030", 2024-08-18T15:57:20.940 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:57:20.941 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:57:20.941 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:57:20.941 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:57:20.941 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:57:20.941 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:57:20.941 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9" 2024-08-18T15:57:20.941 INFO:teuthology.orchestra.run.smithi193.stdout:] 2024-08-18T15:57:20.941 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:57:25.944 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:57:25.944 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:57:25.944 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:57:25.999 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:57:25.999 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:57:26.063 INFO:teuthology.orchestra.run.smithi193.stdout:[ 2024-08-18T15:57:26.063 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:57:26.063 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:57:26.063 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.29", 2024-08-18T15:57:26.063 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000078", 2024-08-18T15:57:26.064 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.history", 2024-08-18T15:57:26.064 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:57:26.064 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:57:26.064 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:57:26.064 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:57:26.064 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.23", 2024-08-18T15:57:26.064 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000070", 2024-08-18T15:57:26.064 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000104", 2024-08-18T15:57:26.064 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:57:26.064 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2.retry", 2024-08-18T15:57:26.064 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:57:26.064 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:57:26.064 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:57:26.065 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:57:26.065 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000026", 2024-08-18T15:57:26.065 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:57:26.065 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:57:26.065 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:57:26.065 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:57:26.065 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:57:26.065 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:57:26.065 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.8", 2024-08-18T15:57:26.065 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:57:26.065 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:57:26.065 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:57:26.065 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:57:26.066 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000028", 2024-08-18T15:57:26.066 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:57:26.066 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:57:26.066 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:57:26.066 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:57:26.066 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:57:26.066 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:57:26.066 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:57:26.066 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2.retry", 2024-08-18T15:57:26.066 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:57:26.066 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.30", 2024-08-18T15:57:26.066 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:57:26.066 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:57:26.067 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:57:26.067 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:57:26.067 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000040", 2024-08-18T15:57:26.067 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:57:26.067 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.6", 2024-08-18T15:57:26.067 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:57:26.067 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:57:26.067 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:57:26.067 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:57:26.067 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:57:26.067 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:57:26.067 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000015", 2024-08-18T15:57:26.067 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:57:26.068 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:57:26.068 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000069", 2024-08-18T15:57:26.068 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:57:26.068 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:57:26.068 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:57:26.068 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000095", 2024-08-18T15:57:26.068 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0", 2024-08-18T15:57:26.068 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000003", 2024-08-18T15:57:26.068 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000047", 2024-08-18T15:57:26.068 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:57:26.068 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:57:26.068 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:57:26.068 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:57:26.068 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:57:26.069 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:57:26.069 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:57:26.069 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:57:26.069 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:57:26.069 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000035", 2024-08-18T15:57:26.069 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:57:26.069 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:57:26.069 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000037", 2024-08-18T15:57:26.069 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:57:26.069 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:57:26.069 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:57:26.069 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:57:26.069 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:57:26.070 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:57:26.070 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:57:26.070 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:57:26.070 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:57:26.070 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:57:26.070 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:57:26.070 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:57:26.070 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:57:26.070 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:57:26.070 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000024", 2024-08-18T15:57:26.070 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:57:26.070 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:57:26.070 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:57:26.071 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:57:26.071 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000000", 2024-08-18T15:57:26.071 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:57:26.071 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000031", 2024-08-18T15:57:26.071 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:57:26.071 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:57:26.071 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:57:26.071 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:57:26.071 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:57:26.071 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:57:26.071 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3", 2024-08-18T15:57:26.071 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:57:26.071 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:57:26.072 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:57:26.072 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:57:26.072 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:57:26.072 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:57:26.072 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:57:26.072 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:57:26.072 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:57:26.072 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:57:26.072 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:57:26.072 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:57:26.072 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:57:26.072 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:57:26.072 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:57:26.073 INFO:teuthology.orchestra.run.smithi193.stdout: "data_loggenerations_metadata", 2024-08-18T15:57:26.073 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2", 2024-08-18T15:57:26.073 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:57:26.073 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:57:26.073 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:57:26.073 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:57:26.073 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T15:57:26.073 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:57:26.073 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:57:26.073 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:57:26.073 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:57:26.073 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:57:26.073 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000076", 2024-08-18T15:57:26.073 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:57:26.074 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:57:26.074 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000113", 2024-08-18T15:57:26.074 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000005", 2024-08-18T15:57:26.074 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:57:26.074 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000011", 2024-08-18T15:57:26.074 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:57:26.074 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000058", 2024-08-18T15:57:26.074 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:57:26.074 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:57:26.074 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:57:26.074 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000052", 2024-08-18T15:57:26.074 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:57:26.075 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:57:26.075 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:57:26.075 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:57:26.075 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.21", 2024-08-18T15:57:26.075 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:57:26.075 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:57:26.075 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:57:26.075 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:57:26.075 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:57:26.075 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:57:26.075 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:57:26.075 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:57:26.075 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:57:26.075 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:57:26.076 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:57:26.076 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:57:26.076 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:57:26.076 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:57:26.076 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:57:26.076 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:57:26.076 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:57:26.076 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:57:26.076 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:57:26.076 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000088", 2024-08-18T15:57:26.076 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000080", 2024-08-18T15:57:26.076 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:57:26.076 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:57:26.077 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000090", 2024-08-18T15:57:26.077 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:57:26.077 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:57:26.077 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000110", 2024-08-18T15:57:26.077 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:57:26.077 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:57:26.077 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:57:26.077 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.0", 2024-08-18T15:57:26.077 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:57:26.077 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000096", 2024-08-18T15:57:26.077 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:57:26.077 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:57:26.077 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:57:26.078 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:57:26.078 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:57:26.078 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000087", 2024-08-18T15:57:26.078 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:57:26.078 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:57:26.078 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:57:26.078 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:57:26.078 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000008", 2024-08-18T15:57:26.078 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:57:26.078 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:57:26.078 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:57:26.078 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:57:26.078 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:57:26.078 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000006", 2024-08-18T15:57:26.079 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000029", 2024-08-18T15:57:26.079 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:57:26.079 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000089", 2024-08-18T15:57:26.079 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:57:26.079 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:57:26.079 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000022", 2024-08-18T15:57:26.079 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:57:26.079 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:57:26.079 INFO:teuthology.orchestra.run.smithi193.stdout: "bilog.trim", 2024-08-18T15:57:26.079 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000016", 2024-08-18T15:57:26.079 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:57:26.079 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:57:26.079 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:57:26.080 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:57:26.080 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:57:26.080 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000041", 2024-08-18T15:57:26.080 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:57:26.080 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:57:26.080 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.log.01050e30-179b-43be-bdbb-3bb23cf37ae8.3", 2024-08-18T15:57:26.080 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:57:26.080 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000018", 2024-08-18T15:57:26.080 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000092", 2024-08-18T15:57:26.080 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:57:26.080 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:57:26.080 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:57:26.080 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:57:26.080 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.27", 2024-08-18T15:57:26.081 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.4", 2024-08-18T15:57:26.081 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:57:26.081 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:57:26.081 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:57:26.081 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:57:26.081 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:57:26.081 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:57:26.081 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:57:26.081 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000014", 2024-08-18T15:57:26.081 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:57:26.081 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3.retry", 2024-08-18T15:57:26.081 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:57:26.081 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:57:26.081 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:57:26.082 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:57:26.082 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:57:26.082 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000112", 2024-08-18T15:57:26.082 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:57:26.082 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000007", 2024-08-18T15:57:26.082 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000021", 2024-08-18T15:57:26.082 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:57:26.082 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:57:26.082 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:57:26.082 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:57:26.082 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:57:26.082 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.16", 2024-08-18T15:57:26.083 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:57:26.083 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000064", 2024-08-18T15:57:26.083 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:57:26.083 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:57:26.083 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:57:26.083 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:57:26.083 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:57:26.083 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:57:26.083 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:57:26.083 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000071", 2024-08-18T15:57:26.083 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:57:26.083 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:57:26.083 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:57:26.084 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000074", 2024-08-18T15:57:26.084 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:57:26.084 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:57:26.084 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:57:26.084 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:57:26.084 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:57:26.084 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:57:26.084 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:57:26.084 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:57:26.084 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:57:26.084 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:57:26.084 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:57:26.084 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:57:26.085 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:57:26.085 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:57:26.085 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:57:26.085 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000081", 2024-08-18T15:57:26.085 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:57:26.085 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000009", 2024-08-18T15:57:26.085 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000121", 2024-08-18T15:57:26.085 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1", 2024-08-18T15:57:26.085 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:57:26.085 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:57:26.085 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:57:26.085 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:57:26.085 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:57:26.086 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:57:26.086 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.17", 2024-08-18T15:57:26.086 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000125", 2024-08-18T15:57:26.086 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:57:26.086 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000082", 2024-08-18T15:57:26.086 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:57:26.086 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:57:26.086 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:57:26.086 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:57:26.086 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:57:26.086 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:57:26.086 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:57:26.086 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:57:26.087 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:57:26.087 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:57:26.087 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:57:26.087 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000105", 2024-08-18T15:57:26.087 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000059", 2024-08-18T15:57:26.087 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:57:26.087 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:57:26.087 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:57:26.087 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:57:26.087 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:57:26.087 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:57:26.087 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:57:26.087 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:57:26.087 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:57:26.088 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:57:26.088 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:57:26.088 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000077", 2024-08-18T15:57:26.088 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000032", 2024-08-18T15:57:26.088 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:57:26.088 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:57:26.088 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:57:26.088 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:57:26.088 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:57:26.088 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:57:26.088 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.24", 2024-08-18T15:57:26.088 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:57:26.088 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:57:26.089 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:57:26.089 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:57:26.089 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3.retry", 2024-08-18T15:57:26.089 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:57:26.089 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:57:26.089 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:57:26.089 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:57:26.089 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:57:26.089 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:57:26.089 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000053", 2024-08-18T15:57:26.089 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000091", 2024-08-18T15:57:26.089 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:57:26.089 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:57:26.090 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:57:26.090 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000065", 2024-08-18T15:57:26.090 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:57:26.090 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:57:26.090 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:57:26.090 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:57:26.090 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000083", 2024-08-18T15:57:26.090 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000010", 2024-08-18T15:57:26.091 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:57:26.091 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:57:26.091 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:57:26.091 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:57:26.091 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:57:26.091 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:57:26.091 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:57:26.091 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:57:26.091 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:57:26.091 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:57:26.091 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:57:26.091 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.7", 2024-08-18T15:57:26.091 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.3", 2024-08-18T15:57:26.091 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:57:26.092 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:57:26.092 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:57:26.092 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:57:26.092 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:57:26.092 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:57:26.092 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:57:26.092 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:57:26.092 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.2", 2024-08-18T15:57:26.092 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:57:26.092 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:57:26.092 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:57:26.092 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:57:26.092 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:57:26.093 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:57:26.093 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:57:26.093 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000045", 2024-08-18T15:57:26.093 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000002", 2024-08-18T15:57:26.093 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:57:26.093 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3.2UbiLidwSTy.0", 2024-08-18T15:57:26.093 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:57:26.093 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.12", 2024-08-18T15:57:26.093 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:57:26.093 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:57:26.093 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:57:26.093 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:57:26.093 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:57:26.093 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:57:26.094 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000116", 2024-08-18T15:57:26.094 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.25", 2024-08-18T15:57:26.094 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:57:26.094 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:57:26.094 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:57:26.094 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:57:26.094 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000034", 2024-08-18T15:57:26.094 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:57:26.094 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:57:26.094 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000101", 2024-08-18T15:57:26.094 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:57:26.094 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:57:26.094 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.13", 2024-08-18T15:57:26.095 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:57:26.095 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:57:26.095 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.1", 2024-08-18T15:57:26.095 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0.retry", 2024-08-18T15:57:26.095 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:57:26.095 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:57:26.095 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000079", 2024-08-18T15:57:26.095 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000049", 2024-08-18T15:57:26.095 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:57:26.095 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0", 2024-08-18T15:57:26.095 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:57:26.095 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:57:26.095 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000117", 2024-08-18T15:57:26.096 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:57:26.096 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000044", 2024-08-18T15:57:26.096 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:57:26.096 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:57:26.096 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:57:26.096 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:57:26.096 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:57:26.096 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:57:26.096 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.14", 2024-08-18T15:57:26.096 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:57:26.096 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000066", 2024-08-18T15:57:26.096 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:57:26.096 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:57:26.097 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:57:26.097 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:57:26.097 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:57:26.097 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000068", 2024-08-18T15:57:26.097 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:57:26.097 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000085", 2024-08-18T15:57:26.097 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:57:26.097 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:57:26.097 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:57:26.097 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:57:26.097 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:57:26.097 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:57:26.097 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:57:26.097 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:57:26.098 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:57:26.098 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:57:26.098 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:57:26.098 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:57:26.098 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000073", 2024-08-18T15:57:26.098 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:57:26.098 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:57:26.098 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:57:26.098 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000038", 2024-08-18T15:57:26.098 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:57:26.098 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:57:26.098 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:57:26.098 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:57:26.098 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.10", 2024-08-18T15:57:26.099 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:57:26.099 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:57:26.099 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:57:26.099 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000118", 2024-08-18T15:57:26.099 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:57:26.099 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3", 2024-08-18T15:57:26.099 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:57:26.099 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000036", 2024-08-18T15:57:26.099 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:57:26.099 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:57:26.099 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:57:26.099 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:57:26.099 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:57:26.099 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000103", 2024-08-18T15:57:26.100 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.3", 2024-08-18T15:57:26.100 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:57:26.100 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000119", 2024-08-18T15:57:26.100 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000098", 2024-08-18T15:57:26.100 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:57:26.100 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:57:26.100 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:57:26.100 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:57:26.100 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:57:26.100 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:57:26.100 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:57:26.100 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:57:26.100 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.0", 2024-08-18T15:57:26.100 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:57:26.101 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:57:26.101 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:57:26.101 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:57:26.101 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:57:26.101 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:57:26.101 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:57:26.101 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:57:26.101 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000027", 2024-08-18T15:57:26.101 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:57:26.101 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status", 2024-08-18T15:57:26.101 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:57:26.101 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000019", 2024-08-18T15:57:26.101 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:57:26.102 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:57:26.102 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:57:26.102 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:57:26.102 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1", 2024-08-18T15:57:26.102 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:57:26.102 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:57:26.102 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:57:26.102 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:57:26.102 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000039", 2024-08-18T15:57:26.102 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:57:26.102 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:57:26.102 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:57:26.102 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:57:26.103 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:57:26.103 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:57:26.103 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:57:26.103 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:57:26.103 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:57:26.103 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:57:26.103 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:57:26.103 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000100", 2024-08-18T15:57:26.103 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:57:26.103 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:57:26.103 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:57:26.103 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:57:26.103 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:57:26.104 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000093", 2024-08-18T15:57:26.104 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:57:26.104 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000004", 2024-08-18T15:57:26.104 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000063", 2024-08-18T15:57:26.104 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000122", 2024-08-18T15:57:26.104 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:57:26.104 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:57:26.104 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:57:26.104 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000057", 2024-08-18T15:57:26.104 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:57:26.104 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:57:26.104 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:57:26.104 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:57:26.105 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000054", 2024-08-18T15:57:26.105 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:57:26.105 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000114", 2024-08-18T15:57:26.105 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:57:26.105 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:57:26.105 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:57:26.105 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.2", 2024-08-18T15:57:26.105 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:57:26.105 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:57:26.105 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0", 2024-08-18T15:57:26.105 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:57:26.105 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:57:26.105 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:57:26.106 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:57:26.106 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:57:26.106 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:57:26.106 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000012", 2024-08-18T15:57:26.106 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:57:26.106 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:57:26.106 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:57:26.106 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000084", 2024-08-18T15:57:26.106 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000043", 2024-08-18T15:57:26.106 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3", 2024-08-18T15:57:26.106 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:57:26.106 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:57:26.106 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.11", 2024-08-18T15:57:26.107 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:57:26.107 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:57:26.107 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:57:26.107 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:57:26.107 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:57:26.107 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:57:26.107 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:57:26.107 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:57:26.107 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000111", 2024-08-18T15:57:26.107 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:57:26.107 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000048", 2024-08-18T15:57:26.107 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:57:26.108 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000020", 2024-08-18T15:57:26.108 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:57:26.108 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:57:26.108 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:57:26.108 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:57:26.108 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:57:26.108 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000099", 2024-08-18T15:57:26.108 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000056", 2024-08-18T15:57:26.108 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:57:26.108 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:57:26.108 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.19", 2024-08-18T15:57:26.108 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:57:26.108 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.28", 2024-08-18T15:57:26.108 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:57:26.109 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:57:26.109 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:57:26.109 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:57:26.109 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:57:26.109 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000072", 2024-08-18T15:57:26.109 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:57:26.109 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:57:26.109 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000062", 2024-08-18T15:57:26.109 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:57:26.109 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:57:26.109 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:57:26.109 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000115", 2024-08-18T15:57:26.110 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:57:26.110 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.1", 2024-08-18T15:57:26.110 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2", 2024-08-18T15:57:26.110 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000033", 2024-08-18T15:57:26.110 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:57:26.110 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:57:26.110 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:57:26.110 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:57:26.110 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:57:26.110 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:57:26.110 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:57:26.110 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:57:26.110 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:57:26.110 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:57:26.111 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:57:26.111 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:57:26.111 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:57:26.111 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:57:26.111 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:57:26.111 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:57:26.111 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:57:26.111 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:57:26.111 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:57:26.111 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000025", 2024-08-18T15:57:26.111 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:57:26.111 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000123", 2024-08-18T15:57:26.111 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.31", 2024-08-18T15:57:26.112 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:57:26.112 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:57:26.112 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:57:26.112 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:57:26.112 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.18", 2024-08-18T15:57:26.112 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:57:26.112 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:57:26.112 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:57:26.112 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:57:26.112 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:57:26.112 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000108", 2024-08-18T15:57:26.112 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:57:26.112 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:57:26.113 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000094", 2024-08-18T15:57:26.113 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:57:26.113 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:57:26.113 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:57:26.113 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:57:26.113 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:57:26.113 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:57:26.113 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000060", 2024-08-18T15:57:26.113 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:57:26.113 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:57:26.113 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:57:26.113 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000109", 2024-08-18T15:57:26.113 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.15", 2024-08-18T15:57:26.113 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:57:26.114 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:57:26.114 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:57:26.114 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:57:26.114 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:57:26.114 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:57:26.114 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:57:26.114 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:57:26.114 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:57:26.114 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000013", 2024-08-18T15:57:26.114 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:57:26.114 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:57:26.114 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:57:26.114 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:57:26.115 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:57:26.115 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:57:26.115 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:57:26.115 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:57:26.115 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:57:26.115 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:57:26.115 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:57:26.115 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:57:26.115 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:57:26.115 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:57:26.115 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000051", 2024-08-18T15:57:26.115 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:57:26.115 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:57:26.115 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:57:26.116 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:57:26.116 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000106", 2024-08-18T15:57:26.116 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:57:26.116 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:57:26.116 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:57:26.116 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:57:26.116 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:57:26.116 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000017", 2024-08-18T15:57:26.116 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:57:26.116 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:57:26.116 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:57:26.116 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000023", 2024-08-18T15:57:26.116 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:57:26.116 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:57:26.117 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:57:26.117 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:57:26.117 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:57:26.117 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:57:26.117 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:57:26.117 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000107", 2024-08-18T15:57:26.117 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:57:26.117 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:57:26.117 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000046", 2024-08-18T15:57:26.117 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:57:26.117 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:57:26.117 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:57:26.117 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.26", 2024-08-18T15:57:26.117 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:57:26.118 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:57:26.118 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9", 2024-08-18T15:57:26.118 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.22", 2024-08-18T15:57:26.118 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:57:26.118 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:57:26.118 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:57:26.118 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:57:26.118 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:57:26.118 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:57:26.118 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:57:26.118 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2", 2024-08-18T15:57:26.118 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:57:26.118 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:57:26.118 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000067", 2024-08-18T15:57:26.119 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:57:26.119 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:57:26.119 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:57:26.119 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:57:26.119 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:57:26.119 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:57:26.119 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:57:26.119 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:57:26.119 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:57:26.119 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:57:26.119 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:57:26.119 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:57:26.119 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:57:26.120 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:57:26.120 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:57:26.120 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:57:26.120 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:57:26.120 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:57:26.120 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:57:26.120 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.9", 2024-08-18T15:57:26.120 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000086", 2024-08-18T15:57:26.120 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.5", 2024-08-18T15:57:26.120 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:57:26.120 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:57:26.120 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:57:26.120 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:57:26.120 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:57:26.121 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:57:26.121 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2.MFXJ583gCr1.0", 2024-08-18T15:57:26.121 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:57:26.121 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1.GPnO16liyZ+.0", 2024-08-18T15:57:26.121 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:57:26.121 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:57:26.121 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:57:26.121 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:57:26.121 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:57:26.121 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:57:26.121 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:57:26.121 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:57:26.121 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:57:26.122 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.20", 2024-08-18T15:57:26.122 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:57:26.122 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:57:26.122 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:57:26.122 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:57:26.122 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:57:26.122 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:57:26.122 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1.retry", 2024-08-18T15:57:26.122 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:57:26.122 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:57:26.122 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:57:26.122 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0.retry", 2024-08-18T15:57:26.122 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000050", 2024-08-18T15:57:26.123 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:57:26.123 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000102", 2024-08-18T15:57:26.123 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:57:26.123 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:57:26.123 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:57:26.123 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:57:26.123 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:57:26.123 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:57:26.123 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:57:26.123 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:57:26.123 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:57:26.123 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:57:26.123 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:57:26.123 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000124", 2024-08-18T15:57:26.124 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:57:26.124 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:57:26.124 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:57:26.124 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000001", 2024-08-18T15:57:26.124 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:57:26.124 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:57:26.124 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:57:26.124 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:57:26.124 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000126", 2024-08-18T15:57:26.124 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:57:26.124 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0.fZhctfLXnSt.0", 2024-08-18T15:57:26.124 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:57:26.124 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000055", 2024-08-18T15:57:26.124 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:57:26.125 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000061", 2024-08-18T15:57:26.125 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1", 2024-08-18T15:57:26.125 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:57:26.125 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:57:26.125 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:57:26.125 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000097", 2024-08-18T15:57:26.125 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:57:26.125 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:57:26.125 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:57:26.125 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:57:26.125 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:57:26.125 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:57:26.125 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:57:26.125 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:57:26.126 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000042", 2024-08-18T15:57:26.126 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:57:26.126 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:57:26.126 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:57:26.126 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1.retry", 2024-08-18T15:57:26.126 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:57:26.126 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:57:26.126 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:57:26.126 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:57:26.126 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:57:26.126 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:57:26.126 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:57:26.127 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000075", 2024-08-18T15:57:26.127 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:57:26.127 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000120", 2024-08-18T15:57:26.127 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000030", 2024-08-18T15:57:26.127 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:57:26.127 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:57:26.127 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:57:26.127 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:57:26.127 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:57:26.127 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:57:26.127 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9" 2024-08-18T15:57:26.127 INFO:teuthology.orchestra.run.smithi193.stdout:] 2024-08-18T15:57:26.127 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:57:31.129 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:57:31.129 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:57:31.130 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:57:31.182 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:57:31.182 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:57:31.245 INFO:teuthology.orchestra.run.smithi193.stdout:[ 2024-08-18T15:57:31.245 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:57:31.246 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:57:31.246 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.29", 2024-08-18T15:57:31.246 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000078", 2024-08-18T15:57:31.246 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.history", 2024-08-18T15:57:31.246 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:57:31.246 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:57:31.246 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:57:31.246 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:57:31.246 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.23", 2024-08-18T15:57:31.246 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000070", 2024-08-18T15:57:31.246 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000104", 2024-08-18T15:57:31.246 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:57:31.247 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2.retry", 2024-08-18T15:57:31.247 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:57:31.247 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:57:31.247 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:57:31.247 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:57:31.247 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000026", 2024-08-18T15:57:31.247 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:57:31.247 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:57:31.247 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:57:31.247 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:57:31.248 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:57:31.248 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:57:31.248 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.8", 2024-08-18T15:57:31.248 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:57:31.248 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:57:31.248 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:57:31.248 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:57:31.248 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000028", 2024-08-18T15:57:31.248 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:57:31.248 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:57:31.248 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:57:31.248 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:57:31.248 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:57:31.248 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:57:31.249 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:57:31.249 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2.retry", 2024-08-18T15:57:31.249 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:57:31.249 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.30", 2024-08-18T15:57:31.249 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:57:31.249 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:57:31.249 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:57:31.249 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:57:31.249 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000040", 2024-08-18T15:57:31.249 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:57:31.249 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.6", 2024-08-18T15:57:31.249 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:57:31.249 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:57:31.249 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:57:31.250 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:57:31.250 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:57:31.250 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:57:31.250 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000015", 2024-08-18T15:57:31.250 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:57:31.250 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:57:31.250 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000069", 2024-08-18T15:57:31.250 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:57:31.250 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:57:31.250 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:57:31.250 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000095", 2024-08-18T15:57:31.250 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0", 2024-08-18T15:57:31.250 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000003", 2024-08-18T15:57:31.251 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000047", 2024-08-18T15:57:31.251 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:57:31.251 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:57:31.251 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:57:31.251 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:57:31.251 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:57:31.251 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:57:31.251 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:57:31.251 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:57:31.251 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:57:31.251 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000035", 2024-08-18T15:57:31.251 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:57:31.251 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:57:31.252 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000037", 2024-08-18T15:57:31.252 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:57:31.252 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:57:31.252 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:57:31.252 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:57:31.252 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:57:31.252 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:57:31.252 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:57:31.252 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:57:31.252 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:57:31.252 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:57:31.252 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:57:31.252 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:57:31.252 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:57:31.253 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:57:31.253 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000024", 2024-08-18T15:57:31.253 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:57:31.253 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:57:31.253 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:57:31.253 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:57:31.253 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000000", 2024-08-18T15:57:31.253 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:57:31.253 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000031", 2024-08-18T15:57:31.253 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:57:31.253 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:57:31.253 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:57:31.253 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:57:31.254 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:57:31.254 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:57:31.254 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3", 2024-08-18T15:57:31.254 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:57:31.254 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:57:31.254 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:57:31.254 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:57:31.254 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:57:31.254 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:57:31.254 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:57:31.254 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:57:31.254 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:57:31.254 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:57:31.255 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:57:31.255 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:57:31.255 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:57:31.255 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:57:31.255 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:57:31.255 INFO:teuthology.orchestra.run.smithi193.stdout: "data_loggenerations_metadata", 2024-08-18T15:57:31.255 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2", 2024-08-18T15:57:31.255 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:57:31.255 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:57:31.255 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:57:31.255 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:57:31.255 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T15:57:31.255 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:57:31.255 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:57:31.256 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:57:31.256 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:57:31.256 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:57:31.256 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000076", 2024-08-18T15:57:31.256 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:57:31.256 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:57:31.256 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000113", 2024-08-18T15:57:31.256 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000005", 2024-08-18T15:57:31.256 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:57:31.256 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000011", 2024-08-18T15:57:31.256 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:57:31.256 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000058", 2024-08-18T15:57:31.256 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:57:31.256 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:57:31.257 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:57:31.257 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000052", 2024-08-18T15:57:31.257 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:57:31.257 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:57:31.257 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:57:31.257 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:57:31.257 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.21", 2024-08-18T15:57:31.257 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:57:31.257 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:57:31.257 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:57:31.257 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:57:31.257 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:57:31.257 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:57:31.257 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:57:31.258 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:57:31.258 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:57:31.258 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:57:31.258 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:57:31.258 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:57:31.258 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:57:31.258 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:57:31.258 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:57:31.258 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:57:31.258 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:57:31.258 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:57:31.258 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:57:31.259 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000088", 2024-08-18T15:57:31.259 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000080", 2024-08-18T15:57:31.259 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:57:31.259 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:57:31.259 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000090", 2024-08-18T15:57:31.259 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:57:31.259 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:57:31.259 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000110", 2024-08-18T15:57:31.259 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:57:31.259 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:57:31.259 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:57:31.259 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.0", 2024-08-18T15:57:31.259 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:57:31.260 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000096", 2024-08-18T15:57:31.260 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:57:31.260 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:57:31.260 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:57:31.260 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:57:31.260 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:57:31.260 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000087", 2024-08-18T15:57:31.261 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:57:31.261 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:57:31.261 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:57:31.261 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:57:31.261 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000008", 2024-08-18T15:57:31.261 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:57:31.261 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:57:31.261 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:57:31.261 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:57:31.261 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:57:31.261 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000006", 2024-08-18T15:57:31.261 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000029", 2024-08-18T15:57:31.261 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:57:31.262 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000089", 2024-08-18T15:57:31.262 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:57:31.262 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:57:31.262 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000022", 2024-08-18T15:57:31.262 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:57:31.262 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:57:31.262 INFO:teuthology.orchestra.run.smithi193.stdout: "bilog.trim", 2024-08-18T15:57:31.262 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000016", 2024-08-18T15:57:31.262 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:57:31.262 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:57:31.262 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:57:31.262 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:57:31.262 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:57:31.262 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000041", 2024-08-18T15:57:31.263 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:57:31.263 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:57:31.263 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.log.01050e30-179b-43be-bdbb-3bb23cf37ae8.3", 2024-08-18T15:57:31.263 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:57:31.263 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000018", 2024-08-18T15:57:31.263 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000092", 2024-08-18T15:57:31.263 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:57:31.263 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:57:31.263 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:57:31.263 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:57:31.263 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.27", 2024-08-18T15:57:31.263 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.4", 2024-08-18T15:57:31.263 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:57:31.264 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:57:31.264 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:57:31.264 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:57:31.264 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:57:31.264 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:57:31.264 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:57:31.264 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000014", 2024-08-18T15:57:31.264 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:57:31.264 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3.retry", 2024-08-18T15:57:31.264 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:57:31.265 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:57:31.265 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:57:31.265 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:57:31.265 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:57:31.265 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000112", 2024-08-18T15:57:31.265 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:57:31.265 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000007", 2024-08-18T15:57:31.265 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000021", 2024-08-18T15:57:31.265 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:57:31.265 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:57:31.265 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:57:31.266 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:57:31.266 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:57:31.266 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.16", 2024-08-18T15:57:31.266 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:57:31.266 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000064", 2024-08-18T15:57:31.266 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:57:31.266 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:57:31.266 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:57:31.266 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:57:31.266 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:57:31.267 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:57:31.267 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:57:31.267 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000071", 2024-08-18T15:57:31.267 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:57:31.267 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:57:31.267 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:57:31.267 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000074", 2024-08-18T15:57:31.267 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:57:31.267 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:57:31.267 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:57:31.267 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:57:31.267 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:57:31.267 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:57:31.267 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:57:31.268 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:57:31.268 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:57:31.268 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:57:31.268 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:57:31.268 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:57:31.268 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:57:31.268 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:57:31.268 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:57:31.268 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000081", 2024-08-18T15:57:31.268 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:57:31.268 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000009", 2024-08-18T15:57:31.268 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000121", 2024-08-18T15:57:31.268 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1", 2024-08-18T15:57:31.269 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:57:31.269 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:57:31.269 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:57:31.269 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:57:31.269 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:57:31.269 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:57:31.269 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.17", 2024-08-18T15:57:31.269 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000125", 2024-08-18T15:57:31.269 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:57:31.269 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000082", 2024-08-18T15:57:31.269 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:57:31.269 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:57:31.269 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:57:31.269 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:57:31.270 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:57:31.270 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:57:31.270 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:57:31.270 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:57:31.270 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:57:31.270 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:57:31.270 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:57:31.270 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000105", 2024-08-18T15:57:31.270 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000059", 2024-08-18T15:57:31.270 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:57:31.270 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:57:31.270 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:57:31.270 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:57:31.271 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:57:31.271 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:57:31.271 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:57:31.271 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:57:31.271 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:57:31.271 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:57:31.271 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:57:31.271 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000077", 2024-08-18T15:57:31.271 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000032", 2024-08-18T15:57:31.271 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:57:31.271 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:57:31.271 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:57:31.271 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:57:31.272 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:57:31.272 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:57:31.272 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.24", 2024-08-18T15:57:31.272 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:57:31.272 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:57:31.272 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:57:31.272 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:57:31.272 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3.retry", 2024-08-18T15:57:31.272 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:57:31.272 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:57:31.272 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:57:31.272 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:57:31.272 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:57:31.272 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:57:31.273 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000053", 2024-08-18T15:57:31.273 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000091", 2024-08-18T15:57:31.273 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:57:31.273 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:57:31.273 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:57:31.273 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000065", 2024-08-18T15:57:31.273 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:57:31.273 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:57:31.273 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:57:31.273 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:57:31.273 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000083", 2024-08-18T15:57:31.273 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000010", 2024-08-18T15:57:31.273 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:57:31.273 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:57:31.274 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:57:31.274 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:57:31.274 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:57:31.274 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:57:31.274 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:57:31.274 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:57:31.274 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:57:31.274 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:57:31.274 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:57:31.274 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.7", 2024-08-18T15:57:31.274 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.3", 2024-08-18T15:57:31.274 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:57:31.274 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:57:31.275 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:57:31.275 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:57:31.275 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:57:31.275 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:57:31.275 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:57:31.275 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:57:31.275 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.2", 2024-08-18T15:57:31.275 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:57:31.275 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:57:31.275 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:57:31.275 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:57:31.275 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:57:31.275 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:57:31.276 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:57:31.276 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000045", 2024-08-18T15:57:31.276 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000002", 2024-08-18T15:57:31.276 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:57:31.276 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3.2UbiLidwSTy.0", 2024-08-18T15:57:31.276 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:57:31.276 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.12", 2024-08-18T15:57:31.276 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:57:31.276 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:57:31.276 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:57:31.276 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:57:31.276 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:57:31.276 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:57:31.276 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000116", 2024-08-18T15:57:31.277 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.25", 2024-08-18T15:57:31.277 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:57:31.277 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:57:31.277 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:57:31.277 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:57:31.277 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000034", 2024-08-18T15:57:31.277 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:57:31.277 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:57:31.277 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000101", 2024-08-18T15:57:31.277 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:57:31.277 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:57:31.277 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.13", 2024-08-18T15:57:31.277 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:57:31.278 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:57:31.278 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.1", 2024-08-18T15:57:31.278 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0.retry", 2024-08-18T15:57:31.278 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:57:31.278 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:57:31.278 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000079", 2024-08-18T15:57:31.278 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000049", 2024-08-18T15:57:31.278 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:57:31.278 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0", 2024-08-18T15:57:31.278 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:57:31.278 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:57:31.278 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000117", 2024-08-18T15:57:31.278 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:57:31.279 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000044", 2024-08-18T15:57:31.279 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:57:31.279 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:57:31.279 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:57:31.279 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:57:31.279 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:57:31.279 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:57:31.279 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.14", 2024-08-18T15:57:31.279 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:57:31.279 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000066", 2024-08-18T15:57:31.279 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:57:31.279 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:57:31.279 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:57:31.279 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:57:31.280 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:57:31.280 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000068", 2024-08-18T15:57:31.280 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:57:31.280 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000085", 2024-08-18T15:57:31.280 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:57:31.280 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:57:31.280 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:57:31.280 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:57:31.280 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:57:31.280 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:57:31.280 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:57:31.280 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:57:31.280 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:57:31.280 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:57:31.281 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:57:31.281 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:57:31.281 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000073", 2024-08-18T15:57:31.281 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:57:31.281 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:57:31.281 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:57:31.281 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000038", 2024-08-18T15:57:31.281 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:57:31.281 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:57:31.281 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:57:31.281 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:57:31.281 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.10", 2024-08-18T15:57:31.281 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:57:31.282 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:57:31.282 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:57:31.282 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000118", 2024-08-18T15:57:31.282 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:57:31.282 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3", 2024-08-18T15:57:31.282 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:57:31.282 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000036", 2024-08-18T15:57:31.282 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:57:31.282 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:57:31.282 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:57:31.282 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:57:31.282 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:57:31.282 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000103", 2024-08-18T15:57:31.282 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.3", 2024-08-18T15:57:31.283 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:57:31.283 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000119", 2024-08-18T15:57:31.283 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000098", 2024-08-18T15:57:31.283 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:57:31.283 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:57:31.283 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:57:31.283 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:57:31.283 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:57:31.283 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:57:31.283 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:57:31.283 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:57:31.283 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.0", 2024-08-18T15:57:31.284 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:57:31.284 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:57:31.284 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:57:31.284 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:57:31.284 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:57:31.284 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:57:31.284 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:57:31.284 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:57:31.284 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000027", 2024-08-18T15:57:31.284 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:57:31.284 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status", 2024-08-18T15:57:31.284 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:57:31.284 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000019", 2024-08-18T15:57:31.284 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:57:31.285 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:57:31.285 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:57:31.285 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:57:31.285 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1", 2024-08-18T15:57:31.285 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:57:31.285 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:57:31.285 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:57:31.285 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:57:31.285 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000039", 2024-08-18T15:57:31.285 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:57:31.285 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:57:31.285 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:57:31.285 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:57:31.286 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:57:31.286 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:57:31.286 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:57:31.286 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:57:31.286 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:57:31.286 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:57:31.286 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:57:31.286 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000100", 2024-08-18T15:57:31.286 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:57:31.286 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:57:31.286 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:57:31.286 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:57:31.286 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:57:31.286 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000093", 2024-08-18T15:57:31.287 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:57:31.287 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000004", 2024-08-18T15:57:31.287 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000063", 2024-08-18T15:57:31.287 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000122", 2024-08-18T15:57:31.287 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:57:31.287 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:57:31.287 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:57:31.287 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000057", 2024-08-18T15:57:31.287 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:57:31.287 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:57:31.287 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:57:31.287 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:57:31.287 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000054", 2024-08-18T15:57:31.288 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:57:31.288 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000114", 2024-08-18T15:57:31.288 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:57:31.288 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:57:31.288 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:57:31.288 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.2", 2024-08-18T15:57:31.288 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:57:31.288 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:57:31.288 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0", 2024-08-18T15:57:31.288 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:57:31.288 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:57:31.288 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:57:31.288 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:57:31.289 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:57:31.289 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:57:31.289 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000012", 2024-08-18T15:57:31.289 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:57:31.289 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:57:31.289 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:57:31.289 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000084", 2024-08-18T15:57:31.289 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000043", 2024-08-18T15:57:31.289 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3", 2024-08-18T15:57:31.289 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:57:31.289 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:57:31.289 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.11", 2024-08-18T15:57:31.289 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:57:31.289 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:57:31.290 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:57:31.290 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:57:31.290 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:57:31.290 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:57:31.290 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:57:31.290 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:57:31.290 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000111", 2024-08-18T15:57:31.290 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:57:31.290 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000048", 2024-08-18T15:57:31.290 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:57:31.290 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000020", 2024-08-18T15:57:31.290 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:57:31.290 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:57:31.291 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:57:31.291 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:57:31.291 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:57:31.291 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000099", 2024-08-18T15:57:31.291 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000056", 2024-08-18T15:57:31.291 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:57:31.291 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:57:31.291 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.19", 2024-08-18T15:57:31.291 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:57:31.291 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.28", 2024-08-18T15:57:31.291 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:57:31.291 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:57:31.291 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:57:31.291 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:57:31.292 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:57:31.292 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000072", 2024-08-18T15:57:31.292 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:57:31.292 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:57:31.292 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000062", 2024-08-18T15:57:31.292 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:57:31.292 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:57:31.292 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:57:31.292 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000115", 2024-08-18T15:57:31.292 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:57:31.292 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.1", 2024-08-18T15:57:31.292 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2", 2024-08-18T15:57:31.292 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000033", 2024-08-18T15:57:31.293 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:57:31.293 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:57:31.293 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:57:31.293 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:57:31.293 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:57:31.293 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:57:31.293 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:57:31.293 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:57:31.293 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:57:31.293 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:57:31.293 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:57:31.293 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:57:31.293 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:57:31.293 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:57:31.294 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:57:31.294 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:57:31.294 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:57:31.294 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:57:31.294 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:57:31.294 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000025", 2024-08-18T15:57:31.294 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:57:31.294 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000123", 2024-08-18T15:57:31.294 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.31", 2024-08-18T15:57:31.294 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:57:31.294 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:57:31.294 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:57:31.294 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:57:31.295 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.18", 2024-08-18T15:57:31.295 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:57:31.295 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:57:31.295 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:57:31.295 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:57:31.295 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:57:31.295 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000108", 2024-08-18T15:57:31.295 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:57:31.295 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:57:31.295 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000094", 2024-08-18T15:57:31.295 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:57:31.295 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:57:31.295 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:57:31.296 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:57:31.296 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:57:31.296 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:57:31.296 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000060", 2024-08-18T15:57:31.296 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:57:31.296 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:57:31.296 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:57:31.296 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000109", 2024-08-18T15:57:31.296 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.15", 2024-08-18T15:57:31.296 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:57:31.296 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:57:31.296 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:57:31.296 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:57:31.297 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:57:31.297 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:57:31.297 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:57:31.297 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:57:31.297 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:57:31.297 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000013", 2024-08-18T15:57:31.297 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:57:31.297 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:57:31.297 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:57:31.297 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:57:31.297 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:57:31.297 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:57:31.297 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:57:31.297 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:57:31.298 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:57:31.298 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:57:31.298 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:57:31.298 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:57:31.298 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:57:31.298 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:57:31.298 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000051", 2024-08-18T15:57:31.298 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:57:31.298 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:57:31.298 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:57:31.298 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:57:31.298 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000106", 2024-08-18T15:57:31.298 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:57:31.299 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:57:31.299 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:57:31.299 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:57:31.299 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:57:31.299 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000017", 2024-08-18T15:57:31.299 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:57:31.299 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:57:31.299 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:57:31.299 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000023", 2024-08-18T15:57:31.299 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:57:31.299 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:57:31.299 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:57:31.299 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:57:31.299 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:57:31.301 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:57:31.301 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:57:31.301 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000107", 2024-08-18T15:57:31.302 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:57:31.302 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:57:31.302 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000046", 2024-08-18T15:57:31.302 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:57:31.302 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:57:31.302 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:57:31.302 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.26", 2024-08-18T15:57:31.302 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:57:31.302 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:57:31.302 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9", 2024-08-18T15:57:31.302 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.22", 2024-08-18T15:57:31.302 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:57:31.302 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:57:31.303 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:57:31.303 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:57:31.303 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:57:31.303 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:57:31.303 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:57:31.303 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2", 2024-08-18T15:57:31.303 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:57:31.303 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:57:31.303 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000067", 2024-08-18T15:57:31.303 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:57:31.303 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:57:31.303 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:57:31.303 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:57:31.303 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:57:31.304 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:57:31.304 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:57:31.304 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:57:31.304 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:57:31.304 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:57:31.304 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:57:31.304 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:57:31.304 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:57:31.304 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:57:31.304 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:57:31.304 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:57:31.304 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:57:31.304 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:57:31.304 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:57:31.305 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.9", 2024-08-18T15:57:31.305 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000086", 2024-08-18T15:57:31.305 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.5", 2024-08-18T15:57:31.305 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:57:31.305 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:57:31.305 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:57:31.305 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:57:31.305 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:57:31.305 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:57:31.305 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2.MFXJ583gCr1.0", 2024-08-18T15:57:31.305 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:57:31.305 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1.GPnO16liyZ+.0", 2024-08-18T15:57:31.306 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:57:31.306 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:57:31.306 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:57:31.306 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:57:31.306 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:57:31.306 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:57:31.306 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:57:31.306 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:57:31.306 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:57:31.306 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.20", 2024-08-18T15:57:31.306 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:57:31.307 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:57:31.307 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:57:31.307 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:57:31.307 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:57:31.307 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:57:31.307 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1.retry", 2024-08-18T15:57:31.307 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:57:31.307 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:57:31.307 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:57:31.307 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0.retry", 2024-08-18T15:57:31.307 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000050", 2024-08-18T15:57:31.307 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:57:31.307 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000102", 2024-08-18T15:57:31.307 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:57:31.308 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:57:31.308 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:57:31.308 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:57:31.308 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:57:31.308 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:57:31.308 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:57:31.308 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:57:31.308 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:57:31.308 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:57:31.308 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:57:31.308 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000124", 2024-08-18T15:57:31.308 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:57:31.308 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:57:31.308 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:57:31.309 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000001", 2024-08-18T15:57:31.309 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:57:31.309 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:57:31.309 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:57:31.309 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:57:31.309 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000126", 2024-08-18T15:57:31.309 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:57:31.309 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0.fZhctfLXnSt.0", 2024-08-18T15:57:31.309 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:57:31.309 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000055", 2024-08-18T15:57:31.309 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:57:31.309 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000061", 2024-08-18T15:57:31.309 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1", 2024-08-18T15:57:31.310 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:57:31.310 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:57:31.310 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:57:31.310 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000097", 2024-08-18T15:57:31.310 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:57:31.310 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:57:31.310 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:57:31.310 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:57:31.310 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:57:31.310 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:57:31.310 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:57:31.310 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:57:31.310 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000042", 2024-08-18T15:57:31.311 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:57:31.311 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:57:31.311 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:57:31.311 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1.retry", 2024-08-18T15:57:31.311 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:57:31.311 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:57:31.311 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:57:31.311 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:57:31.311 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:57:31.311 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:57:31.311 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:57:31.311 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000075", 2024-08-18T15:57:31.311 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:57:31.311 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000120", 2024-08-18T15:57:31.312 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000030", 2024-08-18T15:57:31.312 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:57:31.312 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:57:31.312 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:57:31.312 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:57:31.312 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:57:31.312 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:57:31.312 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9" 2024-08-18T15:57:31.312 INFO:teuthology.orchestra.run.smithi193.stdout:] 2024-08-18T15:57:31.312 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:57:36.314 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:57:36.315 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:57:36.315 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:57:36.368 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:57:36.368 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:57:36.435 INFO:teuthology.orchestra.run.smithi193.stdout:[ 2024-08-18T15:57:36.435 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:57:36.435 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:57:36.435 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.29", 2024-08-18T15:57:36.436 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000078", 2024-08-18T15:57:36.436 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.history", 2024-08-18T15:57:36.436 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:57:36.436 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:57:36.436 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:57:36.436 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:57:36.436 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.23", 2024-08-18T15:57:36.436 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000070", 2024-08-18T15:57:36.436 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000104", 2024-08-18T15:57:36.436 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:57:36.436 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2.retry", 2024-08-18T15:57:36.437 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:57:36.437 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:57:36.437 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:57:36.437 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:57:36.437 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000026", 2024-08-18T15:57:36.437 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:57:36.437 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:57:36.437 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:57:36.437 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:57:36.437 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:57:36.437 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:57:36.438 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.8", 2024-08-18T15:57:36.438 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:57:36.438 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:57:36.438 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:57:36.438 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:57:36.438 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000028", 2024-08-18T15:57:36.438 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:57:36.438 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:57:36.438 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:57:36.438 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:57:36.439 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:57:36.439 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:57:36.439 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:57:36.439 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2.retry", 2024-08-18T15:57:36.439 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:57:36.439 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.30", 2024-08-18T15:57:36.439 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:57:36.439 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:57:36.439 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:57:36.439 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:57:36.440 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000040", 2024-08-18T15:57:36.440 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:57:36.440 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.6", 2024-08-18T15:57:36.440 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:57:36.440 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:57:36.440 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:57:36.440 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:57:36.440 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:57:36.440 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:57:36.440 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000015", 2024-08-18T15:57:36.440 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:57:36.441 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:57:36.441 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000069", 2024-08-18T15:57:36.441 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:57:36.441 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:57:36.441 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:57:36.441 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000095", 2024-08-18T15:57:36.441 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0", 2024-08-18T15:57:36.441 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000003", 2024-08-18T15:57:36.441 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000047", 2024-08-18T15:57:36.441 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:57:36.442 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:57:36.442 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:57:36.442 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:57:36.442 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:57:36.442 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:57:36.442 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:57:36.442 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:57:36.442 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:57:36.442 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000035", 2024-08-18T15:57:36.442 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:57:36.443 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:57:36.443 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000037", 2024-08-18T15:57:36.443 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:57:36.443 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:57:36.443 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:57:36.443 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:57:36.443 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:57:36.443 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:57:36.443 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:57:36.443 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:57:36.443 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:57:36.444 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:57:36.444 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:57:36.444 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:57:36.444 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:57:36.444 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:57:36.444 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000024", 2024-08-18T15:57:36.444 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:57:36.444 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:57:36.444 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:57:36.444 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:57:36.445 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000000", 2024-08-18T15:57:36.445 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:57:36.445 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000031", 2024-08-18T15:57:36.445 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:57:36.445 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:57:36.445 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:57:36.445 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:57:36.445 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:57:36.445 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:57:36.445 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3", 2024-08-18T15:57:36.445 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:57:36.446 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:57:36.446 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:57:36.446 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:57:36.446 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:57:36.446 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:57:36.446 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:57:36.446 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:57:36.446 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:57:36.446 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:57:36.446 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:57:36.446 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:57:36.447 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:57:36.447 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:57:36.447 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:57:36.447 INFO:teuthology.orchestra.run.smithi193.stdout: "data_loggenerations_metadata", 2024-08-18T15:57:36.447 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2", 2024-08-18T15:57:36.447 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:57:36.447 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:57:36.447 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:57:36.447 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:57:36.447 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T15:57:36.447 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:57:36.448 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:57:36.448 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:57:36.448 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:57:36.448 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:57:36.448 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000076", 2024-08-18T15:57:36.448 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:57:36.448 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:57:36.448 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000113", 2024-08-18T15:57:36.448 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000005", 2024-08-18T15:57:36.448 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:57:36.448 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000011", 2024-08-18T15:57:36.449 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:57:36.449 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000058", 2024-08-18T15:57:36.449 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:57:36.449 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:57:36.449 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:57:36.449 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000052", 2024-08-18T15:57:36.449 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:57:36.449 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:57:36.449 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:57:36.449 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:57:36.449 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.21", 2024-08-18T15:57:36.450 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:57:36.450 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:57:36.450 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:57:36.450 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:57:36.450 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:57:36.450 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:57:36.450 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:57:36.450 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:57:36.450 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:57:36.450 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:57:36.450 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:57:36.451 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:57:36.451 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:57:36.451 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:57:36.451 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:57:36.451 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:57:36.451 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:57:36.451 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:57:36.451 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:57:36.451 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000088", 2024-08-18T15:57:36.451 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000080", 2024-08-18T15:57:36.452 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:57:36.452 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:57:36.452 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000090", 2024-08-18T15:57:36.452 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:57:36.452 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:57:36.452 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000110", 2024-08-18T15:57:36.452 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:57:36.452 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:57:36.452 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:57:36.452 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.0", 2024-08-18T15:57:36.452 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:57:36.453 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000096", 2024-08-18T15:57:36.453 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:57:36.453 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:57:36.453 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:57:36.453 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:57:36.453 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:57:36.453 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000087", 2024-08-18T15:57:36.453 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:57:36.453 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:57:36.453 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:57:36.454 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:57:36.454 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000008", 2024-08-18T15:57:36.454 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:57:36.454 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:57:36.454 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:57:36.454 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:57:36.454 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:57:36.454 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000006", 2024-08-18T15:57:36.454 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000029", 2024-08-18T15:57:36.454 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:57:36.455 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000089", 2024-08-18T15:57:36.455 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:57:36.455 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:57:36.455 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000022", 2024-08-18T15:57:36.455 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:57:36.455 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:57:36.455 INFO:teuthology.orchestra.run.smithi193.stdout: "bilog.trim", 2024-08-18T15:57:36.455 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000016", 2024-08-18T15:57:36.455 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:57:36.455 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:57:36.455 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:57:36.456 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:57:36.456 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:57:36.456 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000041", 2024-08-18T15:57:36.456 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:57:36.456 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:57:36.456 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.log.01050e30-179b-43be-bdbb-3bb23cf37ae8.3", 2024-08-18T15:57:36.456 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:57:36.456 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000018", 2024-08-18T15:57:36.456 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000092", 2024-08-18T15:57:36.456 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:57:36.457 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:57:36.457 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:57:36.457 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:57:36.457 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.27", 2024-08-18T15:57:36.457 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.4", 2024-08-18T15:57:36.457 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:57:36.457 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:57:36.457 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:57:36.457 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:57:36.457 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:57:36.457 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:57:36.458 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:57:36.458 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000014", 2024-08-18T15:57:36.458 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:57:36.458 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3.retry", 2024-08-18T15:57:36.458 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:57:36.458 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:57:36.458 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:57:36.458 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:57:36.458 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:57:36.458 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000112", 2024-08-18T15:57:36.459 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:57:36.459 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000007", 2024-08-18T15:57:36.459 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000021", 2024-08-18T15:57:36.459 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:57:36.459 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:57:36.459 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:57:36.459 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:57:36.459 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:57:36.459 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.16", 2024-08-18T15:57:36.459 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:57:36.460 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000064", 2024-08-18T15:57:36.460 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:57:36.460 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:57:36.460 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:57:36.460 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:57:36.460 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:57:36.460 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:57:36.460 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:57:36.460 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000071", 2024-08-18T15:57:36.460 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:57:36.460 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:57:36.461 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:57:36.461 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000074", 2024-08-18T15:57:36.461 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:57:36.461 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:57:36.461 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:57:36.461 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:57:36.461 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:57:36.461 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:57:36.461 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:57:36.461 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:57:36.461 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:57:36.462 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:57:36.462 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:57:36.462 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:57:36.462 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:57:36.462 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:57:36.462 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:57:36.462 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000081", 2024-08-18T15:57:36.462 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:57:36.462 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000009", 2024-08-18T15:57:36.462 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000121", 2024-08-18T15:57:36.462 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1", 2024-08-18T15:57:36.463 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:57:36.463 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:57:36.463 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:57:36.463 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:57:36.463 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:57:36.463 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:57:36.463 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.17", 2024-08-18T15:57:36.463 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000125", 2024-08-18T15:57:36.463 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:57:36.463 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000082", 2024-08-18T15:57:36.463 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:57:36.464 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:57:36.464 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:57:36.464 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:57:36.464 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:57:36.464 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:57:36.464 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:57:36.464 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:57:36.464 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:57:36.464 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:57:36.464 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:57:36.464 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000105", 2024-08-18T15:57:36.465 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000059", 2024-08-18T15:57:36.465 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:57:36.465 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:57:36.465 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:57:36.465 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:57:36.465 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:57:36.465 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:57:36.465 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:57:36.465 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:57:36.465 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:57:36.465 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:57:36.466 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:57:36.466 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000077", 2024-08-18T15:57:36.466 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000032", 2024-08-18T15:57:36.466 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:57:36.466 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:57:36.466 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:57:36.466 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:57:36.466 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:57:36.466 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:57:36.466 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.24", 2024-08-18T15:57:36.467 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:57:36.467 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:57:36.467 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:57:36.467 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:57:36.467 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3.retry", 2024-08-18T15:57:36.467 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:57:36.467 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:57:36.467 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:57:36.467 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:57:36.467 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:57:36.468 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:57:36.468 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000053", 2024-08-18T15:57:36.468 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000091", 2024-08-18T15:57:36.468 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:57:36.468 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:57:36.468 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:57:36.468 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000065", 2024-08-18T15:57:36.468 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:57:36.468 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:57:36.468 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:57:36.468 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:57:36.469 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000083", 2024-08-18T15:57:36.469 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000010", 2024-08-18T15:57:36.469 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:57:36.469 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:57:36.469 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:57:36.469 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:57:36.469 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:57:36.469 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:57:36.469 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:57:36.469 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:57:36.469 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:57:36.470 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:57:36.470 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:57:36.470 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.7", 2024-08-18T15:57:36.470 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.3", 2024-08-18T15:57:36.470 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:57:36.470 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:57:36.470 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:57:36.470 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:57:36.470 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:57:36.470 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:57:36.470 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:57:36.471 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:57:36.471 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.2", 2024-08-18T15:57:36.471 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:57:36.471 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:57:36.471 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:57:36.471 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:57:36.471 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:57:36.471 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:57:36.471 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:57:36.471 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000045", 2024-08-18T15:57:36.471 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000002", 2024-08-18T15:57:36.472 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:57:36.472 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3.2UbiLidwSTy.0", 2024-08-18T15:57:36.472 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:57:36.472 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.12", 2024-08-18T15:57:36.472 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:57:36.472 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:57:36.472 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:57:36.472 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:57:36.472 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:57:36.472 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:57:36.472 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000116", 2024-08-18T15:57:36.472 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.25", 2024-08-18T15:57:36.473 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:57:36.473 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:57:36.473 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:57:36.473 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:57:36.473 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000034", 2024-08-18T15:57:36.473 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:57:36.473 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:57:36.473 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000101", 2024-08-18T15:57:36.473 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:57:36.473 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:57:36.474 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.13", 2024-08-18T15:57:36.474 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:57:36.474 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:57:36.474 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.1", 2024-08-18T15:57:36.474 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0.retry", 2024-08-18T15:57:36.474 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:57:36.474 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:57:36.474 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000079", 2024-08-18T15:57:36.474 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000049", 2024-08-18T15:57:36.474 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:57:36.474 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0", 2024-08-18T15:57:36.475 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:57:36.475 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:57:36.475 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000117", 2024-08-18T15:57:36.475 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:57:36.475 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000044", 2024-08-18T15:57:36.475 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:57:36.475 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:57:36.475 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:57:36.475 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:57:36.475 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:57:36.475 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:57:36.476 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.14", 2024-08-18T15:57:36.476 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:57:36.476 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000066", 2024-08-18T15:57:36.476 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:57:36.476 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:57:36.476 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:57:36.476 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:57:36.476 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:57:36.476 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000068", 2024-08-18T15:57:36.476 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:57:36.476 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000085", 2024-08-18T15:57:36.477 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:57:36.477 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:57:36.477 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:57:36.477 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:57:36.477 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:57:36.477 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:57:36.477 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:57:36.477 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:57:36.477 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:57:36.477 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:57:36.477 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:57:36.478 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:57:36.478 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000073", 2024-08-18T15:57:36.478 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:57:36.478 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:57:36.478 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:57:36.478 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000038", 2024-08-18T15:57:36.478 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:57:36.478 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:57:36.478 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:57:36.478 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:57:36.478 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.10", 2024-08-18T15:57:36.479 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:57:36.479 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:57:36.479 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:57:36.479 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000118", 2024-08-18T15:57:36.479 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:57:36.479 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3", 2024-08-18T15:57:36.479 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:57:36.479 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000036", 2024-08-18T15:57:36.479 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:57:36.479 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:57:36.480 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:57:36.480 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:57:36.480 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:57:36.480 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000103", 2024-08-18T15:57:36.480 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.3", 2024-08-18T15:57:36.480 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:57:36.480 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000119", 2024-08-18T15:57:36.480 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000098", 2024-08-18T15:57:36.480 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:57:36.480 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:57:36.480 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:57:36.481 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:57:36.481 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:57:36.481 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:57:36.481 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:57:36.481 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:57:36.481 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.0", 2024-08-18T15:57:36.481 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:57:36.481 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:57:36.481 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:57:36.481 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:57:36.481 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:57:36.482 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:57:36.482 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:57:36.482 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:57:36.482 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000027", 2024-08-18T15:57:36.482 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:57:36.482 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status", 2024-08-18T15:57:36.482 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:57:36.482 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000019", 2024-08-18T15:57:36.482 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:57:36.482 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:57:36.483 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:57:36.483 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:57:36.483 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1", 2024-08-18T15:57:36.483 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:57:36.483 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:57:36.483 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:57:36.483 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:57:36.483 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000039", 2024-08-18T15:57:36.483 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:57:36.483 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:57:36.483 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:57:36.484 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:57:36.484 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:57:36.484 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:57:36.484 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:57:36.484 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:57:36.484 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:57:36.484 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:57:36.484 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:57:36.484 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000100", 2024-08-18T15:57:36.484 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:57:36.484 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:57:36.485 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:57:36.485 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:57:36.485 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:57:36.485 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000093", 2024-08-18T15:57:36.485 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:57:36.485 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000004", 2024-08-18T15:57:36.485 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000063", 2024-08-18T15:57:36.485 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000122", 2024-08-18T15:57:36.485 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:57:36.485 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:57:36.486 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:57:36.486 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000057", 2024-08-18T15:57:36.486 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:57:36.486 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:57:36.486 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:57:36.486 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:57:36.486 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000054", 2024-08-18T15:57:36.486 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:57:36.486 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000114", 2024-08-18T15:57:36.486 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:57:36.487 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:57:36.487 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:57:36.487 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.2", 2024-08-18T15:57:36.487 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:57:36.487 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:57:36.487 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0", 2024-08-18T15:57:36.487 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:57:36.487 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:57:36.487 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:57:36.487 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:57:36.487 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:57:36.488 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:57:36.488 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000012", 2024-08-18T15:57:36.488 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:57:36.488 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:57:36.488 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:57:36.488 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000084", 2024-08-18T15:57:36.488 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000043", 2024-08-18T15:57:36.488 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3", 2024-08-18T15:57:36.488 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:57:36.488 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:57:36.488 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.11", 2024-08-18T15:57:36.489 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:57:36.489 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:57:36.489 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:57:36.489 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:57:36.489 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:57:36.489 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:57:36.489 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:57:36.489 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:57:36.489 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000111", 2024-08-18T15:57:36.489 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:57:36.489 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000048", 2024-08-18T15:57:36.490 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:57:36.490 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000020", 2024-08-18T15:57:36.490 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:57:36.490 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:57:36.490 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:57:36.490 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:57:36.490 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:57:36.490 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000099", 2024-08-18T15:57:36.490 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000056", 2024-08-18T15:57:36.490 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:57:36.490 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:57:36.491 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.19", 2024-08-18T15:57:36.491 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:57:36.491 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.28", 2024-08-18T15:57:36.491 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:57:36.491 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:57:36.491 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:57:36.491 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:57:36.491 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:57:36.491 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000072", 2024-08-18T15:57:36.491 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:57:36.491 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:57:36.492 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000062", 2024-08-18T15:57:36.492 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:57:36.492 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:57:36.492 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:57:36.492 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000115", 2024-08-18T15:57:36.492 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:57:36.492 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.1", 2024-08-18T15:57:36.492 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2", 2024-08-18T15:57:36.492 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000033", 2024-08-18T15:57:36.492 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:57:36.492 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:57:36.492 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:57:36.493 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:57:36.493 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:57:36.493 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:57:36.493 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:57:36.493 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:57:36.493 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:57:36.493 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:57:36.493 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:57:36.493 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:57:36.493 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:57:36.493 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:57:36.493 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:57:36.494 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:57:36.494 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:57:36.494 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:57:36.494 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:57:36.494 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000025", 2024-08-18T15:57:36.494 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:57:36.494 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000123", 2024-08-18T15:57:36.494 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.31", 2024-08-18T15:57:36.494 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:57:36.494 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:57:36.495 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:57:36.495 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:57:36.495 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.18", 2024-08-18T15:57:36.495 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:57:36.495 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:57:36.495 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:57:36.495 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:57:36.495 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:57:36.495 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000108", 2024-08-18T15:57:36.495 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:57:36.495 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:57:36.496 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000094", 2024-08-18T15:57:36.496 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:57:36.496 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:57:36.496 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:57:36.496 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:57:36.496 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:57:36.496 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:57:36.496 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000060", 2024-08-18T15:57:36.496 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:57:36.496 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:57:36.496 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:57:36.497 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000109", 2024-08-18T15:57:36.497 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.15", 2024-08-18T15:57:36.497 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:57:36.497 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:57:36.497 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:57:36.497 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:57:36.497 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:57:36.497 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:57:36.497 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:57:36.497 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:57:36.497 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:57:36.497 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000013", 2024-08-18T15:57:36.497 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:57:36.498 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:57:36.498 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:57:36.498 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:57:36.498 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:57:36.498 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:57:36.498 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:57:36.498 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:57:36.498 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:57:36.498 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:57:36.498 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:57:36.498 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:57:36.499 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:57:36.499 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:57:36.499 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000051", 2024-08-18T15:57:36.499 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:57:36.499 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:57:36.499 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:57:36.499 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:57:36.499 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000106", 2024-08-18T15:57:36.499 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:57:36.499 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:57:36.499 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:57:36.500 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:57:36.500 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:57:36.500 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000017", 2024-08-18T15:57:36.500 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:57:36.500 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:57:36.500 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:57:36.500 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000023", 2024-08-18T15:57:36.500 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:57:36.500 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:57:36.500 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:57:36.500 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:57:36.500 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:57:36.501 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:57:36.501 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:57:36.501 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000107", 2024-08-18T15:57:36.501 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:57:36.501 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:57:36.501 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000046", 2024-08-18T15:57:36.501 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:57:36.501 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:57:36.501 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:57:36.502 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.26", 2024-08-18T15:57:36.502 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:57:36.502 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:57:36.502 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9", 2024-08-18T15:57:36.502 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.22", 2024-08-18T15:57:36.502 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:57:36.502 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:57:36.502 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:57:36.502 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:57:36.502 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:57:36.502 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:57:36.503 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:57:36.503 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2", 2024-08-18T15:57:36.503 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:57:36.503 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:57:36.503 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000067", 2024-08-18T15:57:36.503 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:57:36.503 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:57:36.503 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:57:36.503 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:57:36.503 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:57:36.504 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:57:36.504 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:57:36.504 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:57:36.504 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:57:36.504 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:57:36.504 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:57:36.504 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:57:36.504 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:57:36.504 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:57:36.504 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:57:36.504 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:57:36.504 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:57:36.505 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:57:36.505 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:57:36.505 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.9", 2024-08-18T15:57:36.505 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000086", 2024-08-18T15:57:36.505 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.5", 2024-08-18T15:57:36.505 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:57:36.505 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:57:36.505 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:57:36.505 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:57:36.505 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:57:36.505 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:57:36.506 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2.MFXJ583gCr1.0", 2024-08-18T15:57:36.506 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:57:36.506 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1.GPnO16liyZ+.0", 2024-08-18T15:57:36.506 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:57:36.506 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:57:36.506 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:57:36.506 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:57:36.507 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:57:36.507 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:57:36.507 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:57:36.507 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:57:36.507 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:57:36.507 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.20", 2024-08-18T15:57:36.507 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:57:36.507 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:57:36.507 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:57:36.507 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:57:36.507 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:57:36.508 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:57:36.508 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1.retry", 2024-08-18T15:57:36.508 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:57:36.508 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:57:36.508 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:57:36.508 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0.retry", 2024-08-18T15:57:36.508 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000050", 2024-08-18T15:57:36.508 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:57:36.508 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000102", 2024-08-18T15:57:36.508 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:57:36.508 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:57:36.509 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:57:36.509 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:57:36.509 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:57:36.509 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:57:36.509 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:57:36.509 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:57:36.509 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:57:36.509 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:57:36.509 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:57:36.509 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000124", 2024-08-18T15:57:36.509 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:57:36.509 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:57:36.510 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:57:36.510 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000001", 2024-08-18T15:57:36.510 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:57:36.510 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:57:36.510 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:57:36.510 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:57:36.510 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000126", 2024-08-18T15:57:36.510 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:57:36.510 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0.fZhctfLXnSt.0", 2024-08-18T15:57:36.510 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:57:36.511 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000055", 2024-08-18T15:57:36.511 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:57:36.511 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000061", 2024-08-18T15:57:36.511 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1", 2024-08-18T15:57:36.511 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:57:36.512 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:57:36.512 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:57:36.512 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000097", 2024-08-18T15:57:36.512 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:57:36.512 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:57:36.512 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:57:36.512 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:57:36.512 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:57:36.512 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:57:36.512 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:57:36.512 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:57:36.513 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000042", 2024-08-18T15:57:36.513 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:57:36.513 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:57:36.513 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:57:36.513 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1.retry", 2024-08-18T15:57:36.513 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:57:36.513 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:57:36.513 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:57:36.513 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:57:36.513 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:57:36.513 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:57:36.514 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:57:36.514 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000075", 2024-08-18T15:57:36.514 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:57:36.514 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000120", 2024-08-18T15:57:36.514 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000030", 2024-08-18T15:57:36.514 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:57:36.514 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:57:36.514 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:57:36.514 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:57:36.514 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:57:36.514 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:57:36.515 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9" 2024-08-18T15:57:36.515 INFO:teuthology.orchestra.run.smithi193.stdout:] 2024-08-18T15:57:36.515 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:57:41.517 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:57:41.518 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:57:41.518 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:57:41.570 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:57:41.570 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:57:41.634 INFO:teuthology.orchestra.run.smithi193.stdout:[ 2024-08-18T15:57:41.635 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:57:41.635 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:57:41.635 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.29", 2024-08-18T15:57:41.635 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000078", 2024-08-18T15:57:41.635 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.history", 2024-08-18T15:57:41.635 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:57:41.635 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:57:41.635 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:57:41.635 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:57:41.635 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.23", 2024-08-18T15:57:41.636 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000070", 2024-08-18T15:57:41.636 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000104", 2024-08-18T15:57:41.636 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:57:41.636 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2.retry", 2024-08-18T15:57:41.636 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:57:41.636 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:57:41.636 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:57:41.636 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:57:41.636 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000026", 2024-08-18T15:57:41.636 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:57:41.637 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:57:41.637 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:57:41.637 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:57:41.637 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:57:41.637 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:57:41.637 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.8", 2024-08-18T15:57:41.637 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:57:41.637 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:57:41.637 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:57:41.637 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:57:41.637 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000028", 2024-08-18T15:57:41.638 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:57:41.638 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:57:41.638 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:57:41.638 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:57:41.638 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:57:41.638 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:57:41.638 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:57:41.638 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2.retry", 2024-08-18T15:57:41.638 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:57:41.638 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.30", 2024-08-18T15:57:41.639 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:57:41.639 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:57:41.639 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:57:41.639 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:57:41.639 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000040", 2024-08-18T15:57:41.639 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:57:41.639 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.6", 2024-08-18T15:57:41.639 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:57:41.639 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:57:41.639 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:57:41.639 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:57:41.640 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:57:41.640 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:57:41.640 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000015", 2024-08-18T15:57:41.640 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:57:41.640 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:57:41.640 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000069", 2024-08-18T15:57:41.640 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:57:41.640 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:57:41.640 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:57:41.640 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000095", 2024-08-18T15:57:41.640 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0", 2024-08-18T15:57:41.641 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000003", 2024-08-18T15:57:41.641 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000047", 2024-08-18T15:57:41.641 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:57:41.641 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:57:41.641 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:57:41.641 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:57:41.641 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:57:41.641 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:57:41.641 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:57:41.641 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:57:41.641 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:57:41.642 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000035", 2024-08-18T15:57:41.642 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:57:41.642 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:57:41.642 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000037", 2024-08-18T15:57:41.642 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:57:41.642 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:57:41.642 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:57:41.642 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:57:41.642 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:57:41.642 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:57:41.642 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:57:41.643 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:57:41.643 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:57:41.643 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:57:41.643 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:57:41.643 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:57:41.643 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:57:41.643 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:57:41.643 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000024", 2024-08-18T15:57:41.643 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:57:41.643 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:57:41.643 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:57:41.644 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:57:41.644 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000000", 2024-08-18T15:57:41.644 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:57:41.644 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000031", 2024-08-18T15:57:41.644 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:57:41.644 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:57:41.644 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:57:41.644 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:57:41.644 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:57:41.644 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:57:41.644 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3", 2024-08-18T15:57:41.645 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:57:41.645 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:57:41.645 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:57:41.645 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:57:41.645 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:57:41.645 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:57:41.645 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:57:41.645 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:57:41.645 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:57:41.645 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:57:41.646 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:57:41.646 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:57:41.646 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:57:41.646 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:57:41.646 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:57:41.646 INFO:teuthology.orchestra.run.smithi193.stdout: "data_loggenerations_metadata", 2024-08-18T15:57:41.646 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2", 2024-08-18T15:57:41.646 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:57:41.646 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:57:41.646 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:57:41.646 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:57:41.647 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T15:57:41.647 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:57:41.647 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:57:41.647 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:57:41.647 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:57:41.647 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:57:41.647 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000076", 2024-08-18T15:57:41.647 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:57:41.647 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:57:41.647 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000113", 2024-08-18T15:57:41.647 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000005", 2024-08-18T15:57:41.648 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:57:41.648 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000011", 2024-08-18T15:57:41.648 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:57:41.648 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000058", 2024-08-18T15:57:41.648 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:57:41.648 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:57:41.648 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:57:41.648 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000052", 2024-08-18T15:57:41.648 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:57:41.648 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:57:41.648 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:57:41.649 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:57:41.649 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.21", 2024-08-18T15:57:41.649 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:57:41.649 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:57:41.649 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:57:41.649 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:57:41.649 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:57:41.649 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:57:41.649 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:57:41.649 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:57:41.649 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:57:41.650 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:57:41.650 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:57:41.650 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:57:41.650 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:57:41.650 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:57:41.650 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:57:41.650 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:57:41.650 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:57:41.650 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:57:41.650 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:57:41.651 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000088", 2024-08-18T15:57:41.651 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000080", 2024-08-18T15:57:41.651 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:57:41.651 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:57:41.651 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000090", 2024-08-18T15:57:41.651 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:57:41.651 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:57:41.651 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000110", 2024-08-18T15:57:41.651 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:57:41.651 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:57:41.651 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:57:41.652 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.0", 2024-08-18T15:57:41.652 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:57:41.652 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000096", 2024-08-18T15:57:41.652 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:57:41.652 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:57:41.652 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:57:41.652 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:57:41.652 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:57:41.652 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000087", 2024-08-18T15:57:41.652 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:57:41.652 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:57:41.652 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:57:41.653 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:57:41.653 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000008", 2024-08-18T15:57:41.653 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:57:41.653 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:57:41.653 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:57:41.653 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:57:41.653 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:57:41.653 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000006", 2024-08-18T15:57:41.653 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000029", 2024-08-18T15:57:41.653 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:57:41.654 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000089", 2024-08-18T15:57:41.654 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:57:41.654 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:57:41.654 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000022", 2024-08-18T15:57:41.654 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:57:41.654 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:57:41.654 INFO:teuthology.orchestra.run.smithi193.stdout: "bilog.trim", 2024-08-18T15:57:41.654 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000016", 2024-08-18T15:57:41.654 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:57:41.654 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:57:41.654 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:57:41.655 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:57:41.655 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:57:41.655 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000041", 2024-08-18T15:57:41.655 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:57:41.655 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:57:41.655 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.log.01050e30-179b-43be-bdbb-3bb23cf37ae8.3", 2024-08-18T15:57:41.655 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:57:41.655 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000018", 2024-08-18T15:57:41.655 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000092", 2024-08-18T15:57:41.655 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:57:41.656 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:57:41.656 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:57:41.656 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:57:41.656 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.27", 2024-08-18T15:57:41.656 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.4", 2024-08-18T15:57:41.656 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:57:41.656 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:57:41.656 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:57:41.656 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:57:41.656 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:57:41.656 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:57:41.657 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:57:41.657 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000014", 2024-08-18T15:57:41.657 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:57:41.657 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3.retry", 2024-08-18T15:57:41.657 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:57:41.657 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:57:41.657 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:57:41.657 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:57:41.657 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:57:41.657 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000112", 2024-08-18T15:57:41.658 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:57:41.658 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000007", 2024-08-18T15:57:41.658 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000021", 2024-08-18T15:57:41.658 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:57:41.658 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:57:41.658 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:57:41.658 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:57:41.658 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:57:41.658 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.16", 2024-08-18T15:57:41.658 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:57:41.658 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000064", 2024-08-18T15:57:41.659 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:57:41.659 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:57:41.659 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:57:41.659 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:57:41.659 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:57:41.659 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:57:41.659 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:57:41.659 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000071", 2024-08-18T15:57:41.659 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:57:41.659 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:57:41.659 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:57:41.660 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000074", 2024-08-18T15:57:41.660 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:57:41.660 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:57:41.660 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:57:41.660 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:57:41.660 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:57:41.660 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:57:41.660 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:57:41.660 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:57:41.660 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:57:41.660 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:57:41.661 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:57:41.661 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:57:41.661 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:57:41.661 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:57:41.661 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:57:41.661 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000081", 2024-08-18T15:57:41.661 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:57:41.661 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000009", 2024-08-18T15:57:41.661 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000121", 2024-08-18T15:57:41.661 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1", 2024-08-18T15:57:41.661 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:57:41.662 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:57:41.662 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:57:41.662 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:57:41.662 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:57:41.662 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:57:41.662 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.17", 2024-08-18T15:57:41.662 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000125", 2024-08-18T15:57:41.662 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:57:41.662 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000082", 2024-08-18T15:57:41.662 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:57:41.662 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:57:41.663 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:57:41.663 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:57:41.663 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:57:41.663 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:57:41.663 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:57:41.663 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:57:41.663 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:57:41.663 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:57:41.663 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:57:41.663 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000105", 2024-08-18T15:57:41.663 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000059", 2024-08-18T15:57:41.664 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:57:41.664 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:57:41.664 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:57:41.664 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:57:41.664 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:57:41.664 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:57:41.664 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:57:41.664 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:57:41.664 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:57:41.664 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:57:41.664 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:57:41.665 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000077", 2024-08-18T15:57:41.665 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000032", 2024-08-18T15:57:41.665 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:57:41.665 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:57:41.665 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:57:41.665 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:57:41.665 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:57:41.665 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:57:41.665 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.24", 2024-08-18T15:57:41.665 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:57:41.665 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:57:41.666 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:57:41.666 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:57:41.666 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3.retry", 2024-08-18T15:57:41.666 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:57:41.666 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:57:41.666 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:57:41.666 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:57:41.666 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:57:41.666 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:57:41.666 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000053", 2024-08-18T15:57:41.667 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000091", 2024-08-18T15:57:41.667 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:57:41.667 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:57:41.667 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:57:41.667 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000065", 2024-08-18T15:57:41.667 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:57:41.667 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:57:41.667 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:57:41.667 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:57:41.667 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000083", 2024-08-18T15:57:41.667 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000010", 2024-08-18T15:57:41.668 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:57:41.668 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:57:41.668 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:57:41.668 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:57:41.668 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:57:41.668 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:57:41.668 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:57:41.668 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:57:41.668 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:57:41.668 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:57:41.668 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:57:41.669 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.7", 2024-08-18T15:57:41.669 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.3", 2024-08-18T15:57:41.669 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:57:41.669 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:57:41.669 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:57:41.669 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:57:41.669 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:57:41.669 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:57:41.669 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:57:41.669 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:57:41.669 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.2", 2024-08-18T15:57:41.670 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:57:41.670 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:57:41.670 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:57:41.670 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:57:41.670 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:57:41.670 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:57:41.670 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:57:41.670 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000045", 2024-08-18T15:57:41.670 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000002", 2024-08-18T15:57:41.670 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:57:41.671 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3.2UbiLidwSTy.0", 2024-08-18T15:57:41.671 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:57:41.671 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.12", 2024-08-18T15:57:41.671 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:57:41.671 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:57:41.671 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:57:41.671 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:57:41.671 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:57:41.671 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:57:41.671 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000116", 2024-08-18T15:57:41.672 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.25", 2024-08-18T15:57:41.672 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:57:41.672 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:57:41.672 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:57:41.672 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:57:41.672 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000034", 2024-08-18T15:57:41.672 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:57:41.672 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:57:41.672 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000101", 2024-08-18T15:57:41.672 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:57:41.672 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:57:41.673 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.13", 2024-08-18T15:57:41.673 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:57:41.673 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:57:41.673 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.1", 2024-08-18T15:57:41.673 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0.retry", 2024-08-18T15:57:41.673 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:57:41.673 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:57:41.673 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000079", 2024-08-18T15:57:41.673 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000049", 2024-08-18T15:57:41.673 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:57:41.673 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0", 2024-08-18T15:57:41.674 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:57:41.674 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:57:41.674 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000117", 2024-08-18T15:57:41.674 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:57:41.674 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000044", 2024-08-18T15:57:41.674 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:57:41.674 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:57:41.674 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:57:41.674 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:57:41.674 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:57:41.674 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:57:41.675 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.14", 2024-08-18T15:57:41.675 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:57:41.675 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000066", 2024-08-18T15:57:41.675 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:57:41.675 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:57:41.675 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:57:41.675 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:57:41.675 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:57:41.675 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000068", 2024-08-18T15:57:41.675 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:57:41.675 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000085", 2024-08-18T15:57:41.676 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:57:41.676 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:57:41.676 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:57:41.676 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:57:41.676 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:57:41.676 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:57:41.676 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:57:41.676 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:57:41.676 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:57:41.676 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:57:41.676 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:57:41.677 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:57:41.677 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000073", 2024-08-18T15:57:41.677 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:57:41.677 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:57:41.677 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:57:41.677 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000038", 2024-08-18T15:57:41.677 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:57:41.677 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:57:41.677 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:57:41.677 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:57:41.678 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.10", 2024-08-18T15:57:41.678 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:57:41.678 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:57:41.678 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:57:41.678 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000118", 2024-08-18T15:57:41.678 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:57:41.678 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3", 2024-08-18T15:57:41.678 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:57:41.678 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000036", 2024-08-18T15:57:41.678 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:57:41.678 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:57:41.679 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:57:41.679 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:57:41.679 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:57:41.679 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000103", 2024-08-18T15:57:41.679 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.3", 2024-08-18T15:57:41.679 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:57:41.679 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000119", 2024-08-18T15:57:41.679 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000098", 2024-08-18T15:57:41.679 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:57:41.679 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:57:41.680 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:57:41.680 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:57:41.680 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:57:41.680 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:57:41.680 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:57:41.680 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:57:41.680 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.0", 2024-08-18T15:57:41.680 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:57:41.680 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:57:41.680 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:57:41.680 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:57:41.681 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:57:41.681 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:57:41.681 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:57:41.681 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:57:41.681 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000027", 2024-08-18T15:57:41.681 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:57:41.681 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status", 2024-08-18T15:57:41.681 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:57:41.681 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000019", 2024-08-18T15:57:41.681 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:57:41.681 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:57:41.681 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:57:41.682 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:57:41.682 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1", 2024-08-18T15:57:41.682 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:57:41.682 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:57:41.682 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:57:41.682 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:57:41.682 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000039", 2024-08-18T15:57:41.682 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:57:41.682 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:57:41.682 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:57:41.682 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:57:41.683 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:57:41.683 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:57:41.683 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:57:41.683 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:57:41.683 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:57:41.683 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:57:41.683 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:57:41.683 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000100", 2024-08-18T15:57:41.683 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:57:41.683 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:57:41.683 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:57:41.684 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:57:41.684 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:57:41.684 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000093", 2024-08-18T15:57:41.684 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:57:41.684 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000004", 2024-08-18T15:57:41.684 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000063", 2024-08-18T15:57:41.684 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000122", 2024-08-18T15:57:41.684 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:57:41.684 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:57:41.684 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:57:41.684 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000057", 2024-08-18T15:57:41.685 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:57:41.685 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:57:41.685 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:57:41.685 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:57:41.685 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000054", 2024-08-18T15:57:41.685 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:57:41.685 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000114", 2024-08-18T15:57:41.685 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:57:41.685 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:57:41.685 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:57:41.685 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.2", 2024-08-18T15:57:41.686 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:57:41.686 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:57:41.686 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0", 2024-08-18T15:57:41.686 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:57:41.686 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:57:41.686 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:57:41.686 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:57:41.686 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:57:41.686 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:57:41.686 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000012", 2024-08-18T15:57:41.687 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:57:41.687 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:57:41.687 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:57:41.687 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000084", 2024-08-18T15:57:41.687 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000043", 2024-08-18T15:57:41.687 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3", 2024-08-18T15:57:41.687 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:57:41.687 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:57:41.687 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.11", 2024-08-18T15:57:41.687 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:57:41.687 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:57:41.688 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:57:41.688 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:57:41.688 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:57:41.688 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:57:41.688 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:57:41.688 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:57:41.688 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000111", 2024-08-18T15:57:41.688 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:57:41.688 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000048", 2024-08-18T15:57:41.688 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:57:41.689 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000020", 2024-08-18T15:57:41.689 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:57:41.689 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:57:41.689 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:57:41.689 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:57:41.689 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:57:41.689 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000099", 2024-08-18T15:57:41.689 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000056", 2024-08-18T15:57:41.689 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:57:41.689 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:57:41.689 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.19", 2024-08-18T15:57:41.690 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:57:41.690 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.28", 2024-08-18T15:57:41.690 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:57:41.690 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:57:41.690 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:57:41.690 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:57:41.690 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:57:41.690 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000072", 2024-08-18T15:57:41.690 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:57:41.690 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:57:41.691 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000062", 2024-08-18T15:57:41.691 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:57:41.691 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:57:41.691 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:57:41.691 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000115", 2024-08-18T15:57:41.691 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:57:41.691 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.1", 2024-08-18T15:57:41.691 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2", 2024-08-18T15:57:41.691 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000033", 2024-08-18T15:57:41.691 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:57:41.692 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:57:41.692 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:57:41.692 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:57:41.692 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:57:41.692 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:57:41.692 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:57:41.692 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:57:41.692 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:57:41.692 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:57:41.692 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:57:41.693 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:57:41.693 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:57:41.693 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:57:41.693 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:57:41.693 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:57:41.693 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:57:41.693 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:57:41.693 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:57:41.693 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000025", 2024-08-18T15:57:41.693 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:57:41.694 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000123", 2024-08-18T15:57:41.694 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.31", 2024-08-18T15:57:41.694 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:57:41.694 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:57:41.694 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:57:41.694 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:57:41.694 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.18", 2024-08-18T15:57:41.694 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:57:41.694 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:57:41.694 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:57:41.695 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:57:41.695 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:57:41.695 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000108", 2024-08-18T15:57:41.695 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:57:41.695 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:57:41.695 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000094", 2024-08-18T15:57:41.695 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:57:41.695 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:57:41.695 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:57:41.695 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:57:41.696 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:57:41.696 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:57:41.696 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000060", 2024-08-18T15:57:41.696 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:57:41.696 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:57:41.696 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:57:41.696 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000109", 2024-08-18T15:57:41.696 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.15", 2024-08-18T15:57:41.696 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:57:41.696 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:57:41.697 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:57:41.697 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:57:41.697 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:57:41.697 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:57:41.697 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:57:41.697 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:57:41.697 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:57:41.697 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000013", 2024-08-18T15:57:41.697 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:57:41.697 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:57:41.698 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:57:41.698 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:57:41.698 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:57:41.698 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:57:41.698 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:57:41.698 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:57:41.698 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:57:41.698 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:57:41.698 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:57:41.699 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:57:41.699 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:57:41.699 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:57:41.699 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000051", 2024-08-18T15:57:41.699 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:57:41.699 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:57:41.699 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:57:41.699 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:57:41.699 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000106", 2024-08-18T15:57:41.699 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:57:41.699 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:57:41.700 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:57:41.700 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:57:41.700 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:57:41.700 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000017", 2024-08-18T15:57:41.700 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:57:41.700 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:57:41.700 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:57:41.700 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000023", 2024-08-18T15:57:41.700 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:57:41.700 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:57:41.701 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:57:41.701 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:57:41.701 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:57:41.701 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:57:41.701 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:57:41.701 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000107", 2024-08-18T15:57:41.701 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:57:41.701 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:57:41.701 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000046", 2024-08-18T15:57:41.701 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:57:41.701 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:57:41.702 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:57:41.702 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.26", 2024-08-18T15:57:41.702 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:57:41.702 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:57:41.702 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9", 2024-08-18T15:57:41.702 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.22", 2024-08-18T15:57:41.702 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:57:41.702 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:57:41.702 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:57:41.702 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:57:41.702 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:57:41.703 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:57:41.703 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:57:41.703 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2", 2024-08-18T15:57:41.703 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:57:41.703 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:57:41.703 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000067", 2024-08-18T15:57:41.703 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:57:41.704 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:57:41.704 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:57:41.704 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:57:41.704 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:57:41.704 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:57:41.704 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:57:41.704 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:57:41.704 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:57:41.704 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:57:41.704 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:57:41.705 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:57:41.705 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:57:41.705 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:57:41.705 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:57:41.705 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:57:41.705 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:57:41.705 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:57:41.705 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:57:41.705 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.9", 2024-08-18T15:57:41.705 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000086", 2024-08-18T15:57:41.705 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.5", 2024-08-18T15:57:41.706 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:57:41.706 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:57:41.706 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:57:41.706 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:57:41.706 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:57:41.706 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:57:41.706 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2.MFXJ583gCr1.0", 2024-08-18T15:57:41.706 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:57:41.706 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1.GPnO16liyZ+.0", 2024-08-18T15:57:41.706 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:57:41.706 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:57:41.707 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:57:41.707 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:57:41.707 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:57:41.707 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:57:41.707 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:57:41.707 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:57:41.707 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:57:41.707 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.20", 2024-08-18T15:57:41.707 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:57:41.707 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:57:41.707 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:57:41.708 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:57:41.708 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:57:41.708 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:57:41.708 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1.retry", 2024-08-18T15:57:41.708 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:57:41.708 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:57:41.708 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:57:41.708 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0.retry", 2024-08-18T15:57:41.708 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000050", 2024-08-18T15:57:41.708 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:57:41.709 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000102", 2024-08-18T15:57:41.709 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:57:41.709 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:57:41.709 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:57:41.709 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:57:41.709 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:57:41.709 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:57:41.709 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:57:41.709 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:57:41.709 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:57:41.710 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:57:41.710 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:57:41.710 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000124", 2024-08-18T15:57:41.710 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:57:41.710 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:57:41.710 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:57:41.710 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000001", 2024-08-18T15:57:41.710 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:57:41.710 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:57:41.710 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:57:41.711 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:57:41.711 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000126", 2024-08-18T15:57:41.711 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:57:41.711 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0.fZhctfLXnSt.0", 2024-08-18T15:57:41.711 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:57:41.711 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000055", 2024-08-18T15:57:41.711 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:57:41.711 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000061", 2024-08-18T15:57:41.711 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1", 2024-08-18T15:57:41.711 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:57:41.712 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:57:41.712 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:57:41.712 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000097", 2024-08-18T15:57:41.712 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:57:41.712 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:57:41.712 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:57:41.712 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:57:41.712 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:57:41.712 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:57:41.712 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:57:41.712 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:57:41.713 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000042", 2024-08-18T15:57:41.713 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:57:41.713 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:57:41.713 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:57:41.713 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1.retry", 2024-08-18T15:57:41.713 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:57:41.713 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:57:41.713 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:57:41.713 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:57:41.713 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:57:41.713 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:57:41.714 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:57:41.714 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000075", 2024-08-18T15:57:41.714 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:57:41.714 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000120", 2024-08-18T15:57:41.714 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000030", 2024-08-18T15:57:41.714 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:57:41.714 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:57:41.714 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:57:41.714 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:57:41.714 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:57:41.715 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:57:41.715 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9" 2024-08-18T15:57:41.715 INFO:teuthology.orchestra.run.smithi193.stdout:] 2024-08-18T15:57:41.715 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:57:46.717 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:57:46.717 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:57:46.717 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:57:46.769 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:57:46.769 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:57:46.835 INFO:teuthology.orchestra.run.smithi193.stdout:[ 2024-08-18T15:57:46.835 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:57:46.835 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:57:46.835 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.29", 2024-08-18T15:57:46.836 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000078", 2024-08-18T15:57:46.836 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.history", 2024-08-18T15:57:46.836 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:57:46.836 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:57:46.836 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:57:46.836 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:57:46.836 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.23", 2024-08-18T15:57:46.836 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000070", 2024-08-18T15:57:46.836 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000104", 2024-08-18T15:57:46.836 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:57:46.836 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2.retry", 2024-08-18T15:57:46.836 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:57:46.837 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:57:46.837 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:57:46.837 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:57:46.837 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000026", 2024-08-18T15:57:46.837 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:57:46.837 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:57:46.837 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:57:46.837 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:57:46.837 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:57:46.837 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:57:46.837 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.8", 2024-08-18T15:57:46.837 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:57:46.837 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:57:46.838 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:57:46.838 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:57:46.838 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000028", 2024-08-18T15:57:46.838 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:57:46.838 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:57:46.838 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:57:46.838 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:57:46.838 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:57:46.838 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:57:46.838 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:57:46.838 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2.retry", 2024-08-18T15:57:46.838 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:57:46.839 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.30", 2024-08-18T15:57:46.839 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:57:46.839 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:57:46.839 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:57:46.839 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:57:46.839 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000040", 2024-08-18T15:57:46.839 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:57:46.839 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.6", 2024-08-18T15:57:46.839 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:57:46.839 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:57:46.839 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:57:46.839 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:57:46.839 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:57:46.840 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:57:46.840 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000015", 2024-08-18T15:57:46.840 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:57:46.840 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:57:46.840 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000069", 2024-08-18T15:57:46.840 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:57:46.840 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:57:46.840 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:57:46.840 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000095", 2024-08-18T15:57:46.840 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0", 2024-08-18T15:57:46.840 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000003", 2024-08-18T15:57:46.840 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000047", 2024-08-18T15:57:46.840 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:57:46.841 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:57:46.841 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:57:46.841 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:57:46.841 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:57:46.841 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:57:46.841 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:57:46.841 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:57:46.841 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:57:46.841 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000035", 2024-08-18T15:57:46.841 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:57:46.841 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:57:46.841 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000037", 2024-08-18T15:57:46.841 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:57:46.841 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:57:46.842 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:57:46.842 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:57:46.842 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:57:46.842 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:57:46.842 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:57:46.842 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:57:46.842 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:57:46.842 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:57:46.842 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:57:46.842 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:57:46.842 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:57:46.842 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:57:46.843 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000024", 2024-08-18T15:57:46.843 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:57:46.843 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:57:46.843 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:57:46.843 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:57:46.843 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000000", 2024-08-18T15:57:46.843 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:57:46.843 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000031", 2024-08-18T15:57:46.843 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:57:46.843 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:57:46.843 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:57:46.843 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:57:46.843 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:57:46.843 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:57:46.844 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3", 2024-08-18T15:57:46.844 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:57:46.844 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:57:46.844 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:57:46.844 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:57:46.844 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:57:46.844 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:57:46.844 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:57:46.844 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:57:46.844 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:57:46.844 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:57:46.844 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:57:46.844 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:57:46.845 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:57:46.845 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:57:46.845 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:57:46.845 INFO:teuthology.orchestra.run.smithi193.stdout: "data_loggenerations_metadata", 2024-08-18T15:57:46.845 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2", 2024-08-18T15:57:46.845 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:57:46.845 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:57:46.845 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:57:46.845 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:57:46.845 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T15:57:46.845 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:57:46.845 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:57:46.845 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:57:46.846 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:57:46.846 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:57:46.846 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000076", 2024-08-18T15:57:46.846 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:57:46.846 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:57:46.846 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000113", 2024-08-18T15:57:46.846 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000005", 2024-08-18T15:57:46.846 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:57:46.846 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000011", 2024-08-18T15:57:46.846 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:57:46.846 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000058", 2024-08-18T15:57:46.846 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:57:46.846 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:57:46.847 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:57:46.847 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000052", 2024-08-18T15:57:46.847 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:57:46.847 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:57:46.847 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:57:46.847 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:57:46.847 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.21", 2024-08-18T15:57:46.847 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:57:46.847 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:57:46.847 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:57:46.847 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:57:46.847 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:57:46.847 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:57:46.847 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:57:46.848 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:57:46.848 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:57:46.848 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:57:46.848 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:57:46.848 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:57:46.848 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:57:46.848 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:57:46.848 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:57:46.848 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:57:46.848 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:57:46.848 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:57:46.849 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:57:46.849 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000088", 2024-08-18T15:57:46.849 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000080", 2024-08-18T15:57:46.849 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:57:46.849 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:57:46.849 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000090", 2024-08-18T15:57:46.849 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:57:46.849 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:57:46.849 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000110", 2024-08-18T15:57:46.849 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:57:46.849 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:57:46.849 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:57:46.850 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.0", 2024-08-18T15:57:46.850 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:57:46.850 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000096", 2024-08-18T15:57:46.850 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:57:46.850 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:57:46.850 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:57:46.850 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:57:46.850 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:57:46.850 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000087", 2024-08-18T15:57:46.850 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:57:46.850 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:57:46.851 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:57:46.851 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:57:46.851 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000008", 2024-08-18T15:57:46.851 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:57:46.851 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:57:46.851 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:57:46.851 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:57:46.851 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:57:46.851 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000006", 2024-08-18T15:57:46.851 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000029", 2024-08-18T15:57:46.851 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:57:46.851 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000089", 2024-08-18T15:57:46.851 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:57:46.852 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:57:46.852 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000022", 2024-08-18T15:57:46.852 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:57:46.852 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:57:46.852 INFO:teuthology.orchestra.run.smithi193.stdout: "bilog.trim", 2024-08-18T15:57:46.852 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000016", 2024-08-18T15:57:46.852 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:57:46.852 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:57:46.852 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:57:46.852 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:57:46.852 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:57:46.852 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000041", 2024-08-18T15:57:46.852 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:57:46.853 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:57:46.853 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.log.01050e30-179b-43be-bdbb-3bb23cf37ae8.3", 2024-08-18T15:57:46.853 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:57:46.853 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000018", 2024-08-18T15:57:46.853 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000092", 2024-08-18T15:57:46.853 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:57:46.853 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:57:46.853 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:57:46.853 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:57:46.853 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.27", 2024-08-18T15:57:46.853 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.4", 2024-08-18T15:57:46.853 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:57:46.853 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:57:46.853 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:57:46.854 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:57:46.854 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:57:46.854 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:57:46.854 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:57:46.854 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000014", 2024-08-18T15:57:46.854 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:57:46.854 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3.retry", 2024-08-18T15:57:46.854 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:57:46.854 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:57:46.854 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:57:46.854 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:57:46.854 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:57:46.855 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000112", 2024-08-18T15:57:46.855 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:57:46.855 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000007", 2024-08-18T15:57:46.855 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000021", 2024-08-18T15:57:46.855 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:57:46.855 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:57:46.855 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:57:46.855 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:57:46.855 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:57:46.855 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.16", 2024-08-18T15:57:46.855 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:57:46.856 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000064", 2024-08-18T15:57:46.856 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:57:46.856 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:57:46.856 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:57:46.856 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:57:46.856 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:57:46.856 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:57:46.856 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:57:46.856 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000071", 2024-08-18T15:57:46.856 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:57:46.856 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:57:46.856 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:57:46.856 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000074", 2024-08-18T15:57:46.857 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:57:46.857 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:57:46.857 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:57:46.857 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:57:46.857 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:57:46.857 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:57:46.857 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:57:46.857 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:57:46.857 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:57:46.857 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:57:46.857 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:57:46.857 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:57:46.857 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:57:46.858 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:57:46.858 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:57:46.858 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000081", 2024-08-18T15:57:46.858 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:57:46.858 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000009", 2024-08-18T15:57:46.858 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000121", 2024-08-18T15:57:46.858 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1", 2024-08-18T15:57:46.858 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:57:46.858 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:57:46.858 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:57:46.858 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:57:46.858 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:57:46.858 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:57:46.859 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.17", 2024-08-18T15:57:46.859 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000125", 2024-08-18T15:57:46.859 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:57:46.859 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000082", 2024-08-18T15:57:46.859 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:57:46.859 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:57:46.859 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:57:46.859 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:57:46.859 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:57:46.859 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:57:46.859 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:57:46.859 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:57:46.859 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:57:46.859 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:57:46.860 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:57:46.860 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000105", 2024-08-18T15:57:46.860 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000059", 2024-08-18T15:57:46.860 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:57:46.860 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:57:46.860 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:57:46.860 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:57:46.860 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:57:46.860 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:57:46.860 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:57:46.860 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:57:46.860 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:57:46.860 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:57:46.860 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:57:46.861 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000077", 2024-08-18T15:57:46.861 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000032", 2024-08-18T15:57:46.861 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:57:46.861 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:57:46.861 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:57:46.861 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:57:46.861 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:57:46.861 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:57:46.861 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.24", 2024-08-18T15:57:46.861 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:57:46.861 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:57:46.861 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:57:46.861 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:57:46.862 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3.retry", 2024-08-18T15:57:46.862 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:57:46.862 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:57:46.862 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:57:46.862 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:57:46.862 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:57:46.862 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:57:46.862 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000053", 2024-08-18T15:57:46.862 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000091", 2024-08-18T15:57:46.862 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:57:46.862 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:57:46.862 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:57:46.862 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000065", 2024-08-18T15:57:46.863 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:57:46.863 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:57:46.863 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:57:46.863 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:57:46.863 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000083", 2024-08-18T15:57:46.863 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000010", 2024-08-18T15:57:46.863 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:57:46.863 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:57:46.863 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:57:46.863 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:57:46.863 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:57:46.863 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:57:46.863 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:57:46.864 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:57:46.864 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:57:46.864 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:57:46.864 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:57:46.864 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.7", 2024-08-18T15:57:46.864 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.3", 2024-08-18T15:57:46.864 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:57:46.864 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:57:46.864 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:57:46.864 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:57:46.864 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:57:46.864 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:57:46.864 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:57:46.865 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:57:46.865 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.2", 2024-08-18T15:57:46.865 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:57:46.865 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:57:46.865 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:57:46.865 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:57:46.865 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:57:46.865 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:57:46.865 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:57:46.865 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000045", 2024-08-18T15:57:46.865 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000002", 2024-08-18T15:57:46.865 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:57:46.866 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3.2UbiLidwSTy.0", 2024-08-18T15:57:46.866 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:57:46.866 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.12", 2024-08-18T15:57:46.866 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:57:46.866 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:57:46.866 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:57:46.866 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:57:46.866 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:57:46.866 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:57:46.866 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000116", 2024-08-18T15:57:46.866 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.25", 2024-08-18T15:57:46.867 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:57:46.867 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:57:46.867 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:57:46.867 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:57:46.867 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000034", 2024-08-18T15:57:46.867 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:57:46.867 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:57:46.867 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000101", 2024-08-18T15:57:46.867 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:57:46.867 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:57:46.867 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.13", 2024-08-18T15:57:46.867 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:57:46.868 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:57:46.868 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.1", 2024-08-18T15:57:46.868 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0.retry", 2024-08-18T15:57:46.868 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:57:46.868 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:57:46.868 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000079", 2024-08-18T15:57:46.868 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000049", 2024-08-18T15:57:46.868 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:57:46.868 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0", 2024-08-18T15:57:46.868 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:57:46.868 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:57:46.868 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000117", 2024-08-18T15:57:46.868 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:57:46.868 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000044", 2024-08-18T15:57:46.869 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:57:46.869 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:57:46.869 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:57:46.869 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:57:46.869 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:57:46.869 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:57:46.869 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.14", 2024-08-18T15:57:46.869 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:57:46.869 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000066", 2024-08-18T15:57:46.869 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:57:46.869 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:57:46.869 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:57:46.869 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:57:46.870 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:57:46.870 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000068", 2024-08-18T15:57:46.870 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:57:46.870 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000085", 2024-08-18T15:57:46.870 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:57:46.870 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:57:46.870 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:57:46.870 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:57:46.870 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:57:46.870 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:57:46.870 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:57:46.870 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:57:46.870 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:57:46.871 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:57:46.871 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:57:46.871 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:57:46.871 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000073", 2024-08-18T15:57:46.871 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:57:46.871 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:57:46.871 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:57:46.871 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000038", 2024-08-18T15:57:46.871 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:57:46.871 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:57:46.871 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:57:46.871 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:57:46.872 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.10", 2024-08-18T15:57:46.872 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:57:46.872 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:57:46.872 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:57:46.872 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000118", 2024-08-18T15:57:46.872 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:57:46.872 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3", 2024-08-18T15:57:46.872 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:57:46.872 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000036", 2024-08-18T15:57:46.872 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:57:46.872 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:57:46.872 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:57:46.873 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:57:46.873 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:57:46.873 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000103", 2024-08-18T15:57:46.873 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.3", 2024-08-18T15:57:46.873 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:57:46.873 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000119", 2024-08-18T15:57:46.873 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000098", 2024-08-18T15:57:46.873 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:57:46.873 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:57:46.873 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:57:46.873 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:57:46.873 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:57:46.873 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:57:46.874 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:57:46.874 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:57:46.874 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.0", 2024-08-18T15:57:46.874 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:57:46.874 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:57:46.874 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:57:46.874 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:57:46.874 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:57:46.874 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:57:46.874 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:57:46.875 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:57:46.875 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000027", 2024-08-18T15:57:46.875 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:57:46.875 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status", 2024-08-18T15:57:46.875 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:57:46.875 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000019", 2024-08-18T15:57:46.875 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:57:46.875 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:57:46.875 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:57:46.875 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:57:46.875 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1", 2024-08-18T15:57:46.875 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:57:46.875 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:57:46.876 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:57:46.876 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:57:46.876 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000039", 2024-08-18T15:57:46.876 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:57:46.876 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:57:46.876 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:57:46.876 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:57:46.876 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:57:46.876 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:57:46.876 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:57:46.876 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:57:46.876 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:57:46.876 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:57:46.876 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:57:46.877 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000100", 2024-08-18T15:57:46.877 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:57:46.877 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:57:46.877 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:57:46.877 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:57:46.877 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:57:46.877 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000093", 2024-08-18T15:57:46.877 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:57:46.877 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000004", 2024-08-18T15:57:46.877 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000063", 2024-08-18T15:57:46.877 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000122", 2024-08-18T15:57:46.877 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:57:46.877 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:57:46.877 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:57:46.878 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000057", 2024-08-18T15:57:46.878 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:57:46.878 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:57:46.878 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:57:46.878 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:57:46.878 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000054", 2024-08-18T15:57:46.878 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:57:46.878 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000114", 2024-08-18T15:57:46.878 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:57:46.878 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:57:46.878 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:57:46.879 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.2", 2024-08-18T15:57:46.879 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:57:46.879 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:57:46.879 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0", 2024-08-18T15:57:46.879 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:57:46.879 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:57:46.879 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:57:46.879 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:57:46.879 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:57:46.879 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:57:46.879 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000012", 2024-08-18T15:57:46.879 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:57:46.879 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:57:46.880 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:57:46.880 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000084", 2024-08-18T15:57:46.880 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000043", 2024-08-18T15:57:46.880 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3", 2024-08-18T15:57:46.880 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:57:46.880 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:57:46.880 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.11", 2024-08-18T15:57:46.880 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:57:46.880 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:57:46.880 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:57:46.880 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:57:46.880 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:57:46.880 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:57:46.881 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:57:46.881 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:57:46.881 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000111", 2024-08-18T15:57:46.881 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:57:46.881 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000048", 2024-08-18T15:57:46.881 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:57:46.881 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000020", 2024-08-18T15:57:46.881 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:57:46.881 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:57:46.881 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:57:46.881 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:57:46.881 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:57:46.881 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000099", 2024-08-18T15:57:46.882 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000056", 2024-08-18T15:57:46.882 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:57:46.882 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:57:46.882 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.19", 2024-08-18T15:57:46.882 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:57:46.882 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.28", 2024-08-18T15:57:46.882 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:57:46.882 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:57:46.882 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:57:46.882 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:57:46.882 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:57:46.882 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000072", 2024-08-18T15:57:46.883 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:57:46.883 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:57:46.883 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000062", 2024-08-18T15:57:46.883 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:57:46.883 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:57:46.883 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:57:46.883 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000115", 2024-08-18T15:57:46.883 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:57:46.883 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.1", 2024-08-18T15:57:46.883 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2", 2024-08-18T15:57:46.883 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000033", 2024-08-18T15:57:46.883 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:57:46.883 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:57:46.883 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:57:46.884 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:57:46.884 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:57:46.884 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:57:46.884 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:57:46.884 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:57:46.884 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:57:46.884 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:57:46.884 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:57:46.884 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:57:46.884 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:57:46.884 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:57:46.884 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:57:46.884 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:57:46.885 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:57:46.885 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:57:46.885 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:57:46.885 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000025", 2024-08-18T15:57:46.885 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:57:46.885 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000123", 2024-08-18T15:57:46.885 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.31", 2024-08-18T15:57:46.885 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:57:46.885 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:57:46.885 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:57:46.885 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:57:46.885 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.18", 2024-08-18T15:57:46.885 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:57:46.886 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:57:46.886 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:57:46.886 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:57:46.886 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:57:46.886 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000108", 2024-08-18T15:57:46.886 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:57:46.886 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:57:46.886 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000094", 2024-08-18T15:57:46.886 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:57:46.886 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:57:46.886 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:57:46.886 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:57:46.886 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:57:46.887 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:57:46.887 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000060", 2024-08-18T15:57:46.887 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:57:46.887 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:57:46.887 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:57:46.887 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000109", 2024-08-18T15:57:46.887 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.15", 2024-08-18T15:57:46.887 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:57:46.887 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:57:46.887 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:57:46.887 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:57:46.887 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:57:46.887 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:57:46.887 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:57:46.888 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:57:46.888 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:57:46.888 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000013", 2024-08-18T15:57:46.888 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:57:46.888 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:57:46.888 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:57:46.888 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:57:46.888 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:57:46.888 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:57:46.888 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:57:46.888 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:57:46.888 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:57:46.888 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:57:46.889 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:57:46.889 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:57:46.889 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:57:46.889 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:57:46.889 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000051", 2024-08-18T15:57:46.889 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:57:46.889 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:57:46.889 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:57:46.889 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:57:46.889 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000106", 2024-08-18T15:57:46.889 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:57:46.889 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:57:46.889 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:57:46.889 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:57:46.890 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:57:46.890 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000017", 2024-08-18T15:57:46.890 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:57:46.890 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:57:46.890 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:57:46.890 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000023", 2024-08-18T15:57:46.890 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:57:46.890 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:57:46.890 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:57:46.890 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:57:46.890 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:57:46.890 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:57:46.891 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:57:46.891 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000107", 2024-08-18T15:57:46.891 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:57:46.891 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:57:46.891 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000046", 2024-08-18T15:57:46.891 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:57:46.891 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:57:46.891 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:57:46.891 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.26", 2024-08-18T15:57:46.891 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:57:46.891 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:57:46.891 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9", 2024-08-18T15:57:46.891 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.22", 2024-08-18T15:57:46.891 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:57:46.892 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:57:46.892 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:57:46.892 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:57:46.892 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:57:46.892 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:57:46.892 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:57:46.892 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2", 2024-08-18T15:57:46.892 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:57:46.892 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:57:46.892 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000067", 2024-08-18T15:57:46.892 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:57:46.892 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:57:46.893 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:57:46.893 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:57:46.893 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:57:46.893 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:57:46.893 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:57:46.893 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:57:46.893 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:57:46.893 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:57:46.893 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:57:46.893 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:57:46.893 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:57:46.893 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:57:46.894 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:57:46.894 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:57:46.894 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:57:46.894 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:57:46.894 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:57:46.894 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.9", 2024-08-18T15:57:46.894 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000086", 2024-08-18T15:57:46.894 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.5", 2024-08-18T15:57:46.894 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:57:46.894 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:57:46.894 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:57:46.894 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:57:46.894 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:57:46.895 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:57:46.895 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2.MFXJ583gCr1.0", 2024-08-18T15:57:46.895 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:57:46.895 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1.GPnO16liyZ+.0", 2024-08-18T15:57:46.895 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:57:46.895 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:57:46.895 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:57:46.895 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:57:46.895 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:57:46.895 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:57:46.895 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:57:46.895 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:57:46.895 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:57:46.896 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.20", 2024-08-18T15:57:46.896 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:57:46.896 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:57:46.896 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:57:46.896 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:57:46.896 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:57:46.896 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:57:46.896 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1.retry", 2024-08-18T15:57:46.896 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:57:46.896 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:57:46.896 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:57:46.896 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0.retry", 2024-08-18T15:57:46.896 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000050", 2024-08-18T15:57:46.897 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:57:46.897 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000102", 2024-08-18T15:57:46.897 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:57:46.897 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:57:46.897 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:57:46.897 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:57:46.897 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:57:46.897 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:57:46.897 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:57:46.897 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:57:46.897 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:57:46.897 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:57:46.897 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:57:46.897 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000124", 2024-08-18T15:57:46.897 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:57:46.898 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:57:46.898 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:57:46.898 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000001", 2024-08-18T15:57:46.898 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:57:46.898 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:57:46.898 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:57:46.898 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:57:46.898 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000126", 2024-08-18T15:57:46.898 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:57:46.898 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0.fZhctfLXnSt.0", 2024-08-18T15:57:46.898 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:57:46.899 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000055", 2024-08-18T15:57:46.899 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:57:46.899 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000061", 2024-08-18T15:57:46.899 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1", 2024-08-18T15:57:46.899 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:57:46.899 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:57:46.899 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:57:46.899 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000097", 2024-08-18T15:57:46.899 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:57:46.899 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:57:46.900 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:57:46.900 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:57:46.900 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:57:46.900 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:57:46.900 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:57:46.900 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:57:46.900 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000042", 2024-08-18T15:57:46.900 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:57:46.900 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:57:46.900 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:57:46.900 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1.retry", 2024-08-18T15:57:46.900 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:57:46.900 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:57:46.901 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:57:46.901 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:57:46.901 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:57:46.901 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:57:46.901 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:57:46.901 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000075", 2024-08-18T15:57:46.901 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:57:46.901 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000120", 2024-08-18T15:57:46.901 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000030", 2024-08-18T15:57:46.901 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:57:46.901 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:57:46.901 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:57:46.901 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:57:46.902 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:57:46.902 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:57:46.902 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9" 2024-08-18T15:57:46.902 INFO:teuthology.orchestra.run.smithi193.stdout:] 2024-08-18T15:57:46.902 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:57:51.904 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:57:51.904 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:57:51.905 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:57:51.957 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:57:51.958 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:57:52.023 INFO:teuthology.orchestra.run.smithi193.stdout:[ 2024-08-18T15:57:52.023 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:57:52.023 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:57:52.023 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.29", 2024-08-18T15:57:52.023 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000078", 2024-08-18T15:57:52.023 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.history", 2024-08-18T15:57:52.023 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:57:52.024 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:57:52.024 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:57:52.024 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:57:52.024 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.23", 2024-08-18T15:57:52.025 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000070", 2024-08-18T15:57:52.025 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000104", 2024-08-18T15:57:52.025 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:57:52.025 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2.retry", 2024-08-18T15:57:52.025 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:57:52.025 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:57:52.025 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:57:52.025 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:57:52.026 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000026", 2024-08-18T15:57:52.026 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:57:52.026 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:57:52.026 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:57:52.026 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:57:52.026 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:57:52.027 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:57:52.027 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.8", 2024-08-18T15:57:52.027 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:57:52.027 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:57:52.027 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:57:52.028 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:57:52.028 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000028", 2024-08-18T15:57:52.028 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:57:52.028 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:57:52.028 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:57:52.028 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:57:52.028 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:57:52.028 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:57:52.029 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:57:52.029 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2.retry", 2024-08-18T15:57:52.029 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:57:52.029 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.30", 2024-08-18T15:57:52.029 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:57:52.029 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:57:52.029 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:57:52.029 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:57:52.030 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000040", 2024-08-18T15:57:52.030 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:57:52.030 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.6", 2024-08-18T15:57:52.030 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:57:52.030 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:57:52.030 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:57:52.030 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:57:52.031 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:57:52.031 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:57:52.031 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000015", 2024-08-18T15:57:52.031 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:57:52.031 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:57:52.031 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000069", 2024-08-18T15:57:52.031 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:57:52.031 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:57:52.032 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:57:52.032 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000095", 2024-08-18T15:57:52.032 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0", 2024-08-18T15:57:52.032 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000003", 2024-08-18T15:57:52.032 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000047", 2024-08-18T15:57:52.032 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:57:52.033 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:57:52.033 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:57:52.033 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:57:52.033 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:57:52.033 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:57:52.033 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:57:52.033 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:57:52.033 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:57:52.034 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000035", 2024-08-18T15:57:52.034 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:57:52.034 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:57:52.034 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000037", 2024-08-18T15:57:52.034 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:57:52.034 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:57:52.034 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:57:52.035 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:57:52.035 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:57:52.035 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:57:52.035 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:57:52.035 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:57:52.035 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:57:52.035 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:57:52.035 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:57:52.036 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:57:52.036 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:57:52.036 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:57:52.036 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000024", 2024-08-18T15:57:52.036 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:57:52.036 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:57:52.036 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:57:52.037 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:57:52.037 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000000", 2024-08-18T15:57:52.037 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:57:52.037 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000031", 2024-08-18T15:57:52.037 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:57:52.037 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:57:52.037 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:57:52.038 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:57:52.038 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:57:52.038 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:57:52.038 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3", 2024-08-18T15:57:52.038 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:57:52.038 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:57:52.038 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:57:52.038 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:57:52.038 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:57:52.038 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:57:52.038 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:57:52.039 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:57:52.039 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:57:52.039 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:57:52.039 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:57:52.039 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:57:52.039 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:57:52.039 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:57:52.039 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:57:52.039 INFO:teuthology.orchestra.run.smithi193.stdout: "data_loggenerations_metadata", 2024-08-18T15:57:52.039 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2", 2024-08-18T15:57:52.039 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:57:52.040 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:57:52.040 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:57:52.040 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:57:52.040 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T15:57:52.040 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:57:52.040 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:57:52.040 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:57:52.040 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:57:52.040 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:57:52.041 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000076", 2024-08-18T15:57:52.041 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:57:52.041 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:57:52.041 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000113", 2024-08-18T15:57:52.041 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000005", 2024-08-18T15:57:52.041 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:57:52.042 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000011", 2024-08-18T15:57:52.042 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:57:52.042 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000058", 2024-08-18T15:57:52.043 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:57:52.043 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:57:52.043 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:57:52.043 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000052", 2024-08-18T15:57:52.043 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:57:52.043 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:57:52.043 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:57:52.043 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:57:52.044 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.21", 2024-08-18T15:57:52.044 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:57:52.044 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:57:52.044 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:57:52.044 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:57:52.044 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:57:52.044 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:57:52.044 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:57:52.044 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:57:52.044 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:57:52.044 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:57:52.044 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:57:52.044 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:57:52.044 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:57:52.045 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:57:52.045 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:57:52.045 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:57:52.045 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:57:52.045 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:57:52.045 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:57:52.045 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000088", 2024-08-18T15:57:52.045 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000080", 2024-08-18T15:57:52.045 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:57:52.045 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:57:52.045 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000090", 2024-08-18T15:57:52.045 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:57:52.045 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:57:52.046 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000110", 2024-08-18T15:57:52.046 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:57:52.046 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:57:52.046 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:57:52.046 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.0", 2024-08-18T15:57:52.046 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:57:52.046 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000096", 2024-08-18T15:57:52.046 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:57:52.046 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:57:52.046 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:57:52.046 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:57:52.046 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:57:52.046 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000087", 2024-08-18T15:57:52.047 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:57:52.047 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:57:52.047 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:57:52.047 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:57:52.047 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000008", 2024-08-18T15:57:52.047 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:57:52.047 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:57:52.047 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:57:52.047 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:57:52.047 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:57:52.047 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000006", 2024-08-18T15:57:52.047 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000029", 2024-08-18T15:57:52.048 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:57:52.048 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000089", 2024-08-18T15:57:52.048 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:57:52.048 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:57:52.048 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000022", 2024-08-18T15:57:52.048 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:57:52.048 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:57:52.048 INFO:teuthology.orchestra.run.smithi193.stdout: "bilog.trim", 2024-08-18T15:57:52.048 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000016", 2024-08-18T15:57:52.049 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:57:52.049 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:57:52.049 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:57:52.049 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:57:52.049 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:57:52.049 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000041", 2024-08-18T15:57:52.049 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:57:52.050 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:57:52.050 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.log.01050e30-179b-43be-bdbb-3bb23cf37ae8.3", 2024-08-18T15:57:52.050 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:57:52.050 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000018", 2024-08-18T15:57:52.050 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000092", 2024-08-18T15:57:52.050 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:57:52.050 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:57:52.050 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:57:52.050 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:57:52.050 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.27", 2024-08-18T15:57:52.050 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.4", 2024-08-18T15:57:52.050 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:57:52.051 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:57:52.051 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:57:52.051 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:57:52.051 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:57:52.051 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:57:52.051 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:57:52.051 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000014", 2024-08-18T15:57:52.051 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:57:52.051 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3.retry", 2024-08-18T15:57:52.051 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:57:52.051 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:57:52.052 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:57:52.052 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:57:52.052 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:57:52.052 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000112", 2024-08-18T15:57:52.052 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:57:52.052 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000007", 2024-08-18T15:57:52.052 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000021", 2024-08-18T15:57:52.052 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:57:52.052 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:57:52.053 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:57:52.053 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:57:52.053 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:57:52.053 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.16", 2024-08-18T15:57:52.053 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:57:52.053 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000064", 2024-08-18T15:57:52.053 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:57:52.053 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:57:52.053 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:57:52.053 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:57:52.053 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:57:52.053 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:57:52.053 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:57:52.053 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000071", 2024-08-18T15:57:52.054 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:57:52.054 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:57:52.054 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:57:52.054 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000074", 2024-08-18T15:57:52.054 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:57:52.054 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:57:52.054 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:57:52.054 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:57:52.054 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:57:52.054 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:57:52.054 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:57:52.054 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:57:52.055 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:57:52.055 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:57:52.055 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:57:52.055 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:57:52.055 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:57:52.055 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:57:52.055 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:57:52.055 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000081", 2024-08-18T15:57:52.055 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:57:52.055 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000009", 2024-08-18T15:57:52.055 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000121", 2024-08-18T15:57:52.055 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1", 2024-08-18T15:57:52.055 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:57:52.056 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:57:52.056 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:57:52.056 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:57:52.056 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:57:52.056 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:57:52.056 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.17", 2024-08-18T15:57:52.056 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000125", 2024-08-18T15:57:52.056 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:57:52.056 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000082", 2024-08-18T15:57:52.056 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:57:52.056 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:57:52.056 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:57:52.056 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:57:52.057 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:57:52.057 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:57:52.057 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:57:52.057 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:57:52.057 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:57:52.057 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:57:52.057 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:57:52.057 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000105", 2024-08-18T15:57:52.057 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000059", 2024-08-18T15:57:52.057 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:57:52.057 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:57:52.057 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:57:52.057 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:57:52.058 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:57:52.058 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:57:52.058 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:57:52.058 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:57:52.058 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:57:52.058 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:57:52.058 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:57:52.058 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000077", 2024-08-18T15:57:52.058 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000032", 2024-08-18T15:57:52.059 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:57:52.059 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:57:52.059 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:57:52.059 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:57:52.059 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:57:52.059 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:57:52.059 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.24", 2024-08-18T15:57:52.059 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:57:52.059 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:57:52.059 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:57:52.059 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:57:52.059 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3.retry", 2024-08-18T15:57:52.059 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:57:52.060 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:57:52.060 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:57:52.060 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:57:52.060 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:57:52.060 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:57:52.060 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000053", 2024-08-18T15:57:52.060 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000091", 2024-08-18T15:57:52.060 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:57:52.060 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:57:52.060 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:57:52.060 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000065", 2024-08-18T15:57:52.060 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:57:52.061 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:57:52.061 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:57:52.061 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:57:52.061 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000083", 2024-08-18T15:57:52.061 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000010", 2024-08-18T15:57:52.061 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:57:52.061 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:57:52.061 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:57:52.061 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:57:52.061 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:57:52.061 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:57:52.061 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:57:52.062 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:57:52.062 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:57:52.062 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:57:52.062 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:57:52.062 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.7", 2024-08-18T15:57:52.062 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.3", 2024-08-18T15:57:52.062 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:57:52.062 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:57:52.062 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:57:52.063 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:57:52.063 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:57:52.063 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:57:52.063 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:57:52.063 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:57:52.063 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.2", 2024-08-18T15:57:52.063 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:57:52.063 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:57:52.063 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:57:52.064 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:57:52.064 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:57:52.064 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:57:52.064 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:57:52.064 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000045", 2024-08-18T15:57:52.064 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000002", 2024-08-18T15:57:52.064 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:57:52.064 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3.2UbiLidwSTy.0", 2024-08-18T15:57:52.065 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:57:52.065 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.12", 2024-08-18T15:57:52.065 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:57:52.065 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:57:52.065 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:57:52.065 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:57:52.065 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:57:52.065 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:57:52.065 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000116", 2024-08-18T15:57:52.065 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.25", 2024-08-18T15:57:52.065 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:57:52.065 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:57:52.065 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:57:52.066 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:57:52.066 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000034", 2024-08-18T15:57:52.066 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:57:52.066 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:57:52.066 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000101", 2024-08-18T15:57:52.066 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:57:52.066 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:57:52.066 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.13", 2024-08-18T15:57:52.067 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:57:52.067 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:57:52.067 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.1", 2024-08-18T15:57:52.067 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0.retry", 2024-08-18T15:57:52.067 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:57:52.067 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:57:52.067 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000079", 2024-08-18T15:57:52.068 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000049", 2024-08-18T15:57:52.068 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:57:52.068 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0", 2024-08-18T15:57:52.068 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:57:52.068 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:57:52.068 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000117", 2024-08-18T15:57:52.069 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:57:52.069 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000044", 2024-08-18T15:57:52.069 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:57:52.069 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:57:52.069 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:57:52.069 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:57:52.070 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:57:52.070 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:57:52.070 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.14", 2024-08-18T15:57:52.070 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:57:52.070 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000066", 2024-08-18T15:57:52.070 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:57:52.070 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:57:52.070 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:57:52.070 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:57:52.070 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:57:52.071 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000068", 2024-08-18T15:57:52.071 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:57:52.071 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000085", 2024-08-18T15:57:52.071 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:57:52.071 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:57:52.071 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:57:52.071 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:57:52.071 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:57:52.071 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:57:52.071 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:57:52.072 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:57:52.072 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:57:52.072 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:57:52.072 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:57:52.072 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:57:52.072 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000073", 2024-08-18T15:57:52.072 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:57:52.072 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:57:52.072 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:57:52.072 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000038", 2024-08-18T15:57:52.072 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:57:52.072 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:57:52.073 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:57:52.073 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:57:52.073 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.10", 2024-08-18T15:57:52.073 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:57:52.073 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:57:52.073 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:57:52.073 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000118", 2024-08-18T15:57:52.073 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:57:52.073 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3", 2024-08-18T15:57:52.073 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:57:52.073 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000036", 2024-08-18T15:57:52.073 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:57:52.073 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:57:52.073 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:57:52.074 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:57:52.074 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:57:52.074 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000103", 2024-08-18T15:57:52.074 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.3", 2024-08-18T15:57:52.074 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:57:52.074 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000119", 2024-08-18T15:57:52.074 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000098", 2024-08-18T15:57:52.074 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:57:52.074 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:57:52.074 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:57:52.074 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:57:52.074 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:57:52.075 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:57:52.075 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:57:52.075 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:57:52.075 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.0", 2024-08-18T15:57:52.075 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:57:52.075 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:57:52.075 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:57:52.075 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:57:52.075 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:57:52.075 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:57:52.075 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:57:52.075 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:57:52.076 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000027", 2024-08-18T15:57:52.076 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:57:52.076 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status", 2024-08-18T15:57:52.076 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:57:52.076 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000019", 2024-08-18T15:57:52.076 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:57:52.077 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:57:52.077 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:57:52.077 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:57:52.077 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1", 2024-08-18T15:57:52.077 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:57:52.077 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:57:52.077 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:57:52.077 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:57:52.077 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000039", 2024-08-18T15:57:52.077 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:57:52.077 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:57:52.077 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:57:52.078 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:57:52.078 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:57:52.078 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:57:52.078 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:57:52.078 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:57:52.078 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:57:52.078 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:57:52.078 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:57:52.078 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000100", 2024-08-18T15:57:52.078 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:57:52.079 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:57:52.079 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:57:52.079 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:57:52.079 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:57:52.079 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000093", 2024-08-18T15:57:52.079 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:57:52.079 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000004", 2024-08-18T15:57:52.079 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000063", 2024-08-18T15:57:52.079 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000122", 2024-08-18T15:57:52.079 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:57:52.079 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:57:52.079 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:57:52.079 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000057", 2024-08-18T15:57:52.080 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:57:52.080 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:57:52.080 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:57:52.080 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:57:52.080 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000054", 2024-08-18T15:57:52.080 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:57:52.080 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000114", 2024-08-18T15:57:52.080 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:57:52.080 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:57:52.080 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:57:52.081 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.2", 2024-08-18T15:57:52.081 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:57:52.081 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:57:52.081 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0", 2024-08-18T15:57:52.081 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:57:52.081 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:57:52.081 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:57:52.081 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:57:52.081 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:57:52.081 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:57:52.081 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000012", 2024-08-18T15:57:52.081 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:57:52.081 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:57:52.081 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:57:52.082 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000084", 2024-08-18T15:57:52.082 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000043", 2024-08-18T15:57:52.082 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3", 2024-08-18T15:57:52.082 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:57:52.082 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:57:52.082 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.11", 2024-08-18T15:57:52.082 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:57:52.082 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:57:52.082 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:57:52.082 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:57:52.082 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:57:52.082 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:57:52.082 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:57:52.083 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:57:52.083 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000111", 2024-08-18T15:57:52.083 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:57:52.083 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000048", 2024-08-18T15:57:52.083 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:57:52.083 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000020", 2024-08-18T15:57:52.083 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:57:52.083 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:57:52.083 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:57:52.083 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:57:52.083 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:57:52.083 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000099", 2024-08-18T15:57:52.083 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000056", 2024-08-18T15:57:52.084 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:57:52.084 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:57:52.084 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.19", 2024-08-18T15:57:52.084 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:57:52.084 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.28", 2024-08-18T15:57:52.084 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:57:52.084 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:57:52.084 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:57:52.084 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:57:52.084 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:57:52.084 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000072", 2024-08-18T15:57:52.084 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:57:52.084 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:57:52.084 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000062", 2024-08-18T15:57:52.085 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:57:52.085 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:57:52.085 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:57:52.085 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000115", 2024-08-18T15:57:52.085 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:57:52.085 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.1", 2024-08-18T15:57:52.085 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2", 2024-08-18T15:57:52.085 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000033", 2024-08-18T15:57:52.085 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:57:52.085 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:57:52.085 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:57:52.085 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:57:52.085 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:57:52.085 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:57:52.086 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:57:52.086 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:57:52.086 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:57:52.086 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:57:52.086 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:57:52.086 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:57:52.086 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:57:52.086 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:57:52.086 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:57:52.087 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:57:52.087 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:57:52.087 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:57:52.087 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:57:52.087 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000025", 2024-08-18T15:57:52.087 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:57:52.087 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000123", 2024-08-18T15:57:52.087 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.31", 2024-08-18T15:57:52.087 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:57:52.087 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:57:52.087 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:57:52.087 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:57:52.087 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.18", 2024-08-18T15:57:52.087 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:57:52.088 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:57:52.088 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:57:52.088 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:57:52.088 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:57:52.088 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000108", 2024-08-18T15:57:52.088 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:57:52.088 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:57:52.088 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000094", 2024-08-18T15:57:52.088 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:57:52.088 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:57:52.088 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:57:52.088 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:57:52.088 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:57:52.088 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:57:52.088 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000060", 2024-08-18T15:57:52.089 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:57:52.089 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:57:52.089 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:57:52.089 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000109", 2024-08-18T15:57:52.089 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.15", 2024-08-18T15:57:52.089 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:57:52.089 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:57:52.089 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:57:52.089 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:57:52.089 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:57:52.089 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:57:52.089 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:57:52.089 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:57:52.089 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:57:52.089 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000013", 2024-08-18T15:57:52.090 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:57:52.090 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:57:52.090 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:57:52.090 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:57:52.090 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:57:52.090 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:57:52.090 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:57:52.090 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:57:52.090 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:57:52.091 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:57:52.091 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:57:52.091 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:57:52.091 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:57:52.091 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:57:52.091 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000051", 2024-08-18T15:57:52.091 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:57:52.091 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:57:52.091 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:57:52.091 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:57:52.091 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000106", 2024-08-18T15:57:52.091 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:57:52.091 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:57:52.091 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:57:52.092 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:57:52.092 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:57:52.092 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000017", 2024-08-18T15:57:52.092 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:57:52.092 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:57:52.092 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:57:52.092 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000023", 2024-08-18T15:57:52.092 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:57:52.092 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:57:52.092 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:57:52.092 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:57:52.092 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:57:52.093 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:57:52.093 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:57:52.093 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000107", 2024-08-18T15:57:52.093 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:57:52.093 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:57:52.093 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000046", 2024-08-18T15:57:52.093 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:57:52.093 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:57:52.093 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:57:52.093 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.26", 2024-08-18T15:57:52.094 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:57:52.094 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:57:52.094 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9", 2024-08-18T15:57:52.094 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.22", 2024-08-18T15:57:52.094 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:57:52.094 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:57:52.094 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:57:52.094 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:57:52.094 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:57:52.094 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:57:52.094 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:57:52.094 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2", 2024-08-18T15:57:52.094 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:57:52.095 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:57:52.095 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000067", 2024-08-18T15:57:52.095 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:57:52.095 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:57:52.095 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:57:52.095 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:57:52.095 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:57:52.095 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:57:52.095 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:57:52.095 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:57:52.095 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:57:52.095 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:57:52.095 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:57:52.095 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:57:52.096 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:57:52.096 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:57:52.096 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:57:52.096 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:57:52.096 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:57:52.096 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:57:52.096 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:57:52.096 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.9", 2024-08-18T15:57:52.096 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000086", 2024-08-18T15:57:52.096 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.5", 2024-08-18T15:57:52.096 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:57:52.096 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:57:52.096 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:57:52.097 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:57:52.097 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:57:52.097 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:57:52.097 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2.MFXJ583gCr1.0", 2024-08-18T15:57:52.097 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:57:52.097 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1.GPnO16liyZ+.0", 2024-08-18T15:57:52.097 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:57:52.097 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:57:52.098 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:57:52.098 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:57:52.098 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:57:52.098 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:57:52.098 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:57:52.098 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:57:52.098 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:57:52.098 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.20", 2024-08-18T15:57:52.098 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:57:52.098 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:57:52.098 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:57:52.098 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:57:52.099 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:57:52.099 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:57:52.099 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1.retry", 2024-08-18T15:57:52.099 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:57:52.099 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:57:52.099 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:57:52.099 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0.retry", 2024-08-18T15:57:52.099 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000050", 2024-08-18T15:57:52.099 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:57:52.099 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000102", 2024-08-18T15:57:52.099 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:57:52.099 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:57:52.100 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:57:52.100 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:57:52.100 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:57:52.100 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:57:52.100 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:57:52.100 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:57:52.100 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:57:52.100 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:57:52.100 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:57:52.100 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000124", 2024-08-18T15:57:52.100 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:57:52.100 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:57:52.100 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:57:52.101 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000001", 2024-08-18T15:57:52.101 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:57:52.101 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:57:52.101 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:57:52.101 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:57:52.101 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000126", 2024-08-18T15:57:52.101 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:57:52.101 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0.fZhctfLXnSt.0", 2024-08-18T15:57:52.101 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:57:52.101 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000055", 2024-08-18T15:57:52.101 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:57:52.101 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000061", 2024-08-18T15:57:52.101 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1", 2024-08-18T15:57:52.102 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:57:52.102 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:57:52.102 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:57:52.102 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000097", 2024-08-18T15:57:52.102 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:57:52.102 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:57:52.102 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:57:52.102 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:57:52.102 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:57:52.102 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:57:52.103 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:57:52.103 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:57:52.103 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000042", 2024-08-18T15:57:52.103 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:57:52.103 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:57:52.103 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:57:52.103 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1.retry", 2024-08-18T15:57:52.103 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:57:52.103 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:57:52.103 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:57:52.103 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:57:52.103 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:57:52.103 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:57:52.103 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:57:52.104 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000075", 2024-08-18T15:57:52.104 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:57:52.104 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000120", 2024-08-18T15:57:52.104 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000030", 2024-08-18T15:57:52.104 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:57:52.104 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:57:52.104 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:57:52.104 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:57:52.104 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:57:52.104 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:57:52.104 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9" 2024-08-18T15:57:52.105 INFO:teuthology.orchestra.run.smithi193.stdout:] 2024-08-18T15:57:52.105 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:57:57.107 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:57:57.107 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:57:57.107 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:57:57.161 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:57:57.162 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:57:57.226 INFO:teuthology.orchestra.run.smithi193.stdout:[ 2024-08-18T15:57:57.226 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:57:57.226 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:57:57.226 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.29", 2024-08-18T15:57:57.226 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000078", 2024-08-18T15:57:57.226 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.history", 2024-08-18T15:57:57.226 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:57:57.226 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:57:57.226 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:57:57.227 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:57:57.227 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.23", 2024-08-18T15:57:57.227 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000070", 2024-08-18T15:57:57.227 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000104", 2024-08-18T15:57:57.227 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:57:57.227 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2.retry", 2024-08-18T15:57:57.227 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:57:57.227 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:57:57.227 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:57:57.227 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:57:57.227 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000026", 2024-08-18T15:57:57.227 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:57:57.227 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:57:57.228 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:57:57.228 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:57:57.228 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:57:57.228 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:57:57.228 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.8", 2024-08-18T15:57:57.228 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:57:57.228 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:57:57.228 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:57:57.228 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:57:57.228 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000028", 2024-08-18T15:57:57.228 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:57:57.228 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:57:57.228 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:57:57.228 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:57:57.229 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:57:57.229 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:57:57.229 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:57:57.229 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2.retry", 2024-08-18T15:57:57.229 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:57:57.229 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.30", 2024-08-18T15:57:57.229 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:57:57.229 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:57:57.229 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:57:57.229 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:57:57.229 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000040", 2024-08-18T15:57:57.229 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:57:57.229 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.6", 2024-08-18T15:57:57.229 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:57:57.230 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:57:57.230 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:57:57.230 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:57:57.230 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:57:57.230 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:57:57.230 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000015", 2024-08-18T15:57:57.230 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:57:57.230 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:57:57.230 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000069", 2024-08-18T15:57:57.230 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:57:57.231 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:57:57.231 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:57:57.231 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000095", 2024-08-18T15:57:57.231 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0", 2024-08-18T15:57:57.231 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000003", 2024-08-18T15:57:57.231 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000047", 2024-08-18T15:57:57.231 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:57:57.231 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:57:57.231 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:57:57.231 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:57:57.231 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:57:57.231 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:57:57.231 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:57:57.231 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:57:57.232 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:57:57.232 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000035", 2024-08-18T15:57:57.232 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:57:57.232 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:57:57.232 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000037", 2024-08-18T15:57:57.232 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:57:57.232 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:57:57.232 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:57:57.232 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:57:57.232 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:57:57.232 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:57:57.232 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:57:57.232 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:57:57.233 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:57:57.233 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:57:57.233 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:57:57.233 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:57:57.233 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:57:57.233 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:57:57.233 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000024", 2024-08-18T15:57:57.233 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:57:57.233 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:57:57.233 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:57:57.233 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:57:57.233 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000000", 2024-08-18T15:57:57.233 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:57:57.233 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000031", 2024-08-18T15:57:57.234 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:57:57.234 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:57:57.234 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:57:57.234 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:57:57.234 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:57:57.234 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:57:57.234 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3", 2024-08-18T15:57:57.234 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:57:57.234 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:57:57.235 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:57:57.235 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:57:57.235 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:57:57.235 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:57:57.235 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:57:57.235 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:57:57.235 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:57:57.235 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:57:57.235 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:57:57.235 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:57:57.235 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:57:57.235 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:57:57.235 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:57:57.235 INFO:teuthology.orchestra.run.smithi193.stdout: "data_loggenerations_metadata", 2024-08-18T15:57:57.236 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2", 2024-08-18T15:57:57.236 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:57:57.236 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:57:57.236 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:57:57.236 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:57:57.236 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T15:57:57.236 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:57:57.236 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:57:57.236 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:57:57.236 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:57:57.236 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:57:57.236 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000076", 2024-08-18T15:57:57.236 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:57:57.236 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:57:57.237 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000113", 2024-08-18T15:57:57.237 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000005", 2024-08-18T15:57:57.237 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:57:57.237 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000011", 2024-08-18T15:57:57.237 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:57:57.237 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000058", 2024-08-18T15:57:57.237 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:57:57.237 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:57:57.237 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:57:57.237 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000052", 2024-08-18T15:57:57.237 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:57:57.237 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:57:57.237 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:57:57.237 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:57:57.237 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.21", 2024-08-18T15:57:57.238 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:57:57.238 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:57:57.238 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:57:57.238 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:57:57.238 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:57:57.238 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:57:57.238 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:57:57.238 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:57:57.238 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:57:57.238 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:57:57.238 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:57:57.238 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:57:57.238 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:57:57.239 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:57:57.239 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:57:57.239 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:57:57.239 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:57:57.239 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:57:57.239 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:57:57.239 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000088", 2024-08-18T15:57:57.239 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000080", 2024-08-18T15:57:57.239 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:57:57.239 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:57:57.239 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000090", 2024-08-18T15:57:57.239 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:57:57.239 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:57:57.239 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000110", 2024-08-18T15:57:57.240 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:57:57.240 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:57:57.240 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:57:57.240 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.0", 2024-08-18T15:57:57.240 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:57:57.240 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000096", 2024-08-18T15:57:57.240 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:57:57.240 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:57:57.240 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:57:57.240 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:57:57.240 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:57:57.240 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000087", 2024-08-18T15:57:57.240 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:57:57.240 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:57:57.241 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:57:57.241 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:57:57.241 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000008", 2024-08-18T15:57:57.241 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:57:57.241 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:57:57.241 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:57:57.241 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:57:57.241 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:57:57.241 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000006", 2024-08-18T15:57:57.241 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000029", 2024-08-18T15:57:57.241 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:57:57.241 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000089", 2024-08-18T15:57:57.241 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:57:57.241 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:57:57.242 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000022", 2024-08-18T15:57:57.242 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:57:57.242 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:57:57.242 INFO:teuthology.orchestra.run.smithi193.stdout: "bilog.trim", 2024-08-18T15:57:57.242 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000016", 2024-08-18T15:57:57.242 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:57:57.242 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:57:57.242 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:57:57.242 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:57:57.242 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:57:57.242 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000041", 2024-08-18T15:57:57.242 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:57:57.242 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:57:57.242 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.log.01050e30-179b-43be-bdbb-3bb23cf37ae8.3", 2024-08-18T15:57:57.243 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:57:57.243 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000018", 2024-08-18T15:57:57.243 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000092", 2024-08-18T15:57:57.243 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:57:57.243 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:57:57.243 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:57:57.243 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:57:57.243 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.27", 2024-08-18T15:57:57.243 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.4", 2024-08-18T15:57:57.243 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:57:57.243 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:57:57.243 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:57:57.243 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:57:57.243 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:57:57.244 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:57:57.244 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:57:57.244 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000014", 2024-08-18T15:57:57.244 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:57:57.244 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3.retry", 2024-08-18T15:57:57.244 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:57:57.244 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:57:57.244 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:57:57.244 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:57:57.245 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:57:57.245 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000112", 2024-08-18T15:57:57.245 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:57:57.245 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000007", 2024-08-18T15:57:57.245 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000021", 2024-08-18T15:57:57.245 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:57:57.245 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:57:57.245 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:57:57.245 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:57:57.245 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:57:57.245 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.16", 2024-08-18T15:57:57.245 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:57:57.245 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000064", 2024-08-18T15:57:57.245 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:57:57.246 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:57:57.246 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:57:57.246 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:57:57.246 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:57:57.246 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:57:57.246 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:57:57.246 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000071", 2024-08-18T15:57:57.246 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:57:57.246 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:57:57.246 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:57:57.246 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000074", 2024-08-18T15:57:57.246 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:57:57.246 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:57:57.247 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:57:57.247 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:57:57.247 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:57:57.247 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:57:57.247 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:57:57.247 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:57:57.247 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:57:57.247 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:57:57.247 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:57:57.247 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:57:57.247 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:57:57.247 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:57:57.247 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:57:57.247 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000081", 2024-08-18T15:57:57.248 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:57:57.248 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000009", 2024-08-18T15:57:57.248 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000121", 2024-08-18T15:57:57.248 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1", 2024-08-18T15:57:57.248 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:57:57.248 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:57:57.248 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:57:57.248 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:57:57.248 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:57:57.248 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:57:57.248 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.17", 2024-08-18T15:57:57.248 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000125", 2024-08-18T15:57:57.248 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:57:57.248 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000082", 2024-08-18T15:57:57.249 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:57:57.249 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:57:57.249 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:57:57.249 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:57:57.249 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:57:57.249 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:57:57.249 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:57:57.249 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:57:57.249 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:57:57.249 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:57:57.249 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:57:57.249 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000105", 2024-08-18T15:57:57.249 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000059", 2024-08-18T15:57:57.249 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:57:57.250 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:57:57.250 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:57:57.250 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:57:57.250 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:57:57.250 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:57:57.250 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:57:57.250 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:57:57.250 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:57:57.250 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:57:57.250 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:57:57.250 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000077", 2024-08-18T15:57:57.250 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000032", 2024-08-18T15:57:57.250 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:57:57.250 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:57:57.251 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:57:57.251 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:57:57.251 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:57:57.251 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:57:57.251 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.24", 2024-08-18T15:57:57.251 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:57:57.251 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:57:57.251 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:57:57.251 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:57:57.251 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3.retry", 2024-08-18T15:57:57.251 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:57:57.251 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:57:57.251 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:57:57.251 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:57:57.252 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:57:57.252 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:57:57.252 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000053", 2024-08-18T15:57:57.252 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000091", 2024-08-18T15:57:57.252 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:57:57.252 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:57:57.252 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:57:57.252 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000065", 2024-08-18T15:57:57.252 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:57:57.252 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:57:57.252 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:57:57.252 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:57:57.252 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000083", 2024-08-18T15:57:57.252 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000010", 2024-08-18T15:57:57.253 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:57:57.253 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:57:57.253 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:57:57.253 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:57:57.253 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:57:57.253 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:57:57.253 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:57:57.253 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:57:57.253 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:57:57.253 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:57:57.253 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:57:57.253 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.7", 2024-08-18T15:57:57.253 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.3", 2024-08-18T15:57:57.254 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:57:57.254 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:57:57.254 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:57:57.254 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:57:57.254 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:57:57.254 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:57:57.254 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:57:57.254 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:57:57.254 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.2", 2024-08-18T15:57:57.254 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:57:57.255 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:57:57.255 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:57:57.255 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:57:57.255 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:57:57.255 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:57:57.255 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:57:57.255 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000045", 2024-08-18T15:57:57.255 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000002", 2024-08-18T15:57:57.255 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:57:57.255 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3.2UbiLidwSTy.0", 2024-08-18T15:57:57.255 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:57:57.255 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.12", 2024-08-18T15:57:57.255 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:57:57.256 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:57:57.256 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:57:57.256 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:57:57.256 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:57:57.256 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:57:57.256 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000116", 2024-08-18T15:57:57.256 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.25", 2024-08-18T15:57:57.256 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:57:57.256 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:57:57.256 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:57:57.256 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:57:57.256 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000034", 2024-08-18T15:57:57.256 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:57:57.257 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:57:57.257 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000101", 2024-08-18T15:57:57.257 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:57:57.257 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:57:57.257 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.13", 2024-08-18T15:57:57.257 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:57:57.257 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:57:57.257 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.1", 2024-08-18T15:57:57.257 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0.retry", 2024-08-18T15:57:57.257 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:57:57.257 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:57:57.257 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000079", 2024-08-18T15:57:57.257 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000049", 2024-08-18T15:57:57.257 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:57:57.258 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0", 2024-08-18T15:57:57.258 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:57:57.258 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:57:57.258 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000117", 2024-08-18T15:57:57.258 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:57:57.258 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000044", 2024-08-18T15:57:57.258 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:57:57.258 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:57:57.258 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:57:57.258 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:57:57.258 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:57:57.258 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:57:57.258 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.14", 2024-08-18T15:57:57.259 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:57:57.259 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000066", 2024-08-18T15:57:57.259 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:57:57.259 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:57:57.259 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:57:57.259 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:57:57.259 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:57:57.259 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000068", 2024-08-18T15:57:57.259 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:57:57.259 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000085", 2024-08-18T15:57:57.259 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:57:57.259 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:57:57.260 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:57:57.260 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:57:57.260 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:57:57.260 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:57:57.260 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:57:57.260 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:57:57.260 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:57:57.260 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:57:57.260 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:57:57.260 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:57:57.260 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000073", 2024-08-18T15:57:57.260 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:57:57.261 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:57:57.261 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:57:57.261 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000038", 2024-08-18T15:57:57.261 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:57:57.261 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:57:57.261 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:57:57.261 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:57:57.261 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.10", 2024-08-18T15:57:57.261 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:57:57.261 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:57:57.261 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:57:57.261 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000118", 2024-08-18T15:57:57.261 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:57:57.261 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3", 2024-08-18T15:57:57.262 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:57:57.262 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000036", 2024-08-18T15:57:57.262 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:57:57.262 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:57:57.262 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:57:57.262 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:57:57.262 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:57:57.262 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000103", 2024-08-18T15:57:57.262 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.3", 2024-08-18T15:57:57.262 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:57:57.262 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000119", 2024-08-18T15:57:57.262 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000098", 2024-08-18T15:57:57.262 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:57:57.263 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:57:57.263 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:57:57.263 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:57:57.263 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:57:57.263 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:57:57.263 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:57:57.263 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:57:57.263 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.0", 2024-08-18T15:57:57.263 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:57:57.263 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:57:57.263 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:57:57.263 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:57:57.263 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:57:57.264 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:57:57.264 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:57:57.264 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:57:57.264 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000027", 2024-08-18T15:57:57.264 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:57:57.264 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status", 2024-08-18T15:57:57.264 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:57:57.264 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000019", 2024-08-18T15:57:57.264 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:57:57.264 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:57:57.264 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:57:57.264 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:57:57.264 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1", 2024-08-18T15:57:57.264 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:57:57.264 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:57:57.265 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:57:57.265 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:57:57.265 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000039", 2024-08-18T15:57:57.265 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:57:57.265 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:57:57.265 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:57:57.265 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:57:57.265 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:57:57.265 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:57:57.266 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:57:57.266 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:57:57.266 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:57:57.266 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:57:57.266 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:57:57.266 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000100", 2024-08-18T15:57:57.266 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:57:57.266 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:57:57.266 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:57:57.266 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:57:57.266 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:57:57.266 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000093", 2024-08-18T15:57:57.266 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:57:57.267 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000004", 2024-08-18T15:57:57.267 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000063", 2024-08-18T15:57:57.267 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000122", 2024-08-18T15:57:57.267 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:57:57.267 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:57:57.267 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:57:57.267 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000057", 2024-08-18T15:57:57.267 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:57:57.267 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:57:57.267 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:57:57.267 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:57:57.267 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000054", 2024-08-18T15:57:57.267 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:57:57.267 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000114", 2024-08-18T15:57:57.268 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:57:57.268 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:57:57.268 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:57:57.268 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.2", 2024-08-18T15:57:57.268 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:57:57.268 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:57:57.268 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0", 2024-08-18T15:57:57.268 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:57:57.268 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:57:57.268 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:57:57.268 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:57:57.268 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:57:57.268 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:57:57.268 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000012", 2024-08-18T15:57:57.269 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:57:57.269 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:57:57.269 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:57:57.269 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000084", 2024-08-18T15:57:57.269 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000043", 2024-08-18T15:57:57.269 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3", 2024-08-18T15:57:57.269 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:57:57.269 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:57:57.269 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.11", 2024-08-18T15:57:57.269 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:57:57.269 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:57:57.269 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:57:57.269 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:57:57.269 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:57:57.270 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:57:57.270 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:57:57.270 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:57:57.270 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000111", 2024-08-18T15:57:57.270 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:57:57.270 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000048", 2024-08-18T15:57:57.270 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:57:57.270 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000020", 2024-08-18T15:57:57.270 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:57:57.270 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:57:57.270 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:57:57.270 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:57:57.271 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:57:57.271 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000099", 2024-08-18T15:57:57.271 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000056", 2024-08-18T15:57:57.271 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:57:57.271 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:57:57.271 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.19", 2024-08-18T15:57:57.271 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:57:57.271 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.28", 2024-08-18T15:57:57.271 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:57:57.271 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:57:57.272 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:57:57.272 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:57:57.272 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:57:57.272 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000072", 2024-08-18T15:57:57.272 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:57:57.272 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:57:57.272 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000062", 2024-08-18T15:57:57.272 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:57:57.272 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:57:57.272 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:57:57.272 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000115", 2024-08-18T15:57:57.272 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:57:57.272 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.1", 2024-08-18T15:57:57.272 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2", 2024-08-18T15:57:57.273 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000033", 2024-08-18T15:57:57.273 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:57:57.273 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:57:57.273 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:57:57.273 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:57:57.273 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:57:57.273 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:57:57.273 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:57:57.273 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:57:57.273 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:57:57.273 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:57:57.273 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:57:57.273 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:57:57.273 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:57:57.274 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:57:57.274 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:57:57.274 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:57:57.274 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:57:57.274 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:57:57.274 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:57:57.274 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000025", 2024-08-18T15:57:57.274 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:57:57.274 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000123", 2024-08-18T15:57:57.274 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.31", 2024-08-18T15:57:57.274 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:57:57.274 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:57:57.275 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:57:57.275 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:57:57.275 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.18", 2024-08-18T15:57:57.275 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:57:57.275 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:57:57.275 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:57:57.275 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:57:57.275 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:57:57.275 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000108", 2024-08-18T15:57:57.275 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:57:57.275 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:57:57.275 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000094", 2024-08-18T15:57:57.275 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:57:57.275 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:57:57.276 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:57:57.276 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:57:57.276 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:57:57.276 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:57:57.276 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000060", 2024-08-18T15:57:57.276 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:57:57.276 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:57:57.276 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:57:57.276 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000109", 2024-08-18T15:57:57.276 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.15", 2024-08-18T15:57:57.276 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:57:57.276 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:57:57.276 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:57:57.277 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:57:57.277 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:57:57.277 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:57:57.277 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:57:57.277 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:57:57.277 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:57:57.277 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000013", 2024-08-18T15:57:57.277 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:57:57.277 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:57:57.277 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:57:57.277 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:57:57.277 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:57:57.277 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:57:57.278 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:57:57.278 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:57:57.278 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:57:57.278 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:57:57.278 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:57:57.278 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:57:57.278 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:57:57.278 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:57:57.278 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000051", 2024-08-18T15:57:57.278 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:57:57.278 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:57:57.278 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:57:57.278 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:57:57.279 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000106", 2024-08-18T15:57:57.279 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:57:57.279 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:57:57.279 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:57:57.279 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:57:57.279 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:57:57.279 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000017", 2024-08-18T15:57:57.279 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:57:57.279 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:57:57.279 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:57:57.279 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000023", 2024-08-18T15:57:57.279 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:57:57.279 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:57:57.279 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:57:57.280 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:57:57.280 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:57:57.280 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:57:57.280 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:57:57.280 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000107", 2024-08-18T15:57:57.280 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:57:57.280 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:57:57.280 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000046", 2024-08-18T15:57:57.280 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:57:57.280 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:57:57.280 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:57:57.280 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.26", 2024-08-18T15:57:57.280 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:57:57.280 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:57:57.281 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9", 2024-08-18T15:57:57.281 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.22", 2024-08-18T15:57:57.281 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:57:57.281 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:57:57.281 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:57:57.281 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:57:57.281 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:57:57.281 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:57:57.281 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:57:57.281 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2", 2024-08-18T15:57:57.281 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:57:57.281 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:57:57.281 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000067", 2024-08-18T15:57:57.281 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:57:57.282 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:57:57.282 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:57:57.282 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:57:57.282 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:57:57.282 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:57:57.282 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:57:57.282 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:57:57.282 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:57:57.282 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:57:57.282 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:57:57.282 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:57:57.282 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:57:57.282 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:57:57.283 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:57:57.283 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:57:57.283 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:57:57.283 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:57:57.283 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:57:57.283 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.9", 2024-08-18T15:57:57.283 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000086", 2024-08-18T15:57:57.283 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.5", 2024-08-18T15:57:57.283 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:57:57.283 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:57:57.283 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:57:57.283 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:57:57.283 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:57:57.284 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:57:57.284 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2.MFXJ583gCr1.0", 2024-08-18T15:57:57.284 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:57:57.284 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1.GPnO16liyZ+.0", 2024-08-18T15:57:57.284 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:57:57.284 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:57:57.284 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:57:57.284 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:57:57.284 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:57:57.284 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:57:57.284 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:57:57.284 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:57:57.284 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:57:57.284 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.20", 2024-08-18T15:57:57.285 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:57:57.285 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:57:57.285 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:57:57.285 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:57:57.285 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:57:57.285 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:57:57.285 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1.retry", 2024-08-18T15:57:57.285 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:57:57.285 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:57:57.285 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:57:57.285 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0.retry", 2024-08-18T15:57:57.285 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000050", 2024-08-18T15:57:57.285 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:57:57.285 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000102", 2024-08-18T15:57:57.286 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:57:57.286 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:57:57.286 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:57:57.286 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:57:57.286 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:57:57.286 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:57:57.286 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:57:57.286 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:57:57.286 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:57:57.286 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:57:57.286 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:57:57.286 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000124", 2024-08-18T15:57:57.286 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:57:57.287 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:57:57.287 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:57:57.287 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000001", 2024-08-18T15:57:57.287 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:57:57.287 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:57:57.287 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:57:57.287 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:57:57.287 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000126", 2024-08-18T15:57:57.287 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:57:57.287 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0.fZhctfLXnSt.0", 2024-08-18T15:57:57.287 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:57:57.287 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000055", 2024-08-18T15:57:57.287 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:57:57.288 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000061", 2024-08-18T15:57:57.288 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1", 2024-08-18T15:57:57.288 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:57:57.288 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:57:57.288 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:57:57.288 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000097", 2024-08-18T15:57:57.288 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:57:57.288 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:57:57.288 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:57:57.288 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:57:57.288 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:57:57.288 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:57:57.288 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:57:57.288 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:57:57.289 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000042", 2024-08-18T15:57:57.289 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:57:57.289 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:57:57.289 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:57:57.289 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1.retry", 2024-08-18T15:57:57.289 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:57:57.289 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:57:57.289 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:57:57.289 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:57:57.289 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:57:57.289 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:57:57.289 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:57:57.289 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000075", 2024-08-18T15:57:57.289 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:57:57.290 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000120", 2024-08-18T15:57:57.290 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000030", 2024-08-18T15:57:57.290 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:57:57.290 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:57:57.290 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:57:57.290 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:57:57.290 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:57:57.290 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:57:57.290 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9" 2024-08-18T15:57:57.290 INFO:teuthology.orchestra.run.smithi193.stdout:] 2024-08-18T15:57:57.290 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:58:02.292 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:58:02.292 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:58:02.293 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:58:02.345 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:58:02.345 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:58:02.410 INFO:teuthology.orchestra.run.smithi193.stdout:[ 2024-08-18T15:58:02.410 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:58:02.410 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:58:02.410 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.29", 2024-08-18T15:58:02.410 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000078", 2024-08-18T15:58:02.410 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.history", 2024-08-18T15:58:02.410 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:58:02.410 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:58:02.411 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:58:02.411 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:58:02.411 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.23", 2024-08-18T15:58:02.411 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000070", 2024-08-18T15:58:02.411 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000104", 2024-08-18T15:58:02.411 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:58:02.411 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2.retry", 2024-08-18T15:58:02.411 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:58:02.411 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:58:02.411 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:58:02.411 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:58:02.411 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000026", 2024-08-18T15:58:02.412 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:58:02.412 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:58:02.412 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:58:02.412 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:58:02.412 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:58:02.412 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:58:02.412 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.8", 2024-08-18T15:58:02.412 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:58:02.412 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:58:02.412 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:58:02.412 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:58:02.412 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000028", 2024-08-18T15:58:02.412 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:58:02.413 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:58:02.413 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:58:02.413 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:58:02.413 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:58:02.413 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:58:02.413 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:58:02.413 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2.retry", 2024-08-18T15:58:02.413 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:58:02.413 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.30", 2024-08-18T15:58:02.413 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:58:02.413 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:58:02.413 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:58:02.413 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:58:02.414 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000040", 2024-08-18T15:58:02.414 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:58:02.414 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.6", 2024-08-18T15:58:02.414 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:58:02.414 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:58:02.414 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:58:02.414 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:58:02.414 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:58:02.414 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:58:02.414 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000015", 2024-08-18T15:58:02.414 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:58:02.414 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:58:02.415 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000069", 2024-08-18T15:58:02.415 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:58:02.415 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:58:02.415 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:58:02.415 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000095", 2024-08-18T15:58:02.415 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0", 2024-08-18T15:58:02.415 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000003", 2024-08-18T15:58:02.415 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000047", 2024-08-18T15:58:02.415 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:58:02.415 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:58:02.416 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:58:02.416 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:58:02.416 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:58:02.416 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:58:02.416 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:58:02.416 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:58:02.416 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:58:02.416 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000035", 2024-08-18T15:58:02.416 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:58:02.416 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:58:02.416 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000037", 2024-08-18T15:58:02.416 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:58:02.417 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:58:02.417 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:58:02.417 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:58:02.417 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:58:02.417 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:58:02.417 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:58:02.417 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:58:02.417 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:58:02.417 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:58:02.417 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:58:02.417 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:58:02.417 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:58:02.418 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:58:02.418 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000024", 2024-08-18T15:58:02.418 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:58:02.418 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:58:02.418 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:58:02.418 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:58:02.418 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000000", 2024-08-18T15:58:02.418 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:58:02.418 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000031", 2024-08-18T15:58:02.418 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:58:02.418 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:58:02.418 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:58:02.419 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:58:02.419 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:58:02.419 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:58:02.419 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3", 2024-08-18T15:58:02.419 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:58:02.419 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:58:02.419 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:58:02.419 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:58:02.419 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:58:02.419 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:58:02.419 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:58:02.419 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:58:02.419 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:58:02.421 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:58:02.421 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:58:02.421 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:58:02.421 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:58:02.422 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:58:02.422 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:58:02.422 INFO:teuthology.orchestra.run.smithi193.stdout: "data_loggenerations_metadata", 2024-08-18T15:58:02.422 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2", 2024-08-18T15:58:02.422 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:58:02.422 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:58:02.422 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:58:02.422 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:58:02.422 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T15:58:02.422 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:58:02.422 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:58:02.422 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:58:02.423 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:58:02.423 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:58:02.423 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000076", 2024-08-18T15:58:02.423 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:58:02.423 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:58:02.423 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000113", 2024-08-18T15:58:02.423 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000005", 2024-08-18T15:58:02.423 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:58:02.423 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000011", 2024-08-18T15:58:02.423 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:58:02.423 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000058", 2024-08-18T15:58:02.423 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:58:02.423 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:58:02.423 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:58:02.423 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000052", 2024-08-18T15:58:02.424 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:58:02.424 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:58:02.424 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:58:02.424 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:58:02.424 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.21", 2024-08-18T15:58:02.424 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:58:02.424 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:58:02.424 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:58:02.424 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:58:02.424 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:58:02.424 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:58:02.424 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:58:02.424 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:58:02.424 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:58:02.425 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:58:02.425 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:58:02.425 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:58:02.425 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:58:02.425 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:58:02.425 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:58:02.425 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:58:02.425 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:58:02.425 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:58:02.425 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:58:02.425 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000088", 2024-08-18T15:58:02.425 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000080", 2024-08-18T15:58:02.425 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:58:02.426 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:58:02.426 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000090", 2024-08-18T15:58:02.426 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:58:02.426 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:58:02.426 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000110", 2024-08-18T15:58:02.426 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:58:02.426 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:58:02.426 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:58:02.426 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.0", 2024-08-18T15:58:02.426 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:58:02.426 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000096", 2024-08-18T15:58:02.426 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:58:02.427 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:58:02.427 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:58:02.427 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:58:02.427 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:58:02.427 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000087", 2024-08-18T15:58:02.427 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:58:02.427 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:58:02.427 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:58:02.427 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:58:02.427 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000008", 2024-08-18T15:58:02.427 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:58:02.427 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:58:02.427 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:58:02.427 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:58:02.428 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:58:02.428 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000006", 2024-08-18T15:58:02.428 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000029", 2024-08-18T15:58:02.428 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:58:02.428 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000089", 2024-08-18T15:58:02.428 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:58:02.428 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:58:02.428 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000022", 2024-08-18T15:58:02.428 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:58:02.428 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:58:02.428 INFO:teuthology.orchestra.run.smithi193.stdout: "bilog.trim", 2024-08-18T15:58:02.428 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000016", 2024-08-18T15:58:02.429 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:58:02.429 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:58:02.429 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:58:02.429 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:58:02.429 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:58:02.429 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000041", 2024-08-18T15:58:02.429 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:58:02.429 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:58:02.429 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.log.01050e30-179b-43be-bdbb-3bb23cf37ae8.3", 2024-08-18T15:58:02.429 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:58:02.429 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000018", 2024-08-18T15:58:02.429 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000092", 2024-08-18T15:58:02.429 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:58:02.429 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:58:02.429 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:58:02.430 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:58:02.430 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.27", 2024-08-18T15:58:02.430 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.4", 2024-08-18T15:58:02.430 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:58:02.430 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:58:02.430 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:58:02.430 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:58:02.430 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:58:02.430 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:58:02.430 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:58:02.430 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000014", 2024-08-18T15:58:02.430 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:58:02.430 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3.retry", 2024-08-18T15:58:02.431 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:58:02.431 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:58:02.431 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:58:02.431 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:58:02.431 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:58:02.431 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000112", 2024-08-18T15:58:02.431 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:58:02.431 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000007", 2024-08-18T15:58:02.431 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000021", 2024-08-18T15:58:02.431 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:58:02.431 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:58:02.431 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:58:02.432 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:58:02.432 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:58:02.432 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.16", 2024-08-18T15:58:02.432 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:58:02.432 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000064", 2024-08-18T15:58:02.432 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:58:02.432 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:58:02.432 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:58:02.432 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:58:02.432 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:58:02.432 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:58:02.432 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:58:02.432 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000071", 2024-08-18T15:58:02.432 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:58:02.433 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:58:02.433 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:58:02.433 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000074", 2024-08-18T15:58:02.433 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:58:02.433 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:58:02.433 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:58:02.433 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:58:02.433 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:58:02.433 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:58:02.433 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:58:02.433 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:58:02.433 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:58:02.434 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:58:02.434 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:58:02.434 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:58:02.434 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:58:02.434 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:58:02.434 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:58:02.434 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000081", 2024-08-18T15:58:02.434 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:58:02.434 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000009", 2024-08-18T15:58:02.434 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000121", 2024-08-18T15:58:02.434 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1", 2024-08-18T15:58:02.434 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:58:02.434 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:58:02.435 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:58:02.435 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:58:02.435 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:58:02.435 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:58:02.435 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.17", 2024-08-18T15:58:02.435 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000125", 2024-08-18T15:58:02.435 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:58:02.435 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000082", 2024-08-18T15:58:02.435 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:58:02.435 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:58:02.435 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:58:02.435 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:58:02.436 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:58:02.436 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:58:02.436 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:58:02.436 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:58:02.436 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:58:02.436 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:58:02.436 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:58:02.436 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000105", 2024-08-18T15:58:02.436 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000059", 2024-08-18T15:58:02.436 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:58:02.436 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:58:02.436 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:58:02.436 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:58:02.436 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:58:02.437 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:58:02.437 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:58:02.437 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:58:02.437 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:58:02.437 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:58:02.437 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:58:02.437 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000077", 2024-08-18T15:58:02.437 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000032", 2024-08-18T15:58:02.437 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:58:02.437 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:58:02.437 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:58:02.437 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:58:02.437 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:58:02.438 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:58:02.438 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.24", 2024-08-18T15:58:02.438 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:58:02.438 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:58:02.438 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:58:02.438 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:58:02.438 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3.retry", 2024-08-18T15:58:02.438 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:58:02.438 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:58:02.438 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:58:02.438 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:58:02.438 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:58:02.439 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:58:02.439 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000053", 2024-08-18T15:58:02.439 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000091", 2024-08-18T15:58:02.439 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:58:02.439 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:58:02.439 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:58:02.439 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000065", 2024-08-18T15:58:02.439 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:58:02.439 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:58:02.439 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:58:02.439 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:58:02.439 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000083", 2024-08-18T15:58:02.440 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000010", 2024-08-18T15:58:02.440 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:58:02.440 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:58:02.440 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:58:02.440 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:58:02.440 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:58:02.440 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:58:02.440 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:58:02.440 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:58:02.440 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:58:02.440 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:58:02.440 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:58:02.440 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.7", 2024-08-18T15:58:02.440 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.3", 2024-08-18T15:58:02.441 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:58:02.441 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:58:02.441 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:58:02.441 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:58:02.441 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:58:02.441 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:58:02.441 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:58:02.441 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:58:02.441 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.2", 2024-08-18T15:58:02.441 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:58:02.441 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:58:02.441 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:58:02.441 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:58:02.442 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:58:02.442 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:58:02.442 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:58:02.442 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000045", 2024-08-18T15:58:02.442 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000002", 2024-08-18T15:58:02.442 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:58:02.442 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3.2UbiLidwSTy.0", 2024-08-18T15:58:02.442 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:58:02.442 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.12", 2024-08-18T15:58:02.442 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:58:02.442 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:58:02.442 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:58:02.443 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:58:02.443 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:58:02.443 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:58:02.443 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000116", 2024-08-18T15:58:02.443 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.25", 2024-08-18T15:58:02.443 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:58:02.443 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:58:02.443 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:58:02.443 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:58:02.443 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000034", 2024-08-18T15:58:02.443 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:58:02.443 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:58:02.443 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000101", 2024-08-18T15:58:02.444 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:58:02.444 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:58:02.444 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.13", 2024-08-18T15:58:02.444 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:58:02.444 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:58:02.444 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.1", 2024-08-18T15:58:02.444 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0.retry", 2024-08-18T15:58:02.444 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:58:02.444 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:58:02.444 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000079", 2024-08-18T15:58:02.444 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000049", 2024-08-18T15:58:02.444 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:58:02.445 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0", 2024-08-18T15:58:02.445 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:58:02.445 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:58:02.445 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000117", 2024-08-18T15:58:02.445 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:58:02.445 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000044", 2024-08-18T15:58:02.445 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:58:02.445 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:58:02.445 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:58:02.445 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:58:02.445 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:58:02.445 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:58:02.445 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.14", 2024-08-18T15:58:02.446 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:58:02.446 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000066", 2024-08-18T15:58:02.446 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:58:02.446 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:58:02.446 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:58:02.446 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:58:02.446 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:58:02.446 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000068", 2024-08-18T15:58:02.446 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:58:02.446 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000085", 2024-08-18T15:58:02.446 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:58:02.446 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:58:02.447 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:58:02.447 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:58:02.447 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:58:02.447 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:58:02.447 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:58:02.447 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:58:02.447 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:58:02.447 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:58:02.447 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:58:02.447 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:58:02.447 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000073", 2024-08-18T15:58:02.447 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:58:02.447 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:58:02.448 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:58:02.448 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000038", 2024-08-18T15:58:02.448 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:58:02.448 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:58:02.448 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:58:02.448 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:58:02.448 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.10", 2024-08-18T15:58:02.448 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:58:02.448 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:58:02.448 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:58:02.448 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000118", 2024-08-18T15:58:02.448 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:58:02.449 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3", 2024-08-18T15:58:02.449 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:58:02.449 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000036", 2024-08-18T15:58:02.449 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:58:02.449 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:58:02.449 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:58:02.449 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:58:02.449 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:58:02.449 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000103", 2024-08-18T15:58:02.449 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.3", 2024-08-18T15:58:02.449 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:58:02.449 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000119", 2024-08-18T15:58:02.450 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000098", 2024-08-18T15:58:02.450 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:58:02.450 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:58:02.450 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:58:02.450 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:58:02.450 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:58:02.450 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:58:02.450 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:58:02.450 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:58:02.450 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.0", 2024-08-18T15:58:02.450 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:58:02.450 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:58:02.451 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:58:02.451 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:58:02.451 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:58:02.451 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:58:02.451 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:58:02.451 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:58:02.451 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000027", 2024-08-18T15:58:02.451 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:58:02.451 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status", 2024-08-18T15:58:02.451 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:58:02.451 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000019", 2024-08-18T15:58:02.451 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:58:02.452 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:58:02.452 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:58:02.452 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:58:02.452 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1", 2024-08-18T15:58:02.452 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:58:02.452 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:58:02.452 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:58:02.452 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:58:02.452 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000039", 2024-08-18T15:58:02.452 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:58:02.452 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:58:02.452 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:58:02.452 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:58:02.453 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:58:02.453 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:58:02.453 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:58:02.453 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:58:02.453 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:58:02.453 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:58:02.453 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:58:02.453 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000100", 2024-08-18T15:58:02.453 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:58:02.453 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:58:02.453 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:58:02.453 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:58:02.453 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:58:02.454 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000093", 2024-08-18T15:58:02.454 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:58:02.454 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000004", 2024-08-18T15:58:02.454 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000063", 2024-08-18T15:58:02.454 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000122", 2024-08-18T15:58:02.454 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:58:02.454 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:58:02.454 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:58:02.454 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000057", 2024-08-18T15:58:02.454 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:58:02.454 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:58:02.454 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:58:02.454 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:58:02.455 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000054", 2024-08-18T15:58:02.455 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:58:02.455 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000114", 2024-08-18T15:58:02.455 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:58:02.455 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:58:02.455 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:58:02.455 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.2", 2024-08-18T15:58:02.455 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:58:02.455 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:58:02.455 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0", 2024-08-18T15:58:02.455 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:58:02.455 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:58:02.455 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:58:02.455 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:58:02.456 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:58:02.456 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:58:02.456 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000012", 2024-08-18T15:58:02.456 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:58:02.456 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:58:02.456 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:58:02.456 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000084", 2024-08-18T15:58:02.456 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000043", 2024-08-18T15:58:02.456 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3", 2024-08-18T15:58:02.456 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:58:02.456 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:58:02.456 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.11", 2024-08-18T15:58:02.456 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:58:02.456 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:58:02.457 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:58:02.457 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:58:02.457 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:58:02.457 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:58:02.457 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:58:02.457 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:58:02.457 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000111", 2024-08-18T15:58:02.457 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:58:02.457 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000048", 2024-08-18T15:58:02.457 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:58:02.457 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000020", 2024-08-18T15:58:02.457 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:58:02.458 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:58:02.458 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:58:02.458 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:58:02.458 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:58:02.458 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000099", 2024-08-18T15:58:02.458 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000056", 2024-08-18T15:58:02.458 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:58:02.458 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:58:02.458 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.19", 2024-08-18T15:58:02.458 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:58:02.458 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.28", 2024-08-18T15:58:02.458 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:58:02.459 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:58:02.459 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:58:02.459 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:58:02.459 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:58:02.459 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000072", 2024-08-18T15:58:02.459 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:58:02.459 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:58:02.459 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000062", 2024-08-18T15:58:02.459 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:58:02.459 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:58:02.459 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:58:02.459 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000115", 2024-08-18T15:58:02.460 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:58:02.460 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.1", 2024-08-18T15:58:02.460 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2", 2024-08-18T15:58:02.460 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000033", 2024-08-18T15:58:02.460 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:58:02.460 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:58:02.460 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:58:02.460 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:58:02.460 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:58:02.460 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:58:02.460 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:58:02.460 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:58:02.460 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:58:02.460 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:58:02.461 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:58:02.461 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:58:02.461 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:58:02.461 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:58:02.461 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:58:02.461 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:58:02.461 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:58:02.461 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:58:02.461 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:58:02.461 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000025", 2024-08-18T15:58:02.461 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:58:02.461 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000123", 2024-08-18T15:58:02.461 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.31", 2024-08-18T15:58:02.461 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:58:02.461 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:58:02.462 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:58:02.462 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:58:02.462 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.18", 2024-08-18T15:58:02.462 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:58:02.462 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:58:02.462 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:58:02.462 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:58:02.462 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:58:02.462 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000108", 2024-08-18T15:58:02.462 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:58:02.462 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:58:02.462 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000094", 2024-08-18T15:58:02.463 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:58:02.463 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:58:02.463 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:58:02.463 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:58:02.463 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:58:02.463 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:58:02.463 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000060", 2024-08-18T15:58:02.463 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:58:02.463 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:58:02.463 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:58:02.463 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000109", 2024-08-18T15:58:02.463 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.15", 2024-08-18T15:58:02.463 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:58:02.464 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:58:02.464 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:58:02.464 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:58:02.464 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:58:02.464 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:58:02.464 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:58:02.464 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:58:02.464 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:58:02.464 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000013", 2024-08-18T15:58:02.464 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:58:02.464 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:58:02.464 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:58:02.465 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:58:02.465 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:58:02.465 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:58:02.465 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:58:02.465 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:58:02.465 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:58:02.465 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:58:02.465 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:58:02.465 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:58:02.465 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:58:02.465 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:58:02.465 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000051", 2024-08-18T15:58:02.465 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:58:02.466 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:58:02.466 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:58:02.466 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:58:02.466 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000106", 2024-08-18T15:58:02.466 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:58:02.466 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:58:02.466 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:58:02.466 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:58:02.466 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:58:02.466 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000017", 2024-08-18T15:58:02.466 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:58:02.466 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:58:02.466 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:58:02.467 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000023", 2024-08-18T15:58:02.467 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:58:02.467 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:58:02.467 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:58:02.467 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:58:02.467 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:58:02.467 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:58:02.467 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:58:02.467 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000107", 2024-08-18T15:58:02.467 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:58:02.467 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:58:02.467 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000046", 2024-08-18T15:58:02.467 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:58:02.468 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:58:02.468 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:58:02.468 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.26", 2024-08-18T15:58:02.468 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:58:02.468 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:58:02.468 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9", 2024-08-18T15:58:02.468 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.22", 2024-08-18T15:58:02.468 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:58:02.468 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:58:02.468 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:58:02.468 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:58:02.468 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:58:02.468 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:58:02.468 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:58:02.469 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2", 2024-08-18T15:58:02.469 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:58:02.469 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:58:02.469 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000067", 2024-08-18T15:58:02.469 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:58:02.469 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:58:02.469 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:58:02.469 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:58:02.469 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:58:02.469 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:58:02.469 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:58:02.469 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:58:02.469 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:58:02.470 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:58:02.470 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:58:02.470 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:58:02.470 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:58:02.470 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:58:02.470 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:58:02.470 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:58:02.470 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:58:02.470 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:58:02.470 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:58:02.470 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.9", 2024-08-18T15:58:02.470 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000086", 2024-08-18T15:58:02.471 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.5", 2024-08-18T15:58:02.471 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:58:02.471 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:58:02.471 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:58:02.471 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:58:02.471 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:58:02.471 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:58:02.471 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2.MFXJ583gCr1.0", 2024-08-18T15:58:02.471 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:58:02.471 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1.GPnO16liyZ+.0", 2024-08-18T15:58:02.471 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:58:02.471 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:58:02.471 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:58:02.472 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:58:02.472 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:58:02.472 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:58:02.472 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:58:02.472 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:58:02.472 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:58:02.472 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.20", 2024-08-18T15:58:02.472 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:58:02.472 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:58:02.472 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:58:02.472 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:58:02.472 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:58:02.473 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:58:02.473 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1.retry", 2024-08-18T15:58:02.473 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:58:02.473 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:58:02.473 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:58:02.473 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0.retry", 2024-08-18T15:58:02.473 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000050", 2024-08-18T15:58:02.473 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:58:02.473 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000102", 2024-08-18T15:58:02.473 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:58:02.473 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:58:02.473 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:58:02.474 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:58:02.474 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:58:02.474 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:58:02.474 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:58:02.474 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:58:02.474 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:58:02.474 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:58:02.474 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:58:02.474 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000124", 2024-08-18T15:58:02.474 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:58:02.474 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:58:02.474 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:58:02.475 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000001", 2024-08-18T15:58:02.475 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:58:02.475 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:58:02.475 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:58:02.475 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:58:02.475 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000126", 2024-08-18T15:58:02.475 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:58:02.475 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0.fZhctfLXnSt.0", 2024-08-18T15:58:02.475 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:58:02.475 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000055", 2024-08-18T15:58:02.475 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:58:02.475 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000061", 2024-08-18T15:58:02.476 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1", 2024-08-18T15:58:02.476 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:58:02.476 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:58:02.476 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:58:02.476 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000097", 2024-08-18T15:58:02.476 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:58:02.476 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:58:02.476 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:58:02.476 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:58:02.476 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:58:02.476 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:58:02.476 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:58:02.476 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:58:02.476 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000042", 2024-08-18T15:58:02.477 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:58:02.477 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:58:02.477 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:58:02.477 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1.retry", 2024-08-18T15:58:02.477 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:58:02.477 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:58:02.477 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:58:02.477 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:58:02.477 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:58:02.477 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:58:02.477 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:58:02.477 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000075", 2024-08-18T15:58:02.477 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:58:02.478 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000120", 2024-08-18T15:58:02.478 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000030", 2024-08-18T15:58:02.478 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:58:02.478 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:58:02.478 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:58:02.478 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:58:02.478 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:58:02.478 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:58:02.478 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9" 2024-08-18T15:58:02.478 INFO:teuthology.orchestra.run.smithi193.stdout:] 2024-08-18T15:58:02.478 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:58:07.480 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:58:07.480 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:58:07.480 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:58:07.533 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:58:07.534 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:58:07.596 INFO:teuthology.orchestra.run.smithi193.stdout:[ 2024-08-18T15:58:07.596 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:58:07.596 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:58:07.596 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.29", 2024-08-18T15:58:07.596 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000078", 2024-08-18T15:58:07.596 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.history", 2024-08-18T15:58:07.596 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:58:07.596 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:58:07.597 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:58:07.597 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:58:07.597 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.23", 2024-08-18T15:58:07.597 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000070", 2024-08-18T15:58:07.597 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000104", 2024-08-18T15:58:07.597 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:58:07.597 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2.retry", 2024-08-18T15:58:07.597 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:58:07.597 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:58:07.597 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:58:07.597 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:58:07.597 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000026", 2024-08-18T15:58:07.597 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:58:07.598 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:58:07.598 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:58:07.598 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:58:07.598 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:58:07.598 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:58:07.598 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.8", 2024-08-18T15:58:07.598 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:58:07.598 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:58:07.598 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:58:07.598 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:58:07.598 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000028", 2024-08-18T15:58:07.599 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:58:07.599 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:58:07.599 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:58:07.599 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:58:07.599 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:58:07.599 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:58:07.600 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:58:07.600 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2.retry", 2024-08-18T15:58:07.600 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:58:07.600 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.30", 2024-08-18T15:58:07.600 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:58:07.600 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:58:07.600 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:58:07.600 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:58:07.600 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000040", 2024-08-18T15:58:07.600 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:58:07.601 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.6", 2024-08-18T15:58:07.601 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:58:07.601 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:58:07.601 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:58:07.601 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:58:07.601 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:58:07.601 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:58:07.601 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000015", 2024-08-18T15:58:07.601 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:58:07.601 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:58:07.601 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000069", 2024-08-18T15:58:07.601 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:58:07.602 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:58:07.602 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:58:07.602 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000095", 2024-08-18T15:58:07.602 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0", 2024-08-18T15:58:07.602 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000003", 2024-08-18T15:58:07.602 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000047", 2024-08-18T15:58:07.602 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:58:07.602 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:58:07.602 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:58:07.602 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:58:07.602 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:58:07.602 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:58:07.602 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:58:07.603 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:58:07.603 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:58:07.603 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000035", 2024-08-18T15:58:07.603 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:58:07.603 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:58:07.603 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000037", 2024-08-18T15:58:07.603 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:58:07.603 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:58:07.603 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:58:07.603 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:58:07.603 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:58:07.603 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:58:07.604 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:58:07.604 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:58:07.604 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:58:07.604 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:58:07.604 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:58:07.604 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:58:07.604 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:58:07.604 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:58:07.604 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000024", 2024-08-18T15:58:07.604 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:58:07.604 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:58:07.604 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:58:07.605 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:58:07.605 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000000", 2024-08-18T15:58:07.605 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:58:07.605 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000031", 2024-08-18T15:58:07.605 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:58:07.605 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:58:07.605 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:58:07.605 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:58:07.605 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:58:07.605 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:58:07.605 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3", 2024-08-18T15:58:07.605 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:58:07.605 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:58:07.605 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:58:07.606 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:58:07.606 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:58:07.606 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:58:07.606 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:58:07.606 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:58:07.606 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:58:07.606 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:58:07.606 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:58:07.606 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:58:07.606 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:58:07.606 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:58:07.606 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:58:07.606 INFO:teuthology.orchestra.run.smithi193.stdout: "data_loggenerations_metadata", 2024-08-18T15:58:07.607 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2", 2024-08-18T15:58:07.607 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:58:07.607 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:58:07.607 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:58:07.607 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:58:07.607 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T15:58:07.607 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:58:07.607 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:58:07.607 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:58:07.607 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:58:07.607 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:58:07.607 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000076", 2024-08-18T15:58:07.607 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:58:07.608 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:58:07.608 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000113", 2024-08-18T15:58:07.608 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000005", 2024-08-18T15:58:07.608 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:58:07.608 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000011", 2024-08-18T15:58:07.608 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:58:07.608 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000058", 2024-08-18T15:58:07.608 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:58:07.608 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:58:07.608 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:58:07.608 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000052", 2024-08-18T15:58:07.608 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:58:07.608 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:58:07.609 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:58:07.609 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:58:07.609 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.21", 2024-08-18T15:58:07.609 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:58:07.609 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:58:07.609 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:58:07.609 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:58:07.609 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:58:07.609 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:58:07.609 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:58:07.609 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:58:07.609 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:58:07.609 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:58:07.610 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:58:07.610 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:58:07.610 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:58:07.610 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:58:07.610 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:58:07.610 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:58:07.610 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:58:07.610 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:58:07.610 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:58:07.610 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000088", 2024-08-18T15:58:07.610 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000080", 2024-08-18T15:58:07.610 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:58:07.611 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:58:07.611 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000090", 2024-08-18T15:58:07.611 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:58:07.611 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:58:07.611 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000110", 2024-08-18T15:58:07.611 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:58:07.611 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:58:07.611 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:58:07.611 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.0", 2024-08-18T15:58:07.611 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:58:07.611 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000096", 2024-08-18T15:58:07.611 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:58:07.611 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:58:07.611 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:58:07.612 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:58:07.612 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:58:07.612 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000087", 2024-08-18T15:58:07.612 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:58:07.612 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:58:07.612 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:58:07.612 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:58:07.612 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000008", 2024-08-18T15:58:07.612 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:58:07.612 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:58:07.612 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:58:07.613 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:58:07.613 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:58:07.613 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000006", 2024-08-18T15:58:07.613 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000029", 2024-08-18T15:58:07.613 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:58:07.613 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000089", 2024-08-18T15:58:07.613 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:58:07.613 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:58:07.613 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000022", 2024-08-18T15:58:07.613 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:58:07.613 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:58:07.614 INFO:teuthology.orchestra.run.smithi193.stdout: "bilog.trim", 2024-08-18T15:58:07.614 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000016", 2024-08-18T15:58:07.614 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:58:07.614 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:58:07.614 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:58:07.614 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:58:07.614 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:58:07.614 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000041", 2024-08-18T15:58:07.614 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:58:07.614 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:58:07.614 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.log.01050e30-179b-43be-bdbb-3bb23cf37ae8.3", 2024-08-18T15:58:07.614 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:58:07.615 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000018", 2024-08-18T15:58:07.615 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000092", 2024-08-18T15:58:07.615 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:58:07.615 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:58:07.615 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:58:07.615 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:58:07.615 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.27", 2024-08-18T15:58:07.615 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.4", 2024-08-18T15:58:07.615 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:58:07.615 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:58:07.615 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:58:07.615 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:58:07.615 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:58:07.615 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:58:07.616 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:58:07.616 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000014", 2024-08-18T15:58:07.616 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:58:07.616 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3.retry", 2024-08-18T15:58:07.616 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:58:07.616 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:58:07.616 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:58:07.616 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:58:07.616 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:58:07.616 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000112", 2024-08-18T15:58:07.616 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:58:07.616 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000007", 2024-08-18T15:58:07.616 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000021", 2024-08-18T15:58:07.617 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:58:07.617 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:58:07.617 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:58:07.617 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:58:07.617 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:58:07.617 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.16", 2024-08-18T15:58:07.617 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:58:07.617 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000064", 2024-08-18T15:58:07.617 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:58:07.617 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:58:07.617 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:58:07.617 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:58:07.617 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:58:07.618 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:58:07.618 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:58:07.618 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000071", 2024-08-18T15:58:07.618 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:58:07.618 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:58:07.618 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:58:07.618 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000074", 2024-08-18T15:58:07.618 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:58:07.618 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:58:07.618 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:58:07.618 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:58:07.618 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:58:07.618 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:58:07.619 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:58:07.619 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:58:07.619 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:58:07.619 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:58:07.619 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:58:07.619 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:58:07.619 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:58:07.619 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:58:07.619 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:58:07.619 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000081", 2024-08-18T15:58:07.619 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:58:07.619 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000009", 2024-08-18T15:58:07.619 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000121", 2024-08-18T15:58:07.620 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1", 2024-08-18T15:58:07.620 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:58:07.620 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:58:07.620 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:58:07.620 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:58:07.620 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:58:07.620 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:58:07.620 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.17", 2024-08-18T15:58:07.620 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000125", 2024-08-18T15:58:07.620 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:58:07.620 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000082", 2024-08-18T15:58:07.620 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:58:07.620 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:58:07.621 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:58:07.621 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:58:07.621 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:58:07.621 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:58:07.621 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:58:07.621 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:58:07.621 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:58:07.621 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:58:07.621 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:58:07.621 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000105", 2024-08-18T15:58:07.621 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000059", 2024-08-18T15:58:07.621 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:58:07.621 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:58:07.621 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:58:07.622 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:58:07.622 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:58:07.622 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:58:07.622 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:58:07.622 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:58:07.622 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:58:07.622 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:58:07.622 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:58:07.622 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000077", 2024-08-18T15:58:07.622 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000032", 2024-08-18T15:58:07.622 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:58:07.622 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:58:07.623 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:58:07.623 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:58:07.623 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:58:07.623 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:58:07.623 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.24", 2024-08-18T15:58:07.623 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:58:07.623 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:58:07.623 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:58:07.623 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:58:07.623 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3.retry", 2024-08-18T15:58:07.623 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:58:07.623 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:58:07.623 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:58:07.624 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:58:07.624 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:58:07.624 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:58:07.624 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000053", 2024-08-18T15:58:07.624 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000091", 2024-08-18T15:58:07.624 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:58:07.624 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:58:07.624 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:58:07.624 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000065", 2024-08-18T15:58:07.624 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:58:07.624 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:58:07.624 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:58:07.624 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:58:07.625 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000083", 2024-08-18T15:58:07.625 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000010", 2024-08-18T15:58:07.625 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:58:07.625 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:58:07.625 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:58:07.625 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:58:07.625 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:58:07.625 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:58:07.625 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:58:07.625 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:58:07.625 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:58:07.625 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:58:07.625 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:58:07.625 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.7", 2024-08-18T15:58:07.626 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.3", 2024-08-18T15:58:07.626 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:58:07.626 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:58:07.626 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:58:07.626 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:58:07.626 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:58:07.626 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:58:07.626 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:58:07.626 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:58:07.626 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.2", 2024-08-18T15:58:07.626 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:58:07.626 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:58:07.626 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:58:07.627 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:58:07.627 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:58:07.627 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:58:07.627 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:58:07.627 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000045", 2024-08-18T15:58:07.627 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000002", 2024-08-18T15:58:07.627 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:58:07.627 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3.2UbiLidwSTy.0", 2024-08-18T15:58:07.627 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:58:07.627 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.12", 2024-08-18T15:58:07.627 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:58:07.627 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:58:07.628 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:58:07.628 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:58:07.628 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:58:07.628 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:58:07.628 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000116", 2024-08-18T15:58:07.628 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.25", 2024-08-18T15:58:07.628 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:58:07.628 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:58:07.628 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:58:07.628 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:58:07.628 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000034", 2024-08-18T15:58:07.628 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:58:07.628 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:58:07.629 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000101", 2024-08-18T15:58:07.629 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:58:07.629 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:58:07.629 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.13", 2024-08-18T15:58:07.629 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:58:07.629 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:58:07.629 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.1", 2024-08-18T15:58:07.629 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0.retry", 2024-08-18T15:58:07.629 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:58:07.629 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:58:07.629 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000079", 2024-08-18T15:58:07.629 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000049", 2024-08-18T15:58:07.630 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:58:07.630 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0", 2024-08-18T15:58:07.630 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:58:07.630 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:58:07.630 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000117", 2024-08-18T15:58:07.630 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:58:07.630 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000044", 2024-08-18T15:58:07.630 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:58:07.630 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:58:07.630 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:58:07.630 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:58:07.630 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:58:07.630 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:58:07.631 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.14", 2024-08-18T15:58:07.631 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:58:07.631 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000066", 2024-08-18T15:58:07.631 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:58:07.631 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:58:07.631 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:58:07.631 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:58:07.631 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:58:07.631 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000068", 2024-08-18T15:58:07.631 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:58:07.631 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000085", 2024-08-18T15:58:07.631 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:58:07.631 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:58:07.632 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:58:07.632 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:58:07.632 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:58:07.632 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:58:07.632 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:58:07.632 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:58:07.632 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:58:07.632 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:58:07.632 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:58:07.632 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:58:07.632 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000073", 2024-08-18T15:58:07.632 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:58:07.632 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:58:07.633 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:58:07.633 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000038", 2024-08-18T15:58:07.633 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:58:07.633 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:58:07.633 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:58:07.633 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:58:07.633 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.10", 2024-08-18T15:58:07.633 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:58:07.633 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:58:07.633 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:58:07.633 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000118", 2024-08-18T15:58:07.633 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:58:07.633 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3", 2024-08-18T15:58:07.633 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:58:07.633 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000036", 2024-08-18T15:58:07.634 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:58:07.634 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:58:07.634 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:58:07.634 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:58:07.634 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:58:07.634 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000103", 2024-08-18T15:58:07.634 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.3", 2024-08-18T15:58:07.634 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:58:07.634 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000119", 2024-08-18T15:58:07.634 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000098", 2024-08-18T15:58:07.634 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:58:07.634 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:58:07.634 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:58:07.635 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:58:07.635 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:58:07.635 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:58:07.635 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:58:07.635 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:58:07.635 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.0", 2024-08-18T15:58:07.635 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:58:07.635 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:58:07.635 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:58:07.635 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:58:07.635 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:58:07.635 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:58:07.635 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:58:07.636 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:58:07.636 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000027", 2024-08-18T15:58:07.636 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:58:07.636 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status", 2024-08-18T15:58:07.636 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:58:07.636 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000019", 2024-08-18T15:58:07.636 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:58:07.636 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:58:07.636 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:58:07.636 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:58:07.636 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1", 2024-08-18T15:58:07.636 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:58:07.636 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:58:07.636 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:58:07.636 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:58:07.637 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000039", 2024-08-18T15:58:07.637 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:58:07.637 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:58:07.637 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:58:07.637 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:58:07.637 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:58:07.637 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:58:07.637 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:58:07.637 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:58:07.637 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:58:07.637 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:58:07.637 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:58:07.637 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000100", 2024-08-18T15:58:07.638 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:58:07.638 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:58:07.638 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:58:07.638 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:58:07.638 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:58:07.638 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000093", 2024-08-18T15:58:07.638 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:58:07.638 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000004", 2024-08-18T15:58:07.638 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000063", 2024-08-18T15:58:07.638 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000122", 2024-08-18T15:58:07.638 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:58:07.638 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:58:07.638 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:58:07.639 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000057", 2024-08-18T15:58:07.639 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:58:07.639 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:58:07.639 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:58:07.639 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:58:07.639 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000054", 2024-08-18T15:58:07.639 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:58:07.639 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000114", 2024-08-18T15:58:07.639 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:58:07.639 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:58:07.639 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:58:07.639 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.2", 2024-08-18T15:58:07.639 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:58:07.640 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:58:07.640 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0", 2024-08-18T15:58:07.640 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:58:07.640 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:58:07.640 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:58:07.640 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:58:07.640 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:58:07.640 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:58:07.640 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000012", 2024-08-18T15:58:07.640 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:58:07.641 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:58:07.641 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:58:07.641 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000084", 2024-08-18T15:58:07.641 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000043", 2024-08-18T15:58:07.641 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3", 2024-08-18T15:58:07.641 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:58:07.641 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:58:07.641 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.11", 2024-08-18T15:58:07.641 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:58:07.641 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:58:07.641 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:58:07.641 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:58:07.641 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:58:07.642 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:58:07.642 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:58:07.642 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:58:07.642 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000111", 2024-08-18T15:58:07.642 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:58:07.642 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000048", 2024-08-18T15:58:07.642 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:58:07.642 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000020", 2024-08-18T15:58:07.642 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:58:07.642 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:58:07.642 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:58:07.642 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:58:07.642 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:58:07.643 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000099", 2024-08-18T15:58:07.643 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000056", 2024-08-18T15:58:07.643 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:58:07.643 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:58:07.643 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.19", 2024-08-18T15:58:07.643 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:58:07.643 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.28", 2024-08-18T15:58:07.643 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:58:07.643 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:58:07.643 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:58:07.643 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:58:07.643 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:58:07.643 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000072", 2024-08-18T15:58:07.644 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:58:07.644 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:58:07.644 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000062", 2024-08-18T15:58:07.644 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:58:07.644 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:58:07.644 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:58:07.644 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000115", 2024-08-18T15:58:07.644 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:58:07.644 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.1", 2024-08-18T15:58:07.644 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2", 2024-08-18T15:58:07.644 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000033", 2024-08-18T15:58:07.644 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:58:07.645 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:58:07.645 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:58:07.645 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:58:07.645 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:58:07.645 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:58:07.645 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:58:07.645 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:58:07.645 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:58:07.645 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:58:07.645 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:58:07.645 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:58:07.645 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:58:07.645 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:58:07.646 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:58:07.646 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:58:07.646 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:58:07.646 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:58:07.646 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:58:07.646 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000025", 2024-08-18T15:58:07.646 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:58:07.646 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000123", 2024-08-18T15:58:07.646 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.31", 2024-08-18T15:58:07.646 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:58:07.646 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:58:07.646 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:58:07.646 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:58:07.647 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.18", 2024-08-18T15:58:07.647 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:58:07.647 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:58:07.647 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:58:07.647 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:58:07.647 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:58:07.647 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000108", 2024-08-18T15:58:07.647 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:58:07.647 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:58:07.647 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000094", 2024-08-18T15:58:07.647 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:58:07.647 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:58:07.647 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:58:07.648 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:58:07.648 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:58:07.648 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:58:07.648 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000060", 2024-08-18T15:58:07.648 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:58:07.648 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:58:07.648 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:58:07.648 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000109", 2024-08-18T15:58:07.648 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.15", 2024-08-18T15:58:07.648 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:58:07.648 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:58:07.648 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:58:07.648 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:58:07.649 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:58:07.649 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:58:07.649 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:58:07.649 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:58:07.649 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:58:07.649 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000013", 2024-08-18T15:58:07.649 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:58:07.649 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:58:07.649 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:58:07.649 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:58:07.649 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:58:07.649 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:58:07.649 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:58:07.650 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:58:07.650 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:58:07.650 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:58:07.650 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:58:07.650 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:58:07.650 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:58:07.650 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:58:07.650 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000051", 2024-08-18T15:58:07.650 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:58:07.650 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:58:07.650 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:58:07.650 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:58:07.650 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000106", 2024-08-18T15:58:07.651 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:58:07.651 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:58:07.651 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:58:07.651 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:58:07.651 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:58:07.651 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000017", 2024-08-18T15:58:07.651 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:58:07.651 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:58:07.651 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:58:07.651 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000023", 2024-08-18T15:58:07.651 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:58:07.651 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:58:07.651 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:58:07.652 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:58:07.652 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:58:07.652 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:58:07.652 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:58:07.652 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000107", 2024-08-18T15:58:07.652 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:58:07.652 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:58:07.652 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000046", 2024-08-18T15:58:07.652 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:58:07.652 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:58:07.652 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:58:07.652 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.26", 2024-08-18T15:58:07.652 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:58:07.653 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:58:07.653 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9", 2024-08-18T15:58:07.653 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.22", 2024-08-18T15:58:07.653 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:58:07.653 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:58:07.653 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:58:07.653 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:58:07.653 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:58:07.653 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:58:07.653 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:58:07.653 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2", 2024-08-18T15:58:07.653 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:58:07.653 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:58:07.654 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000067", 2024-08-18T15:58:07.654 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:58:07.654 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:58:07.654 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:58:07.654 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:58:07.654 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:58:07.654 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:58:07.654 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:58:07.654 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:58:07.654 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:58:07.654 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:58:07.654 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:58:07.654 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:58:07.655 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:58:07.655 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:58:07.655 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:58:07.655 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:58:07.655 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:58:07.655 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:58:07.655 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:58:07.655 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.9", 2024-08-18T15:58:07.655 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000086", 2024-08-18T15:58:07.655 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.5", 2024-08-18T15:58:07.655 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:58:07.655 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:58:07.655 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:58:07.656 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:58:07.656 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:58:07.656 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:58:07.656 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2.MFXJ583gCr1.0", 2024-08-18T15:58:07.656 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:58:07.656 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1.GPnO16liyZ+.0", 2024-08-18T15:58:07.656 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:58:07.656 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:58:07.656 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:58:07.656 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:58:07.656 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:58:07.656 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:58:07.656 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:58:07.656 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:58:07.657 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:58:07.657 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.20", 2024-08-18T15:58:07.657 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:58:07.657 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:58:07.657 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:58:07.657 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:58:07.657 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:58:07.657 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:58:07.657 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1.retry", 2024-08-18T15:58:07.657 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:58:07.657 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:58:07.657 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:58:07.657 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0.retry", 2024-08-18T15:58:07.657 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000050", 2024-08-18T15:58:07.658 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:58:07.658 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000102", 2024-08-18T15:58:07.658 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:58:07.658 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:58:07.658 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:58:07.658 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:58:07.658 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:58:07.658 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:58:07.658 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:58:07.658 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:58:07.658 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:58:07.658 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:58:07.658 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:58:07.659 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000124", 2024-08-18T15:58:07.659 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:58:07.659 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:58:07.659 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:58:07.659 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000001", 2024-08-18T15:58:07.659 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:58:07.659 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:58:07.659 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:58:07.659 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:58:07.659 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000126", 2024-08-18T15:58:07.659 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:58:07.659 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0.fZhctfLXnSt.0", 2024-08-18T15:58:07.659 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:58:07.660 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000055", 2024-08-18T15:58:07.660 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:58:07.660 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000061", 2024-08-18T15:58:07.660 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1", 2024-08-18T15:58:07.660 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:58:07.660 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:58:07.660 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:58:07.660 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000097", 2024-08-18T15:58:07.660 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:58:07.660 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:58:07.660 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:58:07.660 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:58:07.660 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:58:07.661 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:58:07.661 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:58:07.661 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:58:07.661 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000042", 2024-08-18T15:58:07.661 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:58:07.661 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:58:07.661 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:58:07.661 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1.retry", 2024-08-18T15:58:07.661 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:58:07.661 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:58:07.661 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:58:07.661 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:58:07.661 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:58:07.661 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:58:07.662 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:58:07.662 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000075", 2024-08-18T15:58:07.662 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:58:07.662 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000120", 2024-08-18T15:58:07.662 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000030", 2024-08-18T15:58:07.662 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:58:07.662 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:58:07.662 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:58:07.662 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:58:07.662 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:58:07.662 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:58:07.662 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9" 2024-08-18T15:58:07.662 INFO:teuthology.orchestra.run.smithi193.stdout:] 2024-08-18T15:58:07.663 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:58:12.665 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:58:12.665 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:58:12.665 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:58:12.717 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:58:12.718 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:58:12.781 INFO:teuthology.orchestra.run.smithi193.stdout:[ 2024-08-18T15:58:12.781 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:58:12.782 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:58:12.782 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.29", 2024-08-18T15:58:12.782 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000078", 2024-08-18T15:58:12.782 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.history", 2024-08-18T15:58:12.782 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:58:12.782 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:58:12.782 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:58:12.782 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:58:12.782 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.23", 2024-08-18T15:58:12.782 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000070", 2024-08-18T15:58:12.782 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000104", 2024-08-18T15:58:12.783 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:58:12.783 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2.retry", 2024-08-18T15:58:12.783 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:58:12.783 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:58:12.783 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:58:12.783 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:58:12.783 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000026", 2024-08-18T15:58:12.783 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:58:12.783 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:58:12.783 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:58:12.783 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:58:12.783 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:58:12.783 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:58:12.783 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.8", 2024-08-18T15:58:12.784 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:58:12.784 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:58:12.784 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:58:12.784 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:58:12.784 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000028", 2024-08-18T15:58:12.784 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:58:12.784 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:58:12.784 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:58:12.784 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:58:12.784 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:58:12.784 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:58:12.784 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:58:12.784 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2.retry", 2024-08-18T15:58:12.784 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:58:12.785 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.30", 2024-08-18T15:58:12.785 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:58:12.785 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:58:12.785 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:58:12.785 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:58:12.785 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000040", 2024-08-18T15:58:12.785 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:58:12.785 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.6", 2024-08-18T15:58:12.785 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:58:12.785 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:58:12.785 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:58:12.785 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:58:12.785 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:58:12.785 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:58:12.786 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000015", 2024-08-18T15:58:12.786 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:58:12.786 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:58:12.786 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000069", 2024-08-18T15:58:12.786 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:58:12.786 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:58:12.786 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:58:12.786 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000095", 2024-08-18T15:58:12.786 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0", 2024-08-18T15:58:12.786 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000003", 2024-08-18T15:58:12.786 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000047", 2024-08-18T15:58:12.786 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:58:12.787 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:58:12.787 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:58:12.787 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:58:12.787 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:58:12.787 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:58:12.787 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:58:12.787 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:58:12.787 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:58:12.787 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000035", 2024-08-18T15:58:12.787 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:58:12.787 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:58:12.787 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000037", 2024-08-18T15:58:12.787 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:58:12.787 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:58:12.788 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:58:12.788 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:58:12.788 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:58:12.788 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:58:12.788 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:58:12.788 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:58:12.788 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:58:12.788 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:58:12.788 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:58:12.788 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:58:12.788 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:58:12.788 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:58:12.789 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000024", 2024-08-18T15:58:12.789 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:58:12.789 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:58:12.789 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:58:12.789 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:58:12.789 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000000", 2024-08-18T15:58:12.789 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:58:12.789 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000031", 2024-08-18T15:58:12.789 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:58:12.789 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:58:12.789 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:58:12.789 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:58:12.790 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:58:12.790 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:58:12.790 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3", 2024-08-18T15:58:12.790 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:58:12.790 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:58:12.790 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:58:12.790 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:58:12.790 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:58:12.790 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:58:12.790 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:58:12.790 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:58:12.791 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:58:12.791 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:58:12.791 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:58:12.791 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:58:12.791 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:58:12.791 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:58:12.791 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:58:12.791 INFO:teuthology.orchestra.run.smithi193.stdout: "data_loggenerations_metadata", 2024-08-18T15:58:12.791 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2", 2024-08-18T15:58:12.791 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:58:12.791 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:58:12.791 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:58:12.791 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:58:12.792 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T15:58:12.792 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:58:12.792 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:58:12.792 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:58:12.792 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:58:12.792 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:58:12.792 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000076", 2024-08-18T15:58:12.792 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:58:12.792 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:58:12.792 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000113", 2024-08-18T15:58:12.792 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000005", 2024-08-18T15:58:12.792 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:58:12.792 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000011", 2024-08-18T15:58:12.792 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:58:12.792 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000058", 2024-08-18T15:58:12.793 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:58:12.793 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:58:12.793 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:58:12.793 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000052", 2024-08-18T15:58:12.793 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:58:12.793 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:58:12.793 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:58:12.793 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:58:12.793 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.21", 2024-08-18T15:58:12.793 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:58:12.793 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:58:12.793 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:58:12.793 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:58:12.793 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:58:12.793 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:58:12.794 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:58:12.794 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:58:12.794 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:58:12.794 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:58:12.794 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:58:12.794 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:58:12.794 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:58:12.794 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:58:12.794 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:58:12.794 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:58:12.794 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:58:12.794 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:58:12.794 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:58:12.795 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000088", 2024-08-18T15:58:12.795 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000080", 2024-08-18T15:58:12.795 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:58:12.795 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:58:12.795 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000090", 2024-08-18T15:58:12.795 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:58:12.795 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:58:12.795 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000110", 2024-08-18T15:58:12.795 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:58:12.796 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:58:12.796 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:58:12.796 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.0", 2024-08-18T15:58:12.796 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:58:12.796 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000096", 2024-08-18T15:58:12.796 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:58:12.796 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:58:12.796 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:58:12.796 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:58:12.796 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:58:12.796 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000087", 2024-08-18T15:58:12.796 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:58:12.796 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:58:12.797 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:58:12.797 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:58:12.797 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000008", 2024-08-18T15:58:12.797 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:58:12.797 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:58:12.797 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:58:12.797 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:58:12.797 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:58:12.797 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000006", 2024-08-18T15:58:12.797 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000029", 2024-08-18T15:58:12.797 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:58:12.797 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000089", 2024-08-18T15:58:12.797 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:58:12.797 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:58:12.798 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000022", 2024-08-18T15:58:12.798 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:58:12.798 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:58:12.798 INFO:teuthology.orchestra.run.smithi193.stdout: "bilog.trim", 2024-08-18T15:58:12.798 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000016", 2024-08-18T15:58:12.798 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:58:12.798 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:58:12.798 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:58:12.798 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:58:12.798 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:58:12.798 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000041", 2024-08-18T15:58:12.798 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:58:12.799 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:58:12.799 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.log.01050e30-179b-43be-bdbb-3bb23cf37ae8.3", 2024-08-18T15:58:12.799 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:58:12.799 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000018", 2024-08-18T15:58:12.799 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000092", 2024-08-18T15:58:12.799 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:58:12.799 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:58:12.799 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:58:12.799 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:58:12.799 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.27", 2024-08-18T15:58:12.799 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.4", 2024-08-18T15:58:12.799 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:58:12.799 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:58:12.799 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:58:12.800 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:58:12.800 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:58:12.800 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:58:12.800 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:58:12.800 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000014", 2024-08-18T15:58:12.800 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:58:12.800 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3.retry", 2024-08-18T15:58:12.800 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:58:12.800 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:58:12.800 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:58:12.800 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:58:12.800 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:58:12.800 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000112", 2024-08-18T15:58:12.800 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:58:12.801 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000007", 2024-08-18T15:58:12.801 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000021", 2024-08-18T15:58:12.801 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:58:12.801 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:58:12.801 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:58:12.801 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:58:12.801 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:58:12.801 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.16", 2024-08-18T15:58:12.801 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:58:12.801 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000064", 2024-08-18T15:58:12.801 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:58:12.801 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:58:12.801 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:58:12.801 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:58:12.801 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:58:12.802 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:58:12.802 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:58:12.802 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000071", 2024-08-18T15:58:12.802 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:58:12.802 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:58:12.802 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:58:12.802 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000074", 2024-08-18T15:58:12.802 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:58:12.802 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:58:12.802 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:58:12.802 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:58:12.802 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:58:12.802 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:58:12.802 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:58:12.803 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:58:12.803 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:58:12.803 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:58:12.803 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:58:12.803 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:58:12.803 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:58:12.803 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:58:12.803 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:58:12.803 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000081", 2024-08-18T15:58:12.803 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:58:12.803 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000009", 2024-08-18T15:58:12.803 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000121", 2024-08-18T15:58:12.803 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1", 2024-08-18T15:58:12.804 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:58:12.804 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:58:12.804 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:58:12.804 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:58:12.804 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:58:12.804 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:58:12.804 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.17", 2024-08-18T15:58:12.804 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000125", 2024-08-18T15:58:12.804 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:58:12.804 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000082", 2024-08-18T15:58:12.804 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:58:12.804 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:58:12.804 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:58:12.805 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:58:12.805 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:58:12.805 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:58:12.805 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:58:12.805 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:58:12.805 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:58:12.805 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:58:12.805 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:58:12.805 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000105", 2024-08-18T15:58:12.805 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000059", 2024-08-18T15:58:12.805 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:58:12.805 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:58:12.805 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:58:12.806 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:58:12.806 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:58:12.806 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:58:12.806 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:58:12.806 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:58:12.806 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:58:12.806 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:58:12.806 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:58:12.806 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000077", 2024-08-18T15:58:12.806 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000032", 2024-08-18T15:58:12.806 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:58:12.807 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:58:12.807 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:58:12.807 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:58:12.807 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:58:12.807 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:58:12.807 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.24", 2024-08-18T15:58:12.807 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:58:12.807 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:58:12.807 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:58:12.807 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:58:12.807 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3.retry", 2024-08-18T15:58:12.807 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:58:12.807 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:58:12.808 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:58:12.808 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:58:12.808 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:58:12.808 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:58:12.808 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000053", 2024-08-18T15:58:12.808 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000091", 2024-08-18T15:58:12.808 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:58:12.808 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:58:12.808 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:58:12.808 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000065", 2024-08-18T15:58:12.808 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:58:12.808 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:58:12.808 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:58:12.809 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:58:12.809 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000083", 2024-08-18T15:58:12.809 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000010", 2024-08-18T15:58:12.809 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:58:12.809 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:58:12.809 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:58:12.809 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:58:12.809 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:58:12.809 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:58:12.809 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:58:12.809 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:58:12.809 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:58:12.809 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:58:12.810 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:58:12.810 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.7", 2024-08-18T15:58:12.810 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.3", 2024-08-18T15:58:12.810 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:58:12.810 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:58:12.810 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:58:12.810 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:58:12.810 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:58:12.810 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:58:12.810 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:58:12.810 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:58:12.810 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.2", 2024-08-18T15:58:12.810 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:58:12.811 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:58:12.811 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:58:12.811 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:58:12.811 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:58:12.811 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:58:12.811 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:58:12.811 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000045", 2024-08-18T15:58:12.811 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000002", 2024-08-18T15:58:12.811 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:58:12.811 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3.2UbiLidwSTy.0", 2024-08-18T15:58:12.811 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:58:12.811 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.12", 2024-08-18T15:58:12.811 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:58:12.812 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:58:12.812 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:58:12.812 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:58:12.812 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:58:12.812 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:58:12.812 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000116", 2024-08-18T15:58:12.812 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.25", 2024-08-18T15:58:12.812 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:58:12.812 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:58:12.812 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:58:12.812 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:58:12.812 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000034", 2024-08-18T15:58:12.813 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:58:12.813 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:58:12.813 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000101", 2024-08-18T15:58:12.813 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:58:12.813 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:58:12.813 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.13", 2024-08-18T15:58:12.813 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:58:12.813 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:58:12.813 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.1", 2024-08-18T15:58:12.813 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0.retry", 2024-08-18T15:58:12.813 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:58:12.813 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:58:12.813 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000079", 2024-08-18T15:58:12.814 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000049", 2024-08-18T15:58:12.814 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:58:12.814 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0", 2024-08-18T15:58:12.814 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:58:12.814 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:58:12.814 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000117", 2024-08-18T15:58:12.814 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:58:12.814 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000044", 2024-08-18T15:58:12.814 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:58:12.814 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:58:12.814 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:58:12.814 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:58:12.815 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:58:12.815 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:58:12.815 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.14", 2024-08-18T15:58:12.815 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:58:12.815 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000066", 2024-08-18T15:58:12.815 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:58:12.815 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:58:12.815 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:58:12.815 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:58:12.815 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:58:12.815 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000068", 2024-08-18T15:58:12.815 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:58:12.815 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000085", 2024-08-18T15:58:12.815 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:58:12.816 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:58:12.816 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:58:12.816 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:58:12.816 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:58:12.816 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:58:12.816 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:58:12.816 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:58:12.816 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:58:12.816 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:58:12.816 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:58:12.816 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:58:12.816 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000073", 2024-08-18T15:58:12.816 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:58:12.817 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:58:12.817 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:58:12.817 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000038", 2024-08-18T15:58:12.817 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:58:12.817 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:58:12.817 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:58:12.817 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:58:12.817 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.10", 2024-08-18T15:58:12.817 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:58:12.817 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:58:12.818 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:58:12.818 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000118", 2024-08-18T15:58:12.818 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:58:12.818 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3", 2024-08-18T15:58:12.818 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:58:12.818 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000036", 2024-08-18T15:58:12.818 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:58:12.818 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:58:12.818 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:58:12.818 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:58:12.819 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:58:12.819 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000103", 2024-08-18T15:58:12.819 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.3", 2024-08-18T15:58:12.819 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:58:12.819 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000119", 2024-08-18T15:58:12.819 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000098", 2024-08-18T15:58:12.819 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:58:12.819 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:58:12.819 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:58:12.819 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:58:12.819 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:58:12.819 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:58:12.819 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:58:12.820 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:58:12.820 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.0", 2024-08-18T15:58:12.820 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:58:12.820 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:58:12.820 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:58:12.820 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:58:12.820 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:58:12.820 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:58:12.820 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:58:12.820 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:58:12.820 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000027", 2024-08-18T15:58:12.820 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:58:12.821 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status", 2024-08-18T15:58:12.821 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:58:12.821 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000019", 2024-08-18T15:58:12.821 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:58:12.821 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:58:12.821 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:58:12.821 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:58:12.821 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1", 2024-08-18T15:58:12.821 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:58:12.821 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:58:12.821 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:58:12.821 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:58:12.821 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000039", 2024-08-18T15:58:12.821 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:58:12.822 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:58:12.822 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:58:12.822 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:58:12.822 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:58:12.822 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:58:12.822 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:58:12.822 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:58:12.822 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:58:12.822 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:58:12.822 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:58:12.822 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000100", 2024-08-18T15:58:12.822 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:58:12.823 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:58:12.823 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:58:12.823 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:58:12.823 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:58:12.823 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000093", 2024-08-18T15:58:12.823 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:58:12.823 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000004", 2024-08-18T15:58:12.823 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000063", 2024-08-18T15:58:12.823 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000122", 2024-08-18T15:58:12.823 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:58:12.823 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:58:12.823 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:58:12.823 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000057", 2024-08-18T15:58:12.823 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:58:12.824 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:58:12.824 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:58:12.824 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:58:12.824 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000054", 2024-08-18T15:58:12.824 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:58:12.824 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000114", 2024-08-18T15:58:12.824 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:58:12.824 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:58:12.824 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:58:12.824 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.2", 2024-08-18T15:58:12.824 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:58:12.824 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:58:12.824 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0", 2024-08-18T15:58:12.825 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:58:12.825 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:58:12.825 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:58:12.825 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:58:12.825 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:58:12.825 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:58:12.825 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000012", 2024-08-18T15:58:12.825 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:58:12.825 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:58:12.825 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:58:12.825 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000084", 2024-08-18T15:58:12.825 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000043", 2024-08-18T15:58:12.825 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3", 2024-08-18T15:58:12.826 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:58:12.826 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:58:12.826 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.11", 2024-08-18T15:58:12.826 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:58:12.826 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:58:12.826 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:58:12.826 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:58:12.826 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:58:12.826 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:58:12.826 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:58:12.826 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:58:12.826 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000111", 2024-08-18T15:58:12.826 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:58:12.827 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000048", 2024-08-18T15:58:12.827 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:58:12.827 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000020", 2024-08-18T15:58:12.827 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:58:12.827 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:58:12.827 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:58:12.827 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:58:12.827 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:58:12.827 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000099", 2024-08-18T15:58:12.827 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000056", 2024-08-18T15:58:12.827 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:58:12.827 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:58:12.827 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.19", 2024-08-18T15:58:12.828 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:58:12.828 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.28", 2024-08-18T15:58:12.828 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:58:12.828 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:58:12.828 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:58:12.828 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:58:12.828 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:58:12.828 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000072", 2024-08-18T15:58:12.828 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:58:12.828 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:58:12.828 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000062", 2024-08-18T15:58:12.828 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:58:12.828 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:58:12.828 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:58:12.828 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000115", 2024-08-18T15:58:12.829 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:58:12.829 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.1", 2024-08-18T15:58:12.829 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2", 2024-08-18T15:58:12.829 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000033", 2024-08-18T15:58:12.829 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:58:12.829 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:58:12.829 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:58:12.829 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:58:12.829 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:58:12.829 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:58:12.829 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:58:12.829 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:58:12.829 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:58:12.829 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:58:12.830 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:58:12.830 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:58:12.830 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:58:12.830 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:58:12.830 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:58:12.830 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:58:12.830 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:58:12.830 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:58:12.830 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:58:12.830 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000025", 2024-08-18T15:58:12.830 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:58:12.831 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000123", 2024-08-18T15:58:12.831 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.31", 2024-08-18T15:58:12.831 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:58:12.831 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:58:12.831 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:58:12.831 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:58:12.831 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.18", 2024-08-18T15:58:12.831 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:58:12.831 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:58:12.831 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:58:12.831 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:58:12.831 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:58:12.831 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000108", 2024-08-18T15:58:12.831 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:58:12.832 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:58:12.832 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000094", 2024-08-18T15:58:12.832 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:58:12.832 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:58:12.832 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:58:12.832 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:58:12.832 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:58:12.832 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:58:12.832 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000060", 2024-08-18T15:58:12.832 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:58:12.832 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:58:12.832 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:58:12.832 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000109", 2024-08-18T15:58:12.832 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.15", 2024-08-18T15:58:12.833 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:58:12.833 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:58:12.833 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:58:12.833 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:58:12.833 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:58:12.833 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:58:12.833 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:58:12.833 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:58:12.833 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:58:12.833 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000013", 2024-08-18T15:58:12.833 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:58:12.833 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:58:12.833 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:58:12.834 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:58:12.834 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:58:12.834 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:58:12.834 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:58:12.834 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:58:12.834 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:58:12.834 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:58:12.834 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:58:12.834 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:58:12.834 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:58:12.834 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:58:12.834 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000051", 2024-08-18T15:58:12.835 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:58:12.835 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:58:12.835 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:58:12.835 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:58:12.835 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000106", 2024-08-18T15:58:12.835 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:58:12.835 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:58:12.835 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:58:12.835 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:58:12.835 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:58:12.835 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000017", 2024-08-18T15:58:12.835 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:58:12.835 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:58:12.835 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:58:12.836 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000023", 2024-08-18T15:58:12.836 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:58:12.836 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:58:12.836 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:58:12.836 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:58:12.836 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:58:12.836 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:58:12.836 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:58:12.836 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000107", 2024-08-18T15:58:12.836 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:58:12.836 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:58:12.836 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000046", 2024-08-18T15:58:12.836 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:58:12.836 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:58:12.837 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:58:12.837 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.26", 2024-08-18T15:58:12.837 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:58:12.837 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:58:12.837 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9", 2024-08-18T15:58:12.837 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.22", 2024-08-18T15:58:12.837 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:58:12.837 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:58:12.837 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:58:12.837 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:58:12.837 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:58:12.837 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:58:12.837 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:58:12.837 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2", 2024-08-18T15:58:12.837 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:58:12.838 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:58:12.838 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000067", 2024-08-18T15:58:12.838 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:58:12.838 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:58:12.838 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:58:12.838 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:58:12.838 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:58:12.838 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:58:12.838 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:58:12.838 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:58:12.839 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:58:12.839 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:58:12.839 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:58:12.839 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:58:12.839 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:58:12.839 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:58:12.839 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:58:12.839 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:58:12.839 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:58:12.839 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:58:12.839 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:58:12.839 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.9", 2024-08-18T15:58:12.839 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000086", 2024-08-18T15:58:12.839 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.5", 2024-08-18T15:58:12.839 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:58:12.840 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:58:12.840 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:58:12.840 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:58:12.840 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:58:12.840 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:58:12.840 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2.MFXJ583gCr1.0", 2024-08-18T15:58:12.840 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:58:12.840 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1.GPnO16liyZ+.0", 2024-08-18T15:58:12.840 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:58:12.840 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:58:12.840 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:58:12.840 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:58:12.840 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:58:12.841 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:58:12.841 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:58:12.841 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:58:12.841 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:58:12.841 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.20", 2024-08-18T15:58:12.841 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:58:12.841 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:58:12.841 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:58:12.841 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:58:12.841 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:58:12.841 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:58:12.841 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1.retry", 2024-08-18T15:58:12.842 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:58:12.842 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:58:12.842 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:58:12.842 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0.retry", 2024-08-18T15:58:12.842 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000050", 2024-08-18T15:58:12.842 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:58:12.842 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000102", 2024-08-18T15:58:12.842 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:58:12.842 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:58:12.842 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:58:12.842 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:58:12.842 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:58:12.842 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:58:12.842 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:58:12.843 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:58:12.843 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:58:12.843 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:58:12.843 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:58:12.843 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000124", 2024-08-18T15:58:12.843 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:58:12.843 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:58:12.843 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:58:12.843 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000001", 2024-08-18T15:58:12.843 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:58:12.843 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:58:12.843 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:58:12.844 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:58:12.844 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000126", 2024-08-18T15:58:12.844 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:58:12.844 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0.fZhctfLXnSt.0", 2024-08-18T15:58:12.844 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:58:12.844 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000055", 2024-08-18T15:58:12.844 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:58:12.844 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000061", 2024-08-18T15:58:12.844 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1", 2024-08-18T15:58:12.844 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:58:12.844 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:58:12.844 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:58:12.844 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000097", 2024-08-18T15:58:12.845 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:58:12.845 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:58:12.845 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:58:12.845 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:58:12.845 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:58:12.845 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:58:12.845 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:58:12.845 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:58:12.845 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000042", 2024-08-18T15:58:12.845 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:58:12.845 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:58:12.845 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:58:12.845 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1.retry", 2024-08-18T15:58:12.846 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:58:12.846 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:58:12.846 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:58:12.846 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:58:12.846 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:58:12.846 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:58:12.846 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:58:12.846 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000075", 2024-08-18T15:58:12.846 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:58:12.846 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000120", 2024-08-18T15:58:12.846 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000030", 2024-08-18T15:58:12.846 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:58:12.846 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:58:12.847 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:58:12.847 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:58:12.847 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:58:12.847 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:58:12.847 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9" 2024-08-18T15:58:12.847 INFO:teuthology.orchestra.run.smithi193.stdout:] 2024-08-18T15:58:12.847 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:58:17.849 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:58:17.849 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:58:17.850 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:58:17.905 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:58:17.906 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:58:17.971 INFO:teuthology.orchestra.run.smithi193.stdout:[ 2024-08-18T15:58:17.971 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:58:17.971 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:58:17.971 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.29", 2024-08-18T15:58:17.971 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000078", 2024-08-18T15:58:17.972 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.history", 2024-08-18T15:58:17.972 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:58:17.972 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:58:17.972 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:58:17.972 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:58:17.972 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.23", 2024-08-18T15:58:17.972 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000070", 2024-08-18T15:58:17.972 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000104", 2024-08-18T15:58:17.972 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:58:17.972 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2.retry", 2024-08-18T15:58:17.972 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:58:17.972 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:58:17.972 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:58:17.973 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:58:17.973 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000026", 2024-08-18T15:58:17.973 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:58:17.973 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:58:17.973 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:58:17.973 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:58:17.973 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:58:17.973 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:58:17.973 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.8", 2024-08-18T15:58:17.973 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:58:17.973 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:58:17.973 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:58:17.974 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:58:17.974 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000028", 2024-08-18T15:58:17.974 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:58:17.974 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:58:17.974 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:58:17.974 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:58:17.974 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:58:17.974 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:58:17.974 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:58:17.974 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2.retry", 2024-08-18T15:58:17.975 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:58:17.975 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.30", 2024-08-18T15:58:17.975 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:58:17.975 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:58:17.975 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:58:17.975 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:58:17.975 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000040", 2024-08-18T15:58:17.975 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:58:17.975 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.6", 2024-08-18T15:58:17.976 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:58:17.976 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:58:17.976 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:58:17.976 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:58:17.976 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:58:17.976 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:58:17.976 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000015", 2024-08-18T15:58:17.976 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:58:17.977 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:58:17.977 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000069", 2024-08-18T15:58:17.977 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:58:17.977 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:58:17.977 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:58:17.978 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000095", 2024-08-18T15:58:17.978 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0", 2024-08-18T15:58:17.978 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000003", 2024-08-18T15:58:17.978 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000047", 2024-08-18T15:58:17.978 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:58:17.978 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:58:17.978 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:58:17.979 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:58:17.979 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:58:17.979 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:58:17.979 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:58:17.979 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:58:17.979 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:58:17.979 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000035", 2024-08-18T15:58:17.979 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:58:17.979 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:58:17.980 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000037", 2024-08-18T15:58:17.980 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:58:17.980 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:58:17.980 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:58:17.980 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:58:17.980 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:58:17.980 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:58:17.980 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:58:17.981 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:58:17.981 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:58:17.981 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:58:17.981 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:58:17.981 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:58:17.981 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:58:17.981 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:58:17.981 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000024", 2024-08-18T15:58:17.982 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:58:17.982 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:58:17.982 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:58:17.982 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:58:17.982 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000000", 2024-08-18T15:58:17.982 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:58:17.982 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000031", 2024-08-18T15:58:17.982 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:58:17.983 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:58:17.983 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:58:17.983 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:58:17.983 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:58:17.983 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:58:17.983 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3", 2024-08-18T15:58:17.983 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:58:17.983 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:58:17.984 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:58:17.984 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:58:17.984 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:58:17.984 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:58:17.984 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:58:17.984 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:58:17.984 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:58:17.984 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:58:17.985 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:58:17.985 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:58:17.985 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:58:17.985 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:58:17.985 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:58:17.985 INFO:teuthology.orchestra.run.smithi193.stdout: "data_loggenerations_metadata", 2024-08-18T15:58:17.985 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2", 2024-08-18T15:58:17.985 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:58:17.986 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:58:17.986 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:58:17.986 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:58:17.986 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T15:58:17.986 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:58:17.986 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:58:17.986 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:58:17.987 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:58:17.987 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:58:17.987 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000076", 2024-08-18T15:58:17.987 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:58:17.987 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:58:17.987 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000113", 2024-08-18T15:58:17.987 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000005", 2024-08-18T15:58:17.987 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:58:17.987 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000011", 2024-08-18T15:58:17.988 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:58:17.988 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000058", 2024-08-18T15:58:17.988 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:58:17.988 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:58:17.988 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:58:17.988 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000052", 2024-08-18T15:58:17.988 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:58:17.989 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:58:17.989 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:58:17.989 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:58:17.989 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.21", 2024-08-18T15:58:17.989 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:58:17.989 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:58:17.989 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:58:17.989 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:58:17.989 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:58:17.989 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:58:17.989 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:58:17.989 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:58:17.990 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:58:17.990 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:58:17.990 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:58:17.990 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:58:17.990 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:58:17.990 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:58:17.990 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:58:17.990 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:58:17.990 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:58:17.990 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:58:17.990 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:58:17.990 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000088", 2024-08-18T15:58:17.991 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000080", 2024-08-18T15:58:17.991 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:58:17.991 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:58:17.991 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000090", 2024-08-18T15:58:17.991 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:58:17.991 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:58:17.991 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000110", 2024-08-18T15:58:17.991 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:58:17.991 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:58:17.991 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:58:17.991 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.0", 2024-08-18T15:58:17.992 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:58:17.992 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000096", 2024-08-18T15:58:17.992 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:58:17.992 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:58:17.992 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:58:17.992 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:58:17.992 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:58:17.992 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000087", 2024-08-18T15:58:17.992 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:58:17.992 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:58:17.992 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:58:17.992 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:58:17.993 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000008", 2024-08-18T15:58:17.993 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:58:17.993 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:58:17.993 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:58:17.993 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:58:17.993 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:58:17.993 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000006", 2024-08-18T15:58:17.993 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000029", 2024-08-18T15:58:17.993 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:58:17.993 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000089", 2024-08-18T15:58:17.993 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:58:17.993 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:58:17.993 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000022", 2024-08-18T15:58:17.993 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:58:17.994 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:58:17.994 INFO:teuthology.orchestra.run.smithi193.stdout: "bilog.trim", 2024-08-18T15:58:17.994 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000016", 2024-08-18T15:58:17.994 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:58:17.994 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:58:17.994 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:58:17.994 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:58:17.994 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:58:17.994 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000041", 2024-08-18T15:58:17.994 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:58:17.994 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:58:17.995 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.log.01050e30-179b-43be-bdbb-3bb23cf37ae8.3", 2024-08-18T15:58:17.995 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:58:17.995 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000018", 2024-08-18T15:58:17.995 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000092", 2024-08-18T15:58:17.995 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:58:17.995 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:58:17.995 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:58:17.995 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:58:17.995 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.27", 2024-08-18T15:58:17.995 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.4", 2024-08-18T15:58:17.995 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:58:17.995 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:58:17.995 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:58:17.995 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:58:17.995 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:58:17.995 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:58:17.996 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:58:17.996 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000014", 2024-08-18T15:58:17.996 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:58:17.996 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3.retry", 2024-08-18T15:58:17.996 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:58:17.996 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:58:17.996 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:58:17.996 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:58:17.996 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:58:17.996 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000112", 2024-08-18T15:58:17.996 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:58:17.996 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000007", 2024-08-18T15:58:17.996 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000021", 2024-08-18T15:58:17.996 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:58:17.997 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:58:17.997 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:58:17.997 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:58:17.997 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:58:17.997 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.16", 2024-08-18T15:58:17.997 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:58:17.997 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000064", 2024-08-18T15:58:17.997 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:58:17.997 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:58:17.997 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:58:17.997 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:58:17.997 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:58:17.997 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:58:17.998 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:58:17.998 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000071", 2024-08-18T15:58:17.998 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:58:17.998 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:58:17.998 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:58:17.998 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000074", 2024-08-18T15:58:17.998 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:58:17.998 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:58:17.998 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:58:17.998 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:58:17.998 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:58:17.998 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:58:17.999 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:58:17.999 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:58:17.999 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:58:17.999 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:58:17.999 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:58:17.999 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:58:17.999 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:58:17.999 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:58:17.999 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:58:17.999 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000081", 2024-08-18T15:58:17.999 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:58:18.000 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000009", 2024-08-18T15:58:18.000 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000121", 2024-08-18T15:58:18.000 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1", 2024-08-18T15:58:18.000 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:58:18.000 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:58:18.000 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:58:18.000 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:58:18.000 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:58:18.000 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:58:18.000 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.17", 2024-08-18T15:58:18.000 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000125", 2024-08-18T15:58:18.000 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:58:18.001 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000082", 2024-08-18T15:58:18.001 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:58:18.001 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:58:18.001 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:58:18.001 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:58:18.001 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:58:18.001 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:58:18.001 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:58:18.001 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:58:18.001 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:58:18.001 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:58:18.001 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:58:18.002 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000105", 2024-08-18T15:58:18.002 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000059", 2024-08-18T15:58:18.002 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:58:18.002 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:58:18.002 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:58:18.002 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:58:18.002 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:58:18.002 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:58:18.002 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:58:18.002 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:58:18.002 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:58:18.003 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:58:18.003 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:58:18.003 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000077", 2024-08-18T15:58:18.003 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000032", 2024-08-18T15:58:18.003 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:58:18.003 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:58:18.003 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:58:18.003 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:58:18.003 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:58:18.003 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:58:18.003 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.24", 2024-08-18T15:58:18.003 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:58:18.004 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:58:18.004 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:58:18.004 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:58:18.004 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3.retry", 2024-08-18T15:58:18.004 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:58:18.004 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:58:18.004 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:58:18.004 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:58:18.004 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:58:18.004 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:58:18.004 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000053", 2024-08-18T15:58:18.004 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000091", 2024-08-18T15:58:18.004 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:58:18.005 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:58:18.005 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:58:18.005 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000065", 2024-08-18T15:58:18.005 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:58:18.005 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:58:18.005 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:58:18.005 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:58:18.005 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000083", 2024-08-18T15:58:18.005 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000010", 2024-08-18T15:58:18.005 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:58:18.005 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:58:18.006 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:58:18.006 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:58:18.006 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:58:18.006 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:58:18.006 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:58:18.006 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:58:18.006 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:58:18.006 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:58:18.006 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:58:18.006 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.7", 2024-08-18T15:58:18.006 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.3", 2024-08-18T15:58:18.006 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:58:18.007 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:58:18.007 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:58:18.007 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:58:18.007 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:58:18.007 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:58:18.007 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:58:18.007 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:58:18.007 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.2", 2024-08-18T15:58:18.007 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:58:18.007 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:58:18.007 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:58:18.007 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:58:18.007 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:58:18.008 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:58:18.008 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:58:18.008 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000045", 2024-08-18T15:58:18.008 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000002", 2024-08-18T15:58:18.008 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:58:18.008 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3.2UbiLidwSTy.0", 2024-08-18T15:58:18.008 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:58:18.008 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.12", 2024-08-18T15:58:18.008 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:58:18.008 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:58:18.008 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:58:18.008 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:58:18.008 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:58:18.009 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:58:18.009 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000116", 2024-08-18T15:58:18.009 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.25", 2024-08-18T15:58:18.009 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:58:18.009 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:58:18.009 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:58:18.009 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:58:18.009 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000034", 2024-08-18T15:58:18.009 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:58:18.009 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:58:18.009 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000101", 2024-08-18T15:58:18.009 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:58:18.009 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:58:18.010 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.13", 2024-08-18T15:58:18.010 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:58:18.010 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:58:18.010 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.1", 2024-08-18T15:58:18.010 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0.retry", 2024-08-18T15:58:18.010 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:58:18.010 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:58:18.010 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000079", 2024-08-18T15:58:18.010 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000049", 2024-08-18T15:58:18.010 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:58:18.010 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0", 2024-08-18T15:58:18.010 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:58:18.011 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:58:18.011 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000117", 2024-08-18T15:58:18.011 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:58:18.011 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000044", 2024-08-18T15:58:18.011 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:58:18.011 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:58:18.011 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:58:18.011 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:58:18.011 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:58:18.011 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:58:18.011 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.14", 2024-08-18T15:58:18.011 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:58:18.011 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000066", 2024-08-18T15:58:18.012 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:58:18.012 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:58:18.012 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:58:18.012 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:58:18.012 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:58:18.012 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000068", 2024-08-18T15:58:18.012 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:58:18.012 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000085", 2024-08-18T15:58:18.012 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:58:18.012 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:58:18.012 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:58:18.012 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:58:18.012 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:58:18.013 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:58:18.013 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:58:18.013 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:58:18.013 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:58:18.013 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:58:18.013 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:58:18.013 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:58:18.013 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000073", 2024-08-18T15:58:18.013 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:58:18.013 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:58:18.013 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:58:18.013 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000038", 2024-08-18T15:58:18.013 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:58:18.013 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:58:18.014 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:58:18.014 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:58:18.014 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.10", 2024-08-18T15:58:18.014 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:58:18.014 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:58:18.014 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:58:18.014 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000118", 2024-08-18T15:58:18.014 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:58:18.014 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3", 2024-08-18T15:58:18.014 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:58:18.014 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000036", 2024-08-18T15:58:18.014 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:58:18.014 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:58:18.015 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:58:18.015 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:58:18.015 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:58:18.015 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000103", 2024-08-18T15:58:18.015 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.3", 2024-08-18T15:58:18.015 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:58:18.015 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000119", 2024-08-18T15:58:18.015 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000098", 2024-08-18T15:58:18.015 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:58:18.015 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:58:18.015 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:58:18.015 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:58:18.016 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:58:18.016 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:58:18.016 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:58:18.016 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:58:18.016 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.0", 2024-08-18T15:58:18.016 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:58:18.016 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:58:18.016 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:58:18.016 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:58:18.016 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:58:18.016 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:58:18.016 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:58:18.016 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:58:18.016 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000027", 2024-08-18T15:58:18.016 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:58:18.017 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status", 2024-08-18T15:58:18.017 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:58:18.017 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000019", 2024-08-18T15:58:18.017 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:58:18.017 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:58:18.017 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:58:18.017 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:58:18.017 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1", 2024-08-18T15:58:18.017 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:58:18.017 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:58:18.017 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:58:18.017 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:58:18.017 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000039", 2024-08-18T15:58:18.017 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:58:18.018 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:58:18.018 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:58:18.018 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:58:18.018 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:58:18.018 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:58:18.018 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:58:18.018 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:58:18.018 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:58:18.018 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:58:18.018 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:58:18.018 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000100", 2024-08-18T15:58:18.018 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:58:18.019 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:58:18.019 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:58:18.019 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:58:18.019 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:58:18.019 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000093", 2024-08-18T15:58:18.019 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:58:18.019 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000004", 2024-08-18T15:58:18.019 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000063", 2024-08-18T15:58:18.019 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000122", 2024-08-18T15:58:18.019 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:58:18.019 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:58:18.019 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:58:18.019 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000057", 2024-08-18T15:58:18.019 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:58:18.019 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:58:18.020 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:58:18.020 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:58:18.020 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000054", 2024-08-18T15:58:18.020 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:58:18.020 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000114", 2024-08-18T15:58:18.020 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:58:18.020 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:58:18.020 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:58:18.020 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.2", 2024-08-18T15:58:18.020 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:58:18.020 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:58:18.020 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0", 2024-08-18T15:58:18.020 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:58:18.020 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:58:18.020 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:58:18.021 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:58:18.021 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:58:18.021 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:58:18.021 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000012", 2024-08-18T15:58:18.021 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:58:18.021 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:58:18.021 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:58:18.021 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000084", 2024-08-18T15:58:18.021 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000043", 2024-08-18T15:58:18.021 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3", 2024-08-18T15:58:18.021 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:58:18.021 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:58:18.022 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.11", 2024-08-18T15:58:18.022 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:58:18.022 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:58:18.022 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:58:18.022 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:58:18.022 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:58:18.022 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:58:18.022 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:58:18.022 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:58:18.022 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000111", 2024-08-18T15:58:18.022 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:58:18.022 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000048", 2024-08-18T15:58:18.022 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:58:18.022 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000020", 2024-08-18T15:58:18.023 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:58:18.023 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:58:18.023 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:58:18.023 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:58:18.023 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:58:18.023 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000099", 2024-08-18T15:58:18.023 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000056", 2024-08-18T15:58:18.023 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:58:18.023 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:58:18.023 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.19", 2024-08-18T15:58:18.023 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:58:18.023 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.28", 2024-08-18T15:58:18.024 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:58:18.024 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:58:18.024 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:58:18.024 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:58:18.024 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:58:18.024 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000072", 2024-08-18T15:58:18.024 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:58:18.024 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:58:18.024 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000062", 2024-08-18T15:58:18.024 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:58:18.024 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:58:18.024 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:58:18.024 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000115", 2024-08-18T15:58:18.024 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:58:18.025 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.1", 2024-08-18T15:58:18.025 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2", 2024-08-18T15:58:18.025 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000033", 2024-08-18T15:58:18.025 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:58:18.025 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:58:18.025 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:58:18.025 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:58:18.025 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:58:18.025 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:58:18.025 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:58:18.025 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:58:18.025 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:58:18.025 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:58:18.026 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:58:18.026 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:58:18.026 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:58:18.026 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:58:18.026 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:58:18.026 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:58:18.026 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:58:18.026 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:58:18.026 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:58:18.026 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000025", 2024-08-18T15:58:18.026 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:58:18.026 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000123", 2024-08-18T15:58:18.027 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.31", 2024-08-18T15:58:18.027 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:58:18.027 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:58:18.027 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:58:18.027 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:58:18.027 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.18", 2024-08-18T15:58:18.027 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:58:18.027 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:58:18.027 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:58:18.027 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:58:18.027 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:58:18.027 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000108", 2024-08-18T15:58:18.027 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:58:18.027 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:58:18.027 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000094", 2024-08-18T15:58:18.028 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:58:18.028 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:58:18.028 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:58:18.028 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:58:18.028 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:58:18.028 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:58:18.028 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000060", 2024-08-18T15:58:18.028 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:58:18.028 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:58:18.028 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:58:18.028 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000109", 2024-08-18T15:58:18.028 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.15", 2024-08-18T15:58:18.028 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:58:18.028 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:58:18.029 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:58:18.029 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:58:18.029 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:58:18.029 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:58:18.029 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:58:18.029 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:58:18.029 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:58:18.029 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000013", 2024-08-18T15:58:18.029 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:58:18.029 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:58:18.029 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:58:18.029 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:58:18.030 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:58:18.030 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:58:18.030 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:58:18.030 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:58:18.030 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:58:18.030 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:58:18.030 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:58:18.030 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:58:18.030 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:58:18.030 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:58:18.030 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000051", 2024-08-18T15:58:18.030 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:58:18.030 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:58:18.031 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:58:18.031 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:58:18.031 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000106", 2024-08-18T15:58:18.031 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:58:18.031 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:58:18.031 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:58:18.031 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:58:18.031 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:58:18.031 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000017", 2024-08-18T15:58:18.031 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:58:18.031 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:58:18.031 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:58:18.032 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000023", 2024-08-18T15:58:18.032 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:58:18.032 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:58:18.032 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:58:18.032 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:58:18.032 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:58:18.032 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:58:18.032 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:58:18.032 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000107", 2024-08-18T15:58:18.032 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:58:18.032 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:58:18.032 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000046", 2024-08-18T15:58:18.032 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:58:18.033 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:58:18.033 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:58:18.033 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.26", 2024-08-18T15:58:18.033 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:58:18.033 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:58:18.033 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9", 2024-08-18T15:58:18.033 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.22", 2024-08-18T15:58:18.033 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:58:18.033 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:58:18.033 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:58:18.033 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:58:18.034 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:58:18.034 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:58:18.034 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:58:18.034 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2", 2024-08-18T15:58:18.034 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:58:18.034 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:58:18.034 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000067", 2024-08-18T15:58:18.034 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:58:18.034 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:58:18.034 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:58:18.034 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:58:18.034 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:58:18.035 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:58:18.035 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:58:18.035 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:58:18.035 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:58:18.035 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:58:18.035 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:58:18.035 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:58:18.035 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:58:18.035 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:58:18.035 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:58:18.035 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:58:18.035 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:58:18.036 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:58:18.036 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:58:18.036 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.9", 2024-08-18T15:58:18.036 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000086", 2024-08-18T15:58:18.036 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.5", 2024-08-18T15:58:18.036 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:58:18.036 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:58:18.036 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:58:18.036 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:58:18.036 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:58:18.036 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:58:18.036 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2.MFXJ583gCr1.0", 2024-08-18T15:58:18.037 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:58:18.037 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1.GPnO16liyZ+.0", 2024-08-18T15:58:18.037 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:58:18.037 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:58:18.037 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:58:18.037 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:58:18.037 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:58:18.037 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:58:18.037 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:58:18.037 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:58:18.038 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:58:18.038 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.20", 2024-08-18T15:58:18.038 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:58:18.038 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:58:18.038 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:58:18.038 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:58:18.038 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:58:18.038 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:58:18.038 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1.retry", 2024-08-18T15:58:18.038 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:58:18.038 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:58:18.038 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:58:18.038 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0.retry", 2024-08-18T15:58:18.039 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000050", 2024-08-18T15:58:18.039 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:58:18.039 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000102", 2024-08-18T15:58:18.039 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:58:18.039 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:58:18.039 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:58:18.039 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:58:18.039 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:58:18.040 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:58:18.040 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:58:18.040 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:58:18.040 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:58:18.040 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:58:18.040 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:58:18.040 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000124", 2024-08-18T15:58:18.040 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:58:18.040 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:58:18.040 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:58:18.040 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000001", 2024-08-18T15:58:18.040 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:58:18.040 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:58:18.041 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:58:18.041 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:58:18.041 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000126", 2024-08-18T15:58:18.041 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:58:18.041 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0.fZhctfLXnSt.0", 2024-08-18T15:58:18.041 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:58:18.041 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000055", 2024-08-18T15:58:18.041 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:58:18.041 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000061", 2024-08-18T15:58:18.041 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1", 2024-08-18T15:58:18.041 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:58:18.041 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:58:18.041 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:58:18.042 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000097", 2024-08-18T15:58:18.042 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:58:18.042 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:58:18.042 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:58:18.042 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:58:18.042 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:58:18.042 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:58:18.042 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:58:18.042 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:58:18.042 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000042", 2024-08-18T15:58:18.043 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:58:18.043 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:58:18.043 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:58:18.043 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1.retry", 2024-08-18T15:58:18.043 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:58:18.043 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:58:18.043 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:58:18.043 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:58:18.043 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:58:18.043 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:58:18.043 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:58:18.043 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000075", 2024-08-18T15:58:18.044 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:58:18.044 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000120", 2024-08-18T15:58:18.044 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000030", 2024-08-18T15:58:18.044 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:58:18.044 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:58:18.044 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:58:18.044 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:58:18.044 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:58:18.044 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:58:18.044 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9" 2024-08-18T15:58:18.044 INFO:teuthology.orchestra.run.smithi193.stdout:] 2024-08-18T15:58:18.044 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:58:23.047 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:58:23.047 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:58:23.047 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:58:23.100 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:58:23.100 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:58:23.205 INFO:teuthology.orchestra.run.smithi193.stdout:[ 2024-08-18T15:58:23.206 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:58:23.206 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:58:23.206 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.29", 2024-08-18T15:58:23.206 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000078", 2024-08-18T15:58:23.206 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.history", 2024-08-18T15:58:23.206 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:58:23.206 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:58:23.206 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:58:23.206 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:58:23.206 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.23", 2024-08-18T15:58:23.207 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000070", 2024-08-18T15:58:23.207 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000104", 2024-08-18T15:58:23.207 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:58:23.207 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2.retry", 2024-08-18T15:58:23.207 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:58:23.207 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:58:23.207 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:58:23.207 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:58:23.207 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000026", 2024-08-18T15:58:23.207 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:58:23.207 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:58:23.207 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:58:23.207 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:58:23.208 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:58:23.208 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:58:23.208 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.8", 2024-08-18T15:58:23.208 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:58:23.208 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:58:23.208 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:58:23.208 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:58:23.208 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000028", 2024-08-18T15:58:23.208 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:58:23.208 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:58:23.208 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:58:23.208 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:58:23.209 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:58:23.209 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:58:23.209 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:58:23.209 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2.retry", 2024-08-18T15:58:23.209 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:58:23.209 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.30", 2024-08-18T15:58:23.209 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:58:23.209 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:58:23.209 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:58:23.209 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:58:23.209 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000040", 2024-08-18T15:58:23.209 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:58:23.209 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.6", 2024-08-18T15:58:23.209 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:58:23.210 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:58:23.210 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:58:23.210 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:58:23.210 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:58:23.210 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:58:23.210 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000015", 2024-08-18T15:58:23.210 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:58:23.210 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:58:23.210 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000069", 2024-08-18T15:58:23.210 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:58:23.210 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:58:23.211 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:58:23.211 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000095", 2024-08-18T15:58:23.211 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0", 2024-08-18T15:58:23.211 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000003", 2024-08-18T15:58:23.211 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000047", 2024-08-18T15:58:23.211 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:58:23.211 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:58:23.211 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:58:23.211 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:58:23.211 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:58:23.211 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:58:23.211 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:58:23.211 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:58:23.212 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:58:23.212 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000035", 2024-08-18T15:58:23.212 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:58:23.212 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:58:23.212 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000037", 2024-08-18T15:58:23.212 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:58:23.212 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:58:23.212 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:58:23.212 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:58:23.212 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:58:23.212 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:58:23.212 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:58:23.213 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:58:23.213 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:58:23.213 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:58:23.213 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:58:23.213 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:58:23.213 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:58:23.213 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:58:23.213 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000024", 2024-08-18T15:58:23.213 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:58:23.213 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:58:23.213 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:58:23.213 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:58:23.213 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000000", 2024-08-18T15:58:23.214 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:58:23.214 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000031", 2024-08-18T15:58:23.214 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:58:23.214 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:58:23.214 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:58:23.214 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:58:23.214 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:58:23.214 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:58:23.214 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3", 2024-08-18T15:58:23.214 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:58:23.214 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:58:23.214 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:58:23.215 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:58:23.215 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:58:23.215 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:58:23.215 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:58:23.215 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:58:23.215 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:58:23.215 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:58:23.215 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:58:23.215 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:58:23.215 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:58:23.215 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:58:23.215 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:58:23.215 INFO:teuthology.orchestra.run.smithi193.stdout: "data_loggenerations_metadata", 2024-08-18T15:58:23.216 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2", 2024-08-18T15:58:23.216 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:58:23.216 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:58:23.216 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:58:23.216 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:58:23.216 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T15:58:23.216 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:58:23.216 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:58:23.216 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:58:23.216 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:58:23.216 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:58:23.216 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000076", 2024-08-18T15:58:23.216 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:58:23.217 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:58:23.217 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000113", 2024-08-18T15:58:23.217 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000005", 2024-08-18T15:58:23.217 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:58:23.217 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000011", 2024-08-18T15:58:23.217 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:58:23.217 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000058", 2024-08-18T15:58:23.217 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:58:23.217 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:58:23.217 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:58:23.217 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000052", 2024-08-18T15:58:23.217 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:58:23.217 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:58:23.218 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:58:23.218 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:58:23.218 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.21", 2024-08-18T15:58:23.218 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:58:23.218 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:58:23.218 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:58:23.218 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:58:23.218 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:58:23.218 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:58:23.218 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:58:23.218 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:58:23.218 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:58:23.219 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:58:23.219 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:58:23.219 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:58:23.219 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:58:23.219 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:58:23.219 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:58:23.219 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:58:23.220 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:58:23.220 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:58:23.220 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:58:23.220 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000088", 2024-08-18T15:58:23.220 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000080", 2024-08-18T15:58:23.220 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:58:23.220 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:58:23.220 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000090", 2024-08-18T15:58:23.220 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:58:23.220 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:58:23.220 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000110", 2024-08-18T15:58:23.221 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:58:23.221 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:58:23.221 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:58:23.221 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.0", 2024-08-18T15:58:23.221 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:58:23.221 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000096", 2024-08-18T15:58:23.221 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:58:23.221 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:58:23.221 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:58:23.221 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:58:23.221 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:58:23.221 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000087", 2024-08-18T15:58:23.221 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:58:23.222 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:58:23.222 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:58:23.222 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:58:23.222 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000008", 2024-08-18T15:58:23.222 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:58:23.222 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:58:23.222 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:58:23.222 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:58:23.222 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:58:23.222 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000006", 2024-08-18T15:58:23.222 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000029", 2024-08-18T15:58:23.222 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:58:23.223 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000089", 2024-08-18T15:58:23.223 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:58:23.223 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:58:23.223 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000022", 2024-08-18T15:58:23.223 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:58:23.223 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:58:23.223 INFO:teuthology.orchestra.run.smithi193.stdout: "bilog.trim", 2024-08-18T15:58:23.223 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000016", 2024-08-18T15:58:23.223 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:58:23.223 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:58:23.223 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:58:23.223 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:58:23.224 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:58:23.224 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000041", 2024-08-18T15:58:23.224 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:58:23.224 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:58:23.224 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.log.01050e30-179b-43be-bdbb-3bb23cf37ae8.3", 2024-08-18T15:58:23.224 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:58:23.224 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000018", 2024-08-18T15:58:23.224 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000092", 2024-08-18T15:58:23.224 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:58:23.224 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:58:23.224 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:58:23.224 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:58:23.224 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.27", 2024-08-18T15:58:23.225 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.4", 2024-08-18T15:58:23.225 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:58:23.225 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:58:23.225 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:58:23.225 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:58:23.225 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:58:23.225 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:58:23.225 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:58:23.225 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000014", 2024-08-18T15:58:23.225 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:58:23.225 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3.retry", 2024-08-18T15:58:23.225 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:58:23.225 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:58:23.226 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:58:23.226 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:58:23.226 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:58:23.226 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000112", 2024-08-18T15:58:23.226 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:58:23.226 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000007", 2024-08-18T15:58:23.226 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000021", 2024-08-18T15:58:23.226 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:58:23.226 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:58:23.226 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:58:23.226 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:58:23.227 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:58:23.227 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.16", 2024-08-18T15:58:23.227 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:58:23.227 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000064", 2024-08-18T15:58:23.227 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:58:23.227 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:58:23.227 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:58:23.227 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:58:23.227 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:58:23.227 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:58:23.227 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:58:23.227 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000071", 2024-08-18T15:58:23.227 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:58:23.228 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:58:23.228 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:58:23.228 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000074", 2024-08-18T15:58:23.228 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:58:23.228 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:58:23.228 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:58:23.228 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:58:23.228 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:58:23.228 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:58:23.228 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:58:23.228 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:58:23.228 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:58:23.229 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:58:23.229 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:58:23.229 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:58:23.229 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:58:23.229 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:58:23.229 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:58:23.229 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000081", 2024-08-18T15:58:23.229 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:58:23.229 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000009", 2024-08-18T15:58:23.229 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000121", 2024-08-18T15:58:23.229 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1", 2024-08-18T15:58:23.229 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:58:23.229 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:58:23.230 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:58:23.230 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:58:23.230 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:58:23.230 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:58:23.230 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.17", 2024-08-18T15:58:23.230 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000125", 2024-08-18T15:58:23.230 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:58:23.230 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000082", 2024-08-18T15:58:23.230 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:58:23.230 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:58:23.230 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:58:23.230 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:58:23.231 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:58:23.231 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:58:23.231 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:58:23.231 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:58:23.231 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:58:23.231 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:58:23.231 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:58:23.231 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000105", 2024-08-18T15:58:23.231 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000059", 2024-08-18T15:58:23.231 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:58:23.231 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:58:23.231 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:58:23.231 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:58:23.232 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:58:23.232 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:58:23.232 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:58:23.232 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:58:23.232 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:58:23.232 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:58:23.232 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:58:23.232 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000077", 2024-08-18T15:58:23.232 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000032", 2024-08-18T15:58:23.232 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:58:23.232 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:58:23.232 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:58:23.233 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:58:23.233 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:58:23.233 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:58:23.233 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.24", 2024-08-18T15:58:23.233 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:58:23.233 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:58:23.233 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:58:23.233 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:58:23.233 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3.retry", 2024-08-18T15:58:23.233 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:58:23.233 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:58:23.233 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:58:23.233 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:58:23.234 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:58:23.234 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:58:23.234 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000053", 2024-08-18T15:58:23.234 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000091", 2024-08-18T15:58:23.234 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:58:23.234 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:58:23.234 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:58:23.234 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000065", 2024-08-18T15:58:23.234 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:58:23.234 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:58:23.234 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:58:23.234 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:58:23.234 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000083", 2024-08-18T15:58:23.235 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000010", 2024-08-18T15:58:23.235 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:58:23.235 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:58:23.235 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:58:23.235 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:58:23.235 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:58:23.235 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:58:23.235 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:58:23.235 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:58:23.235 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:58:23.235 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:58:23.235 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:58:23.235 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.7", 2024-08-18T15:58:23.236 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.3", 2024-08-18T15:58:23.236 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:58:23.236 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:58:23.236 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:58:23.236 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:58:23.236 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:58:23.236 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:58:23.236 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:58:23.236 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:58:23.236 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.2", 2024-08-18T15:58:23.236 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:58:23.236 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:58:23.236 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:58:23.236 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:58:23.237 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:58:23.237 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:58:23.237 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:58:23.237 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000045", 2024-08-18T15:58:23.237 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000002", 2024-08-18T15:58:23.237 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:58:23.237 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3.2UbiLidwSTy.0", 2024-08-18T15:58:23.237 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:58:23.237 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.12", 2024-08-18T15:58:23.237 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:58:23.237 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:58:23.237 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:58:23.237 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:58:23.238 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:58:23.238 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:58:23.238 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000116", 2024-08-18T15:58:23.238 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.25", 2024-08-18T15:58:23.238 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:58:23.238 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:58:23.238 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:58:23.238 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:58:23.238 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000034", 2024-08-18T15:58:23.238 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:58:23.238 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:58:23.238 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000101", 2024-08-18T15:58:23.239 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:58:23.239 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:58:23.239 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.13", 2024-08-18T15:58:23.239 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:58:23.239 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:58:23.239 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.1", 2024-08-18T15:58:23.239 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0.retry", 2024-08-18T15:58:23.239 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:58:23.239 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:58:23.239 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000079", 2024-08-18T15:58:23.239 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000049", 2024-08-18T15:58:23.239 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:58:23.240 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0", 2024-08-18T15:58:23.240 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:58:23.240 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:58:23.240 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000117", 2024-08-18T15:58:23.240 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:58:23.240 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000044", 2024-08-18T15:58:23.240 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:58:23.240 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:58:23.240 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:58:23.240 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:58:23.240 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:58:23.240 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:58:23.240 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.14", 2024-08-18T15:58:23.241 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:58:23.241 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000066", 2024-08-18T15:58:23.241 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:58:23.241 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:58:23.241 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:58:23.241 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:58:23.241 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:58:23.241 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000068", 2024-08-18T15:58:23.241 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:58:23.241 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000085", 2024-08-18T15:58:23.241 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:58:23.241 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:58:23.241 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:58:23.242 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:58:23.242 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:58:23.242 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:58:23.242 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:58:23.242 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:58:23.242 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:58:23.242 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:58:23.242 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:58:23.242 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:58:23.242 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000073", 2024-08-18T15:58:23.242 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:58:23.242 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:58:23.242 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:58:23.243 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000038", 2024-08-18T15:58:23.243 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:58:23.243 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:58:23.243 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:58:23.243 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:58:23.243 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.10", 2024-08-18T15:58:23.243 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:58:23.243 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:58:23.243 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:58:23.243 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000118", 2024-08-18T15:58:23.243 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:58:23.243 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3", 2024-08-18T15:58:23.244 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:58:23.244 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000036", 2024-08-18T15:58:23.244 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:58:23.244 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:58:23.244 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:58:23.244 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:58:23.244 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:58:23.244 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000103", 2024-08-18T15:58:23.244 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.3", 2024-08-18T15:58:23.244 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:58:23.244 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000119", 2024-08-18T15:58:23.244 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000098", 2024-08-18T15:58:23.244 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:58:23.245 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:58:23.245 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:58:23.245 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:58:23.245 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:58:23.245 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:58:23.245 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:58:23.245 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:58:23.245 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.0", 2024-08-18T15:58:23.245 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:58:23.245 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:58:23.245 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:58:23.245 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:58:23.245 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:58:23.246 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:58:23.246 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:58:23.246 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:58:23.246 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000027", 2024-08-18T15:58:23.246 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:58:23.246 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status", 2024-08-18T15:58:23.246 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:58:23.246 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000019", 2024-08-18T15:58:23.246 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:58:23.246 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:58:23.246 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:58:23.246 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:58:23.246 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1", 2024-08-18T15:58:23.247 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:58:23.247 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:58:23.247 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:58:23.247 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:58:23.247 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000039", 2024-08-18T15:58:23.247 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:58:23.247 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:58:23.247 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:58:23.247 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:58:23.247 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:58:23.247 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:58:23.247 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:58:23.247 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:58:23.248 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:58:23.248 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:58:23.248 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:58:23.248 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000100", 2024-08-18T15:58:23.248 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:58:23.248 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:58:23.248 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:58:23.248 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:58:23.248 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:58:23.248 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000093", 2024-08-18T15:58:23.248 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:58:23.248 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000004", 2024-08-18T15:58:23.248 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000063", 2024-08-18T15:58:23.249 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000122", 2024-08-18T15:58:23.249 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:58:23.249 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:58:23.249 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:58:23.249 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000057", 2024-08-18T15:58:23.249 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:58:23.249 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:58:23.249 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:58:23.249 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:58:23.249 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000054", 2024-08-18T15:58:23.249 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:58:23.249 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000114", 2024-08-18T15:58:23.249 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:58:23.250 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:58:23.250 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:58:23.250 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.2", 2024-08-18T15:58:23.250 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:58:23.250 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:58:23.250 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0", 2024-08-18T15:58:23.250 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:58:23.250 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:58:23.250 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:58:23.250 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:58:23.250 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:58:23.250 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:58:23.250 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000012", 2024-08-18T15:58:23.251 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:58:23.251 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:58:23.251 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:58:23.251 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000084", 2024-08-18T15:58:23.251 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000043", 2024-08-18T15:58:23.251 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3", 2024-08-18T15:58:23.251 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:58:23.251 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:58:23.251 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.11", 2024-08-18T15:58:23.251 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:58:23.251 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:58:23.251 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:58:23.251 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:58:23.252 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:58:23.252 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:58:23.252 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:58:23.252 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:58:23.252 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000111", 2024-08-18T15:58:23.252 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:58:23.252 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000048", 2024-08-18T15:58:23.252 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:58:23.252 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000020", 2024-08-18T15:58:23.252 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:58:23.252 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:58:23.252 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:58:23.252 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:58:23.253 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:58:23.253 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000099", 2024-08-18T15:58:23.253 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000056", 2024-08-18T15:58:23.253 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:58:23.253 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:58:23.253 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.19", 2024-08-18T15:58:23.253 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:58:23.253 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.28", 2024-08-18T15:58:23.253 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:58:23.253 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:58:23.253 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:58:23.253 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:58:23.253 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:58:23.253 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000072", 2024-08-18T15:58:23.254 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:58:23.254 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:58:23.254 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000062", 2024-08-18T15:58:23.254 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:58:23.254 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:58:23.254 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:58:23.254 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000115", 2024-08-18T15:58:23.254 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:58:23.254 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.1", 2024-08-18T15:58:23.254 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2", 2024-08-18T15:58:23.254 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000033", 2024-08-18T15:58:23.254 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:58:23.255 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:58:23.255 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:58:23.255 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:58:23.255 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:58:23.255 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:58:23.255 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:58:23.255 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:58:23.255 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:58:23.255 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:58:23.255 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:58:23.255 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:58:23.255 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:58:23.255 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:58:23.256 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:58:23.256 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:58:23.256 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:58:23.256 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:58:23.256 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:58:23.256 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000025", 2024-08-18T15:58:23.256 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:58:23.256 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000123", 2024-08-18T15:58:23.256 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.31", 2024-08-18T15:58:23.256 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:58:23.256 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:58:23.256 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:58:23.256 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:58:23.256 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.18", 2024-08-18T15:58:23.257 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:58:23.257 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:58:23.257 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:58:23.257 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:58:23.257 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:58:23.257 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000108", 2024-08-18T15:58:23.257 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:58:23.257 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:58:23.257 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000094", 2024-08-18T15:58:23.257 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:58:23.257 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:58:23.257 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:58:23.257 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:58:23.258 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:58:23.258 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:58:23.258 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000060", 2024-08-18T15:58:23.258 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:58:23.258 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:58:23.258 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:58:23.258 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000109", 2024-08-18T15:58:23.258 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.15", 2024-08-18T15:58:23.258 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:58:23.258 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:58:23.258 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:58:23.258 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:58:23.259 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:58:23.259 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:58:23.259 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:58:23.259 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:58:23.259 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:58:23.259 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000013", 2024-08-18T15:58:23.259 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:58:23.259 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:58:23.259 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:58:23.259 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:58:23.259 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:58:23.259 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:58:23.259 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:58:23.260 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:58:23.260 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:58:23.260 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:58:23.260 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:58:23.260 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:58:23.260 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:58:23.260 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:58:23.260 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000051", 2024-08-18T15:58:23.260 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:58:23.260 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:58:23.260 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:58:23.260 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:58:23.260 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000106", 2024-08-18T15:58:23.261 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:58:23.261 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:58:23.261 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:58:23.261 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:58:23.261 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:58:23.261 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000017", 2024-08-18T15:58:23.261 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:58:23.261 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:58:23.261 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:58:23.261 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000023", 2024-08-18T15:58:23.261 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:58:23.261 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:58:23.261 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:58:23.261 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:58:23.262 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:58:23.262 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:58:23.262 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:58:23.262 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000107", 2024-08-18T15:58:23.262 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:58:23.262 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:58:23.262 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000046", 2024-08-18T15:58:23.262 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:58:23.262 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:58:23.262 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:58:23.262 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.26", 2024-08-18T15:58:23.262 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:58:23.262 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:58:23.263 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9", 2024-08-18T15:58:23.263 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.22", 2024-08-18T15:58:23.263 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:58:23.263 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:58:23.263 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:58:23.263 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:58:23.263 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:58:23.263 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:58:23.263 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:58:23.263 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2", 2024-08-18T15:58:23.263 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:58:23.263 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:58:23.263 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000067", 2024-08-18T15:58:23.264 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:58:23.264 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:58:23.264 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:58:23.264 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:58:23.264 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:58:23.264 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:58:23.264 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:58:23.264 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:58:23.264 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:58:23.264 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:58:23.264 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:58:23.264 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:58:23.265 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:58:23.265 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:58:23.265 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:58:23.265 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:58:23.265 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:58:23.265 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:58:23.265 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:58:23.265 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.9", 2024-08-18T15:58:23.265 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000086", 2024-08-18T15:58:23.265 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.5", 2024-08-18T15:58:23.265 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:58:23.265 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:58:23.265 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:58:23.265 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:58:23.266 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:58:23.266 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:58:23.266 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2.MFXJ583gCr1.0", 2024-08-18T15:58:23.266 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:58:23.266 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1.GPnO16liyZ+.0", 2024-08-18T15:58:23.266 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:58:23.266 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:58:23.266 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:58:23.266 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:58:23.266 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:58:23.266 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:58:23.266 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:58:23.266 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:58:23.267 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:58:23.267 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.20", 2024-08-18T15:58:23.267 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:58:23.267 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:58:23.267 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:58:23.267 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:58:23.267 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:58:23.267 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:58:23.267 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1.retry", 2024-08-18T15:58:23.267 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:58:23.267 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:58:23.267 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:58:23.268 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0.retry", 2024-08-18T15:58:23.268 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000050", 2024-08-18T15:58:23.268 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:58:23.268 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000102", 2024-08-18T15:58:23.268 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:58:23.268 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:58:23.268 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:58:23.268 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:58:23.268 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:58:23.268 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:58:23.268 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:58:23.268 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:58:23.269 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:58:23.269 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:58:23.269 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:58:23.269 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000124", 2024-08-18T15:58:23.269 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:58:23.269 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:58:23.269 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:58:23.269 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000001", 2024-08-18T15:58:23.269 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:58:23.269 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:58:23.269 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:58:23.269 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:58:23.269 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000126", 2024-08-18T15:58:23.270 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:58:23.270 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0.fZhctfLXnSt.0", 2024-08-18T15:58:23.270 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:58:23.270 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000055", 2024-08-18T15:58:23.270 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:58:23.270 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000061", 2024-08-18T15:58:23.270 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1", 2024-08-18T15:58:23.270 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:58:23.270 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:58:23.270 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:58:23.270 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000097", 2024-08-18T15:58:23.270 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:58:23.270 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:58:23.271 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:58:23.271 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:58:23.271 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:58:23.271 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:58:23.271 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:58:23.271 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:58:23.271 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000042", 2024-08-18T15:58:23.271 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:58:23.271 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:58:23.271 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:58:23.271 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1.retry", 2024-08-18T15:58:23.271 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:58:23.271 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:58:23.271 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:58:23.272 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:58:23.272 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:58:23.272 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:58:23.272 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:58:23.272 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000075", 2024-08-18T15:58:23.272 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:58:23.272 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000120", 2024-08-18T15:58:23.272 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000030", 2024-08-18T15:58:23.272 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:58:23.272 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:58:23.272 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:58:23.272 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:58:23.272 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:58:23.273 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:58:23.273 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9" 2024-08-18T15:58:23.273 INFO:teuthology.orchestra.run.smithi193.stdout:] 2024-08-18T15:58:23.273 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:58:28.275 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:58:28.275 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:58:28.275 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:58:28.329 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:58:28.329 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:58:28.392 INFO:teuthology.orchestra.run.smithi193.stdout:[ 2024-08-18T15:58:28.393 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:58:28.393 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:58:28.393 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.29", 2024-08-18T15:58:28.393 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000078", 2024-08-18T15:58:28.393 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.history", 2024-08-18T15:58:28.393 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:58:28.393 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:58:28.393 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:58:28.393 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:58:28.394 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.23", 2024-08-18T15:58:28.394 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000070", 2024-08-18T15:58:28.394 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000104", 2024-08-18T15:58:28.394 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:58:28.394 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2.retry", 2024-08-18T15:58:28.394 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:58:28.394 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:58:28.394 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:58:28.394 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:58:28.394 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000026", 2024-08-18T15:58:28.395 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:58:28.395 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:58:28.395 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:58:28.395 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:58:28.395 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:58:28.395 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:58:28.395 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.8", 2024-08-18T15:58:28.395 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:58:28.395 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:58:28.395 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:58:28.395 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:58:28.395 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000028", 2024-08-18T15:58:28.396 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:58:28.396 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:58:28.396 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:58:28.396 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:58:28.396 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:58:28.396 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:58:28.396 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:58:28.396 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2.retry", 2024-08-18T15:58:28.396 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:58:28.396 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.30", 2024-08-18T15:58:28.396 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:58:28.396 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:58:28.396 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:58:28.396 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:58:28.397 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000040", 2024-08-18T15:58:28.397 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:58:28.397 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.6", 2024-08-18T15:58:28.397 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:58:28.397 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:58:28.397 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:58:28.397 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:58:28.397 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:58:28.397 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:58:28.397 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000015", 2024-08-18T15:58:28.397 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:58:28.397 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:58:28.397 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000069", 2024-08-18T15:58:28.397 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:58:28.398 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:58:28.398 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:58:28.398 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000095", 2024-08-18T15:58:28.398 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0", 2024-08-18T15:58:28.398 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000003", 2024-08-18T15:58:28.398 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000047", 2024-08-18T15:58:28.398 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:58:28.398 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:58:28.398 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:58:28.398 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:58:28.398 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:58:28.398 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:58:28.398 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:58:28.399 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:58:28.399 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:58:28.399 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000035", 2024-08-18T15:58:28.399 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:58:28.399 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:58:28.399 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000037", 2024-08-18T15:58:28.399 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:58:28.399 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:58:28.399 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:58:28.399 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:58:28.399 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:58:28.399 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:58:28.399 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:58:28.400 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:58:28.400 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:58:28.400 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:58:28.400 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:58:28.400 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:58:28.400 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:58:28.400 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:58:28.400 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000024", 2024-08-18T15:58:28.400 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:58:28.400 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:58:28.401 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:58:28.401 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:58:28.401 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000000", 2024-08-18T15:58:28.401 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:58:28.401 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000031", 2024-08-18T15:58:28.401 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:58:28.401 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:58:28.401 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:58:28.401 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:58:28.401 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:58:28.401 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:58:28.401 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3", 2024-08-18T15:58:28.401 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:58:28.401 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:58:28.402 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:58:28.402 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:58:28.402 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:58:28.402 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:58:28.402 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:58:28.402 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:58:28.402 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:58:28.402 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:58:28.402 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:58:28.402 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:58:28.402 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:58:28.402 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:58:28.402 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:58:28.403 INFO:teuthology.orchestra.run.smithi193.stdout: "data_loggenerations_metadata", 2024-08-18T15:58:28.403 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2", 2024-08-18T15:58:28.403 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:58:28.403 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:58:28.403 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:58:28.403 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:58:28.403 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T15:58:28.403 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:58:28.403 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:58:28.403 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:58:28.403 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:58:28.403 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:58:28.403 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000076", 2024-08-18T15:58:28.403 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:58:28.404 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:58:28.404 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000113", 2024-08-18T15:58:28.404 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000005", 2024-08-18T15:58:28.404 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:58:28.404 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000011", 2024-08-18T15:58:28.404 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:58:28.404 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000058", 2024-08-18T15:58:28.404 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:58:28.404 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:58:28.405 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:58:28.405 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000052", 2024-08-18T15:58:28.405 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:58:28.405 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:58:28.405 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:58:28.405 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:58:28.405 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.21", 2024-08-18T15:58:28.405 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:58:28.405 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:58:28.405 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:58:28.405 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:58:28.405 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:58:28.406 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:58:28.406 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:58:28.406 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:58:28.406 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:58:28.406 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:58:28.406 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:58:28.406 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:58:28.406 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:58:28.406 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:58:28.406 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:58:28.406 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:58:28.407 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:58:28.407 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:58:28.407 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:58:28.407 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000088", 2024-08-18T15:58:28.407 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000080", 2024-08-18T15:58:28.407 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:58:28.407 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:58:28.407 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000090", 2024-08-18T15:58:28.407 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:58:28.407 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:58:28.407 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000110", 2024-08-18T15:58:28.407 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:58:28.407 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:58:28.408 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:58:28.408 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.0", 2024-08-18T15:58:28.408 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:58:28.408 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000096", 2024-08-18T15:58:28.408 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:58:28.408 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:58:28.408 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:58:28.408 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:58:28.408 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:58:28.408 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000087", 2024-08-18T15:58:28.408 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:58:28.408 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:58:28.409 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:58:28.409 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:58:28.409 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000008", 2024-08-18T15:58:28.409 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:58:28.409 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:58:28.409 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:58:28.409 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:58:28.409 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:58:28.409 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000006", 2024-08-18T15:58:28.409 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000029", 2024-08-18T15:58:28.409 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:58:28.409 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000089", 2024-08-18T15:58:28.410 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:58:28.410 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:58:28.410 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000022", 2024-08-18T15:58:28.410 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:58:28.410 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:58:28.410 INFO:teuthology.orchestra.run.smithi193.stdout: "bilog.trim", 2024-08-18T15:58:28.410 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000016", 2024-08-18T15:58:28.410 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:58:28.411 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:58:28.411 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:58:28.411 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:58:28.411 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:58:28.411 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000041", 2024-08-18T15:58:28.411 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:58:28.411 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:58:28.411 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.log.01050e30-179b-43be-bdbb-3bb23cf37ae8.3", 2024-08-18T15:58:28.411 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:58:28.411 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000018", 2024-08-18T15:58:28.412 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000092", 2024-08-18T15:58:28.412 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:58:28.412 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:58:28.412 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:58:28.412 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:58:28.412 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.27", 2024-08-18T15:58:28.412 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.4", 2024-08-18T15:58:28.412 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:58:28.412 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:58:28.412 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:58:28.412 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:58:28.412 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:58:28.412 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:58:28.413 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:58:28.413 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000014", 2024-08-18T15:58:28.413 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:58:28.413 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3.retry", 2024-08-18T15:58:28.413 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:58:28.413 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:58:28.413 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:58:28.413 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:58:28.413 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:58:28.413 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000112", 2024-08-18T15:58:28.413 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:58:28.413 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000007", 2024-08-18T15:58:28.414 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000021", 2024-08-18T15:58:28.414 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:58:28.414 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:58:28.414 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:58:28.414 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:58:28.414 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:58:28.414 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.16", 2024-08-18T15:58:28.414 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:58:28.417 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000064", 2024-08-18T15:58:28.417 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:58:28.418 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:58:28.418 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:58:28.418 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:58:28.418 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:58:28.418 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:58:28.419 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:58:28.419 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000071", 2024-08-18T15:58:28.419 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:58:28.419 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:58:28.419 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:58:28.419 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000074", 2024-08-18T15:58:28.420 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:58:28.420 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:58:28.420 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:58:28.420 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:58:28.420 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:58:28.420 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:58:28.420 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:58:28.420 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:58:28.420 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:58:28.420 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:58:28.420 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:58:28.420 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:58:28.420 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:58:28.421 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:58:28.421 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:58:28.421 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000081", 2024-08-18T15:58:28.421 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:58:28.421 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000009", 2024-08-18T15:58:28.421 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000121", 2024-08-18T15:58:28.421 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1", 2024-08-18T15:58:28.421 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:58:28.421 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:58:28.421 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:58:28.421 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:58:28.421 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:58:28.422 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:58:28.422 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.17", 2024-08-18T15:58:28.422 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000125", 2024-08-18T15:58:28.422 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:58:28.422 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000082", 2024-08-18T15:58:28.422 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:58:28.422 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:58:28.422 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:58:28.422 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:58:28.422 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:58:28.422 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:58:28.423 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:58:28.423 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:58:28.423 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:58:28.423 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:58:28.423 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:58:28.423 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000105", 2024-08-18T15:58:28.423 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000059", 2024-08-18T15:58:28.423 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:58:28.423 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:58:28.423 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:58:28.423 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:58:28.423 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:58:28.423 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:58:28.424 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:58:28.424 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:58:28.424 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:58:28.424 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:58:28.424 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:58:28.424 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000077", 2024-08-18T15:58:28.424 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000032", 2024-08-18T15:58:28.424 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:58:28.424 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:58:28.424 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:58:28.424 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:58:28.424 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:58:28.425 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:58:28.425 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.24", 2024-08-18T15:58:28.425 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:58:28.425 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:58:28.425 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:58:28.425 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:58:28.425 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3.retry", 2024-08-18T15:58:28.425 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:58:28.426 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:58:28.426 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:58:28.426 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:58:28.426 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:58:28.426 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:58:28.426 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000053", 2024-08-18T15:58:28.426 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000091", 2024-08-18T15:58:28.426 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:58:28.426 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:58:28.426 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:58:28.426 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000065", 2024-08-18T15:58:28.426 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:58:28.427 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:58:28.427 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:58:28.427 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:58:28.427 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000083", 2024-08-18T15:58:28.427 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000010", 2024-08-18T15:58:28.427 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:58:28.427 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:58:28.427 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:58:28.427 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:58:28.427 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:58:28.427 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:58:28.427 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:58:28.427 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:58:28.428 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:58:28.428 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:58:28.428 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:58:28.428 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.7", 2024-08-18T15:58:28.428 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.3", 2024-08-18T15:58:28.428 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:58:28.428 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:58:28.428 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:58:28.428 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:58:28.428 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:58:28.428 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:58:28.428 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:58:28.429 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:58:28.429 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.2", 2024-08-18T15:58:28.429 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:58:28.429 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:58:28.429 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:58:28.429 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:58:28.429 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:58:28.429 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:58:28.429 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:58:28.429 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000045", 2024-08-18T15:58:28.429 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000002", 2024-08-18T15:58:28.429 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:58:28.429 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3.2UbiLidwSTy.0", 2024-08-18T15:58:28.429 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:58:28.430 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.12", 2024-08-18T15:58:28.430 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:58:28.430 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:58:28.430 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:58:28.430 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:58:28.430 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:58:28.430 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:58:28.430 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000116", 2024-08-18T15:58:28.430 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.25", 2024-08-18T15:58:28.430 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:58:28.430 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:58:28.430 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:58:28.431 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:58:28.431 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000034", 2024-08-18T15:58:28.431 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:58:28.431 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:58:28.431 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000101", 2024-08-18T15:58:28.432 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:58:28.432 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:58:28.432 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.13", 2024-08-18T15:58:28.432 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:58:28.432 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:58:28.432 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.1", 2024-08-18T15:58:28.432 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0.retry", 2024-08-18T15:58:28.432 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:58:28.433 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:58:28.433 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000079", 2024-08-18T15:58:28.433 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000049", 2024-08-18T15:58:28.433 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:58:28.433 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0", 2024-08-18T15:58:28.433 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:58:28.433 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:58:28.433 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000117", 2024-08-18T15:58:28.433 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:58:28.433 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000044", 2024-08-18T15:58:28.433 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:58:28.433 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:58:28.433 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:58:28.434 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:58:28.434 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:58:28.434 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:58:28.434 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.14", 2024-08-18T15:58:28.434 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:58:28.434 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000066", 2024-08-18T15:58:28.434 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:58:28.434 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:58:28.434 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:58:28.434 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:58:28.435 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:58:28.435 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000068", 2024-08-18T15:58:28.435 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:58:28.435 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000085", 2024-08-18T15:58:28.435 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:58:28.435 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:58:28.435 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:58:28.435 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:58:28.435 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:58:28.435 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:58:28.435 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:58:28.436 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:58:28.436 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:58:28.436 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:58:28.436 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:58:28.436 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:58:28.436 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000073", 2024-08-18T15:58:28.436 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:58:28.436 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:58:28.436 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:58:28.436 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000038", 2024-08-18T15:58:28.436 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:58:28.437 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:58:28.437 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:58:28.437 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:58:28.437 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.10", 2024-08-18T15:58:28.437 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:58:28.437 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:58:28.437 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:58:28.437 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000118", 2024-08-18T15:58:28.437 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:58:28.437 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3", 2024-08-18T15:58:28.437 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:58:28.437 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000036", 2024-08-18T15:58:28.437 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:58:28.437 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:58:28.438 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:58:28.438 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:58:28.438 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:58:28.438 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000103", 2024-08-18T15:58:28.438 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.3", 2024-08-18T15:58:28.438 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:58:28.438 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000119", 2024-08-18T15:58:28.438 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000098", 2024-08-18T15:58:28.438 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:58:28.438 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:58:28.438 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:58:28.438 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:58:28.439 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:58:28.439 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:58:28.439 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:58:28.439 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:58:28.439 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.0", 2024-08-18T15:58:28.439 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:58:28.439 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:58:28.439 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:58:28.439 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:58:28.439 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:58:28.439 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:58:28.439 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:58:28.440 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:58:28.440 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000027", 2024-08-18T15:58:28.440 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:58:28.440 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status", 2024-08-18T15:58:28.440 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:58:28.440 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000019", 2024-08-18T15:58:28.440 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:58:28.440 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:58:28.440 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:58:28.440 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:58:28.440 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1", 2024-08-18T15:58:28.440 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:58:28.440 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:58:28.441 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:58:28.441 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:58:28.441 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000039", 2024-08-18T15:58:28.441 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:58:28.441 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:58:28.441 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:58:28.441 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:58:28.441 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:58:28.441 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:58:28.441 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:58:28.441 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:58:28.441 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:58:28.441 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:58:28.442 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:58:28.442 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000100", 2024-08-18T15:58:28.442 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:58:28.442 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:58:28.442 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:58:28.442 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:58:28.442 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:58:28.442 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000093", 2024-08-18T15:58:28.442 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:58:28.442 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000004", 2024-08-18T15:58:28.442 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000063", 2024-08-18T15:58:28.442 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000122", 2024-08-18T15:58:28.443 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:58:28.443 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:58:28.443 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:58:28.443 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000057", 2024-08-18T15:58:28.443 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:58:28.443 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:58:28.443 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:58:28.443 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:58:28.443 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000054", 2024-08-18T15:58:28.443 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:58:28.443 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000114", 2024-08-18T15:58:28.443 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:58:28.443 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:58:28.444 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:58:28.444 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.2", 2024-08-18T15:58:28.444 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:58:28.444 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:58:28.444 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0", 2024-08-18T15:58:28.444 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:58:28.444 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:58:28.444 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:58:28.444 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:58:28.444 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:58:28.444 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:58:28.444 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000012", 2024-08-18T15:58:28.445 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:58:28.445 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:58:28.445 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:58:28.445 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000084", 2024-08-18T15:58:28.445 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000043", 2024-08-18T15:58:28.445 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3", 2024-08-18T15:58:28.445 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:58:28.445 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:58:28.445 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.11", 2024-08-18T15:58:28.445 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:58:28.445 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:58:28.446 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:58:28.446 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:58:28.446 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:58:28.446 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:58:28.446 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:58:28.446 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:58:28.446 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000111", 2024-08-18T15:58:28.446 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:58:28.447 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000048", 2024-08-18T15:58:28.447 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:58:28.447 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000020", 2024-08-18T15:58:28.447 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:58:28.447 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:58:28.447 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:58:28.447 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:58:28.447 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:58:28.447 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000099", 2024-08-18T15:58:28.447 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000056", 2024-08-18T15:58:28.447 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:58:28.447 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:58:28.447 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.19", 2024-08-18T15:58:28.447 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:58:28.448 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.28", 2024-08-18T15:58:28.448 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:58:28.448 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:58:28.448 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:58:28.448 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:58:28.448 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:58:28.448 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000072", 2024-08-18T15:58:28.448 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:58:28.448 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:58:28.448 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000062", 2024-08-18T15:58:28.448 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:58:28.448 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:58:28.448 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:58:28.449 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000115", 2024-08-18T15:58:28.449 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:58:28.449 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.1", 2024-08-18T15:58:28.449 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2", 2024-08-18T15:58:28.449 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000033", 2024-08-18T15:58:28.449 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:58:28.449 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:58:28.449 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:58:28.449 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:58:28.449 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:58:28.449 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:58:28.449 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:58:28.449 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:58:28.449 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:58:28.450 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:58:28.450 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:58:28.450 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:58:28.450 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:58:28.450 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:58:28.450 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:58:28.450 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:58:28.450 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:58:28.450 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:58:28.450 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:58:28.450 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000025", 2024-08-18T15:58:28.450 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:58:28.451 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000123", 2024-08-18T15:58:28.451 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.31", 2024-08-18T15:58:28.451 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:58:28.451 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:58:28.451 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:58:28.451 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:58:28.451 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.18", 2024-08-18T15:58:28.451 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:58:28.451 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:58:28.451 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:58:28.451 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:58:28.451 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:58:28.451 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000108", 2024-08-18T15:58:28.452 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:58:28.452 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:58:28.452 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000094", 2024-08-18T15:58:28.452 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:58:28.452 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:58:28.452 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:58:28.452 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:58:28.452 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:58:28.452 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:58:28.452 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000060", 2024-08-18T15:58:28.452 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:58:28.452 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:58:28.453 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:58:28.453 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000109", 2024-08-18T15:58:28.453 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.15", 2024-08-18T15:58:28.453 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:58:28.453 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:58:28.453 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:58:28.453 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:58:28.453 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:58:28.453 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:58:28.453 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:58:28.453 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:58:28.453 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:58:28.453 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000013", 2024-08-18T15:58:28.453 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:58:28.454 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:58:28.454 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:58:28.454 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:58:28.454 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:58:28.454 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:58:28.454 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:58:28.454 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:58:28.454 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:58:28.454 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:58:28.454 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:58:28.454 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:58:28.454 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:58:28.455 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:58:28.455 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000051", 2024-08-18T15:58:28.455 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:58:28.455 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:58:28.455 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:58:28.455 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:58:28.455 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000106", 2024-08-18T15:58:28.455 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:58:28.455 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:58:28.455 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:58:28.455 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:58:28.455 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:58:28.455 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000017", 2024-08-18T15:58:28.456 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:58:28.456 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:58:28.456 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:58:28.456 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000023", 2024-08-18T15:58:28.456 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:58:28.456 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:58:28.456 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:58:28.457 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:58:28.457 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:58:28.457 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:58:28.457 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:58:28.457 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000107", 2024-08-18T15:58:28.457 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:58:28.457 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:58:28.457 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000046", 2024-08-18T15:58:28.457 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:58:28.457 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:58:28.457 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:58:28.457 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.26", 2024-08-18T15:58:28.458 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:58:28.458 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:58:28.458 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9", 2024-08-18T15:58:28.458 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.22", 2024-08-18T15:58:28.458 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:58:28.458 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:58:28.458 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:58:28.458 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:58:28.458 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:58:28.458 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:58:28.458 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:58:28.459 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2", 2024-08-18T15:58:28.459 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:58:28.459 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:58:28.459 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000067", 2024-08-18T15:58:28.459 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:58:28.459 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:58:28.459 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:58:28.459 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:58:28.459 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:58:28.459 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:58:28.459 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:58:28.460 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:58:28.460 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:58:28.460 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:58:28.460 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:58:28.460 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:58:28.460 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:58:28.460 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:58:28.460 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:58:28.460 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:58:28.460 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:58:28.460 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:58:28.460 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:58:28.460 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.9", 2024-08-18T15:58:28.460 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000086", 2024-08-18T15:58:28.460 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.5", 2024-08-18T15:58:28.461 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:58:28.461 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:58:28.461 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:58:28.461 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:58:28.461 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:58:28.461 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:58:28.461 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2.MFXJ583gCr1.0", 2024-08-18T15:58:28.461 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:58:28.461 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1.GPnO16liyZ+.0", 2024-08-18T15:58:28.461 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:58:28.461 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:58:28.461 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:58:28.461 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:58:28.462 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:58:28.462 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:58:28.462 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:58:28.462 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:58:28.462 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:58:28.462 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.20", 2024-08-18T15:58:28.462 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:58:28.462 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:58:28.462 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:58:28.462 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:58:28.462 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:58:28.463 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:58:28.463 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1.retry", 2024-08-18T15:58:28.463 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:58:28.463 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:58:28.463 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:58:28.463 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0.retry", 2024-08-18T15:58:28.463 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000050", 2024-08-18T15:58:28.463 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:58:28.463 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000102", 2024-08-18T15:58:28.463 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:58:28.463 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:58:28.463 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:58:28.463 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:58:28.463 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:58:28.464 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:58:28.464 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:58:28.464 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:58:28.464 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:58:28.464 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:58:28.464 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:58:28.464 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000124", 2024-08-18T15:58:28.464 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:58:28.464 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:58:28.464 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:58:28.464 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000001", 2024-08-18T15:58:28.464 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:58:28.464 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:58:28.465 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:58:28.465 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:58:28.465 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000126", 2024-08-18T15:58:28.465 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:58:28.465 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0.fZhctfLXnSt.0", 2024-08-18T15:58:28.465 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:58:28.465 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000055", 2024-08-18T15:58:28.465 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:58:28.465 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000061", 2024-08-18T15:58:28.465 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1", 2024-08-18T15:58:28.465 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:58:28.465 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:58:28.465 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:58:28.466 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000097", 2024-08-18T15:58:28.466 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:58:28.466 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:58:28.466 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:58:28.466 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:58:28.466 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:58:28.466 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:58:28.466 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:58:28.466 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:58:28.466 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000042", 2024-08-18T15:58:28.466 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:58:28.467 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:58:28.467 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:58:28.467 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1.retry", 2024-08-18T15:58:28.467 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:58:28.467 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:58:28.467 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:58:28.467 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:58:28.468 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:58:28.468 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:58:28.468 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:58:28.468 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000075", 2024-08-18T15:58:28.468 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:58:28.468 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000120", 2024-08-18T15:58:28.468 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000030", 2024-08-18T15:58:28.468 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:58:28.468 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:58:28.468 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:58:28.468 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:58:28.468 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:58:28.468 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:58:28.469 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9" 2024-08-18T15:58:28.469 INFO:teuthology.orchestra.run.smithi193.stdout:] 2024-08-18T15:58:28.469 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:58:33.471 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:58:33.471 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:58:33.472 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:58:33.519 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:58:33.519 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:58:33.578 INFO:teuthology.orchestra.run.smithi193.stdout:[ 2024-08-18T15:58:33.578 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:58:33.578 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:58:33.578 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.29", 2024-08-18T15:58:33.578 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000078", 2024-08-18T15:58:33.578 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.history", 2024-08-18T15:58:33.578 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:58:33.579 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:58:33.579 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:58:33.579 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:58:33.579 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.23", 2024-08-18T15:58:33.579 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000070", 2024-08-18T15:58:33.579 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000104", 2024-08-18T15:58:33.579 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:58:33.579 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2.retry", 2024-08-18T15:58:33.579 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:58:33.579 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:58:33.579 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:58:33.579 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:58:33.580 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000026", 2024-08-18T15:58:33.580 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:58:33.580 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:58:33.580 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:58:33.580 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:58:33.580 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:58:33.580 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:58:33.580 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.8", 2024-08-18T15:58:33.580 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:58:33.580 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:58:33.581 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:58:33.581 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:58:33.581 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000028", 2024-08-18T15:58:33.581 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:58:33.581 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:58:33.581 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:58:33.581 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:58:33.581 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:58:33.581 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:58:33.581 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:58:33.582 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2.retry", 2024-08-18T15:58:33.582 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:58:33.582 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.30", 2024-08-18T15:58:33.582 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:58:33.582 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:58:33.582 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:58:33.582 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:58:33.582 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000040", 2024-08-18T15:58:33.582 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:58:33.582 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.6", 2024-08-18T15:58:33.582 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:58:33.582 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:58:33.582 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:58:33.583 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:58:33.583 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:58:33.583 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:58:33.583 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000015", 2024-08-18T15:58:33.583 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:58:33.583 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:58:33.583 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000069", 2024-08-18T15:58:33.583 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:58:33.583 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:58:33.583 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:58:33.583 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000095", 2024-08-18T15:58:33.583 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0", 2024-08-18T15:58:33.583 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000003", 2024-08-18T15:58:33.583 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000047", 2024-08-18T15:58:33.584 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:58:33.584 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:58:33.584 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:58:33.584 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:58:33.584 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:58:33.584 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:58:33.584 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:58:33.584 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:58:33.584 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:58:33.584 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000035", 2024-08-18T15:58:33.584 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:58:33.584 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:58:33.584 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000037", 2024-08-18T15:58:33.584 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:58:33.585 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:58:33.585 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:58:33.585 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:58:33.585 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:58:33.585 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:58:33.585 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:58:33.585 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:58:33.585 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:58:33.585 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:58:33.585 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:58:33.585 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:58:33.585 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:58:33.585 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:58:33.586 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000024", 2024-08-18T15:58:33.586 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:58:33.586 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:58:33.586 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:58:33.586 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:58:33.586 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000000", 2024-08-18T15:58:33.586 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:58:33.586 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000031", 2024-08-18T15:58:33.586 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:58:33.586 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:58:33.586 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:58:33.586 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:58:33.587 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:58:33.587 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:58:33.587 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3", 2024-08-18T15:58:33.587 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:58:33.587 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:58:33.587 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:58:33.587 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:58:33.587 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:58:33.587 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:58:33.587 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:58:33.587 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:58:33.587 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:58:33.587 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:58:33.588 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:58:33.588 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:58:33.588 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:58:33.588 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:58:33.588 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:58:33.588 INFO:teuthology.orchestra.run.smithi193.stdout: "data_loggenerations_metadata", 2024-08-18T15:58:33.588 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2", 2024-08-18T15:58:33.588 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:58:33.588 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:58:33.589 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:58:33.589 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:58:33.589 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T15:58:33.589 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:58:33.589 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:58:33.589 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:58:33.589 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:58:33.589 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:58:33.589 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000076", 2024-08-18T15:58:33.589 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:58:33.589 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:58:33.589 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000113", 2024-08-18T15:58:33.590 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000005", 2024-08-18T15:58:33.590 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:58:33.590 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000011", 2024-08-18T15:58:33.590 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:58:33.590 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000058", 2024-08-18T15:58:33.590 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:58:33.590 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:58:33.590 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:58:33.590 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000052", 2024-08-18T15:58:33.590 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:58:33.590 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:58:33.590 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:58:33.591 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:58:33.591 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.21", 2024-08-18T15:58:33.591 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:58:33.591 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:58:33.591 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:58:33.591 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:58:33.591 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:58:33.591 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:58:33.591 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:58:33.591 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:58:33.591 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:58:33.591 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:58:33.591 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:58:33.592 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:58:33.592 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:58:33.592 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:58:33.592 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:58:33.592 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:58:33.592 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:58:33.592 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:58:33.593 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:58:33.593 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000088", 2024-08-18T15:58:33.593 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000080", 2024-08-18T15:58:33.593 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:58:33.593 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:58:33.593 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000090", 2024-08-18T15:58:33.593 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:58:33.593 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:58:33.594 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000110", 2024-08-18T15:58:33.594 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:58:33.594 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:58:33.594 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:58:33.594 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.0", 2024-08-18T15:58:33.594 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:58:33.594 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000096", 2024-08-18T15:58:33.594 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:58:33.594 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:58:33.594 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:58:33.594 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:58:33.595 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:58:33.595 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000087", 2024-08-18T15:58:33.595 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:58:33.595 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:58:33.595 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:58:33.595 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:58:33.595 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000008", 2024-08-18T15:58:33.595 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:58:33.595 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:58:33.595 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:58:33.596 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:58:33.596 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:58:33.596 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000006", 2024-08-18T15:58:33.596 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000029", 2024-08-18T15:58:33.596 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:58:33.596 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000089", 2024-08-18T15:58:33.596 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:58:33.596 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:58:33.596 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000022", 2024-08-18T15:58:33.596 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:58:33.596 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:58:33.596 INFO:teuthology.orchestra.run.smithi193.stdout: "bilog.trim", 2024-08-18T15:58:33.596 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000016", 2024-08-18T15:58:33.596 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:58:33.597 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:58:33.597 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:58:33.597 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:58:33.597 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:58:33.597 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000041", 2024-08-18T15:58:33.597 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:58:33.597 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:58:33.597 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.log.01050e30-179b-43be-bdbb-3bb23cf37ae8.3", 2024-08-18T15:58:33.597 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:58:33.597 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000018", 2024-08-18T15:58:33.598 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000092", 2024-08-18T15:58:33.598 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:58:33.598 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:58:33.598 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:58:33.598 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:58:33.598 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.27", 2024-08-18T15:58:33.598 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.4", 2024-08-18T15:58:33.598 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:58:33.598 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:58:33.598 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:58:33.599 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:58:33.599 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:58:33.599 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:58:33.599 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:58:33.599 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000014", 2024-08-18T15:58:33.599 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:58:33.599 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3.retry", 2024-08-18T15:58:33.599 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:58:33.599 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:58:33.599 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:58:33.599 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:58:33.599 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:58:33.600 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000112", 2024-08-18T15:58:33.600 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:58:33.600 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000007", 2024-08-18T15:58:33.600 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000021", 2024-08-18T15:58:33.600 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:58:33.600 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:58:33.600 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:58:33.600 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:58:33.600 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:58:33.600 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.16", 2024-08-18T15:58:33.600 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:58:33.600 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000064", 2024-08-18T15:58:33.600 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:58:33.601 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:58:33.601 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:58:33.601 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:58:33.601 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:58:33.601 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:58:33.601 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:58:33.601 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000071", 2024-08-18T15:58:33.601 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:58:33.601 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:58:33.601 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:58:33.601 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000074", 2024-08-18T15:58:33.601 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:58:33.601 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:58:33.602 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:58:33.602 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:58:33.602 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:58:33.602 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:58:33.602 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:58:33.602 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:58:33.602 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:58:33.602 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:58:33.602 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:58:33.602 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:58:33.602 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:58:33.602 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:58:33.603 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:58:33.603 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000081", 2024-08-18T15:58:33.603 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:58:33.603 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000009", 2024-08-18T15:58:33.603 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000121", 2024-08-18T15:58:33.603 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1", 2024-08-18T15:58:33.603 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:58:33.603 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:58:33.603 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:58:33.603 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:58:33.603 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:58:33.604 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:58:33.604 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.17", 2024-08-18T15:58:33.604 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000125", 2024-08-18T15:58:33.604 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:58:33.604 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000082", 2024-08-18T15:58:33.604 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:58:33.604 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:58:33.604 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:58:33.604 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:58:33.604 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:58:33.604 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:58:33.604 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:58:33.605 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:58:33.605 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:58:33.605 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:58:33.605 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:58:33.605 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000105", 2024-08-18T15:58:33.605 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000059", 2024-08-18T15:58:33.605 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:58:33.605 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:58:33.605 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:58:33.605 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:58:33.605 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:58:33.605 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:58:33.606 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:58:33.606 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:58:33.606 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:58:33.606 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:58:33.606 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:58:33.606 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000077", 2024-08-18T15:58:33.606 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000032", 2024-08-18T15:58:33.606 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:58:33.606 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:58:33.606 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:58:33.606 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:58:33.606 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:58:33.607 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:58:33.607 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.24", 2024-08-18T15:58:33.607 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:58:33.607 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:58:33.607 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:58:33.607 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:58:33.607 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3.retry", 2024-08-18T15:58:33.607 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:58:33.607 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:58:33.607 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:58:33.607 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:58:33.607 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:58:33.607 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:58:33.608 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000053", 2024-08-18T15:58:33.608 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000091", 2024-08-18T15:58:33.608 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:58:33.608 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:58:33.608 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:58:33.608 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000065", 2024-08-18T15:58:33.608 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:58:33.608 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:58:33.608 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:58:33.608 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:58:33.608 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000083", 2024-08-18T15:58:33.608 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000010", 2024-08-18T15:58:33.608 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:58:33.608 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:58:33.609 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:58:33.609 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:58:33.609 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:58:33.609 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:58:33.609 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:58:33.609 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:58:33.609 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:58:33.609 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:58:33.609 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:58:33.609 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.7", 2024-08-18T15:58:33.609 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.3", 2024-08-18T15:58:33.609 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:58:33.609 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:58:33.610 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:58:33.610 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:58:33.610 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:58:33.610 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:58:33.610 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:58:33.610 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:58:33.610 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.2", 2024-08-18T15:58:33.610 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:58:33.610 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:58:33.610 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:58:33.610 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:58:33.610 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:58:33.611 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:58:33.611 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:58:33.611 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000045", 2024-08-18T15:58:33.611 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000002", 2024-08-18T15:58:33.611 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:58:33.611 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3.2UbiLidwSTy.0", 2024-08-18T15:58:33.611 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:58:33.611 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.12", 2024-08-18T15:58:33.611 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:58:33.611 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:58:33.611 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:58:33.611 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:58:33.611 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:58:33.611 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:58:33.612 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000116", 2024-08-18T15:58:33.612 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.25", 2024-08-18T15:58:33.612 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:58:33.612 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:58:33.612 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:58:33.612 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:58:33.612 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000034", 2024-08-18T15:58:33.612 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:58:33.612 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:58:33.612 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000101", 2024-08-18T15:58:33.612 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:58:33.612 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:58:33.613 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.13", 2024-08-18T15:58:33.613 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:58:33.613 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:58:33.613 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.1", 2024-08-18T15:58:33.613 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0.retry", 2024-08-18T15:58:33.613 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:58:33.613 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:58:33.613 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000079", 2024-08-18T15:58:33.613 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000049", 2024-08-18T15:58:33.613 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:58:33.613 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0", 2024-08-18T15:58:33.613 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:58:33.613 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:58:33.614 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000117", 2024-08-18T15:58:33.614 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:58:33.614 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000044", 2024-08-18T15:58:33.614 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:58:33.614 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:58:33.614 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:58:33.614 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:58:33.614 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:58:33.614 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:58:33.614 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.14", 2024-08-18T15:58:33.614 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:58:33.615 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000066", 2024-08-18T15:58:33.615 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:58:33.615 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:58:33.615 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:58:33.615 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:58:33.615 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:58:33.615 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000068", 2024-08-18T15:58:33.615 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:58:33.615 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000085", 2024-08-18T15:58:33.615 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:58:33.615 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:58:33.615 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:58:33.616 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:58:33.616 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:58:33.616 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:58:33.616 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:58:33.616 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:58:33.616 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:58:33.616 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:58:33.616 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:58:33.616 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:58:33.616 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000073", 2024-08-18T15:58:33.616 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:58:33.616 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:58:33.616 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:58:33.617 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000038", 2024-08-18T15:58:33.617 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:58:33.617 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:58:33.617 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:58:33.617 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:58:33.617 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.10", 2024-08-18T15:58:33.617 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:58:33.617 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:58:33.617 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:58:33.617 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000118", 2024-08-18T15:58:33.617 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:58:33.618 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3", 2024-08-18T15:58:33.618 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:58:33.618 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000036", 2024-08-18T15:58:33.618 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:58:33.618 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:58:33.618 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:58:33.618 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:58:33.618 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:58:33.618 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000103", 2024-08-18T15:58:33.618 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.3", 2024-08-18T15:58:33.618 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:58:33.618 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000119", 2024-08-18T15:58:33.618 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000098", 2024-08-18T15:58:33.619 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:58:33.619 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:58:33.619 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:58:33.619 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:58:33.619 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:58:33.619 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:58:33.619 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:58:33.619 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:58:33.620 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.0", 2024-08-18T15:58:33.620 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:58:33.620 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:58:33.620 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:58:33.620 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:58:33.620 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:58:33.620 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:58:33.620 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:58:33.620 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:58:33.620 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000027", 2024-08-18T15:58:33.620 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:58:33.620 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status", 2024-08-18T15:58:33.621 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:58:33.621 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000019", 2024-08-18T15:58:33.621 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:58:33.621 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:58:33.621 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:58:33.621 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:58:33.621 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1", 2024-08-18T15:58:33.621 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:58:33.621 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:58:33.621 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:58:33.621 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:58:33.621 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000039", 2024-08-18T15:58:33.621 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:58:33.621 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:58:33.622 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:58:33.622 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:58:33.622 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:58:33.622 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:58:33.622 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:58:33.622 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:58:33.622 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:58:33.622 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:58:33.622 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:58:33.622 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000100", 2024-08-18T15:58:33.622 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:58:33.622 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:58:33.623 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:58:33.623 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:58:33.623 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:58:33.623 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000093", 2024-08-18T15:58:33.623 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:58:33.623 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000004", 2024-08-18T15:58:33.623 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000063", 2024-08-18T15:58:33.623 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000122", 2024-08-18T15:58:33.624 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:58:33.624 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:58:33.624 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:58:33.624 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000057", 2024-08-18T15:58:33.624 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:58:33.624 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:58:33.624 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:58:33.624 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:58:33.624 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000054", 2024-08-18T15:58:33.624 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:58:33.624 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000114", 2024-08-18T15:58:33.625 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:58:33.625 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:58:33.625 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:58:33.625 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.2", 2024-08-18T15:58:33.625 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:58:33.625 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:58:33.625 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0", 2024-08-18T15:58:33.625 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:58:33.625 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:58:33.625 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:58:33.625 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:58:33.625 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:58:33.626 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:58:33.626 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000012", 2024-08-18T15:58:33.626 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:58:33.626 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:58:33.626 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:58:33.626 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000084", 2024-08-18T15:58:33.626 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000043", 2024-08-18T15:58:33.626 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3", 2024-08-18T15:58:33.626 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:58:33.626 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:58:33.626 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.11", 2024-08-18T15:58:33.626 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:58:33.627 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:58:33.627 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:58:33.627 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:58:33.627 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:58:33.627 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:58:33.627 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:58:33.627 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:58:33.627 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000111", 2024-08-18T15:58:33.627 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:58:33.628 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000048", 2024-08-18T15:58:33.628 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:58:33.628 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000020", 2024-08-18T15:58:33.628 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:58:33.628 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:58:33.628 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:58:33.628 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:58:33.628 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:58:33.628 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000099", 2024-08-18T15:58:33.628 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000056", 2024-08-18T15:58:33.628 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:58:33.628 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:58:33.628 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.19", 2024-08-18T15:58:33.628 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:58:33.629 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.28", 2024-08-18T15:58:33.629 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:58:33.629 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:58:33.629 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:58:33.629 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:58:33.629 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:58:33.629 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000072", 2024-08-18T15:58:33.629 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:58:33.629 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:58:33.629 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000062", 2024-08-18T15:58:33.630 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:58:33.630 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:58:33.630 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:58:33.630 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000115", 2024-08-18T15:58:33.630 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:58:33.630 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.1", 2024-08-18T15:58:33.630 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2", 2024-08-18T15:58:33.630 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000033", 2024-08-18T15:58:33.631 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:58:33.631 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:58:33.631 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:58:33.631 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:58:33.631 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:58:33.631 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:58:33.631 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:58:33.631 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:58:33.631 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:58:33.631 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:58:33.631 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:58:33.631 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:58:33.632 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:58:33.632 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:58:33.632 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:58:33.632 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:58:33.632 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:58:33.632 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:58:33.632 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:58:33.632 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000025", 2024-08-18T15:58:33.632 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:58:33.632 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000123", 2024-08-18T15:58:33.632 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.31", 2024-08-18T15:58:33.632 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:58:33.632 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:58:33.633 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:58:33.633 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:58:33.633 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.18", 2024-08-18T15:58:33.633 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:58:33.633 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:58:33.633 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:58:33.633 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:58:33.633 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:58:33.633 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000108", 2024-08-18T15:58:33.633 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:58:33.633 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:58:33.633 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000094", 2024-08-18T15:58:33.633 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:58:33.634 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:58:33.634 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:58:33.634 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:58:33.634 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:58:33.634 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:58:33.634 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000060", 2024-08-18T15:58:33.634 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:58:33.634 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:58:33.634 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:58:33.634 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000109", 2024-08-18T15:58:33.634 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.15", 2024-08-18T15:58:33.634 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:58:33.635 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:58:33.635 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:58:33.635 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:58:33.635 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:58:33.635 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:58:33.635 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:58:33.635 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:58:33.635 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:58:33.636 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000013", 2024-08-18T15:58:33.636 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:58:33.636 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:58:33.636 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:58:33.636 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:58:33.636 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:58:33.636 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:58:33.636 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:58:33.637 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:58:33.637 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:58:33.637 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:58:33.637 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:58:33.637 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:58:33.637 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:58:33.637 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:58:33.637 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000051", 2024-08-18T15:58:33.637 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:58:33.637 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:58:33.638 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:58:33.638 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:58:33.638 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000106", 2024-08-18T15:58:33.638 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:58:33.638 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:58:33.638 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:58:33.638 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:58:33.638 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:58:33.638 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000017", 2024-08-18T15:58:33.638 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:58:33.638 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:58:33.639 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:58:33.639 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000023", 2024-08-18T15:58:33.639 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:58:33.639 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:58:33.639 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:58:33.639 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:58:33.640 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:58:33.640 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:58:33.640 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:58:33.640 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000107", 2024-08-18T15:58:33.640 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:58:33.640 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:58:33.640 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000046", 2024-08-18T15:58:33.640 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:58:33.640 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:58:33.640 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:58:33.640 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.26", 2024-08-18T15:58:33.640 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:58:33.641 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:58:33.641 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9", 2024-08-18T15:58:33.641 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.22", 2024-08-18T15:58:33.641 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:58:33.641 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:58:33.641 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:58:33.642 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:58:33.642 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:58:33.642 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:58:33.642 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:58:33.642 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2", 2024-08-18T15:58:33.642 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:58:33.643 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:58:33.643 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000067", 2024-08-18T15:58:33.643 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:58:33.643 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:58:33.643 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:58:33.643 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:58:33.644 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:58:33.644 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:58:33.644 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:58:33.644 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:58:33.644 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:58:33.644 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:58:33.645 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:58:33.645 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:58:33.645 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:58:33.646 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:58:33.646 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:58:33.646 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:58:33.646 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:58:33.646 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:58:33.647 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:58:33.647 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.9", 2024-08-18T15:58:33.647 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000086", 2024-08-18T15:58:33.647 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.5", 2024-08-18T15:58:33.647 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:58:33.647 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:58:33.647 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:58:33.648 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:58:33.648 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:58:33.648 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:58:33.648 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2.MFXJ583gCr1.0", 2024-08-18T15:58:33.648 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:58:33.649 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1.GPnO16liyZ+.0", 2024-08-18T15:58:33.649 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:58:33.649 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:58:33.649 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:58:33.649 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:58:33.649 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:58:33.650 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:58:33.650 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:58:33.650 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:58:33.650 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:58:33.650 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.20", 2024-08-18T15:58:33.650 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:58:33.651 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:58:33.651 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:58:33.651 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:58:33.651 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:58:33.651 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:58:33.651 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1.retry", 2024-08-18T15:58:33.652 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:58:33.652 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:58:33.652 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:58:33.652 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0.retry", 2024-08-18T15:58:33.652 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000050", 2024-08-18T15:58:33.652 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:58:33.653 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000102", 2024-08-18T15:58:33.653 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:58:33.653 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:58:33.653 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:58:33.653 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:58:33.654 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:58:33.654 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:58:33.654 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:58:33.654 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:58:33.654 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:58:33.655 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:58:33.655 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:58:33.655 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000124", 2024-08-18T15:58:33.655 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:58:33.655 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:58:33.655 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:58:33.656 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000001", 2024-08-18T15:58:33.656 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:58:33.656 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:58:33.656 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:58:33.656 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:58:33.657 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000126", 2024-08-18T15:58:33.657 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:58:33.657 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0.fZhctfLXnSt.0", 2024-08-18T15:58:33.657 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:58:33.657 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000055", 2024-08-18T15:58:33.657 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:58:33.658 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000061", 2024-08-18T15:58:33.658 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1", 2024-08-18T15:58:33.658 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:58:33.659 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:58:33.659 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:58:33.659 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000097", 2024-08-18T15:58:33.659 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:58:33.659 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:58:33.660 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:58:33.660 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:58:33.660 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:58:33.660 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:58:33.660 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:58:33.660 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:58:33.660 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000042", 2024-08-18T15:58:33.661 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:58:33.661 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:58:33.661 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:58:33.661 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1.retry", 2024-08-18T15:58:33.661 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:58:33.662 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:58:33.662 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:58:33.662 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:58:33.662 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:58:33.662 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:58:33.663 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:58:33.663 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000075", 2024-08-18T15:58:33.663 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:58:33.663 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000120", 2024-08-18T15:58:33.663 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000030", 2024-08-18T15:58:33.663 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:58:33.664 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:58:33.664 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:58:33.664 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:58:33.664 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:58:33.664 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:58:33.664 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9" 2024-08-18T15:58:33.665 INFO:teuthology.orchestra.run.smithi193.stdout:] 2024-08-18T15:58:33.665 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:58:38.667 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:58:38.667 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:58:38.668 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:58:38.720 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:58:38.720 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:58:38.784 INFO:teuthology.orchestra.run.smithi193.stdout:[ 2024-08-18T15:58:38.784 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:58:38.784 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:58:38.784 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.29", 2024-08-18T15:58:38.784 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000078", 2024-08-18T15:58:38.784 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.history", 2024-08-18T15:58:38.784 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:58:38.784 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:58:38.784 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:58:38.785 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:58:38.785 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.23", 2024-08-18T15:58:38.785 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000070", 2024-08-18T15:58:38.785 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000104", 2024-08-18T15:58:38.785 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:58:38.785 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2.retry", 2024-08-18T15:58:38.785 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:58:38.785 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:58:38.785 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:58:38.785 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:58:38.785 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000026", 2024-08-18T15:58:38.785 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:58:38.786 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:58:38.786 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:58:38.786 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:58:38.786 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:58:38.786 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:58:38.786 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.8", 2024-08-18T15:58:38.786 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:58:38.786 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:58:38.786 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:58:38.786 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:58:38.786 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000028", 2024-08-18T15:58:38.786 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:58:38.787 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:58:38.787 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:58:38.787 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:58:38.787 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:58:38.787 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:58:38.787 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:58:38.787 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2.retry", 2024-08-18T15:58:38.787 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:58:38.787 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.30", 2024-08-18T15:58:38.787 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:58:38.787 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:58:38.787 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:58:38.787 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:58:38.788 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000040", 2024-08-18T15:58:38.788 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:58:38.788 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.6", 2024-08-18T15:58:38.788 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:58:38.788 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:58:38.788 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:58:38.788 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:58:38.788 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:58:38.788 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:58:38.788 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000015", 2024-08-18T15:58:38.788 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:58:38.788 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:58:38.788 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000069", 2024-08-18T15:58:38.789 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:58:38.789 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:58:38.789 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:58:38.789 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000095", 2024-08-18T15:58:38.789 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0", 2024-08-18T15:58:38.789 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000003", 2024-08-18T15:58:38.789 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000047", 2024-08-18T15:58:38.789 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:58:38.789 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:58:38.789 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:58:38.789 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:58:38.789 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:58:38.789 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:58:38.789 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:58:38.790 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:58:38.790 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:58:38.790 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000035", 2024-08-18T15:58:38.790 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:58:38.790 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:58:38.790 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000037", 2024-08-18T15:58:38.790 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:58:38.790 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:58:38.790 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:58:38.790 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:58:38.790 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:58:38.790 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:58:38.791 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:58:38.791 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:58:38.791 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:58:38.791 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:58:38.791 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:58:38.791 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:58:38.791 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:58:38.791 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:58:38.791 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000024", 2024-08-18T15:58:38.791 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:58:38.791 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:58:38.791 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:58:38.791 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:58:38.792 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000000", 2024-08-18T15:58:38.792 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:58:38.792 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000031", 2024-08-18T15:58:38.792 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:58:38.792 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:58:38.792 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:58:38.792 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:58:38.792 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:58:38.792 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:58:38.792 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3", 2024-08-18T15:58:38.792 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:58:38.792 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:58:38.792 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:58:38.793 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:58:38.793 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:58:38.793 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:58:38.793 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:58:38.793 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:58:38.793 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:58:38.793 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:58:38.793 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:58:38.793 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:58:38.793 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:58:38.793 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:58:38.793 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:58:38.793 INFO:teuthology.orchestra.run.smithi193.stdout: "data_loggenerations_metadata", 2024-08-18T15:58:38.793 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2", 2024-08-18T15:58:38.794 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:58:38.794 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:58:38.794 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:58:38.794 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:58:38.794 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T15:58:38.794 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:58:38.794 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:58:38.794 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:58:38.794 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:58:38.794 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:58:38.794 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000076", 2024-08-18T15:58:38.794 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:58:38.795 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:58:38.795 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000113", 2024-08-18T15:58:38.795 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000005", 2024-08-18T15:58:38.795 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:58:38.795 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000011", 2024-08-18T15:58:38.795 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:58:38.795 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000058", 2024-08-18T15:58:38.795 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:58:38.795 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:58:38.795 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:58:38.795 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000052", 2024-08-18T15:58:38.795 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:58:38.796 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:58:38.796 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:58:38.796 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:58:38.796 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.21", 2024-08-18T15:58:38.796 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:58:38.796 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:58:38.796 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:58:38.796 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:58:38.796 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:58:38.796 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:58:38.796 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:58:38.796 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:58:38.796 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:58:38.797 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:58:38.797 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:58:38.797 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:58:38.797 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:58:38.797 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:58:38.797 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:58:38.797 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:58:38.797 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:58:38.797 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:58:38.797 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:58:38.797 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000088", 2024-08-18T15:58:38.797 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000080", 2024-08-18T15:58:38.797 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:58:38.798 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:58:38.798 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000090", 2024-08-18T15:58:38.798 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:58:38.798 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:58:38.798 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000110", 2024-08-18T15:58:38.798 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:58:38.798 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:58:38.798 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:58:38.798 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.0", 2024-08-18T15:58:38.798 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:58:38.798 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000096", 2024-08-18T15:58:38.798 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:58:38.799 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:58:38.799 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:58:38.799 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:58:38.799 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:58:38.799 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000087", 2024-08-18T15:58:38.799 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:58:38.799 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:58:38.799 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:58:38.799 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:58:38.799 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000008", 2024-08-18T15:58:38.799 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:58:38.799 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:58:38.799 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:58:38.800 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:58:38.800 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:58:38.800 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000006", 2024-08-18T15:58:38.800 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000029", 2024-08-18T15:58:38.800 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:58:38.800 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000089", 2024-08-18T15:58:38.800 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:58:38.800 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:58:38.800 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000022", 2024-08-18T15:58:38.800 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:58:38.800 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:58:38.800 INFO:teuthology.orchestra.run.smithi193.stdout: "bilog.trim", 2024-08-18T15:58:38.800 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000016", 2024-08-18T15:58:38.800 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:58:38.801 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:58:38.801 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:58:38.801 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:58:38.801 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:58:38.801 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000041", 2024-08-18T15:58:38.801 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:58:38.801 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:58:38.801 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.log.01050e30-179b-43be-bdbb-3bb23cf37ae8.3", 2024-08-18T15:58:38.801 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:58:38.801 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000018", 2024-08-18T15:58:38.801 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000092", 2024-08-18T15:58:38.801 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:58:38.801 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:58:38.801 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:58:38.802 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:58:38.802 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.27", 2024-08-18T15:58:38.802 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.4", 2024-08-18T15:58:38.802 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:58:38.802 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:58:38.802 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:58:38.802 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:58:38.802 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:58:38.802 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:58:38.802 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:58:38.802 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000014", 2024-08-18T15:58:38.802 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:58:38.802 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3.retry", 2024-08-18T15:58:38.803 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:58:38.803 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:58:38.803 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:58:38.803 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:58:38.803 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:58:38.803 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000112", 2024-08-18T15:58:38.803 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:58:38.803 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000007", 2024-08-18T15:58:38.803 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000021", 2024-08-18T15:58:38.803 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:58:38.803 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:58:38.803 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:58:38.803 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:58:38.803 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:58:38.804 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.16", 2024-08-18T15:58:38.804 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:58:38.804 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000064", 2024-08-18T15:58:38.804 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:58:38.804 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:58:38.804 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:58:38.804 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:58:38.804 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:58:38.804 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:58:38.804 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:58:38.804 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000071", 2024-08-18T15:58:38.804 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:58:38.804 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:58:38.805 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:58:38.805 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000074", 2024-08-18T15:58:38.805 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:58:38.805 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:58:38.805 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:58:38.805 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:58:38.805 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:58:38.805 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:58:38.805 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:58:38.805 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:58:38.805 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:58:38.805 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:58:38.806 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:58:38.806 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:58:38.806 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:58:38.806 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:58:38.806 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:58:38.806 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000081", 2024-08-18T15:58:38.806 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:58:38.806 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000009", 2024-08-18T15:58:38.806 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000121", 2024-08-18T15:58:38.806 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1", 2024-08-18T15:58:38.806 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:58:38.806 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:58:38.807 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:58:38.807 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:58:38.807 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:58:38.807 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:58:38.807 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.17", 2024-08-18T15:58:38.807 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000125", 2024-08-18T15:58:38.807 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:58:38.807 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000082", 2024-08-18T15:58:38.807 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:58:38.807 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:58:38.807 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:58:38.807 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:58:38.807 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:58:38.807 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:58:38.807 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:58:38.808 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:58:38.808 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:58:38.808 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:58:38.808 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:58:38.808 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000105", 2024-08-18T15:58:38.808 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000059", 2024-08-18T15:58:38.808 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:58:38.808 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:58:38.808 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:58:38.808 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:58:38.808 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:58:38.808 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:58:38.808 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:58:38.808 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:58:38.809 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:58:38.809 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:58:38.809 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:58:38.809 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000077", 2024-08-18T15:58:38.809 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000032", 2024-08-18T15:58:38.809 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:58:38.809 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:58:38.809 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:58:38.809 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:58:38.809 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:58:38.809 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:58:38.809 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.24", 2024-08-18T15:58:38.809 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:58:38.810 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:58:38.810 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:58:38.810 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:58:38.810 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3.retry", 2024-08-18T15:58:38.810 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:58:38.810 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:58:38.810 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:58:38.810 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:58:38.810 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:58:38.810 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:58:38.810 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000053", 2024-08-18T15:58:38.810 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000091", 2024-08-18T15:58:38.810 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:58:38.811 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:58:38.811 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:58:38.811 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000065", 2024-08-18T15:58:38.811 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:58:38.811 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:58:38.811 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:58:38.811 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:58:38.811 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000083", 2024-08-18T15:58:38.811 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000010", 2024-08-18T15:58:38.811 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:58:38.811 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:58:38.811 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:58:38.811 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:58:38.811 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:58:38.812 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:58:38.812 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:58:38.812 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:58:38.812 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:58:38.812 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:58:38.812 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:58:38.812 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.7", 2024-08-18T15:58:38.812 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.3", 2024-08-18T15:58:38.812 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:58:38.812 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:58:38.812 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:58:38.812 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:58:38.812 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:58:38.813 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:58:38.813 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:58:38.813 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:58:38.813 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.2", 2024-08-18T15:58:38.813 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:58:38.813 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:58:38.813 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:58:38.813 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:58:38.813 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:58:38.813 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:58:38.813 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:58:38.813 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000045", 2024-08-18T15:58:38.813 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000002", 2024-08-18T15:58:38.814 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:58:38.814 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3.2UbiLidwSTy.0", 2024-08-18T15:58:38.814 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:58:38.814 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.12", 2024-08-18T15:58:38.814 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:58:38.814 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:58:38.814 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:58:38.814 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:58:38.814 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:58:38.814 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:58:38.814 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000116", 2024-08-18T15:58:38.815 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.25", 2024-08-18T15:58:38.815 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:58:38.815 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:58:38.815 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:58:38.815 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:58:38.815 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000034", 2024-08-18T15:58:38.815 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:58:38.815 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:58:38.815 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000101", 2024-08-18T15:58:38.815 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:58:38.815 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:58:38.816 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.13", 2024-08-18T15:58:38.816 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:58:38.816 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:58:38.816 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.1", 2024-08-18T15:58:38.816 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0.retry", 2024-08-18T15:58:38.816 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:58:38.816 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:58:38.816 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000079", 2024-08-18T15:58:38.817 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000049", 2024-08-18T15:58:38.817 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:58:38.817 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0", 2024-08-18T15:58:38.817 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:58:38.817 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:58:38.817 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000117", 2024-08-18T15:58:38.817 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:58:38.817 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000044", 2024-08-18T15:58:38.817 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:58:38.817 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:58:38.817 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:58:38.817 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:58:38.817 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:58:38.818 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:58:38.818 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.14", 2024-08-18T15:58:38.818 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:58:38.818 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000066", 2024-08-18T15:58:38.818 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:58:38.818 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:58:38.818 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:58:38.818 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:58:38.818 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:58:38.818 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000068", 2024-08-18T15:58:38.818 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:58:38.818 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000085", 2024-08-18T15:58:38.818 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:58:38.819 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:58:38.819 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:58:38.819 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:58:38.819 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:58:38.819 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:58:38.819 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:58:38.819 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:58:38.819 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:58:38.819 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:58:38.819 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:58:38.819 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:58:38.819 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000073", 2024-08-18T15:58:38.819 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:58:38.819 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:58:38.819 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:58:38.820 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000038", 2024-08-18T15:58:38.820 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:58:38.820 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:58:38.820 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:58:38.820 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:58:38.820 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.10", 2024-08-18T15:58:38.820 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:58:38.820 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:58:38.820 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:58:38.820 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000118", 2024-08-18T15:58:38.820 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:58:38.820 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3", 2024-08-18T15:58:38.820 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:58:38.820 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000036", 2024-08-18T15:58:38.821 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:58:38.821 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:58:38.821 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:58:38.821 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:58:38.821 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:58:38.821 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000103", 2024-08-18T15:58:38.821 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.3", 2024-08-18T15:58:38.821 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:58:38.821 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000119", 2024-08-18T15:58:38.821 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000098", 2024-08-18T15:58:38.821 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:58:38.821 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:58:38.821 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:58:38.822 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:58:38.822 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:58:38.822 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:58:38.822 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:58:38.822 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:58:38.822 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.0", 2024-08-18T15:58:38.822 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:58:38.822 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:58:38.823 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:58:38.823 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:58:38.823 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:58:38.823 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:58:38.823 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:58:38.823 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:58:38.823 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000027", 2024-08-18T15:58:38.823 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:58:38.823 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status", 2024-08-18T15:58:38.823 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:58:38.823 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000019", 2024-08-18T15:58:38.823 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:58:38.823 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:58:38.824 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:58:38.824 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:58:38.824 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1", 2024-08-18T15:58:38.824 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:58:38.824 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:58:38.824 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:58:38.824 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:58:38.824 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000039", 2024-08-18T15:58:38.824 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:58:38.824 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:58:38.824 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:58:38.824 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:58:38.824 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:58:38.825 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:58:38.825 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:58:38.825 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:58:38.825 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:58:38.825 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:58:38.825 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:58:38.825 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000100", 2024-08-18T15:58:38.825 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:58:38.825 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:58:38.825 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:58:38.825 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:58:38.826 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:58:38.826 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000093", 2024-08-18T15:58:38.826 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:58:38.826 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000004", 2024-08-18T15:58:38.826 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000063", 2024-08-18T15:58:38.826 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000122", 2024-08-18T15:58:38.826 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:58:38.826 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:58:38.826 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:58:38.826 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000057", 2024-08-18T15:58:38.826 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:58:38.826 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:58:38.826 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:58:38.827 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:58:38.827 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000054", 2024-08-18T15:58:38.827 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:58:38.827 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000114", 2024-08-18T15:58:38.827 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:58:38.827 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:58:38.827 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:58:38.827 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.2", 2024-08-18T15:58:38.827 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:58:38.827 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:58:38.827 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0", 2024-08-18T15:58:38.827 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:58:38.827 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:58:38.827 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:58:38.828 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:58:38.828 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:58:38.828 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:58:38.828 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000012", 2024-08-18T15:58:38.828 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:58:38.828 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:58:38.828 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:58:38.828 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000084", 2024-08-18T15:58:38.828 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000043", 2024-08-18T15:58:38.828 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3", 2024-08-18T15:58:38.828 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:58:38.828 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:58:38.828 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.11", 2024-08-18T15:58:38.829 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:58:38.829 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:58:38.829 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:58:38.829 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:58:38.829 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:58:38.829 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:58:38.829 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:58:38.829 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:58:38.829 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000111", 2024-08-18T15:58:38.829 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:58:38.829 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000048", 2024-08-18T15:58:38.829 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:58:38.829 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000020", 2024-08-18T15:58:38.830 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:58:38.830 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:58:38.830 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:58:38.830 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:58:38.830 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:58:38.830 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000099", 2024-08-18T15:58:38.830 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000056", 2024-08-18T15:58:38.830 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:58:38.830 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:58:38.830 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.19", 2024-08-18T15:58:38.830 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:58:38.830 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.28", 2024-08-18T15:58:38.830 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:58:38.831 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:58:38.831 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:58:38.831 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:58:38.831 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:58:38.831 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000072", 2024-08-18T15:58:38.831 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:58:38.831 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:58:38.831 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000062", 2024-08-18T15:58:38.831 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:58:38.831 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:58:38.831 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:58:38.831 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000115", 2024-08-18T15:58:38.832 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:58:38.832 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.1", 2024-08-18T15:58:38.832 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2", 2024-08-18T15:58:38.832 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000033", 2024-08-18T15:58:38.832 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:58:38.832 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:58:38.832 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:58:38.832 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:58:38.832 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:58:38.832 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:58:38.832 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:58:38.832 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:58:38.832 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:58:38.832 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:58:38.833 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:58:38.833 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:58:38.833 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:58:38.833 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:58:38.833 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:58:38.833 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:58:38.833 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:58:38.833 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:58:38.833 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:58:38.833 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000025", 2024-08-18T15:58:38.833 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:58:38.833 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000123", 2024-08-18T15:58:38.833 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.31", 2024-08-18T15:58:38.834 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:58:38.834 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:58:38.834 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:58:38.834 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:58:38.834 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.18", 2024-08-18T15:58:38.834 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:58:38.834 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:58:38.834 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:58:38.834 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:58:38.834 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:58:38.834 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000108", 2024-08-18T15:58:38.834 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:58:38.834 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:58:38.835 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000094", 2024-08-18T15:58:38.835 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:58:38.835 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:58:38.835 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:58:38.835 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:58:38.835 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:58:38.835 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:58:38.835 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000060", 2024-08-18T15:58:38.835 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:58:38.835 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:58:38.835 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:58:38.835 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000109", 2024-08-18T15:58:38.835 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.15", 2024-08-18T15:58:38.836 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:58:38.836 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:58:38.836 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:58:38.836 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:58:38.836 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:58:38.836 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:58:38.836 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:58:38.836 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:58:38.836 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:58:38.836 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000013", 2024-08-18T15:58:38.836 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:58:38.836 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:58:38.836 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:58:38.837 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:58:38.837 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:58:38.837 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:58:38.837 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:58:38.837 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:58:38.837 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:58:38.837 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:58:38.837 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:58:38.837 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:58:38.837 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:58:38.838 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:58:38.838 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000051", 2024-08-18T15:58:38.838 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:58:38.838 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:58:38.838 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:58:38.838 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:58:38.838 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000106", 2024-08-18T15:58:38.838 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:58:38.838 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:58:38.838 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:58:38.838 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:58:38.838 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:58:38.838 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000017", 2024-08-18T15:58:38.839 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:58:38.839 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:58:38.839 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:58:38.839 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000023", 2024-08-18T15:58:38.839 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:58:38.839 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:58:38.839 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:58:38.839 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:58:38.839 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:58:38.839 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:58:38.839 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:58:38.839 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000107", 2024-08-18T15:58:38.839 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:58:38.839 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:58:38.840 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000046", 2024-08-18T15:58:38.840 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:58:38.840 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:58:38.840 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:58:38.840 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.26", 2024-08-18T15:58:38.840 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:58:38.840 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:58:38.840 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9", 2024-08-18T15:58:38.840 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.22", 2024-08-18T15:58:38.840 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:58:38.840 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:58:38.840 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:58:38.840 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:58:38.841 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:58:38.841 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:58:38.841 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:58:38.841 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2", 2024-08-18T15:58:38.841 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:58:38.841 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:58:38.841 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000067", 2024-08-18T15:58:38.841 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:58:38.841 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:58:38.841 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:58:38.841 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:58:38.841 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:58:38.841 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:58:38.842 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:58:38.842 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:58:38.842 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:58:38.842 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:58:38.842 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:58:38.842 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:58:38.842 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:58:38.842 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:58:38.842 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:58:38.842 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:58:38.842 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:58:38.842 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:58:38.842 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:58:38.843 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.9", 2024-08-18T15:58:38.843 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000086", 2024-08-18T15:58:38.843 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.5", 2024-08-18T15:58:38.843 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:58:38.843 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:58:38.843 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:58:38.843 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:58:38.845 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:58:38.846 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:58:38.846 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2.MFXJ583gCr1.0", 2024-08-18T15:58:38.846 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:58:38.846 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1.GPnO16liyZ+.0", 2024-08-18T15:58:38.846 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:58:38.846 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:58:38.846 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:58:38.846 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:58:38.846 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:58:38.846 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:58:38.846 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:58:38.846 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:58:38.847 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:58:38.847 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.20", 2024-08-18T15:58:38.847 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:58:38.847 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:58:38.847 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:58:38.847 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:58:38.847 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:58:38.847 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:58:38.847 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1.retry", 2024-08-18T15:58:38.847 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:58:38.847 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:58:38.847 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:58:38.847 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0.retry", 2024-08-18T15:58:38.848 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000050", 2024-08-18T15:58:38.848 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:58:38.848 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000102", 2024-08-18T15:58:38.848 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:58:38.848 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:58:38.848 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:58:38.848 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:58:38.848 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:58:38.848 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:58:38.848 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:58:38.848 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:58:38.848 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:58:38.848 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:58:38.849 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:58:38.849 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000124", 2024-08-18T15:58:38.849 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:58:38.849 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:58:38.849 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:58:38.849 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000001", 2024-08-18T15:58:38.849 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:58:38.849 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:58:38.849 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:58:38.849 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:58:38.849 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000126", 2024-08-18T15:58:38.849 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:58:38.849 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0.fZhctfLXnSt.0", 2024-08-18T15:58:38.850 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:58:38.850 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000055", 2024-08-18T15:58:38.850 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:58:38.850 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000061", 2024-08-18T15:58:38.850 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1", 2024-08-18T15:58:38.850 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:58:38.850 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:58:38.850 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:58:38.850 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000097", 2024-08-18T15:58:38.850 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:58:38.850 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:58:38.850 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:58:38.851 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:58:38.851 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:58:38.851 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:58:38.851 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:58:38.851 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:58:38.851 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000042", 2024-08-18T15:58:38.851 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:58:38.851 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:58:38.851 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:58:38.851 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1.retry", 2024-08-18T15:58:38.851 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:58:38.851 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:58:38.851 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:58:38.852 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:58:38.852 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:58:38.852 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:58:38.852 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:58:38.852 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000075", 2024-08-18T15:58:38.852 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:58:38.852 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000120", 2024-08-18T15:58:38.852 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000030", 2024-08-18T15:58:38.852 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:58:38.852 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:58:38.852 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:58:38.852 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:58:38.852 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:58:38.853 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:58:38.853 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9" 2024-08-18T15:58:38.853 INFO:teuthology.orchestra.run.smithi193.stdout:] 2024-08-18T15:58:38.853 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:58:43.855 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:58:43.855 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:58:43.856 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:58:43.909 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:58:43.910 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:58:43.976 INFO:teuthology.orchestra.run.smithi193.stdout:[ 2024-08-18T15:58:43.976 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:58:43.976 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:58:43.976 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.29", 2024-08-18T15:58:43.976 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000078", 2024-08-18T15:58:43.977 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.history", 2024-08-18T15:58:43.977 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:58:43.977 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:58:43.977 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:58:43.977 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:58:43.977 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.23", 2024-08-18T15:58:43.977 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000070", 2024-08-18T15:58:43.977 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000104", 2024-08-18T15:58:43.977 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:58:43.977 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2.retry", 2024-08-18T15:58:43.977 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:58:43.977 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:58:43.978 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:58:43.978 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:58:43.978 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000026", 2024-08-18T15:58:43.978 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:58:43.978 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:58:43.978 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:58:43.978 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:58:43.978 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:58:43.978 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:58:43.978 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.8", 2024-08-18T15:58:43.978 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:58:43.978 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:58:43.978 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:58:43.979 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:58:43.979 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000028", 2024-08-18T15:58:43.979 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:58:43.979 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:58:43.979 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:58:43.979 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:58:43.979 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:58:43.979 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:58:43.979 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:58:43.979 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2.retry", 2024-08-18T15:58:43.979 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:58:43.979 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.30", 2024-08-18T15:58:43.979 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:58:43.980 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:58:43.980 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:58:43.980 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:58:43.980 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000040", 2024-08-18T15:58:43.980 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:58:43.980 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.6", 2024-08-18T15:58:43.980 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:58:43.980 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:58:43.980 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:58:43.980 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:58:43.980 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:58:43.980 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:58:43.980 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000015", 2024-08-18T15:58:43.981 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:58:43.981 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:58:43.981 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000069", 2024-08-18T15:58:43.981 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:58:43.981 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:58:43.981 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:58:43.981 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000095", 2024-08-18T15:58:43.981 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0", 2024-08-18T15:58:43.981 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000003", 2024-08-18T15:58:43.981 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000047", 2024-08-18T15:58:43.981 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:58:43.981 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:58:43.981 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:58:43.982 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:58:43.982 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:58:43.982 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:58:43.982 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:58:43.982 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:58:43.982 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:58:43.982 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000035", 2024-08-18T15:58:43.982 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:58:43.982 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:58:43.982 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000037", 2024-08-18T15:58:43.982 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:58:43.982 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:58:43.982 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:58:43.983 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:58:43.983 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:58:43.983 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:58:43.983 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:58:43.983 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:58:43.983 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:58:43.983 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:58:43.983 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:58:43.983 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:58:43.983 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:58:43.983 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:58:43.984 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000024", 2024-08-18T15:58:43.984 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:58:43.984 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:58:43.984 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:58:43.984 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:58:43.984 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000000", 2024-08-18T15:58:43.984 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:58:43.984 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000031", 2024-08-18T15:58:43.984 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:58:43.984 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:58:43.984 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:58:43.984 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:58:43.984 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:58:43.985 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:58:43.985 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3", 2024-08-18T15:58:43.985 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:58:43.985 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:58:43.985 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:58:43.985 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:58:43.985 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:58:43.985 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:58:43.985 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:58:43.985 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:58:43.985 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:58:43.985 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:58:43.985 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:58:43.986 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:58:43.986 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:58:43.986 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:58:43.986 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:58:43.986 INFO:teuthology.orchestra.run.smithi193.stdout: "data_loggenerations_metadata", 2024-08-18T15:58:43.986 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2", 2024-08-18T15:58:43.986 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:58:43.986 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:58:43.986 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:58:43.986 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:58:43.987 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T15:58:43.987 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:58:43.987 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:58:43.987 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:58:43.987 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:58:43.987 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:58:43.987 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000076", 2024-08-18T15:58:43.987 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:58:43.987 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:58:43.987 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000113", 2024-08-18T15:58:43.987 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000005", 2024-08-18T15:58:43.987 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:58:43.987 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000011", 2024-08-18T15:58:43.988 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:58:43.988 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000058", 2024-08-18T15:58:43.988 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:58:43.988 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:58:43.988 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:58:43.988 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000052", 2024-08-18T15:58:43.988 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:58:43.988 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:58:43.988 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:58:43.988 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:58:43.988 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.21", 2024-08-18T15:58:43.988 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:58:43.988 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:58:43.989 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:58:43.989 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:58:43.989 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:58:43.989 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:58:43.989 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:58:43.989 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:58:43.989 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:58:43.989 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:58:43.989 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:58:43.989 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:58:43.989 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:58:43.989 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:58:43.989 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:58:43.990 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:58:43.990 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:58:43.990 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:58:43.990 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:58:43.990 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000088", 2024-08-18T15:58:43.990 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000080", 2024-08-18T15:58:43.990 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:58:43.990 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:58:43.990 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000090", 2024-08-18T15:58:43.990 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:58:43.990 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:58:43.990 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000110", 2024-08-18T15:58:43.991 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:58:43.991 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:58:43.991 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:58:43.991 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.0", 2024-08-18T15:58:43.991 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:58:43.991 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000096", 2024-08-18T15:58:43.991 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:58:43.991 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:58:43.991 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:58:43.991 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:58:43.991 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:58:43.991 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000087", 2024-08-18T15:58:43.991 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:58:43.992 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:58:43.992 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:58:43.992 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:58:43.992 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000008", 2024-08-18T15:58:43.992 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:58:43.992 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:58:43.992 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:58:43.992 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:58:43.992 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:58:43.992 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000006", 2024-08-18T15:58:43.992 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000029", 2024-08-18T15:58:43.993 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:58:43.993 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000089", 2024-08-18T15:58:43.993 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:58:43.993 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:58:43.993 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000022", 2024-08-18T15:58:43.993 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:58:43.993 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:58:43.993 INFO:teuthology.orchestra.run.smithi193.stdout: "bilog.trim", 2024-08-18T15:58:43.993 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000016", 2024-08-18T15:58:43.993 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:58:43.993 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:58:43.993 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:58:43.993 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:58:43.994 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:58:43.994 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000041", 2024-08-18T15:58:43.994 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:58:43.994 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:58:43.994 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.log.01050e30-179b-43be-bdbb-3bb23cf37ae8.3", 2024-08-18T15:58:43.994 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:58:43.994 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000018", 2024-08-18T15:58:43.994 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000092", 2024-08-18T15:58:43.994 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:58:43.994 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:58:43.994 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:58:43.994 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:58:43.995 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.27", 2024-08-18T15:58:43.995 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.4", 2024-08-18T15:58:43.995 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:58:43.995 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:58:43.995 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:58:43.995 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:58:43.995 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:58:43.995 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:58:43.995 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:58:43.995 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000014", 2024-08-18T15:58:43.995 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:58:43.995 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3.retry", 2024-08-18T15:58:43.995 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:58:43.996 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:58:43.996 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:58:43.996 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:58:43.996 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:58:43.996 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000112", 2024-08-18T15:58:43.996 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:58:43.996 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000007", 2024-08-18T15:58:43.996 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000021", 2024-08-18T15:58:43.996 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:58:43.996 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:58:43.996 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:58:43.996 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:58:43.997 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:58:43.997 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.16", 2024-08-18T15:58:43.997 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:58:43.997 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000064", 2024-08-18T15:58:43.997 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:58:43.997 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:58:43.997 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:58:43.997 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:58:43.997 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:58:43.997 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:58:43.997 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:58:43.997 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000071", 2024-08-18T15:58:43.997 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:58:43.998 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:58:43.998 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:58:43.998 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000074", 2024-08-18T15:58:43.998 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:58:43.998 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:58:43.998 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:58:43.998 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:58:43.998 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:58:43.998 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:58:43.998 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:58:43.998 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:58:43.999 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:58:43.999 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:58:43.999 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:58:43.999 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:58:43.999 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:58:43.999 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:58:43.999 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:58:43.999 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000081", 2024-08-18T15:58:43.999 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:58:43.999 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000009", 2024-08-18T15:58:43.999 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000121", 2024-08-18T15:58:43.999 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1", 2024-08-18T15:58:44.000 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:58:44.000 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:58:44.000 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:58:44.000 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:58:44.000 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:58:44.000 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:58:44.000 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.17", 2024-08-18T15:58:44.000 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000125", 2024-08-18T15:58:44.000 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:58:44.000 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000082", 2024-08-18T15:58:44.000 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:58:44.000 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:58:44.000 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:58:44.001 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:58:44.001 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:58:44.001 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:58:44.001 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:58:44.001 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:58:44.001 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:58:44.001 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:58:44.001 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:58:44.001 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000105", 2024-08-18T15:58:44.001 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000059", 2024-08-18T15:58:44.001 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:58:44.001 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:58:44.001 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:58:44.002 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:58:44.002 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:58:44.002 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:58:44.002 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:58:44.002 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:58:44.002 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:58:44.002 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:58:44.002 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:58:44.002 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000077", 2024-08-18T15:58:44.002 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000032", 2024-08-18T15:58:44.003 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:58:44.003 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:58:44.003 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:58:44.003 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:58:44.003 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:58:44.003 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:58:44.003 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.24", 2024-08-18T15:58:44.003 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:58:44.003 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:58:44.003 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:58:44.003 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:58:44.003 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3.retry", 2024-08-18T15:58:44.004 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:58:44.004 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:58:44.004 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:58:44.004 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:58:44.004 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:58:44.004 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:58:44.004 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000053", 2024-08-18T15:58:44.004 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000091", 2024-08-18T15:58:44.004 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:58:44.004 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:58:44.004 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:58:44.004 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000065", 2024-08-18T15:58:44.004 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:58:44.004 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:58:44.005 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:58:44.005 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:58:44.005 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000083", 2024-08-18T15:58:44.005 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000010", 2024-08-18T15:58:44.005 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:58:44.005 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:58:44.005 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:58:44.005 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:58:44.005 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:58:44.005 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:58:44.005 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:58:44.005 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:58:44.005 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:58:44.005 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:58:44.006 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:58:44.006 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.7", 2024-08-18T15:58:44.006 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.3", 2024-08-18T15:58:44.006 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:58:44.006 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:58:44.006 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:58:44.006 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:58:44.006 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:58:44.006 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:58:44.006 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:58:44.006 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:58:44.006 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.2", 2024-08-18T15:58:44.006 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:58:44.007 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:58:44.007 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:58:44.007 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:58:44.007 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:58:44.007 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:58:44.007 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:58:44.007 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000045", 2024-08-18T15:58:44.007 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000002", 2024-08-18T15:58:44.007 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:58:44.007 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3.2UbiLidwSTy.0", 2024-08-18T15:58:44.007 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:58:44.007 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.12", 2024-08-18T15:58:44.007 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:58:44.008 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:58:44.008 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:58:44.008 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:58:44.008 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:58:44.008 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:58:44.008 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000116", 2024-08-18T15:58:44.008 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.25", 2024-08-18T15:58:44.008 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:58:44.008 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:58:44.008 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:58:44.008 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:58:44.008 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000034", 2024-08-18T15:58:44.008 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:58:44.009 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:58:44.009 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000101", 2024-08-18T15:58:44.009 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:58:44.009 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:58:44.009 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.13", 2024-08-18T15:58:44.009 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:58:44.009 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:58:44.009 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.1", 2024-08-18T15:58:44.009 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0.retry", 2024-08-18T15:58:44.009 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:58:44.009 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:58:44.009 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000079", 2024-08-18T15:58:44.009 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000049", 2024-08-18T15:58:44.010 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:58:44.010 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0", 2024-08-18T15:58:44.010 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:58:44.010 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:58:44.010 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000117", 2024-08-18T15:58:44.010 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:58:44.010 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000044", 2024-08-18T15:58:44.010 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:58:44.010 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:58:44.010 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:58:44.010 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:58:44.010 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:58:44.010 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:58:44.011 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.14", 2024-08-18T15:58:44.011 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:58:44.011 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000066", 2024-08-18T15:58:44.011 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:58:44.011 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:58:44.011 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:58:44.011 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:58:44.011 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:58:44.011 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000068", 2024-08-18T15:58:44.011 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:58:44.011 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000085", 2024-08-18T15:58:44.011 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:58:44.011 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:58:44.012 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:58:44.012 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:58:44.012 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:58:44.012 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:58:44.012 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:58:44.012 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:58:44.012 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:58:44.012 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:58:44.012 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:58:44.012 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:58:44.012 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000073", 2024-08-18T15:58:44.012 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:58:44.012 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:58:44.013 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:58:44.013 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000038", 2024-08-18T15:58:44.013 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:58:44.013 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:58:44.013 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:58:44.013 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:58:44.013 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.10", 2024-08-18T15:58:44.013 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:58:44.013 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:58:44.013 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:58:44.013 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000118", 2024-08-18T15:58:44.013 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:58:44.013 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3", 2024-08-18T15:58:44.013 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:58:44.014 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000036", 2024-08-18T15:58:44.014 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:58:44.014 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:58:44.014 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:58:44.014 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:58:44.014 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:58:44.014 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000103", 2024-08-18T15:58:44.014 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.3", 2024-08-18T15:58:44.014 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:58:44.014 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000119", 2024-08-18T15:58:44.014 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000098", 2024-08-18T15:58:44.014 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:58:44.015 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:58:44.015 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:58:44.015 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:58:44.015 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:58:44.015 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:58:44.015 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:58:44.015 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:58:44.015 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.0", 2024-08-18T15:58:44.015 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:58:44.015 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:58:44.015 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:58:44.015 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:58:44.015 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:58:44.016 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:58:44.016 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:58:44.016 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:58:44.016 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000027", 2024-08-18T15:58:44.016 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:58:44.016 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status", 2024-08-18T15:58:44.016 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:58:44.016 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000019", 2024-08-18T15:58:44.016 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:58:44.016 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:58:44.016 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:58:44.016 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:58:44.016 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1", 2024-08-18T15:58:44.017 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:58:44.017 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:58:44.017 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:58:44.017 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:58:44.017 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000039", 2024-08-18T15:58:44.017 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:58:44.017 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:58:44.017 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:58:44.017 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:58:44.017 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:58:44.017 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:58:44.017 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:58:44.017 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:58:44.018 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:58:44.018 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:58:44.018 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:58:44.018 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000100", 2024-08-18T15:58:44.018 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:58:44.018 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:58:44.018 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:58:44.018 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:58:44.018 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:58:44.018 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000093", 2024-08-18T15:58:44.018 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:58:44.018 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000004", 2024-08-18T15:58:44.018 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000063", 2024-08-18T15:58:44.019 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000122", 2024-08-18T15:58:44.019 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:58:44.019 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:58:44.019 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:58:44.019 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000057", 2024-08-18T15:58:44.019 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:58:44.019 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:58:44.019 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:58:44.019 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:58:44.019 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000054", 2024-08-18T15:58:44.019 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:58:44.019 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000114", 2024-08-18T15:58:44.019 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:58:44.019 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:58:44.020 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:58:44.020 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.2", 2024-08-18T15:58:44.020 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:58:44.020 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:58:44.020 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0", 2024-08-18T15:58:44.020 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:58:44.020 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:58:44.020 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:58:44.020 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:58:44.020 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:58:44.020 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:58:44.020 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000012", 2024-08-18T15:58:44.020 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:58:44.021 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:58:44.021 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:58:44.021 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000084", 2024-08-18T15:58:44.021 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000043", 2024-08-18T15:58:44.021 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3", 2024-08-18T15:58:44.021 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:58:44.021 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:58:44.021 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.11", 2024-08-18T15:58:44.021 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:58:44.021 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:58:44.021 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:58:44.021 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:58:44.022 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:58:44.022 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:58:44.022 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:58:44.022 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:58:44.022 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000111", 2024-08-18T15:58:44.022 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:58:44.022 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000048", 2024-08-18T15:58:44.022 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:58:44.022 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000020", 2024-08-18T15:58:44.022 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:58:44.022 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:58:44.022 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:58:44.022 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:58:44.023 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:58:44.023 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000099", 2024-08-18T15:58:44.023 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000056", 2024-08-18T15:58:44.023 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:58:44.023 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:58:44.023 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.19", 2024-08-18T15:58:44.023 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:58:44.023 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.28", 2024-08-18T15:58:44.023 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:58:44.023 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:58:44.023 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:58:44.023 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:58:44.023 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:58:44.023 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000072", 2024-08-18T15:58:44.024 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:58:44.024 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:58:44.024 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000062", 2024-08-18T15:58:44.024 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:58:44.024 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:58:44.024 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:58:44.024 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000115", 2024-08-18T15:58:44.024 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:58:44.024 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.1", 2024-08-18T15:58:44.024 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2", 2024-08-18T15:58:44.024 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000033", 2024-08-18T15:58:44.024 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:58:44.024 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:58:44.025 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:58:44.025 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:58:44.025 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:58:44.025 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:58:44.025 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:58:44.025 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:58:44.025 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:58:44.025 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:58:44.025 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:58:44.025 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:58:44.025 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:58:44.025 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:58:44.025 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:58:44.025 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:58:44.026 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:58:44.026 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:58:44.026 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:58:44.026 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000025", 2024-08-18T15:58:44.026 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:58:44.026 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000123", 2024-08-18T15:58:44.026 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.31", 2024-08-18T15:58:44.026 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:58:44.027 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:58:44.027 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:58:44.027 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:58:44.027 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.18", 2024-08-18T15:58:44.027 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:58:44.027 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:58:44.027 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:58:44.027 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:58:44.027 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:58:44.027 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000108", 2024-08-18T15:58:44.027 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:58:44.027 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:58:44.027 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000094", 2024-08-18T15:58:44.028 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:58:44.028 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:58:44.028 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:58:44.028 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:58:44.028 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:58:44.028 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:58:44.028 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000060", 2024-08-18T15:58:44.028 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:58:44.028 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:58:44.028 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:58:44.028 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000109", 2024-08-18T15:58:44.028 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.15", 2024-08-18T15:58:44.029 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:58:44.029 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:58:44.029 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:58:44.029 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:58:44.029 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:58:44.029 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:58:44.029 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:58:44.029 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:58:44.029 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:58:44.029 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000013", 2024-08-18T15:58:44.029 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:58:44.029 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:58:44.029 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:58:44.030 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:58:44.030 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:58:44.030 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:58:44.030 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:58:44.030 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:58:44.030 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:58:44.030 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:58:44.030 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:58:44.030 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:58:44.030 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:58:44.030 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:58:44.030 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000051", 2024-08-18T15:58:44.030 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:58:44.031 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:58:44.031 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:58:44.031 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:58:44.031 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000106", 2024-08-18T15:58:44.031 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:58:44.031 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:58:44.031 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:58:44.031 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:58:44.031 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:58:44.031 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000017", 2024-08-18T15:58:44.031 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:58:44.031 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:58:44.031 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:58:44.032 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000023", 2024-08-18T15:58:44.032 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:58:44.032 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:58:44.032 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:58:44.032 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:58:44.032 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:58:44.032 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:58:44.032 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:58:44.032 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000107", 2024-08-18T15:58:44.032 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:58:44.032 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:58:44.032 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000046", 2024-08-18T15:58:44.032 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:58:44.032 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:58:44.033 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:58:44.033 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.26", 2024-08-18T15:58:44.033 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:58:44.033 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:58:44.033 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9", 2024-08-18T15:58:44.033 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.22", 2024-08-18T15:58:44.033 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:58:44.033 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:58:44.033 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:58:44.033 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:58:44.033 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:58:44.033 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:58:44.033 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:58:44.034 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2", 2024-08-18T15:58:44.034 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:58:44.034 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:58:44.034 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000067", 2024-08-18T15:58:44.034 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:58:44.034 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:58:44.034 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:58:44.034 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:58:44.034 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:58:44.034 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:58:44.034 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:58:44.034 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:58:44.035 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:58:44.035 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:58:44.035 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:58:44.035 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:58:44.035 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:58:44.035 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:58:44.035 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:58:44.035 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:58:44.035 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:58:44.035 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:58:44.035 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:58:44.035 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.9", 2024-08-18T15:58:44.035 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000086", 2024-08-18T15:58:44.036 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.5", 2024-08-18T15:58:44.036 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:58:44.036 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:58:44.036 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:58:44.036 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:58:44.036 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:58:44.036 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:58:44.036 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2.MFXJ583gCr1.0", 2024-08-18T15:58:44.036 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:58:44.036 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1.GPnO16liyZ+.0", 2024-08-18T15:58:44.036 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:58:44.036 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:58:44.037 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:58:44.037 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:58:44.037 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:58:44.037 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:58:44.037 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:58:44.037 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:58:44.037 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:58:44.037 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.20", 2024-08-18T15:58:44.037 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:58:44.037 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:58:44.037 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:58:44.037 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:58:44.037 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:58:44.038 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:58:44.038 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1.retry", 2024-08-18T15:58:44.038 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:58:44.038 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:58:44.038 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:58:44.038 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0.retry", 2024-08-18T15:58:44.038 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000050", 2024-08-18T15:58:44.038 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:58:44.038 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000102", 2024-08-18T15:58:44.038 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:58:44.038 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:58:44.038 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:58:44.038 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:58:44.039 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:58:44.039 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:58:44.039 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:58:44.039 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:58:44.039 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:58:44.039 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:58:44.039 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:58:44.039 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000124", 2024-08-18T15:58:44.039 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:58:44.039 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:58:44.039 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:58:44.039 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000001", 2024-08-18T15:58:44.039 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:58:44.040 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:58:44.040 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:58:44.040 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:58:44.040 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000126", 2024-08-18T15:58:44.040 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:58:44.040 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0.fZhctfLXnSt.0", 2024-08-18T15:58:44.040 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:58:44.040 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000055", 2024-08-18T15:58:44.040 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:58:44.040 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000061", 2024-08-18T15:58:44.040 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1", 2024-08-18T15:58:44.040 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:58:44.040 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:58:44.041 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:58:44.041 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000097", 2024-08-18T15:58:44.041 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:58:44.041 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:58:44.041 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:58:44.041 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:58:44.041 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:58:44.041 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:58:44.041 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:58:44.041 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:58:44.041 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000042", 2024-08-18T15:58:44.041 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:58:44.041 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:58:44.042 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:58:44.042 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1.retry", 2024-08-18T15:58:44.042 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:58:44.042 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:58:44.042 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:58:44.042 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:58:44.042 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:58:44.042 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:58:44.042 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:58:44.042 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000075", 2024-08-18T15:58:44.042 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:58:44.042 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000120", 2024-08-18T15:58:44.042 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000030", 2024-08-18T15:58:44.043 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:58:44.043 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:58:44.043 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:58:44.043 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:58:44.043 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:58:44.043 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:58:44.043 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9" 2024-08-18T15:58:44.043 INFO:teuthology.orchestra.run.smithi193.stdout:] 2024-08-18T15:58:44.043 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:58:49.045 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:58:49.045 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:58:49.045 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:58:49.097 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:58:49.097 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:58:49.163 INFO:teuthology.orchestra.run.smithi193.stdout:[ 2024-08-18T15:58:49.163 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:58:49.163 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:58:49.163 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.29", 2024-08-18T15:58:49.163 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000078", 2024-08-18T15:58:49.163 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.history", 2024-08-18T15:58:49.163 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:58:49.164 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:58:49.164 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:58:49.164 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:58:49.164 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.23", 2024-08-18T15:58:49.164 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000070", 2024-08-18T15:58:49.164 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000104", 2024-08-18T15:58:49.164 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:58:49.164 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2.retry", 2024-08-18T15:58:49.164 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:58:49.164 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:58:49.165 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:58:49.165 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:58:49.165 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000026", 2024-08-18T15:58:49.165 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:58:49.165 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:58:49.165 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:58:49.165 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:58:49.165 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:58:49.165 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:58:49.165 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.8", 2024-08-18T15:58:49.166 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:58:49.166 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:58:49.166 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:58:49.166 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:58:49.166 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000028", 2024-08-18T15:58:49.166 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:58:49.166 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:58:49.166 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:58:49.166 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:58:49.166 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:58:49.167 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:58:49.167 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:58:49.167 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2.retry", 2024-08-18T15:58:49.167 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:58:49.167 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.30", 2024-08-18T15:58:49.167 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:58:49.167 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:58:49.167 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:58:49.167 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:58:49.167 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000040", 2024-08-18T15:58:49.168 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:58:49.168 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.6", 2024-08-18T15:58:49.168 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:58:49.168 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:58:49.168 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:58:49.168 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:58:49.168 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:58:49.168 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:58:49.168 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000015", 2024-08-18T15:58:49.168 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:58:49.169 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:58:49.169 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000069", 2024-08-18T15:58:49.169 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:58:49.169 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:58:49.169 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:58:49.169 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000095", 2024-08-18T15:58:49.169 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0", 2024-08-18T15:58:49.169 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000003", 2024-08-18T15:58:49.169 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000047", 2024-08-18T15:58:49.169 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:58:49.169 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:58:49.170 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:58:49.170 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:58:49.170 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:58:49.170 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:58:49.170 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:58:49.170 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:58:49.170 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:58:49.170 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000035", 2024-08-18T15:58:49.170 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:58:49.170 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:58:49.171 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000037", 2024-08-18T15:58:49.171 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:58:49.171 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:58:49.171 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:58:49.171 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:58:49.171 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:58:49.171 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:58:49.171 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:58:49.171 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:58:49.171 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:58:49.172 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:58:49.172 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:58:49.172 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:58:49.172 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:58:49.172 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:58:49.172 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000024", 2024-08-18T15:58:49.172 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:58:49.172 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:58:49.172 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:58:49.172 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:58:49.172 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000000", 2024-08-18T15:58:49.172 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:58:49.173 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000031", 2024-08-18T15:58:49.173 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:58:49.173 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:58:49.173 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:58:49.173 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:58:49.173 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:58:49.173 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:58:49.173 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3", 2024-08-18T15:58:49.173 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:58:49.173 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:58:49.174 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:58:49.174 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:58:49.174 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:58:49.174 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:58:49.174 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:58:49.174 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:58:49.174 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:58:49.174 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:58:49.174 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:58:49.174 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:58:49.175 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:58:49.175 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:58:49.175 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:58:49.175 INFO:teuthology.orchestra.run.smithi193.stdout: "data_loggenerations_metadata", 2024-08-18T15:58:49.175 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2", 2024-08-18T15:58:49.175 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:58:49.175 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:58:49.175 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:58:49.175 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:58:49.175 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T15:58:49.175 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:58:49.175 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:58:49.176 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:58:49.176 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:58:49.176 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:58:49.176 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000076", 2024-08-18T15:58:49.176 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:58:49.176 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:58:49.176 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000113", 2024-08-18T15:58:49.176 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000005", 2024-08-18T15:58:49.176 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:58:49.176 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000011", 2024-08-18T15:58:49.176 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:58:49.177 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000058", 2024-08-18T15:58:49.177 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:58:49.177 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:58:49.177 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:58:49.177 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000052", 2024-08-18T15:58:49.177 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:58:49.177 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:58:49.177 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:58:49.177 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:58:49.177 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.21", 2024-08-18T15:58:49.177 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:58:49.177 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:58:49.178 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:58:49.178 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:58:49.178 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:58:49.178 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:58:49.178 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:58:49.178 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:58:49.178 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:58:49.178 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:58:49.178 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:58:49.178 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:58:49.178 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:58:49.179 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:58:49.179 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:58:49.179 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:58:49.179 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:58:49.179 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:58:49.179 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:58:49.179 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000088", 2024-08-18T15:58:49.179 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000080", 2024-08-18T15:58:49.179 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:58:49.179 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:58:49.180 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000090", 2024-08-18T15:58:49.180 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:58:49.180 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:58:49.180 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000110", 2024-08-18T15:58:49.180 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:58:49.180 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:58:49.180 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:58:49.180 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.0", 2024-08-18T15:58:49.180 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:58:49.180 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000096", 2024-08-18T15:58:49.181 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:58:49.181 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:58:49.181 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:58:49.181 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:58:49.181 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:58:49.181 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000087", 2024-08-18T15:58:49.181 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:58:49.181 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:58:49.181 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:58:49.181 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:58:49.181 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000008", 2024-08-18T15:58:49.182 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:58:49.182 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:58:49.182 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:58:49.182 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:58:49.182 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:58:49.182 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000006", 2024-08-18T15:58:49.182 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000029", 2024-08-18T15:58:49.182 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:58:49.182 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000089", 2024-08-18T15:58:49.182 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:58:49.183 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:58:49.183 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000022", 2024-08-18T15:58:49.183 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:58:49.183 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:58:49.183 INFO:teuthology.orchestra.run.smithi193.stdout: "bilog.trim", 2024-08-18T15:58:49.183 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000016", 2024-08-18T15:58:49.183 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:58:49.183 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:58:49.183 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:58:49.183 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:58:49.183 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:58:49.183 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000041", 2024-08-18T15:58:49.184 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:58:49.184 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:58:49.184 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.log.01050e30-179b-43be-bdbb-3bb23cf37ae8.3", 2024-08-18T15:58:49.184 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:58:49.184 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000018", 2024-08-18T15:58:49.184 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000092", 2024-08-18T15:58:49.184 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:58:49.184 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:58:49.184 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:58:49.184 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:58:49.185 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.27", 2024-08-18T15:58:49.185 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.4", 2024-08-18T15:58:49.185 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:58:49.185 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:58:49.185 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:58:49.185 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:58:49.185 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:58:49.185 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:58:49.185 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:58:49.186 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000014", 2024-08-18T15:58:49.186 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:58:49.186 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3.retry", 2024-08-18T15:58:49.186 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:58:49.186 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:58:49.186 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:58:49.186 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:58:49.186 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:58:49.186 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000112", 2024-08-18T15:58:49.187 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:58:49.187 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000007", 2024-08-18T15:58:49.187 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000021", 2024-08-18T15:58:49.187 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:58:49.187 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:58:49.187 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:58:49.187 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:58:49.187 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:58:49.187 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.16", 2024-08-18T15:58:49.187 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:58:49.187 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000064", 2024-08-18T15:58:49.188 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:58:49.188 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:58:49.188 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:58:49.188 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:58:49.188 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:58:49.188 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:58:49.188 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:58:49.188 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000071", 2024-08-18T15:58:49.188 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:58:49.188 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:58:49.188 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:58:49.189 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000074", 2024-08-18T15:58:49.189 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:58:49.189 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:58:49.189 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:58:49.189 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:58:49.189 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:58:49.189 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:58:49.189 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:58:49.189 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:58:49.189 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:58:49.189 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:58:49.190 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:58:49.190 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:58:49.190 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:58:49.190 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:58:49.190 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:58:49.190 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000081", 2024-08-18T15:58:49.190 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:58:49.190 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000009", 2024-08-18T15:58:49.190 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000121", 2024-08-18T15:58:49.190 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1", 2024-08-18T15:58:49.190 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:58:49.191 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:58:49.191 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:58:49.191 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:58:49.191 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:58:49.191 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:58:49.191 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.17", 2024-08-18T15:58:49.191 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000125", 2024-08-18T15:58:49.191 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:58:49.191 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000082", 2024-08-18T15:58:49.191 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:58:49.192 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:58:49.192 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:58:49.192 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:58:49.192 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:58:49.192 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:58:49.192 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:58:49.192 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:58:49.192 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:58:49.192 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:58:49.192 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:58:49.192 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000105", 2024-08-18T15:58:49.193 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000059", 2024-08-18T15:58:49.193 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:58:49.193 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:58:49.193 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:58:49.193 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:58:49.193 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:58:49.193 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:58:49.193 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:58:49.193 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:58:49.193 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:58:49.193 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:58:49.194 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:58:49.194 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000077", 2024-08-18T15:58:49.194 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000032", 2024-08-18T15:58:49.194 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:58:49.194 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:58:49.194 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:58:49.194 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:58:49.194 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:58:49.194 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:58:49.194 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.24", 2024-08-18T15:58:49.195 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:58:49.195 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:58:49.195 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:58:49.195 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:58:49.195 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3.retry", 2024-08-18T15:58:49.195 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:58:49.195 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:58:49.195 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:58:49.195 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:58:49.195 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:58:49.195 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:58:49.195 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000053", 2024-08-18T15:58:49.196 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000091", 2024-08-18T15:58:49.196 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:58:49.196 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:58:49.196 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:58:49.196 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000065", 2024-08-18T15:58:49.196 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:58:49.196 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:58:49.196 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:58:49.196 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:58:49.196 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000083", 2024-08-18T15:58:49.196 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000010", 2024-08-18T15:58:49.196 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:58:49.196 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:58:49.197 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:58:49.197 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:58:49.197 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:58:49.197 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:58:49.197 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:58:49.197 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:58:49.197 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:58:49.197 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:58:49.197 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:58:49.197 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.7", 2024-08-18T15:58:49.197 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.3", 2024-08-18T15:58:49.198 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:58:49.198 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:58:49.198 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:58:49.198 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:58:49.198 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:58:49.198 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:58:49.198 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:58:49.198 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:58:49.198 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.2", 2024-08-18T15:58:49.198 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:58:49.198 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:58:49.199 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:58:49.199 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:58:49.199 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:58:49.199 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:58:49.199 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:58:49.199 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000045", 2024-08-18T15:58:49.199 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000002", 2024-08-18T15:58:49.199 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:58:49.199 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3.2UbiLidwSTy.0", 2024-08-18T15:58:49.199 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:58:49.199 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.12", 2024-08-18T15:58:49.200 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:58:49.200 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:58:49.200 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:58:49.200 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:58:49.200 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:58:49.200 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:58:49.200 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000116", 2024-08-18T15:58:49.200 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.25", 2024-08-18T15:58:49.200 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:58:49.200 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:58:49.200 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:58:49.200 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:58:49.201 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000034", 2024-08-18T15:58:49.201 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:58:49.201 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:58:49.201 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000101", 2024-08-18T15:58:49.201 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:58:49.201 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:58:49.201 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.13", 2024-08-18T15:58:49.201 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:58:49.201 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:58:49.201 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.1", 2024-08-18T15:58:49.201 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0.retry", 2024-08-18T15:58:49.201 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:58:49.201 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:58:49.202 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000079", 2024-08-18T15:58:49.202 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000049", 2024-08-18T15:58:49.202 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:58:49.202 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0", 2024-08-18T15:58:49.202 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:58:49.202 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:58:49.202 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000117", 2024-08-18T15:58:49.202 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:58:49.202 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000044", 2024-08-18T15:58:49.202 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:58:49.202 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:58:49.203 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:58:49.203 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:58:49.203 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:58:49.203 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:58:49.203 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.14", 2024-08-18T15:58:49.203 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:58:49.203 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000066", 2024-08-18T15:58:49.203 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:58:49.203 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:58:49.203 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:58:49.203 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:58:49.203 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:58:49.203 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000068", 2024-08-18T15:58:49.204 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:58:49.204 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000085", 2024-08-18T15:58:49.204 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:58:49.204 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:58:49.204 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:58:49.204 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:58:49.204 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:58:49.204 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:58:49.204 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:58:49.204 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:58:49.204 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:58:49.204 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:58:49.205 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:58:49.205 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:58:49.205 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000073", 2024-08-18T15:58:49.205 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:58:49.205 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:58:49.205 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:58:49.205 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000038", 2024-08-18T15:58:49.205 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:58:49.205 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:58:49.205 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:58:49.205 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:58:49.205 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.10", 2024-08-18T15:58:49.205 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:58:49.206 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:58:49.206 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:58:49.206 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000118", 2024-08-18T15:58:49.206 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:58:49.206 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3", 2024-08-18T15:58:49.206 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:58:49.206 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000036", 2024-08-18T15:58:49.206 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:58:49.206 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:58:49.206 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:58:49.206 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:58:49.207 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:58:49.207 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000103", 2024-08-18T15:58:49.207 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.3", 2024-08-18T15:58:49.207 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:58:49.207 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000119", 2024-08-18T15:58:49.207 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000098", 2024-08-18T15:58:49.207 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:58:49.207 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:58:49.207 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:58:49.207 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:58:49.208 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:58:49.208 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:58:49.208 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:58:49.208 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:58:49.208 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.0", 2024-08-18T15:58:49.208 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:58:49.208 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:58:49.208 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:58:49.208 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:58:49.208 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:58:49.208 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:58:49.209 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:58:49.209 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:58:49.209 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000027", 2024-08-18T15:58:49.209 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:58:49.209 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status", 2024-08-18T15:58:49.209 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:58:49.209 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000019", 2024-08-18T15:58:49.209 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:58:49.209 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:58:49.209 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:58:49.209 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:58:49.209 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1", 2024-08-18T15:58:49.210 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:58:49.210 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:58:49.210 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:58:49.210 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:58:49.210 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000039", 2024-08-18T15:58:49.210 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:58:49.210 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:58:49.210 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:58:49.210 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:58:49.210 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:58:49.210 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:58:49.211 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:58:49.211 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:58:49.211 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:58:49.211 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:58:49.211 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:58:49.211 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000100", 2024-08-18T15:58:49.211 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:58:49.211 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:58:49.211 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:58:49.211 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:58:49.211 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:58:49.212 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000093", 2024-08-18T15:58:49.212 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:58:49.212 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000004", 2024-08-18T15:58:49.212 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000063", 2024-08-18T15:58:49.212 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000122", 2024-08-18T15:58:49.212 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:58:49.212 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:58:49.212 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:58:49.212 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000057", 2024-08-18T15:58:49.212 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:58:49.212 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:58:49.212 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:58:49.213 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:58:49.213 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000054", 2024-08-18T15:58:49.213 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:58:49.213 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000114", 2024-08-18T15:58:49.213 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:58:49.213 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:58:49.213 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:58:49.213 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.2", 2024-08-18T15:58:49.213 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:58:49.213 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:58:49.213 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0", 2024-08-18T15:58:49.214 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:58:49.214 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:58:49.214 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:58:49.214 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:58:49.214 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:58:49.214 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:58:49.214 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000012", 2024-08-18T15:58:49.214 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:58:49.214 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:58:49.214 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:58:49.214 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000084", 2024-08-18T15:58:49.215 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000043", 2024-08-18T15:58:49.215 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3", 2024-08-18T15:58:49.215 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:58:49.215 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:58:49.215 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.11", 2024-08-18T15:58:49.215 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:58:49.215 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:58:49.215 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:58:49.215 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:58:49.215 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:58:49.215 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:58:49.215 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:58:49.215 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:58:49.216 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000111", 2024-08-18T15:58:49.216 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:58:49.216 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000048", 2024-08-18T15:58:49.216 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:58:49.216 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000020", 2024-08-18T15:58:49.216 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:58:49.216 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:58:49.216 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:58:49.216 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:58:49.217 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:58:49.217 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000099", 2024-08-18T15:58:49.217 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000056", 2024-08-18T15:58:49.217 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:58:49.217 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:58:49.217 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.19", 2024-08-18T15:58:49.217 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:58:49.217 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.28", 2024-08-18T15:58:49.217 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:58:49.217 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:58:49.217 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:58:49.217 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:58:49.218 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:58:49.218 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000072", 2024-08-18T15:58:49.218 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:58:49.218 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:58:49.218 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000062", 2024-08-18T15:58:49.218 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:58:49.218 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:58:49.218 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:58:49.218 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000115", 2024-08-18T15:58:49.219 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:58:49.219 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.1", 2024-08-18T15:58:49.219 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2", 2024-08-18T15:58:49.219 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000033", 2024-08-18T15:58:49.219 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:58:49.219 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:58:49.219 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:58:49.219 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:58:49.219 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:58:49.219 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:58:49.219 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:58:49.219 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:58:49.220 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:58:49.220 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:58:49.220 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:58:49.220 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:58:49.220 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:58:49.220 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:58:49.220 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:58:49.220 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:58:49.220 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:58:49.220 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:58:49.220 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:58:49.221 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000025", 2024-08-18T15:58:49.221 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:58:49.221 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000123", 2024-08-18T15:58:49.221 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.31", 2024-08-18T15:58:49.221 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:58:49.221 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:58:49.221 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:58:49.221 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:58:49.221 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.18", 2024-08-18T15:58:49.221 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:58:49.221 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:58:49.221 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:58:49.221 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:58:49.222 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:58:49.222 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000108", 2024-08-18T15:58:49.222 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:58:49.222 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:58:49.222 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000094", 2024-08-18T15:58:49.222 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:58:49.222 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:58:49.222 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:58:49.222 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:58:49.222 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:58:49.222 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:58:49.223 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000060", 2024-08-18T15:58:49.223 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:58:49.223 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:58:49.223 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:58:49.223 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000109", 2024-08-18T15:58:49.223 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.15", 2024-08-18T15:58:49.223 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:58:49.223 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:58:49.223 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:58:49.223 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:58:49.223 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:58:49.223 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:58:49.224 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:58:49.224 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:58:49.224 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:58:49.224 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000013", 2024-08-18T15:58:49.224 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:58:49.224 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:58:49.224 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:58:49.224 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:58:49.224 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:58:49.224 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:58:49.224 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:58:49.225 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:58:49.225 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:58:49.225 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:58:49.225 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:58:49.225 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:58:49.225 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:58:49.225 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:58:49.226 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000051", 2024-08-18T15:58:49.226 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:58:49.226 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:58:49.226 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:58:49.226 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:58:49.226 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000106", 2024-08-18T15:58:49.226 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:58:49.226 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:58:49.226 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:58:49.226 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:58:49.227 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:58:49.227 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000017", 2024-08-18T15:58:49.227 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:58:49.227 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:58:49.227 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:58:49.227 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000023", 2024-08-18T15:58:49.227 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:58:49.227 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:58:49.227 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:58:49.227 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:58:49.227 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:58:49.228 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:58:49.228 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:58:49.228 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000107", 2024-08-18T15:58:49.228 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:58:49.228 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:58:49.228 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000046", 2024-08-18T15:58:49.228 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:58:49.228 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:58:49.228 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:58:49.228 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.26", 2024-08-18T15:58:49.228 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:58:49.229 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:58:49.229 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9", 2024-08-18T15:58:49.229 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.22", 2024-08-18T15:58:49.229 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:58:49.229 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:58:49.229 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:58:49.229 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:58:49.229 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:58:49.229 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:58:49.229 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:58:49.230 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2", 2024-08-18T15:58:49.230 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:58:49.230 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:58:49.230 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000067", 2024-08-18T15:58:49.230 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:58:49.230 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:58:49.230 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:58:49.230 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:58:49.230 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:58:49.230 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:58:49.230 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:58:49.231 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:58:49.231 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:58:49.231 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:58:49.231 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:58:49.231 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:58:49.231 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:58:49.231 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:58:49.231 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:58:49.231 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:58:49.231 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:58:49.232 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:58:49.232 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:58:49.232 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.9", 2024-08-18T15:58:49.232 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000086", 2024-08-18T15:58:49.232 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.5", 2024-08-18T15:58:49.232 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:58:49.232 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:58:49.232 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:58:49.232 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:58:49.232 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:58:49.233 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:58:49.233 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2.MFXJ583gCr1.0", 2024-08-18T15:58:49.233 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:58:49.233 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1.GPnO16liyZ+.0", 2024-08-18T15:58:49.233 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:58:49.233 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:58:49.233 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:58:49.233 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:58:49.233 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:58:49.233 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:58:49.233 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:58:49.234 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:58:49.234 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:58:49.234 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.20", 2024-08-18T15:58:49.234 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:58:49.234 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:58:49.234 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:58:49.234 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:58:49.234 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:58:49.234 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:58:49.234 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1.retry", 2024-08-18T15:58:49.235 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:58:49.235 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:58:49.235 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:58:49.235 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0.retry", 2024-08-18T15:58:49.235 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000050", 2024-08-18T15:58:49.235 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:58:49.235 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000102", 2024-08-18T15:58:49.235 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:58:49.235 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:58:49.235 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:58:49.235 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:58:49.236 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:58:49.236 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:58:49.236 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:58:49.236 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:58:49.236 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:58:49.236 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:58:49.236 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:58:49.236 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000124", 2024-08-18T15:58:49.236 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:58:49.236 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:58:49.236 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:58:49.237 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000001", 2024-08-18T15:58:49.237 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:58:49.237 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:58:49.237 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:58:49.237 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:58:49.237 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000126", 2024-08-18T15:58:49.237 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:58:49.237 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0.fZhctfLXnSt.0", 2024-08-18T15:58:49.237 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:58:49.237 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000055", 2024-08-18T15:58:49.237 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:58:49.237 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000061", 2024-08-18T15:58:49.238 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1", 2024-08-18T15:58:49.238 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:58:49.238 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:58:49.238 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:58:49.238 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000097", 2024-08-18T15:58:49.238 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:58:49.238 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:58:49.238 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:58:49.238 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:58:49.238 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:58:49.239 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:58:49.239 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:58:49.239 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:58:49.239 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000042", 2024-08-18T15:58:49.239 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:58:49.239 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:58:49.239 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:58:49.239 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1.retry", 2024-08-18T15:58:49.239 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:58:49.239 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:58:49.240 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:58:49.240 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:58:49.240 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:58:49.240 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:58:49.240 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:58:49.240 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000075", 2024-08-18T15:58:49.240 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:58:49.240 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000120", 2024-08-18T15:58:49.240 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000030", 2024-08-18T15:58:49.240 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:58:49.240 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:58:49.241 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:58:49.241 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:58:49.241 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:58:49.241 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:58:49.241 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9" 2024-08-18T15:58:49.241 INFO:teuthology.orchestra.run.smithi193.stdout:] 2024-08-18T15:58:49.241 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:58:54.243 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:58:54.243 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:58:54.244 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:58:54.296 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:58:54.296 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:58:54.359 INFO:teuthology.orchestra.run.smithi193.stdout:[ 2024-08-18T15:58:54.359 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:58:54.359 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:58:54.360 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.29", 2024-08-18T15:58:54.360 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000078", 2024-08-18T15:58:54.360 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.history", 2024-08-18T15:58:54.360 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:58:54.360 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:58:54.360 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:58:54.360 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:58:54.360 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.23", 2024-08-18T15:58:54.360 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000070", 2024-08-18T15:58:54.360 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000104", 2024-08-18T15:58:54.360 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:58:54.360 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2.retry", 2024-08-18T15:58:54.361 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:58:54.361 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:58:54.361 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:58:54.361 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:58:54.361 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000026", 2024-08-18T15:58:54.361 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:58:54.361 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:58:54.361 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:58:54.361 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:58:54.361 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:58:54.361 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:58:54.361 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.8", 2024-08-18T15:58:54.361 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:58:54.362 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:58:54.362 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:58:54.362 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:58:54.362 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000028", 2024-08-18T15:58:54.362 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:58:54.362 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:58:54.362 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:58:54.362 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:58:54.362 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:58:54.362 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:58:54.363 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:58:54.363 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2.retry", 2024-08-18T15:58:54.363 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:58:54.363 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.30", 2024-08-18T15:58:54.363 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:58:54.363 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:58:54.363 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:58:54.363 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:58:54.363 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000040", 2024-08-18T15:58:54.363 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:58:54.364 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.6", 2024-08-18T15:58:54.364 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:58:54.364 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:58:54.364 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:58:54.364 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:58:54.364 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:58:54.364 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:58:54.364 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000015", 2024-08-18T15:58:54.364 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:58:54.365 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:58:54.365 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000069", 2024-08-18T15:58:54.365 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:58:54.365 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:58:54.365 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:58:54.365 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000095", 2024-08-18T15:58:54.365 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0", 2024-08-18T15:58:54.365 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000003", 2024-08-18T15:58:54.366 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000047", 2024-08-18T15:58:54.366 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:58:54.366 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:58:54.366 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:58:54.366 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:58:54.366 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:58:54.366 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:58:54.366 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:58:54.366 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:58:54.366 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:58:54.366 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000035", 2024-08-18T15:58:54.366 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:58:54.367 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:58:54.367 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000037", 2024-08-18T15:58:54.367 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:58:54.367 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:58:54.367 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:58:54.367 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:58:54.367 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:58:54.367 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:58:54.367 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:58:54.367 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:58:54.367 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:58:54.367 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:58:54.368 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:58:54.368 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:58:54.368 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:58:54.368 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:58:54.368 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000024", 2024-08-18T15:58:54.368 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:58:54.368 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:58:54.368 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:58:54.368 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:58:54.368 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000000", 2024-08-18T15:58:54.368 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:58:54.368 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000031", 2024-08-18T15:58:54.368 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:58:54.369 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:58:54.369 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:58:54.369 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:58:54.369 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:58:54.369 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:58:54.369 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3", 2024-08-18T15:58:54.369 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:58:54.369 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:58:54.369 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:58:54.369 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:58:54.369 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:58:54.369 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:58:54.370 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:58:54.370 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:58:54.370 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:58:54.370 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:58:54.370 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:58:54.370 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:58:54.370 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:58:54.370 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:58:54.370 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:58:54.370 INFO:teuthology.orchestra.run.smithi193.stdout: "data_loggenerations_metadata", 2024-08-18T15:58:54.370 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2", 2024-08-18T15:58:54.370 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:58:54.370 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:58:54.371 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:58:54.371 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:58:54.371 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T15:58:54.371 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:58:54.371 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:58:54.371 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:58:54.371 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:58:54.371 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:58:54.371 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000076", 2024-08-18T15:58:54.371 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:58:54.371 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:58:54.371 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000113", 2024-08-18T15:58:54.371 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000005", 2024-08-18T15:58:54.372 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:58:54.372 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000011", 2024-08-18T15:58:54.372 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:58:54.372 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000058", 2024-08-18T15:58:54.372 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:58:54.372 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:58:54.372 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:58:54.372 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000052", 2024-08-18T15:58:54.372 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:58:54.372 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:58:54.372 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:58:54.372 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:58:54.372 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.21", 2024-08-18T15:58:54.372 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:58:54.373 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:58:54.373 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:58:54.373 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:58:54.373 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:58:54.373 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:58:54.373 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:58:54.373 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:58:54.373 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:58:54.373 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:58:54.373 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:58:54.373 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:58:54.373 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:58:54.373 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:58:54.374 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:58:54.374 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:58:54.374 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:58:54.374 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:58:54.374 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:58:54.374 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000088", 2024-08-18T15:58:54.374 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000080", 2024-08-18T15:58:54.374 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:58:54.374 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:58:54.374 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000090", 2024-08-18T15:58:54.374 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:58:54.374 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:58:54.374 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000110", 2024-08-18T15:58:54.375 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:58:54.375 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:58:54.375 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:58:54.375 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.0", 2024-08-18T15:58:54.375 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:58:54.375 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000096", 2024-08-18T15:58:54.375 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:58:54.375 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:58:54.375 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:58:54.375 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:58:54.375 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:58:54.375 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000087", 2024-08-18T15:58:54.375 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:58:54.375 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:58:54.376 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:58:54.376 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:58:54.376 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000008", 2024-08-18T15:58:54.376 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:58:54.376 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:58:54.376 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:58:54.376 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:58:54.376 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:58:54.376 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000006", 2024-08-18T15:58:54.376 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000029", 2024-08-18T15:58:54.376 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:58:54.376 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000089", 2024-08-18T15:58:54.376 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:58:54.377 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:58:54.377 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000022", 2024-08-18T15:58:54.377 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:58:54.377 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:58:54.377 INFO:teuthology.orchestra.run.smithi193.stdout: "bilog.trim", 2024-08-18T15:58:54.377 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000016", 2024-08-18T15:58:54.377 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:58:54.377 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:58:54.377 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:58:54.377 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:58:54.377 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:58:54.377 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000041", 2024-08-18T15:58:54.377 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:58:54.377 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:58:54.378 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.log.01050e30-179b-43be-bdbb-3bb23cf37ae8.3", 2024-08-18T15:58:54.378 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:58:54.378 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000018", 2024-08-18T15:58:54.378 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000092", 2024-08-18T15:58:54.378 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:58:54.378 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:58:54.378 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:58:54.378 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:58:54.378 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.27", 2024-08-18T15:58:54.378 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.4", 2024-08-18T15:58:54.378 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:58:54.379 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:58:54.379 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:58:54.379 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:58:54.379 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:58:54.379 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:58:54.379 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:58:54.379 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000014", 2024-08-18T15:58:54.379 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:58:54.379 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3.retry", 2024-08-18T15:58:54.380 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:58:54.380 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:58:54.380 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:58:54.380 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:58:54.380 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:58:54.380 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000112", 2024-08-18T15:58:54.380 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:58:54.380 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000007", 2024-08-18T15:58:54.380 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000021", 2024-08-18T15:58:54.380 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:58:54.380 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:58:54.380 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:58:54.381 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:58:54.381 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:58:54.381 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.16", 2024-08-18T15:58:54.381 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:58:54.381 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000064", 2024-08-18T15:58:54.381 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:58:54.381 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:58:54.381 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:58:54.381 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:58:54.381 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:58:54.381 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:58:54.381 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:58:54.381 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000071", 2024-08-18T15:58:54.381 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:58:54.382 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:58:54.382 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:58:54.382 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000074", 2024-08-18T15:58:54.382 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:58:54.382 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:58:54.382 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:58:54.382 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:58:54.382 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:58:54.382 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:58:54.382 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:58:54.382 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:58:54.382 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:58:54.382 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:58:54.383 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:58:54.383 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:58:54.383 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:58:54.383 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:58:54.383 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:58:54.383 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000081", 2024-08-18T15:58:54.383 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:58:54.383 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000009", 2024-08-18T15:58:54.383 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000121", 2024-08-18T15:58:54.383 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1", 2024-08-18T15:58:54.383 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:58:54.383 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:58:54.383 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:58:54.384 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:58:54.384 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:58:54.384 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:58:54.384 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.17", 2024-08-18T15:58:54.384 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000125", 2024-08-18T15:58:54.384 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:58:54.384 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000082", 2024-08-18T15:58:54.384 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:58:54.384 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:58:54.384 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:58:54.384 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:58:54.384 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:58:54.384 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:58:54.384 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:58:54.385 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:58:54.385 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:58:54.385 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:58:54.385 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:58:54.385 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000105", 2024-08-18T15:58:54.385 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000059", 2024-08-18T15:58:54.385 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:58:54.385 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:58:54.385 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:58:54.385 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:58:54.385 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:58:54.385 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:58:54.385 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:58:54.386 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:58:54.386 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:58:54.386 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:58:54.386 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:58:54.386 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000077", 2024-08-18T15:58:54.386 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000032", 2024-08-18T15:58:54.386 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:58:54.386 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:58:54.386 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:58:54.386 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:58:54.386 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:58:54.386 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:58:54.386 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.24", 2024-08-18T15:58:54.387 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:58:54.387 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:58:54.387 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:58:54.387 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:58:54.387 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3.retry", 2024-08-18T15:58:54.387 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:58:54.387 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:58:54.387 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:58:54.387 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:58:54.387 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:58:54.387 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:58:54.387 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000053", 2024-08-18T15:58:54.387 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000091", 2024-08-18T15:58:54.388 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:58:54.388 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:58:54.388 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:58:54.388 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000065", 2024-08-18T15:58:54.388 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:58:54.388 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:58:54.388 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:58:54.388 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:58:54.388 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000083", 2024-08-18T15:58:54.388 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000010", 2024-08-18T15:58:54.388 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:58:54.388 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:58:54.388 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:58:54.388 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:58:54.389 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:58:54.389 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:58:54.389 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:58:54.389 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:58:54.389 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:58:54.389 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:58:54.389 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:58:54.389 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.7", 2024-08-18T15:58:54.389 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.3", 2024-08-18T15:58:54.389 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:58:54.389 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:58:54.389 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:58:54.389 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:58:54.389 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:58:54.390 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:58:54.390 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:58:54.390 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:58:54.390 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.2", 2024-08-18T15:58:54.390 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:58:54.390 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:58:54.390 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:58:54.390 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:58:54.390 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:58:54.390 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:58:54.390 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:58:54.390 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000045", 2024-08-18T15:58:54.390 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000002", 2024-08-18T15:58:54.391 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:58:54.391 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3.2UbiLidwSTy.0", 2024-08-18T15:58:54.391 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:58:54.391 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.12", 2024-08-18T15:58:54.391 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:58:54.391 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:58:54.391 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:58:54.391 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:58:54.391 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:58:54.391 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:58:54.391 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000116", 2024-08-18T15:58:54.391 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.25", 2024-08-18T15:58:54.392 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:58:54.392 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:58:54.392 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:58:54.392 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:58:54.392 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000034", 2024-08-18T15:58:54.392 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:58:54.392 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:58:54.392 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000101", 2024-08-18T15:58:54.392 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:58:54.392 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:58:54.392 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.13", 2024-08-18T15:58:54.392 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:58:54.392 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:58:54.393 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.1", 2024-08-18T15:58:54.393 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0.retry", 2024-08-18T15:58:54.393 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:58:54.393 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:58:54.393 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000079", 2024-08-18T15:58:54.393 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000049", 2024-08-18T15:58:54.393 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:58:54.393 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0", 2024-08-18T15:58:54.393 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:58:54.393 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:58:54.393 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000117", 2024-08-18T15:58:54.393 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:58:54.393 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000044", 2024-08-18T15:58:54.394 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:58:54.394 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:58:54.394 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:58:54.394 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:58:54.394 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:58:54.394 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:58:54.394 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.14", 2024-08-18T15:58:54.394 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:58:54.394 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000066", 2024-08-18T15:58:54.394 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:58:54.394 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:58:54.394 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:58:54.395 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:58:54.395 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:58:54.395 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000068", 2024-08-18T15:58:54.395 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:58:54.395 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000085", 2024-08-18T15:58:54.395 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:58:54.395 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:58:54.395 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:58:54.395 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:58:54.395 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:58:54.395 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:58:54.395 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:58:54.396 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:58:54.396 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:58:54.396 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:58:54.396 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:58:54.396 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:58:54.396 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000073", 2024-08-18T15:58:54.396 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:58:54.396 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:58:54.396 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:58:54.396 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000038", 2024-08-18T15:58:54.396 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:58:54.396 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:58:54.397 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:58:54.397 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:58:54.397 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.10", 2024-08-18T15:58:54.397 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:58:54.397 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:58:54.397 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:58:54.397 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000118", 2024-08-18T15:58:54.397 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:58:54.397 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3", 2024-08-18T15:58:54.397 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:58:54.397 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000036", 2024-08-18T15:58:54.397 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:58:54.397 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:58:54.397 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:58:54.398 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:58:54.398 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:58:54.398 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000103", 2024-08-18T15:58:54.398 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.3", 2024-08-18T15:58:54.398 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:58:54.398 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000119", 2024-08-18T15:58:54.398 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000098", 2024-08-18T15:58:54.398 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:58:54.398 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:58:54.398 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:58:54.398 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:58:54.398 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:58:54.398 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:58:54.399 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:58:54.399 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:58:54.399 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.0", 2024-08-18T15:58:54.399 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:58:54.399 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:58:54.399 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:58:54.399 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:58:54.399 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:58:54.399 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:58:54.399 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:58:54.399 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:58:54.399 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000027", 2024-08-18T15:58:54.399 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:58:54.399 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status", 2024-08-18T15:58:54.399 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:58:54.400 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000019", 2024-08-18T15:58:54.400 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:58:54.400 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:58:54.400 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:58:54.400 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:58:54.400 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1", 2024-08-18T15:58:54.400 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:58:54.400 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:58:54.400 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:58:54.400 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:58:54.400 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000039", 2024-08-18T15:58:54.400 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:58:54.400 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:58:54.400 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:58:54.400 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:58:54.401 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:58:54.401 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:58:54.401 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:58:54.401 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:58:54.401 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:58:54.401 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:58:54.401 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:58:54.401 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000100", 2024-08-18T15:58:54.401 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:58:54.401 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:58:54.401 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:58:54.401 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:58:54.401 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:58:54.401 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000093", 2024-08-18T15:58:54.402 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:58:54.402 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000004", 2024-08-18T15:58:54.402 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000063", 2024-08-18T15:58:54.402 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000122", 2024-08-18T15:58:54.402 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:58:54.402 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:58:54.402 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:58:54.402 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000057", 2024-08-18T15:58:54.402 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:58:54.402 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:58:54.402 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:58:54.402 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:58:54.402 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000054", 2024-08-18T15:58:54.403 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:58:54.403 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000114", 2024-08-18T15:58:54.403 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:58:54.403 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:58:54.403 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:58:54.403 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.2", 2024-08-18T15:58:54.403 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:58:54.403 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:58:54.403 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0", 2024-08-18T15:58:54.403 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:58:54.403 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:58:54.403 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:58:54.403 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:58:54.404 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:58:54.404 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:58:54.404 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000012", 2024-08-18T15:58:54.404 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:58:54.404 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:58:54.404 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:58:54.404 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000084", 2024-08-18T15:58:54.404 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000043", 2024-08-18T15:58:54.404 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3", 2024-08-18T15:58:54.404 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:58:54.404 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:58:54.404 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.11", 2024-08-18T15:58:54.404 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:58:54.405 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:58:54.405 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:58:54.405 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:58:54.405 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:58:54.405 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:58:54.405 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:58:54.405 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:58:54.405 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000111", 2024-08-18T15:58:54.405 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:58:54.405 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000048", 2024-08-18T15:58:54.405 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:58:54.405 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000020", 2024-08-18T15:58:54.405 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:58:54.406 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:58:54.406 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:58:54.406 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:58:54.406 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:58:54.406 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000099", 2024-08-18T15:58:54.406 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000056", 2024-08-18T15:58:54.406 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:58:54.406 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:58:54.406 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.19", 2024-08-18T15:58:54.406 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:58:54.406 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.28", 2024-08-18T15:58:54.406 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:58:54.407 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:58:54.407 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:58:54.407 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:58:54.407 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:58:54.407 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000072", 2024-08-18T15:58:54.407 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:58:54.407 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:58:54.407 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000062", 2024-08-18T15:58:54.407 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:58:54.407 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:58:54.407 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:58:54.407 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000115", 2024-08-18T15:58:54.407 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:58:54.407 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.1", 2024-08-18T15:58:54.408 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2", 2024-08-18T15:58:54.408 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000033", 2024-08-18T15:58:54.408 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:58:54.408 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:58:54.408 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:58:54.408 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:58:54.408 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:58:54.408 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:58:54.408 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:58:54.408 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:58:54.408 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:58:54.408 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:58:54.408 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:58:54.409 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:58:54.409 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:58:54.409 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:58:54.409 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:58:54.409 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:58:54.409 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:58:54.409 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:58:54.409 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:58:54.409 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000025", 2024-08-18T15:58:54.409 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:58:54.409 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000123", 2024-08-18T15:58:54.409 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.31", 2024-08-18T15:58:54.409 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:58:54.409 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:58:54.410 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:58:54.410 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:58:54.410 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.18", 2024-08-18T15:58:54.410 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:58:54.410 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:58:54.410 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:58:54.410 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:58:54.410 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:58:54.410 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000108", 2024-08-18T15:58:54.410 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:58:54.410 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:58:54.410 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000094", 2024-08-18T15:58:54.410 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:58:54.411 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:58:54.411 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:58:54.411 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:58:54.411 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:58:54.411 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:58:54.411 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000060", 2024-08-18T15:58:54.411 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:58:54.411 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:58:54.411 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:58:54.411 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000109", 2024-08-18T15:58:54.411 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.15", 2024-08-18T15:58:54.411 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:58:54.411 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:58:54.411 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:58:54.412 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:58:54.412 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:58:54.412 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:58:54.412 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:58:54.412 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:58:54.412 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:58:54.412 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000013", 2024-08-18T15:58:54.412 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:58:54.412 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:58:54.412 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:58:54.412 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:58:54.412 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:58:54.412 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:58:54.413 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:58:54.413 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:58:54.413 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:58:54.413 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:58:54.413 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:58:54.413 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:58:54.413 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:58:54.413 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:58:54.413 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000051", 2024-08-18T15:58:54.413 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:58:54.413 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:58:54.413 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:58:54.413 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:58:54.413 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000106", 2024-08-18T15:58:54.414 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:58:54.414 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:58:54.414 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:58:54.414 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:58:54.414 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:58:54.414 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000017", 2024-08-18T15:58:54.414 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:58:54.414 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:58:54.414 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:58:54.414 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000023", 2024-08-18T15:58:54.414 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:58:54.414 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:58:54.414 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:58:54.415 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:58:54.415 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:58:54.415 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:58:54.415 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:58:54.415 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000107", 2024-08-18T15:58:54.415 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:58:54.415 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:58:54.415 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000046", 2024-08-18T15:58:54.415 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:58:54.415 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:58:54.415 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:58:54.415 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.26", 2024-08-18T15:58:54.415 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:58:54.416 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:58:54.416 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9", 2024-08-18T15:58:54.416 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.22", 2024-08-18T15:58:54.416 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:58:54.416 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:58:54.416 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:58:54.416 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:58:54.416 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:58:54.416 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:58:54.416 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:58:54.416 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2", 2024-08-18T15:58:54.416 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:58:54.416 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:58:54.416 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000067", 2024-08-18T15:58:54.417 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:58:54.417 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:58:54.417 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:58:54.417 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:58:54.417 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:58:54.417 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:58:54.417 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:58:54.417 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:58:54.417 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:58:54.417 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:58:54.417 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:58:54.417 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:58:54.417 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:58:54.418 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:58:54.418 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:58:54.418 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:58:54.418 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:58:54.418 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:58:54.418 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:58:54.418 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.9", 2024-08-18T15:58:54.418 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000086", 2024-08-18T15:58:54.418 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.5", 2024-08-18T15:58:54.418 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:58:54.418 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:58:54.418 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:58:54.419 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:58:54.419 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:58:54.419 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:58:54.419 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2.MFXJ583gCr1.0", 2024-08-18T15:58:54.419 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:58:54.419 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1.GPnO16liyZ+.0", 2024-08-18T15:58:54.419 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:58:54.419 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:58:54.419 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:58:54.419 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:58:54.419 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:58:54.419 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:58:54.419 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:58:54.420 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:58:54.420 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:58:54.420 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.20", 2024-08-18T15:58:54.420 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:58:54.420 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:58:54.420 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:58:54.420 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:58:54.420 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:58:54.420 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:58:54.420 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1.retry", 2024-08-18T15:58:54.420 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:58:54.420 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:58:54.420 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:58:54.420 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0.retry", 2024-08-18T15:58:54.421 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000050", 2024-08-18T15:58:54.421 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:58:54.421 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000102", 2024-08-18T15:58:54.421 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:58:54.421 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:58:54.421 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:58:54.421 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:58:54.421 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:58:54.421 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:58:54.421 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:58:54.421 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:58:54.421 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:58:54.421 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:58:54.422 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:58:54.422 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000124", 2024-08-18T15:58:54.422 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:58:54.422 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:58:54.422 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:58:54.422 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000001", 2024-08-18T15:58:54.422 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:58:54.422 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:58:54.422 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:58:54.422 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:58:54.422 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000126", 2024-08-18T15:58:54.422 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:58:54.422 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0.fZhctfLXnSt.0", 2024-08-18T15:58:54.422 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:58:54.423 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000055", 2024-08-18T15:58:54.423 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:58:54.423 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000061", 2024-08-18T15:58:54.423 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1", 2024-08-18T15:58:54.423 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:58:54.423 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:58:54.423 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:58:54.423 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000097", 2024-08-18T15:58:54.423 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:58:54.423 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:58:54.423 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:58:54.423 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:58:54.424 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:58:54.424 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:58:54.424 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:58:54.424 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:58:54.424 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000042", 2024-08-18T15:58:54.424 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:58:54.424 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:58:54.424 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:58:54.424 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1.retry", 2024-08-18T15:58:54.424 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:58:54.424 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:58:54.424 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:58:54.425 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:58:54.425 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:58:54.425 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:58:54.425 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:58:54.425 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000075", 2024-08-18T15:58:54.425 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:58:54.425 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000120", 2024-08-18T15:58:54.425 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000030", 2024-08-18T15:58:54.425 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:58:54.425 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:58:54.425 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:58:54.425 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:58:54.425 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:58:54.426 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:58:54.426 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9" 2024-08-18T15:58:54.426 INFO:teuthology.orchestra.run.smithi193.stdout:] 2024-08-18T15:58:54.426 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:58:59.427 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:58:59.428 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:58:59.428 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:58:59.480 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:58:59.480 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:58:59.545 INFO:teuthology.orchestra.run.smithi193.stdout:[ 2024-08-18T15:58:59.545 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:58:59.545 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:58:59.545 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.29", 2024-08-18T15:58:59.545 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000078", 2024-08-18T15:58:59.545 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.history", 2024-08-18T15:58:59.545 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:58:59.545 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:58:59.546 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:58:59.546 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:58:59.546 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.23", 2024-08-18T15:58:59.546 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000070", 2024-08-18T15:58:59.546 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000104", 2024-08-18T15:58:59.546 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:58:59.546 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2.retry", 2024-08-18T15:58:59.546 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:58:59.546 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:58:59.546 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:58:59.546 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:58:59.546 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000026", 2024-08-18T15:58:59.546 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:58:59.547 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:58:59.547 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:58:59.547 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:58:59.547 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:58:59.547 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:58:59.547 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.8", 2024-08-18T15:58:59.547 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21", 2024-08-18T15:58:59.547 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:58:59.547 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:58:59.547 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:5", 2024-08-18T15:58:59.547 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000028", 2024-08-18T15:58:59.547 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:58:59.547 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:58:59.547 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:58:59.548 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:58:59.548 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:58:59.548 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:58:59.548 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:58:59.548 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2.retry", 2024-08-18T15:58:59.548 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:8", 2024-08-18T15:58:59.548 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.30", 2024-08-18T15:58:59.548 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:58:59.548 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:58:59.548 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:58:59.548 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:58:59.548 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000040", 2024-08-18T15:58:59.548 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:58:59.549 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.6", 2024-08-18T15:58:59.549 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:58:59.549 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:58:59.549 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:58:59.549 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:58:59.549 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:58:59.549 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:58:59.549 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000015", 2024-08-18T15:58:59.549 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:0", 2024-08-18T15:58:59.549 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:58:59.549 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000069", 2024-08-18T15:58:59.549 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:58:59.549 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:58:59.549 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:58:59.549 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000095", 2024-08-18T15:58:59.550 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0", 2024-08-18T15:58:59.550 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000003", 2024-08-18T15:58:59.550 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000047", 2024-08-18T15:58:59.550 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:58:59.550 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:58:59.550 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:58:59.550 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:58:59.550 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:58:59.551 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:58:59.551 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:58:59.551 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:58:59.551 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:58:59.551 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000035", 2024-08-18T15:58:59.551 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:58:59.551 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:58:59.551 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000037", 2024-08-18T15:58:59.551 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:58:59.551 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:58:59.552 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:58:59.552 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:58:59.552 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:58:59.552 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:58:59.552 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:58:59.552 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:58:59.552 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:58:59.552 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:58:59.552 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:58:59.552 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:58:59.552 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:58:59.553 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:58:59.553 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000024", 2024-08-18T15:58:59.553 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:58:59.553 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:1", 2024-08-18T15:58:59.553 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:58:59.553 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:58:59.553 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000000", 2024-08-18T15:58:59.553 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:58:59.553 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000031", 2024-08-18T15:58:59.553 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:58:59.553 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:58:59.553 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:58:59.553 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:58:59.553 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:58:59.554 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:58:59.554 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3", 2024-08-18T15:58:59.554 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:58:59.554 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:58:59.554 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:2", 2024-08-18T15:58:59.554 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:58:59.554 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:58:59.554 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:58:59.554 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:58:59.554 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:58:59.554 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:58:59.554 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:58:59.554 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:0", 2024-08-18T15:58:59.555 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:4", 2024-08-18T15:58:59.555 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:58:59.555 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:58:59.555 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:58:59.555 INFO:teuthology.orchestra.run.smithi193.stdout: "data_loggenerations_metadata", 2024-08-18T15:58:59.555 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.2", 2024-08-18T15:58:59.555 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:58:59.555 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:58:59.555 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:58:59.555 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:58:59.555 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T15:58:59.555 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:5", 2024-08-18T15:58:59.555 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:58:59.556 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:2", 2024-08-18T15:58:59.556 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:58:59.556 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:58:59.556 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000076", 2024-08-18T15:58:59.556 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:58:59.556 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:58:59.556 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000113", 2024-08-18T15:58:59.556 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000005", 2024-08-18T15:58:59.556 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:58:59.556 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000011", 2024-08-18T15:58:59.556 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:58:59.556 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000058", 2024-08-18T15:58:59.556 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:58:59.557 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:58:59.557 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:58:59.557 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000052", 2024-08-18T15:58:59.557 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:58:59.557 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:58:59.557 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:58:59.557 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:58:59.557 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.21", 2024-08-18T15:58:59.557 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:58:59.557 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:58:59.557 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:58:59.557 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:58:59.557 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29", 2024-08-18T15:58:59.557 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:58:59.557 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:58:59.558 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:58:59.558 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:58:59.558 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:58:59.558 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15", 2024-08-18T15:58:59.558 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:58:59.558 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:58:59.558 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:58:59.558 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:58:59.558 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:58:59.558 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:58:59.558 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:1", 2024-08-18T15:58:59.558 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:58:59.559 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000088", 2024-08-18T15:58:59.559 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000080", 2024-08-18T15:58:59.559 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:58:59.559 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:58:59.559 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000090", 2024-08-18T15:58:59.559 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:58:59.559 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:58:59.559 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000110", 2024-08-18T15:58:59.559 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:58:59.559 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:58:59.559 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:58:59.559 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.0", 2024-08-18T15:58:59.559 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:58:59.560 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000096", 2024-08-18T15:58:59.560 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:58:59.560 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:58:59.560 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:58:59.560 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:58:59.560 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:7", 2024-08-18T15:58:59.560 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000087", 2024-08-18T15:58:59.560 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:58:59.560 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:8", 2024-08-18T15:58:59.560 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:4", 2024-08-18T15:58:59.560 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:6", 2024-08-18T15:58:59.560 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000008", 2024-08-18T15:58:59.560 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:9", 2024-08-18T15:58:59.561 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:58:59.561 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:58:59.561 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:4", 2024-08-18T15:58:59.561 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:58:59.561 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000006", 2024-08-18T15:58:59.561 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000029", 2024-08-18T15:58:59.561 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:58:59.561 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000089", 2024-08-18T15:58:59.561 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:58:59.561 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:8", 2024-08-18T15:58:59.561 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000022", 2024-08-18T15:58:59.561 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:58:59.561 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:58:59.561 INFO:teuthology.orchestra.run.smithi193.stdout: "bilog.trim", 2024-08-18T15:58:59.562 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000016", 2024-08-18T15:58:59.562 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:9", 2024-08-18T15:58:59.562 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:58:59.562 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:58:59.562 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:58:59.562 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:58:59.562 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000041", 2024-08-18T15:58:59.562 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:58:59.563 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:58:59.563 INFO:teuthology.orchestra.run.smithi193.stdout: "meta.log.01050e30-179b-43be-bdbb-3bb23cf37ae8.3", 2024-08-18T15:58:59.563 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:58:59.563 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000018", 2024-08-18T15:58:59.563 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000092", 2024-08-18T15:58:59.563 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:58:59.563 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:10", 2024-08-18T15:58:59.563 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:58:59.563 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:58:59.563 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.27", 2024-08-18T15:58:59.563 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.4", 2024-08-18T15:58:59.563 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:1", 2024-08-18T15:58:59.563 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:58:59.564 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:58:59.564 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:2", 2024-08-18T15:58:59.564 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:2", 2024-08-18T15:58:59.564 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:0", 2024-08-18T15:58:59.564 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:58:59.564 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000014", 2024-08-18T15:58:59.564 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:9", 2024-08-18T15:58:59.564 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3.retry", 2024-08-18T15:58:59.564 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:58:59.564 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:58:59.564 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:0", 2024-08-18T15:58:59.564 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:3", 2024-08-18T15:58:59.565 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:58:59.565 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000112", 2024-08-18T15:58:59.565 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:10", 2024-08-18T15:58:59.565 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000007", 2024-08-18T15:58:59.565 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000021", 2024-08-18T15:58:59.565 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:58:59.565 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:58:59.565 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:4", 2024-08-18T15:58:59.565 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:6", 2024-08-18T15:58:59.565 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:3", 2024-08-18T15:58:59.565 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.16", 2024-08-18T15:58:59.565 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:58:59.565 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000064", 2024-08-18T15:58:59.566 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:0", 2024-08-18T15:58:59.566 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:58:59.566 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:58:59.566 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:58:59.566 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:9", 2024-08-18T15:58:59.566 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:58:59.566 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:58:59.566 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000071", 2024-08-18T15:58:59.566 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:58:59.566 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:6", 2024-08-18T15:58:59.566 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:58:59.566 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000074", 2024-08-18T15:58:59.567 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:58:59.567 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:10", 2024-08-18T15:58:59.567 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:58:59.567 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:58:59.567 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:58:59.567 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:9", 2024-08-18T15:58:59.567 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:58:59.567 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:58:59.567 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:58:59.567 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:58:59.567 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:7", 2024-08-18T15:58:59.567 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:58:59.568 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:58:59.568 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:58:59.568 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:8", 2024-08-18T15:58:59.568 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000081", 2024-08-18T15:58:59.568 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:58:59.568 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000009", 2024-08-18T15:58:59.568 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000121", 2024-08-18T15:58:59.568 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1", 2024-08-18T15:58:59.568 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:58:59.568 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:58:59.568 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:10", 2024-08-18T15:58:59.568 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:58:59.568 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27", 2024-08-18T15:58:59.568 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:58:59.568 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.17", 2024-08-18T15:58:59.569 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000125", 2024-08-18T15:58:59.569 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:58:59.569 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000082", 2024-08-18T15:58:59.569 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:10", 2024-08-18T15:58:59.569 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:10", 2024-08-18T15:58:59.569 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:58:59.569 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:58:59.569 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:58:59.569 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:58:59.569 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19", 2024-08-18T15:58:59.569 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:10", 2024-08-18T15:58:59.569 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:58:59.569 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:58:59.569 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:4", 2024-08-18T15:58:59.570 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000105", 2024-08-18T15:58:59.570 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000059", 2024-08-18T15:58:59.570 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:58:59.570 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:58:59.570 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:58:59.570 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:58:59.570 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:58:59.570 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:58:59.570 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:58:59.570 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:58:59.570 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:58:59.570 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:6", 2024-08-18T15:58:59.570 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:58:59.571 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000077", 2024-08-18T15:58:59.571 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000032", 2024-08-18T15:58:59.571 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:58:59.571 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16", 2024-08-18T15:58:59.571 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:58:59.571 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:58:59.571 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:7", 2024-08-18T15:58:59.571 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:58:59.571 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.24", 2024-08-18T15:58:59.571 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:5", 2024-08-18T15:58:59.571 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:6", 2024-08-18T15:58:59.571 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:9", 2024-08-18T15:58:59.572 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:10", 2024-08-18T15:58:59.572 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3.retry", 2024-08-18T15:58:59.572 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:58:59.572 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:58:59.572 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:58:59.572 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:58:59.572 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:8", 2024-08-18T15:58:59.572 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:58:59.572 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000053", 2024-08-18T15:58:59.572 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000091", 2024-08-18T15:58:59.572 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:5", 2024-08-18T15:58:59.573 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:58:59.573 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:58:59.573 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000065", 2024-08-18T15:58:59.573 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:58:59.573 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:58:59.573 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:9", 2024-08-18T15:58:59.573 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:58:59.573 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000083", 2024-08-18T15:58:59.573 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000010", 2024-08-18T15:58:59.573 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:5", 2024-08-18T15:58:59.573 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:8", 2024-08-18T15:58:59.573 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:0", 2024-08-18T15:58:59.573 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:8", 2024-08-18T15:58:59.574 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:58:59.574 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:7", 2024-08-18T15:58:59.574 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:58:59.574 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:58:59.574 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39", 2024-08-18T15:58:59.574 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:58:59.574 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:58:59.574 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.7", 2024-08-18T15:58:59.574 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.3", 2024-08-18T15:58:59.574 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:58:59.574 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:6", 2024-08-18T15:58:59.574 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:58:59.574 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:58:59.575 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:1", 2024-08-18T15:58:59.575 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:58:59.575 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:9", 2024-08-18T15:58:59.575 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:58:59.575 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.2", 2024-08-18T15:58:59.575 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:10", 2024-08-18T15:58:59.575 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:8", 2024-08-18T15:58:59.575 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:58:59.575 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:58:59.575 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:9", 2024-08-18T15:58:59.576 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:8", 2024-08-18T15:58:59.576 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:58:59.576 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000045", 2024-08-18T15:58:59.576 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000002", 2024-08-18T15:58:59.576 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:58:59.576 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.3.2UbiLidwSTy.0", 2024-08-18T15:58:59.576 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:58:59.576 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.12", 2024-08-18T15:58:59.576 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:58:59.576 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:58:59.576 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:58:59.576 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:58:59.577 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:58:59.577 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:8", 2024-08-18T15:58:59.577 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000116", 2024-08-18T15:58:59.577 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.25", 2024-08-18T15:58:59.577 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:3", 2024-08-18T15:58:59.577 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:2", 2024-08-18T15:58:59.577 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:58:59.577 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:1", 2024-08-18T15:58:59.577 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000034", 2024-08-18T15:58:59.577 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:58:59.577 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:0", 2024-08-18T15:58:59.577 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000101", 2024-08-18T15:58:59.578 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:7", 2024-08-18T15:58:59.578 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:2", 2024-08-18T15:58:59.578 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.13", 2024-08-18T15:58:59.578 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:58:59.578 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:4", 2024-08-18T15:58:59.578 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.1", 2024-08-18T15:58:59.578 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0.retry", 2024-08-18T15:58:59.578 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:3", 2024-08-18T15:58:59.578 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:2", 2024-08-18T15:58:59.578 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000079", 2024-08-18T15:58:59.578 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000049", 2024-08-18T15:58:59.578 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:6", 2024-08-18T15:58:59.579 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.0", 2024-08-18T15:58:59.579 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:58:59.579 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:58:59.579 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000117", 2024-08-18T15:58:59.579 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:8", 2024-08-18T15:58:59.579 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000044", 2024-08-18T15:58:59.579 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:10", 2024-08-18T15:58:59.579 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26", 2024-08-18T15:58:59.579 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:2", 2024-08-18T15:58:59.579 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:1", 2024-08-18T15:58:59.579 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:4", 2024-08-18T15:58:59.579 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:58:59.579 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.14", 2024-08-18T15:58:59.580 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:58:59.580 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000066", 2024-08-18T15:58:59.580 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:58:59.580 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18", 2024-08-18T15:58:59.580 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:0", 2024-08-18T15:58:59.580 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:9", 2024-08-18T15:58:59.580 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:58:59.580 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000068", 2024-08-18T15:58:59.580 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:58:59.580 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000085", 2024-08-18T15:58:59.580 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:58:59.580 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:6", 2024-08-18T15:58:59.581 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:58:59.581 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:58:59.581 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:58:59.581 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:6", 2024-08-18T15:58:59.581 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:58:59.581 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:58:59.581 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:58:59.581 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:58:59.581 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:58:59.582 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:58:59.582 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000073", 2024-08-18T15:58:59.582 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:0", 2024-08-18T15:58:59.582 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:7", 2024-08-18T15:58:59.582 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:58:59.582 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000038", 2024-08-18T15:58:59.582 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:2", 2024-08-18T15:58:59.582 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:3", 2024-08-18T15:58:59.582 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:58:59.583 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:2", 2024-08-18T15:58:59.583 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.10", 2024-08-18T15:58:59.583 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:3", 2024-08-18T15:58:59.583 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:4", 2024-08-18T15:58:59.583 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:58:59.583 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000118", 2024-08-18T15:58:59.583 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:10", 2024-08-18T15:58:59.584 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.3", 2024-08-18T15:58:59.584 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:6", 2024-08-18T15:58:59.584 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000036", 2024-08-18T15:58:59.584 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:58:59.584 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:58:59.584 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:0", 2024-08-18T15:58:59.584 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:0", 2024-08-18T15:58:59.584 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:1", 2024-08-18T15:58:59.584 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000103", 2024-08-18T15:58:59.584 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.3", 2024-08-18T15:58:59.584 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:1", 2024-08-18T15:58:59.584 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000119", 2024-08-18T15:58:59.584 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000098", 2024-08-18T15:58:59.584 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:7", 2024-08-18T15:58:59.585 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:58:59.585 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:2", 2024-08-18T15:58:59.585 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:58:59.585 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:3", 2024-08-18T15:58:59.585 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:58:59.585 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:58:59.585 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:5", 2024-08-18T15:58:59.585 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status.shard.0", 2024-08-18T15:58:59.585 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:7", 2024-08-18T15:58:59.585 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:3", 2024-08-18T15:58:59.585 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:58:59.586 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:6", 2024-08-18T15:58:59.586 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12", 2024-08-18T15:58:59.586 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:58:59.586 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:4", 2024-08-18T15:58:59.586 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:9", 2024-08-18T15:58:59.586 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000027", 2024-08-18T15:58:59.586 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:58:59.586 INFO:teuthology.orchestra.run.smithi193.stdout: "mdlog.sync-status", 2024-08-18T15:58:59.586 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:8", 2024-08-18T15:58:59.586 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000019", 2024-08-18T15:58:59.586 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:58:59.586 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:4", 2024-08-18T15:58:59.587 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:9", 2024-08-18T15:58:59.587 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:58:59.587 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1", 2024-08-18T15:58:59.587 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:3", 2024-08-18T15:58:59.587 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:2", 2024-08-18T15:58:59.587 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:58:59.587 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:58:59.587 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000039", 2024-08-18T15:58:59.587 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:9", 2024-08-18T15:58:59.587 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:5", 2024-08-18T15:58:59.587 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:58:59.588 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:0", 2024-08-18T15:58:59.588 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:4", 2024-08-18T15:58:59.588 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:58:59.588 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:6", 2024-08-18T15:58:59.588 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:58:59.588 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:58:59.588 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:3", 2024-08-18T15:58:59.588 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:3", 2024-08-18T15:58:59.588 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000100", 2024-08-18T15:58:59.588 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:0", 2024-08-18T15:58:59.588 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:5", 2024-08-18T15:58:59.588 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:58:59.588 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:58:59.588 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:0", 2024-08-18T15:58:59.589 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000093", 2024-08-18T15:58:59.589 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:10", 2024-08-18T15:58:59.589 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000004", 2024-08-18T15:58:59.589 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000063", 2024-08-18T15:58:59.589 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000122", 2024-08-18T15:58:59.589 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:58:59.589 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:1", 2024-08-18T15:58:59.589 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:5", 2024-08-18T15:58:59.589 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000057", 2024-08-18T15:58:59.589 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:8", 2024-08-18T15:58:59.589 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:10", 2024-08-18T15:58:59.589 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:5", 2024-08-18T15:58:59.589 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:7", 2024-08-18T15:58:59.589 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000054", 2024-08-18T15:58:59.590 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:58:59.590 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000114", 2024-08-18T15:58:59.590 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:4", 2024-08-18T15:58:59.590 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:58:59.590 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:4", 2024-08-18T15:58:59.590 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.2", 2024-08-18T15:58:59.590 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:9", 2024-08-18T15:58:59.590 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:1", 2024-08-18T15:58:59.590 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0", 2024-08-18T15:58:59.590 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:4", 2024-08-18T15:58:59.590 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:58:59.590 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:10", 2024-08-18T15:58:59.591 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:58:59.591 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:58:59.591 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:58:59.591 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000012", 2024-08-18T15:58:59.591 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:58:59.591 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:10", 2024-08-18T15:58:59.591 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:0", 2024-08-18T15:58:59.591 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000084", 2024-08-18T15:58:59.591 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000043", 2024-08-18T15:58:59.592 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.3", 2024-08-18T15:58:59.592 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:1", 2024-08-18T15:58:59.592 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:6", 2024-08-18T15:58:59.592 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.11", 2024-08-18T15:58:59.592 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:7", 2024-08-18T15:58:59.592 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:5", 2024-08-18T15:58:59.592 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:58:59.592 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:58:59.592 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:58:59.592 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:6", 2024-08-18T15:58:59.592 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:3", 2024-08-18T15:58:59.592 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:8", 2024-08-18T15:58:59.592 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000111", 2024-08-18T15:58:59.593 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5", 2024-08-18T15:58:59.593 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000048", 2024-08-18T15:58:59.593 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:1", 2024-08-18T15:58:59.593 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000020", 2024-08-18T15:58:59.593 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:58:59.593 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:8", 2024-08-18T15:58:59.593 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:58:59.593 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:58:59.593 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1", 2024-08-18T15:58:59.593 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000099", 2024-08-18T15:58:59.593 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000056", 2024-08-18T15:58:59.593 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:58:59.593 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:10", 2024-08-18T15:58:59.594 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.19", 2024-08-18T15:58:59.594 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:5", 2024-08-18T15:58:59.594 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.28", 2024-08-18T15:58:59.594 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:9", 2024-08-18T15:58:59.594 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-61:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.9", 2024-08-18T15:58:59.594 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:5", 2024-08-18T15:58:59.594 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:9", 2024-08-18T15:58:59.594 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:10", 2024-08-18T15:58:59.594 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000072", 2024-08-18T15:58:59.594 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:5", 2024-08-18T15:58:59.594 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:58:59.594 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000062", 2024-08-18T15:58:59.594 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:58:59.595 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:7", 2024-08-18T15:58:59.595 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:58:59.595 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000115", 2024-08-18T15:58:59.595 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:58:59.595 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.1", 2024-08-18T15:58:59.595 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.2", 2024-08-18T15:58:59.595 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000033", 2024-08-18T15:58:59.595 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:58:59.595 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:58:59.595 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:58:59.595 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:4", 2024-08-18T15:58:59.595 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:6", 2024-08-18T15:58:59.596 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:58:59.596 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11", 2024-08-18T15:58:59.596 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:6", 2024-08-18T15:58:59.596 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:7", 2024-08-18T15:58:59.596 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:7", 2024-08-18T15:58:59.596 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:58:59.596 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:3", 2024-08-18T15:58:59.596 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:3", 2024-08-18T15:58:59.596 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:58:59.596 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:7", 2024-08-18T15:58:59.596 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:5", 2024-08-18T15:58:59.596 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:58:59.596 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:3", 2024-08-18T15:58:59.597 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:1", 2024-08-18T15:58:59.597 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000025", 2024-08-18T15:58:59.597 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:58:59.597 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000123", 2024-08-18T15:58:59.597 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.31", 2024-08-18T15:58:59.597 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:58:59.597 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:58:59.597 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:58:59.597 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:9", 2024-08-18T15:58:59.597 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.18", 2024-08-18T15:58:59.597 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:2", 2024-08-18T15:58:59.597 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:6", 2024-08-18T15:58:59.597 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:8", 2024-08-18T15:58:59.597 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:9", 2024-08-18T15:58:59.598 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:10", 2024-08-18T15:58:59.598 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000108", 2024-08-18T15:58:59.598 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:5", 2024-08-18T15:58:59.598 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:2", 2024-08-18T15:58:59.598 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000094", 2024-08-18T15:58:59.598 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:5", 2024-08-18T15:58:59.598 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:6", 2024-08-18T15:58:59.598 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:9", 2024-08-18T15:58:59.598 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:58:59.598 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:1", 2024-08-18T15:58:59.598 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:7", 2024-08-18T15:58:59.598 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000060", 2024-08-18T15:58:59.598 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:58:59.599 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:10", 2024-08-18T15:58:59.599 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:2", 2024-08-18T15:58:59.599 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000109", 2024-08-18T15:58:59.599 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.15", 2024-08-18T15:58:59.599 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:58:59.599 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:9", 2024-08-18T15:58:59.599 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:58:59.599 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:58:59.599 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17", 2024-08-18T15:58:59.599 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:8", 2024-08-18T15:58:59.599 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:0", 2024-08-18T15:58:59.599 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:3", 2024-08-18T15:58:59.599 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:7", 2024-08-18T15:58:59.599 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000013", 2024-08-18T15:58:59.600 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:7", 2024-08-18T15:58:59.600 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:4", 2024-08-18T15:58:59.600 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:8", 2024-08-18T15:58:59.600 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:3", 2024-08-18T15:58:59.600 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:58:59.600 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:5", 2024-08-18T15:58:59.600 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:58:59.600 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:7", 2024-08-18T15:58:59.600 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:3", 2024-08-18T15:58:59.600 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:7", 2024-08-18T15:58:59.600 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:7", 2024-08-18T15:58:59.600 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:58:59.600 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:58:59.600 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:4", 2024-08-18T15:58:59.601 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000051", 2024-08-18T15:58:59.601 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:7", 2024-08-18T15:58:59.601 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:2", 2024-08-18T15:58:59.601 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:4", 2024-08-18T15:58:59.601 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:58:59.601 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000106", 2024-08-18T15:58:59.601 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:0", 2024-08-18T15:58:59.601 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:10", 2024-08-18T15:58:59.601 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:10", 2024-08-18T15:58:59.601 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:58:59.601 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:58:59.601 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000017", 2024-08-18T15:58:59.601 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:3", 2024-08-18T15:58:59.601 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:9", 2024-08-18T15:58:59.602 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:2", 2024-08-18T15:58:59.602 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000023", 2024-08-18T15:58:59.602 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:8", 2024-08-18T15:58:59.602 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:58:59.602 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:3", 2024-08-18T15:58:59.602 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22", 2024-08-18T15:58:59.602 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:5", 2024-08-18T15:58:59.602 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:1", 2024-08-18T15:58:59.602 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:58:59.602 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000107", 2024-08-18T15:58:59.602 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:8", 2024-08-18T15:58:59.602 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:2", 2024-08-18T15:58:59.602 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000046", 2024-08-18T15:58:59.603 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:5", 2024-08-18T15:58:59.603 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:0", 2024-08-18T15:58:59.603 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:0", 2024-08-18T15:58:59.603 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.26", 2024-08-18T15:58:59.603 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:7", 2024-08-18T15:58:59.603 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-59:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.7", 2024-08-18T15:58:59.603 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9", 2024-08-18T15:58:59.603 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.22", 2024-08-18T15:58:59.603 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3:5", 2024-08-18T15:58:59.603 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:9", 2024-08-18T15:58:59.603 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23", 2024-08-18T15:58:59.603 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:4", 2024-08-18T15:58:59.603 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:0", 2024-08-18T15:58:59.604 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:2", 2024-08-18T15:58:59.604 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:8", 2024-08-18T15:58:59.604 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2", 2024-08-18T15:58:59.604 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:58:59.604 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:4", 2024-08-18T15:58:59.604 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000067", 2024-08-18T15:58:59.604 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14", 2024-08-18T15:58:59.604 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:3", 2024-08-18T15:58:59.604 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:1", 2024-08-18T15:58:59.604 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-49:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.3", 2024-08-18T15:58:59.604 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:58:59.604 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:3", 2024-08-18T15:58:59.604 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:5", 2024-08-18T15:58:59.605 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:58:59.605 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T15:58:59.605 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:9", 2024-08-18T15:58:59.605 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41", 2024-08-18T15:58:59.605 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-60:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.8", 2024-08-18T15:58:59.605 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:0", 2024-08-18T15:58:59.605 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:5", 2024-08-18T15:58:59.605 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:5", 2024-08-18T15:58:59.605 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:8", 2024-08-18T15:58:59.605 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24", 2024-08-18T15:58:59.605 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:3", 2024-08-18T15:58:59.605 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:9", 2024-08-18T15:58:59.605 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.9", 2024-08-18T15:58:59.605 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000086", 2024-08-18T15:58:59.606 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.5", 2024-08-18T15:58:59.606 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:10", 2024-08-18T15:58:59.606 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:1", 2024-08-18T15:58:59.606 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-12:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.12:2", 2024-08-18T15:58:59.606 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:4", 2024-08-18T15:58:59.606 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:8", 2024-08-18T15:58:59.606 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:2", 2024-08-18T15:58:59.606 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.2.MFXJ583gCr1.0", 2024-08-18T15:58:59.606 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:10", 2024-08-18T15:58:59.606 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.1.GPnO16liyZ+.0", 2024-08-18T15:58:59.606 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:6", 2024-08-18T15:58:59.606 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40", 2024-08-18T15:58:59.606 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:6", 2024-08-18T15:58:59.607 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:1", 2024-08-18T15:58:59.607 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:6", 2024-08-18T15:58:59.607 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:58:59.607 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:0", 2024-08-18T15:58:59.607 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-16:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.16:8", 2024-08-18T15:58:59.607 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:10", 2024-08-18T15:58:59.607 INFO:teuthology.orchestra.run.smithi193.stdout: "sync.error-log.20", 2024-08-18T15:58:59.607 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:7", 2024-08-18T15:58:59.607 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:2", 2024-08-18T15:58:59.607 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20", 2024-08-18T15:58:59.607 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:7", 2024-08-18T15:58:59.607 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-14:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.14:0", 2024-08-18T15:58:59.607 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:3", 2024-08-18T15:58:59.607 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1.retry", 2024-08-18T15:58:59.608 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-17:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.17:2", 2024-08-18T15:58:59.608 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:2", 2024-08-18T15:58:59.608 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:1", 2024-08-18T15:58:59.608 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.0.retry", 2024-08-18T15:58:59.608 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000050", 2024-08-18T15:58:59.608 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:2", 2024-08-18T15:58:59.608 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000102", 2024-08-18T15:58:59.608 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:6", 2024-08-18T15:58:59.608 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:3", 2024-08-18T15:58:59.608 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4:1", 2024-08-18T15:58:59.608 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:10", 2024-08-18T15:58:59.608 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-50:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.4", 2024-08-18T15:58:59.608 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13", 2024-08-18T15:58:59.608 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:9", 2024-08-18T15:58:59.609 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:6", 2024-08-18T15:58:59.609 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-22:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.22:4", 2024-08-18T15:58:59.609 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:6", 2024-08-18T15:58:59.609 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:7", 2024-08-18T15:58:59.609 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000124", 2024-08-18T15:58:59.609 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-11:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.11:1", 2024-08-18T15:58:59.609 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:0", 2024-08-18T15:58:59.609 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:0", 2024-08-18T15:58:59.609 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000001", 2024-08-18T15:58:59.609 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:58:59.609 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25:8", 2024-08-18T15:58:59.609 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:8", 2024-08-18T15:58:59.609 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:4", 2024-08-18T15:58:59.609 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000126", 2024-08-18T15:58:59.610 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:2", 2024-08-18T15:58:59.610 INFO:teuthology.orchestra.run.smithi193.stdout: "data_log.0.fZhctfLXnSt.0", 2024-08-18T15:58:59.610 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:7", 2024-08-18T15:58:59.610 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000055", 2024-08-18T15:58:59.610 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:2", 2024-08-18T15:58:59.610 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000061", 2024-08-18T15:58:59.610 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.018ea73b-7528-48da-aafa-0118123784f7.1", 2024-08-18T15:58:59.610 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-29:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.29:7", 2024-08-18T15:58:59.610 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28", 2024-08-18T15:58:59.610 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:5", 2024-08-18T15:58:59.610 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000097", 2024-08-18T15:58:59.610 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:6", 2024-08-18T15:58:59.611 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10", 2024-08-18T15:58:59.611 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-39:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.39:4", 2024-08-18T15:58:59.611 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-51:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.5:3", 2024-08-18T15:58:59.611 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-24:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.24:0", 2024-08-18T15:58:59.611 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:6", 2024-08-18T15:58:59.611 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-28:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.28:5", 2024-08-18T15:58:59.611 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:0", 2024-08-18T15:58:59.611 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000042", 2024-08-18T15:58:59.611 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-52:96bf8502-2f39-4d53-8196-3378c28da3b4.5859.1:4", 2024-08-18T15:58:59.611 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:1", 2024-08-18T15:58:59.611 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-25:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.25", 2024-08-18T15:58:59.611 INFO:teuthology.orchestra.run.smithi193.stdout: "datalog.sync-status.shard.96bf8502-2f39-4d53-8196-3378c28da3b4.1.retry", 2024-08-18T15:58:59.611 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:10", 2024-08-18T15:58:59.612 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:1", 2024-08-18T15:58:59.612 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-15:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.15:1", 2024-08-18T15:58:59.612 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-20:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.20:4", 2024-08-18T15:58:59.612 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-41:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.41:3", 2024-08-18T15:58:59.612 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:4", 2024-08-18T15:58:59.612 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-19:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.19:1", 2024-08-18T15:58:59.612 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000075", 2024-08-18T15:58:59.612 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-23:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.23:5", 2024-08-18T15:58:59.612 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000120", 2024-08-18T15:58:59.612 INFO:teuthology.orchestra.run.smithi193.stdout: "obj_delete_at_hint.0000000030", 2024-08-18T15:58:59.612 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-13:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.13:6", 2024-08-18T15:58:59.612 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-18:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.18:1", 2024-08-18T15:58:59.612 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-21:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.21:10", 2024-08-18T15:58:59.612 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-26:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.26:8", 2024-08-18T15:58:59.613 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-27:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.27:9", 2024-08-18T15:58:59.613 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-10:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.10:1", 2024-08-18T15:58:59.613 INFO:teuthology.orchestra.run.smithi193.stdout: "bucket.sync-status.018ea73b-7528-48da-aafa-0118123784f7:kibgmb-40:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.40:9" 2024-08-18T15:58:59.613 INFO:teuthology.orchestra.run.smithi193.stdout:] 2024-08-18T15:58:59.613 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-08-18T15:59:04.615 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_disable ... FAIL 2024-08-18T15:59:04.616 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=kibgmb-62 2024-08-18T15:59:04.637 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=kibgmb-63 2024-08-18T15:59:04.672 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=kibgmb-64 2024-08-18T15:59:04.911 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:59:04.911 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:59:04.911 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:59:04.958 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:59:04.958 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:59:04.988 INFO:teuthology.orchestra.run.smithi134.stderr:No --period given, using current period=01050e30-179b-43be-bdbb-3bb23cf37ae8 2024-08-18T15:59:05.000 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T15:59:05.001 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:59:05.001 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T15:59:05.001 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T15:59:05.001 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:59:05.001 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:59:05.001 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T15:59:05.001 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T15:59:05.001 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:59:05.001 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:59:05.001 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T15:59:05.002 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T15:59:05.002 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:59:05.002 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:59:05.002 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723996744.690922_511.1", 2024-08-18T15:59:05.002 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:59:04.690922Z" 2024-08-18T15:59:05.002 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:59:05.002 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T15:59:05.003 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-08-18T15:59:05.003 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:59:05.004 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:59:05.004 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:59:05.059 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:59:05.059 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:59:05.092 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:59:05.091+0000 7fe7f2a0f880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:59:05.093 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:59:05.092+0000 7fe7f2a0f880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:59:05.107 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:59:05.107 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:59:05.107 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:59:05.107 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:59:05.107 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:59:05.108 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T15:59:05.108 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:59:05.108 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:59:05.108 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:59:05.108 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:59:05.108 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:59:05.108 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:59:05.108 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:59:05.108 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:59:05.108 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:59:05.109 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:59:05.109 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:59:05.109 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:59:05.109 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:59:05.109 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:59:05.109 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:59:05.109 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:59:05.109 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:59:05.109 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:59:05.109 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:59:05.109 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:59:05.110 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:59:05.110 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:59:05.110 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:59:05.110 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:59:05.110 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:59:05.110 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:59:05.110 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:59:05.110 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:59:05.110 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:59:05.110 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:59:05.110 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:59:05.111 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:59:05.111 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:59:05.111 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:59:05.111 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:59:05.111 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:59:05.111 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:59:05.111 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:59:05.111 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:59:05.111 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:59:05.111 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:59:05.112 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:59:05.112 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:59:05.112 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723996744.628517_500.1", 2024-08-18T15:59:05.112 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:59:05.112 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:59:05.112 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:59:05.112 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:59:04.628517Z", 2024-08-18T15:59:05.112 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:59:05.112 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:59:05.112 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:59:05.112 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:59:05.113 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:59:05.113 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:59:05.113 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:59:05.113 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:59:05.113 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:59:05.113 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:59:05Z" 2024-08-18T15:59:05.113 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:59:05.114 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1723996744.690922_511.1 target=1_1723996744.628517_500.1 2024-08-18T15:59:10.116 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:59:10.116 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:59:10.116 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:59:10.172 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:59:10.172 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:59:10.205 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:59:10.204+0000 7fd0ba738880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:59:10.206 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:59:10.205+0000 7fd0ba738880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:59:10.221 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T15:59:10.221 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T15:59:10.221 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T15:59:10.221 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T15:59:10.221 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T15:59:10.221 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T15:59:10.221 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:59:10.221 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:59:10.221 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T15:59:10.221 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:59:10.222 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T15:59:10.222 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:59:10.222 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:59:10.222 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:59:10.222 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:59:10.222 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T15:59:10.222 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:59:10.222 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:59:10.222 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:59:10.222 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:59:10.222 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:59:10.222 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:59:10.222 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T15:59:10.223 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:59:10.223 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:59:10.223 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T15:59:10.223 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:59:10.223 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:59:10.223 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:59:10.223 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T15:59:10.223 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:59:10.223 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:59:10.223 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:59:10.223 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:59:10.223 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T15:59:10.223 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:59:10.223 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:59:10.224 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:59:10.224 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:59:10.224 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:59:10.224 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:59:10.224 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:59:10.224 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T15:59:10.224 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:59:10.224 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:59:10.224 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T15:59:10.224 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T15:59:10.224 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T15:59:10.224 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T15:59:10.224 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723996744.690922_511.1", 2024-08-18T15:59:10.224 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T15:59:10.225 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T15:59:10.225 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T15:59:10.225 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:59:04.690922Z", 2024-08-18T15:59:10.225 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T15:59:10.225 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:59:10.225 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T15:59:10.225 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T15:59:10.225 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:59:10.225 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T15:59:10.225 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T15:59:10.226 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T15:59:10.226 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T15:59:10.226 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T15:59:10Z" 2024-08-18T15:59:10.226 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T15:59:10.227 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-08-18T15:59:10.227 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-08-18T15:59:10.227 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:59:10.227 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T15:59:10.227 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T15:59:10.283 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:59:10.283 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:59:10.315 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:59:10.314+0000 7f4d68886880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:59:10.316 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:59:10.315+0000 7f4d68886880 20 RGW-SYNC:meta: read sync status 2024-08-18T15:59:10.329 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T15:59:10.329 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T15:59:10.329 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T15:59:10.330 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T15:59:10.330 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T15:59:10.330 INFO:teuthology.orchestra.run.smithi134.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T15:59:10.330 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T15:59:10.330 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:59:10.330 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T15:59:10.331 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:59:10.331 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T15:59:10.331 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:59:10.331 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:59:10.331 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T15:59:10.331 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:59:10.332 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 1, 2024-08-18T15:59:10.332 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:59:10.332 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T15:59:10.332 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:59:10.332 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:59:10.333 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:59:10.333 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:59:10.333 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T15:59:10.334 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:59:10.334 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:59:10.334 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T15:59:10.334 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:59:10.334 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:59:10.335 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:59:10.335 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T15:59:10.335 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:59:10.335 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:59:10.335 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:59:10.335 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:59:10.336 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T15:59:10.336 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:59:10.336 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:59:10.336 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T15:59:10.336 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:59:10.336 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:59:10.337 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:59:10.337 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T15:59:10.337 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T15:59:10.337 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:59:10.337 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:59:10.337 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T15:59:10.337 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T15:59:10.337 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T15:59:10.337 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T15:59:10.337 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723996744.690922_511.1", 2024-08-18T15:59:10.337 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T15:59:10.337 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T15:59:10.338 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T15:59:10.338 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:59:04.690922Z", 2024-08-18T15:59:10.338 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T15:59:10.338 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:59:10.338 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T15:59:10.338 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T15:59:10.338 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:59:10.338 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T15:59:10.338 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 1, 2024-08-18T15:59:10.338 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 1 2024-08-18T15:59:10.338 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T15:59:10.338 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T15:59:10Z" 2024-08-18T15:59:10.338 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T15:59:10.339 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-08-18T15:59:10.340 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:59:10.340 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:59:10.340 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-62 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T15:59:10.397 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:59:10.397 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:59:10.432 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:59:10.431+0000 7f006d426880 1 waiting to reach incremental sync.. 2024-08-18T15:59:15.434 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:59:15.434+0000 7f006d426880 1 bucket sync caught up with source: 2024-08-18T15:59:15.435 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000001.52.6, , 00000000001.33.6, , 00000000001.96.6, , , 00000000001.49.6, , , ] 2024-08-18T15:59:15.435 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000001.52.6, , 00000000001.33.6, , 00000000001.96.6, , , 00000000001.49.6, , , ] 2024-08-18T15:59:15.435 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:59:15.434+0000 7f006d426880 0 bucket checkpoint complete 2024-08-18T15:59:15.453 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:59:15.453 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:59:15.453 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-62 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T15:59:15.513 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:59:15.513 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:59:15.551 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:59:15.550+0000 7f1f3f722880 1 bucket sync caught up with source: 2024-08-18T15:59:15.551 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000001.52.6, , 00000000001.33.6, , 00000000001.96.6, , , 00000000001.49.6, , , ] 2024-08-18T15:59:15.551 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000001.52.6, , 00000000001.33.6, , 00000000001.96.6, , , 00000000001.49.6, , , ] 2024-08-18T15:59:15.551 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:59:15.550+0000 7f1f3f722880 0 bucket checkpoint complete 2024-08-18T15:59:15.564 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-62', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:59:15.564 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-62', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:59:15.565 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-62 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T15:59:15.619 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:59:15.619 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:59:15.654 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:59:15.654+0000 7f0eb6738880 1 bucket sync caught up with source: 2024-08-18T15:59:15.654 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000001.207.6, , 00000000001.211.6, , 00000000001.418.6, , , 00000000001.96.6, , , ] 2024-08-18T15:59:15.655 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000001.207.6, , 00000000001.211.6, , 00000000001.418.6, , , 00000000001.96.6, , , ] 2024-08-18T15:59:15.655 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:59:15.654+0000 7f0eb6738880 0 bucket checkpoint complete 2024-08-18T15:59:15.667 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-62', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:59:15.668 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-62', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:59:15.668 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-62 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T15:59:15.721 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:59:15.721 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:59:15.754 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:59:15.754+0000 7f7d4d010880 1 bucket sync caught up with source: 2024-08-18T15:59:15.754 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000001.207.6, , 00000000001.211.6, , 00000000001.418.6, , , 00000000001.96.6, , , ] 2024-08-18T15:59:15.755 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000001.207.6, , 00000000001.211.6, , 00000000001.418.6, , , 00000000001.96.6, , , ] 2024-08-18T15:59:15.755 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:59:15.754+0000 7f7d4d010880 0 bucket checkpoint complete 2024-08-18T15:59:15.767 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-62', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:59:15.767 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-62', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:59:15.767 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-62 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T15:59:15.820 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:59:15.820 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:59:15.853 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:59:15.853+0000 7fd6f8435880 1 waiting to reach incremental sync.. 2024-08-18T15:59:20.854 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:59:20.853+0000 7fd6f8435880 1 waiting to reach incremental sync.. 2024-08-18T15:59:25.856 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:59:25.855+0000 7fd6f8435880 1 bucket sync caught up with source: 2024-08-18T15:59:25.856 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000001.134.6, , 00000000001.150.6, , 00000000001.217.6, , , 00000000001.129.6, , , ] 2024-08-18T15:59:25.856 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000001.134.6, , 00000000001.150.6, , 00000000001.217.6, , , 00000000001.129.6, , , ] 2024-08-18T15:59:25.856 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T15:59:25.855+0000 7fd6f8435880 0 bucket checkpoint complete 2024-08-18T15:59:25.873 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-62', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:59:25.873 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-62', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:59:25.874 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-62 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T15:59:25.937 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:59:25.938 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:59:25.978 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:59:25.978+0000 7f2ab1132880 1 bucket sync caught up with source: 2024-08-18T15:59:25.979 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000001.134.6, , 00000000001.150.6, , 00000000001.217.6, , , 00000000001.129.6, , , ] 2024-08-18T15:59:25.979 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000001.134.6, , 00000000001.150.6, , 00000000001.217.6, , , 00000000001.129.6, , , ] 2024-08-18T15:59:25.979 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:59:25.978+0000 7f2ab1132880 0 bucket checkpoint complete 2024-08-18T15:59:25.993 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-62 zones={test-zone2, test-zone1} 2024-08-18T15:59:25.999 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:59:26.002 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-08-18T15:59:26.002 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-08-18T15:59:26.002 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-08-18T15:59:26.003 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-08-18T15:59:26.003 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:59:26.006 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-08-18T15:59:26.006 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-08-18T15:59:26.006 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-08-18T15:59:26.006 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-08-18T15:59:26.011 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T15:59:26.062 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T15:59:26.074 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T15:59:26.083 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T15:59:26.134 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T15:59:26.149 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T15:59:26.203 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T15:59:26.210 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T15:59:26.263 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T15:59:26.316 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T15:59:26.323 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T15:59:26.378 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T15:59:26.432 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-62 zones={test-zone2, test-zone1} 2024-08-18T15:59:26.432 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-62 zones={test-zone3, test-zone1} 2024-08-18T15:59:26.437 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:59:26.439 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-08-18T15:59:26.439 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-08-18T15:59:26.440 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-08-18T15:59:26.440 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-08-18T15:59:26.440 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:59:26.442 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-08-18T15:59:26.442 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-08-18T15:59:26.443 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-08-18T15:59:26.443 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-08-18T15:59:26.448 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T15:59:26.498 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T15:59:26.560 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T15:59:26.661 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T15:59:26.712 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T15:59:26.765 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T15:59:26.823 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T15:59:26.872 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T15:59:26.973 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T15:59:27.079 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T15:59:27.129 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T15:59:27.184 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T15:59:27.238 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-62 zones={test-zone3, test-zone1} 2024-08-18T15:59:27.239 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-62 zones={test-zone3, test-zone2} 2024-08-18T15:59:27.243 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T15:59:27.246 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-08-18T15:59:27.246 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-08-18T15:59:27.247 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-08-18T15:59:27.247 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-08-18T15:59:27.247 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T15:59:27.249 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-08-18T15:59:27.250 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-08-18T15:59:27.250 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-08-18T15:59:27.250 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-08-18T15:59:27.257 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T15:59:27.306 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T15:59:27.405 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T15:59:27.502 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T15:59:27.552 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T15:59:27.649 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T15:59:27.750 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T15:59:27.804 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T15:59:27.858 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T15:59:27.953 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T15:59:28.008 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T15:59:28.109 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T15:59:28.206 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-62 zones={test-zone3, test-zone2} 2024-08-18T15:59:28.206 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:59:28.206 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T15:59:28.206 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-63 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T15:59:28.268 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T15:59:28.269 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T15:59:28.307 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:59:28.306+0000 7f5269936880 1 waiting to reach incremental sync.. 2024-08-18T15:59:33.307 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:59:33.307+0000 7f5269936880 1 waiting to reach incremental sync.. 2024-08-18T15:59:38.308 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:59:38.308+0000 7f5269936880 1 waiting to reach incremental sync.. 2024-08-18T15:59:43.309 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:59:43.308+0000 7f5269936880 1 waiting to reach incremental sync.. 2024-08-18T15:59:48.309 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:59:48.309+0000 7f5269936880 1 waiting to reach incremental sync.. 2024-08-18T15:59:53.310 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:59:53.309+0000 7f5269936880 1 waiting to reach incremental sync.. 2024-08-18T15:59:58.310 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T15:59:58.310+0000 7f5269936880 1 waiting to reach incremental sync.. 2024-08-18T16:00:03.311 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:00:03.311+0000 7f5269936880 1 waiting to reach incremental sync.. 2024-08-18T16:00:08.312 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:00:08.311+0000 7f5269936880 1 waiting to reach incremental sync.. 2024-08-18T16:00:13.313 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:00:13.311+0000 7f5269936880 1 waiting to reach incremental sync.. 2024-08-18T16:00:18.314 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:00:18.314+0000 7f5269936880 1 bucket sync caught up with source: 2024-08-18T16:00:18.315 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000001.67.6, , 00000000001.72.6, , 00000000001.74.6, , , 00000000001.42.6, , , ] 2024-08-18T16:00:18.315 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000001.67.6, , 00000000001.72.6, , 00000000001.74.6, , , 00000000001.42.6, , , ] 2024-08-18T16:00:18.315 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:00:18.314+0000 7f5269936880 0 bucket checkpoint complete 2024-08-18T16:00:18.333 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:00:18.334 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:00:18.334 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-63 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:00:18.395 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:00:18.395 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:00:18.432 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:00:18.431+0000 7f1502013880 1 bucket sync caught up with source: 2024-08-18T16:00:18.432 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000001.67.6, , 00000000001.72.6, , 00000000001.74.6, , , 00000000001.42.6, , , ] 2024-08-18T16:00:18.432 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000001.67.6, , 00000000001.72.6, , 00000000001.74.6, , , 00000000001.42.6, , , ] 2024-08-18T16:00:18.432 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:00:18.431+0000 7f1502013880 0 bucket checkpoint complete 2024-08-18T16:00:18.446 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-63', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:00:18.446 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-63', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:00:18.446 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-63 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:00:18.500 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:00:18.501 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:00:18.536 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:00:18.536+0000 7fe056626880 1 bucket sync caught up with source: 2024-08-18T16:00:18.536 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000001.171.6, , 00000000001.321.6, , 00000000001.390.6, , , 00000000001.266.6, , , ] 2024-08-18T16:00:18.536 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000001.171.6, , 00000000001.321.6, , 00000000001.390.6, , , 00000000001.266.6, , , ] 2024-08-18T16:00:18.537 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:00:18.536+0000 7fe056626880 0 bucket checkpoint complete 2024-08-18T16:00:18.549 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-63', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:00:18.550 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-63', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:00:18.550 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-63 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:00:18.603 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:00:18.603 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:00:18.637 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:00:18.636+0000 7f882ee98880 1 bucket sync caught up with source: 2024-08-18T16:00:18.637 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000001.171.6, , 00000000001.321.6, , 00000000001.390.6, , , 00000000001.266.6, , , ] 2024-08-18T16:00:18.637 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000001.171.6, , 00000000001.321.6, , 00000000001.390.6, , , 00000000001.266.6, , , ] 2024-08-18T16:00:18.637 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:00:18.636+0000 7f882ee98880 0 bucket checkpoint complete 2024-08-18T16:00:18.650 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-63', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:00:18.650 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-63', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:00:18.650 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-63 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:00:18.704 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:00:18.704 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:00:18.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:00:18.736+0000 7f5c33d19880 1 bucket sync caught up with source: 2024-08-18T16:00:18.737 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000001.192.6, , 00000000001.148.6, , 00000000001.180.6, , , 00000000001.142.6, , , ] 2024-08-18T16:00:18.737 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000001.192.6, , 00000000001.148.6, , 00000000001.180.6, , , 00000000001.142.6, , , ] 2024-08-18T16:00:18.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:00:18.736+0000 7f5c33d19880 0 bucket checkpoint complete 2024-08-18T16:00:18.750 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-63', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:00:18.750 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-63', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:00:18.751 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-63 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T16:00:18.812 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:00:18.812 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:00:18.851 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:00:18.851+0000 7fac2eb42880 1 bucket sync caught up with source: 2024-08-18T16:00:18.851 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000001.192.6, , 00000000001.148.6, , 00000000001.180.6, , , 00000000001.142.6, , , ] 2024-08-18T16:00:18.852 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000001.192.6, , 00000000001.148.6, , 00000000001.180.6, , , 00000000001.142.6, , , ] 2024-08-18T16:00:18.852 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:00:18.851+0000 7fac2eb42880 0 bucket checkpoint complete 2024-08-18T16:00:18.867 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-63 zones={test-zone2, test-zone1} 2024-08-18T16:00:18.872 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:00:18.877 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-08-18T16:00:18.877 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-08-18T16:00:18.878 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-08-18T16:00:18.878 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-08-18T16:00:18.878 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:00:18.881 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-08-18T16:00:18.881 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-08-18T16:00:18.881 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-08-18T16:00:18.881 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-08-18T16:00:18.885 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:00:18.940 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:00:18.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:00:19.002 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:00:19.009 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:00:19.019 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:00:19.071 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:00:19.077 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:00:19.138 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:00:19.145 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:00:19.193 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:00:19.245 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:00:19.293 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-63 zones={test-zone2, test-zone1} 2024-08-18T16:00:19.293 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-63 zones={test-zone3, test-zone1} 2024-08-18T16:00:19.297 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:00:19.300 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-08-18T16:00:19.300 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-08-18T16:00:19.300 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-08-18T16:00:19.300 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-08-18T16:00:19.300 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:00:19.303 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-08-18T16:00:19.303 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-08-18T16:00:19.303 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-08-18T16:00:19.303 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-08-18T16:00:19.307 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:00:19.354 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:00:19.364 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:00:19.413 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:00:19.419 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:00:19.471 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:00:19.479 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:00:19.485 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:00:19.539 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:00:19.587 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:00:19.594 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:00:19.646 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:00:19.653 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-63 zones={test-zone3, test-zone1} 2024-08-18T16:00:19.653 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-63 zones={test-zone3, test-zone2} 2024-08-18T16:00:19.657 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:00:19.659 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-08-18T16:00:19.659 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-08-18T16:00:19.660 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-08-18T16:00:19.660 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-08-18T16:00:19.660 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:00:19.662 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-08-18T16:00:19.662 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-08-18T16:00:19.662 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-08-18T16:00:19.662 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-08-18T16:00:19.667 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:00:19.715 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:00:19.727 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:00:19.775 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:00:19.781 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:00:19.792 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:00:19.840 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:00:19.846 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:00:19.857 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:00:19.909 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:00:19.916 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:00:19.968 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:00:19.976 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-63 zones={test-zone3, test-zone2} 2024-08-18T16:00:19.977 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:00:19.977 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:00:19.977 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-64 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T16:00:20.034 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:00:20.034 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:00:20.068 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:00:20.068+0000 7f92695af880 1 bucket sync caught up with source: 2024-08-18T16:00:20.068 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000001.98.6, , 00000000001.61.6, , 00000000001.59.6, , , 00000000001.47.6, , , ] 2024-08-18T16:00:20.068 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000001.98.6, , 00000000001.61.6, , 00000000001.59.6, , , 00000000001.47.6, , , ] 2024-08-18T16:00:20.068 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:00:20.068+0000 7f92695af880 0 bucket checkpoint complete 2024-08-18T16:00:20.083 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:00:20.083 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:00:20.083 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-64 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:00:20.136 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:00:20.137 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:00:20.169 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:00:20.169+0000 7fb79a812880 1 bucket sync caught up with source: 2024-08-18T16:00:20.170 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000001.98.6, , 00000000001.61.6, , 00000000001.59.6, , , 00000000001.47.6, , , ] 2024-08-18T16:00:20.170 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000001.98.6, , 00000000001.61.6, , 00000000001.59.6, , , 00000000001.47.6, , , ] 2024-08-18T16:00:20.170 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:00:20.169+0000 7fb79a812880 0 bucket checkpoint complete 2024-08-18T16:00:20.183 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-64', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:00:20.183 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-64', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:00:20.183 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-64 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:00:20.239 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:00:20.239 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:00:20.274 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:00:20.273+0000 7fb309f20880 1 bucket sync caught up with source: 2024-08-18T16:00:20.274 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000001.416.6, , 00000000001.316.6, , 00000000001.317.6, , , 00000000001.297.6, , , ] 2024-08-18T16:00:20.274 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000001.416.6, , 00000000001.316.6, , 00000000001.317.6, , , 00000000001.297.6, , , ] 2024-08-18T16:00:20.274 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:00:20.273+0000 7fb309f20880 0 bucket checkpoint complete 2024-08-18T16:00:20.287 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-64', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:00:20.287 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-64', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:00:20.288 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-64 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:00:20.341 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:00:20.341 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:00:20.374 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:00:20.374+0000 7fd6c080f880 1 bucket sync caught up with source: 2024-08-18T16:00:20.375 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000001.416.6, , 00000000001.316.6, , 00000000001.317.6, , , 00000000001.297.6, , , ] 2024-08-18T16:00:20.375 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000001.416.6, , 00000000001.316.6, , 00000000001.317.6, , , 00000000001.297.6, , , ] 2024-08-18T16:00:20.375 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:00:20.374+0000 7fd6c080f880 0 bucket checkpoint complete 2024-08-18T16:00:20.388 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-64', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:00:20.388 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-64', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:00:20.388 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-64 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:00:20.441 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:00:20.441 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:00:20.475 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:00:20.475+0000 7f0ba820f880 1 bucket sync caught up with source: 2024-08-18T16:00:20.476 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000001.213.6, , 00000000001.161.6, , 00000000001.271.6, , , 00000000001.182.6, , , ] 2024-08-18T16:00:20.476 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000001.213.6, , 00000000001.161.6, , 00000000001.271.6, , , 00000000001.182.6, , , ] 2024-08-18T16:00:20.476 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:00:20.475+0000 7f0ba820f880 0 bucket checkpoint complete 2024-08-18T16:00:20.489 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-64', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:00:20.489 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-64', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:00:20.489 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-64 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T16:00:20.545 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:00:20.545 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:00:20.580 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:00:20.580+0000 7fefc1826880 1 bucket sync caught up with source: 2024-08-18T16:00:20.581 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000001.213.6, , 00000000001.161.6, , 00000000001.271.6, , , 00000000001.182.6, , , ] 2024-08-18T16:00:20.581 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000001.213.6, , 00000000001.161.6, , 00000000001.271.6, , , 00000000001.182.6, , , ] 2024-08-18T16:00:20.581 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:00:20.580+0000 7fefc1826880 0 bucket checkpoint complete 2024-08-18T16:00:20.595 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-64 zones={test-zone2, test-zone1} 2024-08-18T16:00:20.599 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:00:20.602 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-08-18T16:00:20.602 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-08-18T16:00:20.602 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-08-18T16:00:20.602 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-08-18T16:00:20.602 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:00:20.604 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-08-18T16:00:20.604 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-08-18T16:00:20.604 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-08-18T16:00:20.605 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-08-18T16:00:20.609 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:00:20.659 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:00:20.670 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:00:20.678 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:00:20.726 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:00:20.736 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:00:20.785 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:00:20.791 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:00:20.844 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:00:20.897 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:00:20.904 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:00:20.914 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:00:20.970 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-64 zones={test-zone2, test-zone1} 2024-08-18T16:00:20.970 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-64 zones={test-zone3, test-zone1} 2024-08-18T16:00:20.975 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:00:20.977 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-08-18T16:00:20.978 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-08-18T16:00:20.978 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-08-18T16:00:20.978 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-08-18T16:00:20.978 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:00:20.980 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-08-18T16:00:20.980 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-08-18T16:00:20.980 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-08-18T16:00:20.980 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-08-18T16:00:20.986 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:00:21.035 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:00:21.090 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:00:21.097 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:00:21.104 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:00:21.156 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:00:21.163 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:00:21.218 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:00:21.228 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:00:21.276 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:00:21.282 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:00:21.293 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:00:21.300 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-64 zones={test-zone3, test-zone1} 2024-08-18T16:00:21.300 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-64 zones={test-zone3, test-zone2} 2024-08-18T16:00:21.303 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:00:21.306 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-08-18T16:00:21.306 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-08-18T16:00:21.306 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-08-18T16:00:21.306 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-08-18T16:00:21.306 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:00:21.308 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-08-18T16:00:21.309 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-08-18T16:00:21.309 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-08-18T16:00:21.309 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-08-18T16:00:21.313 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:00:21.364 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:00:21.374 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:00:21.423 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:00:21.429 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:00:21.482 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:00:21.531 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:00:21.538 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:00:21.589 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:00:21.597 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:00:21.649 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:00:21.710 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:00:21.717 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-64 zones={test-zone3, test-zone2} 2024-08-18T16:00:21.718 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'kibgmb-62', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:00:21.718 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'kibgmb-62', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:00:21.718 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket kibgmb-62 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:00:21.766 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:00:21.766 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:00:21.834 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'kibgmb-62', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:00:21.834 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'kibgmb-62', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:00:21.834 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket kibgmb-62 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:00:21.882 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:00:21.882 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:00:21.950 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'kibgmb-63', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:00:21.950 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'kibgmb-63', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:00:21.951 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket kibgmb-63 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:00:21.998 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:00:21.999 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:00:22.068 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'kibgmb-63', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:00:22.068 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'kibgmb-63', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:00:22.068 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket kibgmb-63 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:00:22.117 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:00:22.117 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:00:22.186 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'kibgmb-64', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:00:22.187 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'kibgmb-64', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:00:22.187 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket kibgmb-64 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:00:22.236 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:00:22.236 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:00:22.303 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'kibgmb-64', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:00:22.303 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'kibgmb-64', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:00:22.303 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket kibgmb-64 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:00:22.351 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:00:22.352 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:00:22.665 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:00:22.666 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:00:22.666 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-62 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T16:00:22.721 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:00:22.721 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:00:22.757 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:00:22.757+0000 7fbea4a31880 1 waiting for incremental sync to catch up: 2024-08-18T16:00:22.758 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000001.52.6, , 00000000001.33.6, , 00000000001.96.6, , , 00000000001.49.6, , , ] 2024-08-18T16:00:22.758 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000004.56.6, 00000000002.46.1, 00000000004.39.6, 00000000002.40.1, 00000000004.104.6, 00000000002.64.1, 00000000002.74.1, 00000000004.53.6, 00000000002.63.1, 00000000002.69.1, 00000000002.101.1] 2024-08-18T16:00:27.759 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:00:27.758+0000 7fbea4a31880 1 waiting for incremental sync to catch up: 2024-08-18T16:00:27.759 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000001.52.6, , 00000000001.33.6, , 00000000001.96.6, , , 00000000001.49.6, , , ] 2024-08-18T16:00:27.759 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000004.56.6, 00000000002.46.1, 00000000004.39.6, 00000000002.40.1, 00000000004.104.6, 00000000002.64.1, 00000000002.74.1, 00000000004.53.6, 00000000002.63.1, 00000000002.69.1, 00000000002.101.1] 2024-08-18T16:00:32.760 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:00:32.759+0000 7fbea4a31880 1 waiting for incremental sync to catch up: 2024-08-18T16:00:32.760 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000001.52.6, , 00000000001.33.6, , 00000000001.96.6, , , 00000000001.49.6, , , ] 2024-08-18T16:00:32.760 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000004.56.6, 00000000002.46.1, 00000000004.39.6, 00000000002.40.1, 00000000004.104.6, 00000000002.64.1, 00000000002.74.1, 00000000004.53.6, 00000000002.63.1, 00000000002.69.1, 00000000002.101.1] 2024-08-18T16:00:37.761 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:00:37.761+0000 7fbea4a31880 1 bucket sync caught up with source: 2024-08-18T16:00:37.762 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000004.56.6, 00000000002.46.1, 00000000004.39.6, 00000000002.40.1, 00000000004.104.6, 00000000002.64.1, 00000000002.74.1, 00000000004.53.6, 00000000002.63.1, 00000000002.69.1, 00000000002.101.1] 2024-08-18T16:00:37.762 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000004.56.6, 00000000002.46.1, 00000000004.39.6, 00000000002.40.1, 00000000004.104.6, 00000000002.64.1, 00000000002.74.1, 00000000004.53.6, 00000000002.63.1, 00000000002.69.1, 00000000002.101.1] 2024-08-18T16:00:37.762 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:00:37.761+0000 7fbea4a31880 0 bucket checkpoint complete 2024-08-18T16:00:37.781 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:00:37.781 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:00:37.781 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-62 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:00:37.842 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:00:37.842 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:00:37.879 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:00:37.877+0000 7f4b82211880 1 bucket sync caught up with source: 2024-08-18T16:00:37.879 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000004.56.6, 00000000002.46.1, 00000000004.39.6, 00000000002.40.1, 00000000004.104.6, 00000000002.64.1, 00000000002.74.1, 00000000004.53.6, 00000000002.63.1, 00000000002.69.1, 00000000002.101.1] 2024-08-18T16:00:37.879 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000004.56.6, 00000000002.46.1, 00000000004.39.6, 00000000002.40.1, 00000000004.104.6, 00000000002.64.1, 00000000002.74.1, 00000000004.53.6, 00000000002.63.1, 00000000002.69.1, 00000000002.101.1] 2024-08-18T16:00:37.879 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:00:37.877+0000 7f4b82211880 0 bucket checkpoint complete 2024-08-18T16:00:37.892 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-62', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:00:37.892 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-62', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:00:37.893 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-62 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:00:37.946 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:00:37.946 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:00:37.980 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:00:37.979+0000 7f2cd1542880 1 waiting for incremental sync to catch up: 2024-08-18T16:00:37.980 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000001.207.6, , 00000000002.235.6, , 00000000001.418.6, , , 00000000002.109.6, , , ] 2024-08-18T16:00:37.981 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000002.220.6, , 00000000002.235.6, , 00000000002.443.6, , , 00000000002.109.6, , , ] 2024-08-18T16:00:42.982 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:00:42.981+0000 7f2cd1542880 1 waiting for incremental sync to catch up: 2024-08-18T16:00:42.982 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000001.207.6, , 00000000002.235.6, , 00000000001.418.6, , , 00000000002.109.6, , , ] 2024-08-18T16:00:42.982 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000002.220.6, , 00000000002.235.6, , 00000000002.443.6, , , 00000000002.109.6, , , ] 2024-08-18T16:00:47.983 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:00:47.982+0000 7f2cd1542880 1 bucket sync caught up with source: 2024-08-18T16:00:47.983 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000002.220.6, , 00000000002.235.6, , 00000000002.443.6, , , 00000000002.109.6, , , ] 2024-08-18T16:00:47.983 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000002.220.6, , 00000000002.235.6, , 00000000002.443.6, , , 00000000002.109.6, , , ] 2024-08-18T16:00:47.983 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:00:47.982+0000 7f2cd1542880 0 bucket checkpoint complete 2024-08-18T16:00:48.000 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-62', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:00:48.000 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-62', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:00:48.000 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-62 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:00:48.059 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:00:48.060 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:00:48.096 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:00:48.095+0000 7f2783919880 1 bucket sync caught up with source: 2024-08-18T16:00:48.096 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000002.220.6, , 00000000002.235.6, , 00000000002.443.6, , , 00000000002.109.6, , , ] 2024-08-18T16:00:48.096 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000002.220.6, , 00000000002.235.6, , 00000000002.443.6, , , 00000000002.109.6, , , ] 2024-08-18T16:00:48.096 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:00:48.095+0000 7f2783919880 0 bucket checkpoint complete 2024-08-18T16:00:48.109 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-62', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:00:48.109 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-62', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:00:48.110 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-62 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:00:48.163 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:00:48.164 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:00:48.198 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:00:48.197+0000 7fe409934880 1 bucket sync caught up with source: 2024-08-18T16:00:48.198 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000002.139.6, , 00000000002.158.6, , 00000000002.228.6, , , 00000000002.134.6, , , ] 2024-08-18T16:00:48.198 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000002.139.6, , 00000000002.158.6, , 00000000002.228.6, , , 00000000002.134.6, , , ] 2024-08-18T16:00:48.198 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:00:48.197+0000 7fe409934880 0 bucket checkpoint complete 2024-08-18T16:00:48.211 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-62', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:00:48.211 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-62', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:00:48.211 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-62 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T16:00:48.272 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:00:48.272 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:00:48.311 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:00:48.310+0000 7fc03c60e880 1 waiting for incremental sync to catch up: 2024-08-18T16:00:48.311 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000001.134.6, , 00000000001.150.6, , 00000000002.228.6, , , 00000000002.134.6, , , ] 2024-08-18T16:00:48.311 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000002.139.6, , 00000000002.158.6, , 00000000002.228.6, , , 00000000002.134.6, , , ] 2024-08-18T16:00:53.312 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:00:53.312+0000 7fc03c60e880 1 waiting for incremental sync to catch up: 2024-08-18T16:00:53.312 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000001.134.6, , 00000000001.150.6, , 00000000002.228.6, , , 00000000002.134.6, , , ] 2024-08-18T16:00:53.312 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000002.139.6, , 00000000002.158.6, , 00000000002.228.6, , , 00000000002.134.6, , , ] 2024-08-18T16:00:58.313 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:00:58.313+0000 7fc03c60e880 1 bucket sync caught up with source: 2024-08-18T16:00:58.314 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000002.139.6, , 00000000002.158.6, , 00000000002.228.6, , , 00000000002.134.6, , , ] 2024-08-18T16:00:58.314 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000002.139.6, , 00000000002.158.6, , 00000000002.228.6, , , 00000000002.134.6, , , ] 2024-08-18T16:00:58.314 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:00:58.313+0000 7fc03c60e880 0 bucket checkpoint complete 2024-08-18T16:00:58.332 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-62 zones={test-zone2, test-zone1} 2024-08-18T16:00:58.340 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:00:58.344 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-08-18T16:00:58.344 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-08-18T16:00:58.344 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-08-18T16:00:58.344 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-08-18T16:00:58.344 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-08-18T16:00:58.344 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-08-18T16:00:58.344 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-08-18T16:00:58.344 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-08-18T16:00:58.344 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:00:58.347 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-08-18T16:00:58.347 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-08-18T16:00:58.347 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-08-18T16:00:58.347 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-08-18T16:00:58.348 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-08-18T16:00:58.348 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-08-18T16:00:58.348 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-08-18T16:00:58.348 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-08-18T16:00:58.353 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:00:58.405 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:00:58.461 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:00:58.513 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:00:58.521 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:00:58.583 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:00:58.640 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:00:58.648 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:00:58.661 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:00:58.713 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:00:58.721 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:00:58.777 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:00:58.787 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-08-18T16:00:58.836 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-08-18T16:00:58.893 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-08-18T16:00:58.945 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-08-18T16:00:58.953 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-08-18T16:00:59.013 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-08-18T16:00:59.069 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-08-18T16:00:59.077 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-08-18T16:00:59.133 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-08-18T16:00:59.185 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-08-18T16:00:59.193 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-08-18T16:00:59.254 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-08-18T16:00:59.263 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-62 zones={test-zone2, test-zone1} 2024-08-18T16:00:59.263 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-62 zones={test-zone3, test-zone1} 2024-08-18T16:00:59.268 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:00:59.270 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-08-18T16:00:59.270 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-08-18T16:00:59.271 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-08-18T16:00:59.271 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-08-18T16:00:59.271 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-08-18T16:00:59.271 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-08-18T16:00:59.271 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-08-18T16:00:59.271 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-08-18T16:00:59.271 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:00:59.274 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-08-18T16:00:59.274 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-08-18T16:00:59.274 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-08-18T16:00:59.274 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-08-18T16:00:59.274 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-08-18T16:00:59.274 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-08-18T16:00:59.274 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-08-18T16:00:59.274 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-08-18T16:00:59.280 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:00:59.330 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:00:59.433 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:00:59.530 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:00:59.584 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:00:59.681 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:00:59.777 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:00:59.832 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:00:59.929 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:01:00.021 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:01:00.072 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:01:00.169 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:01:00.265 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-08-18T16:01:00.273 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-08-18T16:01:00.328 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-08-18T16:01:00.382 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-08-18T16:01:00.389 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-08-18T16:01:00.449 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-08-18T16:01:00.501 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-08-18T16:01:00.508 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-08-18T16:01:00.561 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-08-18T16:01:00.614 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-08-18T16:01:00.623 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-08-18T16:01:00.679 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-08-18T16:01:00.727 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-62 zones={test-zone3, test-zone1} 2024-08-18T16:01:00.727 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-62 zones={test-zone3, test-zone2} 2024-08-18T16:01:00.732 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:01:00.735 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-08-18T16:01:00.735 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-08-18T16:01:00.735 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-08-18T16:01:00.735 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-08-18T16:01:00.735 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-08-18T16:01:00.735 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-08-18T16:01:00.735 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-08-18T16:01:00.735 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-08-18T16:01:00.735 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:01:00.738 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-08-18T16:01:00.738 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-08-18T16:01:00.739 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-08-18T16:01:00.739 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-08-18T16:01:00.739 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-08-18T16:01:00.739 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-08-18T16:01:00.739 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-08-18T16:01:00.739 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-08-18T16:01:00.744 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:01:00.792 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:01:00.889 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:01:00.947 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:01:01.096 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:01:01.154 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:01:01.250 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:01:01.306 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:01:01.362 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:01:01.468 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:01:01.532 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:01:01.598 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:01:01.658 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-08-18T16:01:01.720 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-08-18T16:01:01.782 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-08-18T16:01:01.792 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-08-18T16:01:01.843 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-08-18T16:01:01.860 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-08-18T16:01:01.913 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-08-18T16:01:01.921 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-08-18T16:01:01.979 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-08-18T16:01:02.033 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-08-18T16:01:02.043 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-08-18T16:01:02.110 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-08-18T16:01:02.168 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-62 zones={test-zone3, test-zone2} 2024-08-18T16:01:02.169 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:01:02.169 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:01:02.169 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-63 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T16:01:02.232 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:01:02.232 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:01:02.271 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:01:02.271+0000 7fc941527880 1 bucket sync caught up with source: 2024-08-18T16:01:02.271 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000004.75.6, 00000000002.65.1, 00000000004.78.6, 00000000002.26.1, 00000000004.78.6, 00000000002.37.1, 00000000002.73.1, 00000000004.46.6, 00000000002.48.1, 00000000002.27.1, 00000000002.49.1] 2024-08-18T16:01:02.272 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000004.75.6, 00000000002.65.1, 00000000004.78.6, 00000000002.26.1, 00000000004.78.6, 00000000002.37.1, 00000000002.73.1, 00000000004.46.6, 00000000002.48.1, 00000000002.27.1, 00000000002.49.1] 2024-08-18T16:01:02.272 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:01:02.271+0000 7fc941527880 0 bucket checkpoint complete 2024-08-18T16:01:02.287 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:01:02.287 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:01:02.287 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-63 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:01:02.349 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:01:02.349 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:01:02.386 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:01:02.385+0000 7ff254498880 1 bucket sync caught up with source: 2024-08-18T16:01:02.386 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000004.75.6, 00000000002.65.1, 00000000004.78.6, 00000000002.26.1, 00000000004.78.6, 00000000002.37.1, 00000000002.73.1, 00000000004.46.6, 00000000002.48.1, 00000000002.27.1, 00000000002.49.1] 2024-08-18T16:01:02.386 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000004.75.6, 00000000002.65.1, 00000000004.78.6, 00000000002.26.1, 00000000004.78.6, 00000000002.37.1, 00000000002.73.1, 00000000004.46.6, 00000000002.48.1, 00000000002.27.1, 00000000002.49.1] 2024-08-18T16:01:02.387 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:01:02.385+0000 7ff254498880 0 bucket checkpoint complete 2024-08-18T16:01:02.399 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-63', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:01:02.400 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-63', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:01:02.400 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-63 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:01:02.454 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:01:02.454 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:01:02.488 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:01:02.488+0000 7f5588c0f880 1 bucket sync caught up with source: 2024-08-18T16:01:02.488 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000002.177.6, , 00000000002.326.6, , 00000000002.393.6, , , 00000000002.269.6, , , ] 2024-08-18T16:01:02.489 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000002.177.6, , 00000000002.326.6, , 00000000002.393.6, , , 00000000002.269.6, , , ] 2024-08-18T16:01:02.489 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:01:02.488+0000 7f5588c0f880 0 bucket checkpoint complete 2024-08-18T16:01:02.502 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-63', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:01:02.503 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-63', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:01:02.503 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-63 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:01:02.557 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:01:02.557 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:01:02.591 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:01:02.591+0000 7f2ea0a24880 1 bucket sync caught up with source: 2024-08-18T16:01:02.591 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000002.177.6, , 00000000002.326.6, , 00000000002.393.6, , , 00000000002.269.6, , , ] 2024-08-18T16:01:02.591 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000002.177.6, , 00000000002.326.6, , 00000000002.393.6, , , 00000000002.269.6, , , ] 2024-08-18T16:01:02.592 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:01:02.591+0000 7f2ea0a24880 0 bucket checkpoint complete 2024-08-18T16:01:02.604 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-63', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:01:02.605 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-63', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:01:02.605 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-63 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:01:02.658 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:01:02.658 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:01:02.691 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:01:02.691+0000 7f8160130880 1 bucket sync caught up with source: 2024-08-18T16:01:02.692 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000002.203.6, , 00000000002.156.6, , 00000000002.185.6, , , 00000000002.147.6, , , ] 2024-08-18T16:01:02.692 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000002.203.6, , 00000000002.156.6, , 00000000002.185.6, , , 00000000002.147.6, , , ] 2024-08-18T16:01:02.692 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:01:02.691+0000 7f8160130880 0 bucket checkpoint complete 2024-08-18T16:01:02.705 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-63', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:01:02.705 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-63', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:01:02.705 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-63 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T16:01:02.763 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:01:02.763 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:01:02.801 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:01:02.800+0000 7fbc92010880 1 bucket sync caught up with source: 2024-08-18T16:01:02.801 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000002.203.6, , 00000000002.156.6, , 00000000002.185.6, , , 00000000002.147.6, , , ] 2024-08-18T16:01:02.801 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000002.203.6, , 00000000002.156.6, , 00000000002.185.6, , , 00000000002.147.6, , , ] 2024-08-18T16:01:02.801 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:01:02.800+0000 7fbc92010880 0 bucket checkpoint complete 2024-08-18T16:01:02.816 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-63 zones={test-zone2, test-zone1} 2024-08-18T16:01:02.821 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:01:02.825 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-08-18T16:01:02.825 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-08-18T16:01:02.825 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-08-18T16:01:02.825 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-08-18T16:01:02.825 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-08-18T16:01:02.826 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-08-18T16:01:02.826 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-08-18T16:01:02.826 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-08-18T16:01:02.826 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:01:02.829 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-08-18T16:01:02.830 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-08-18T16:01:02.830 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-08-18T16:01:02.830 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-08-18T16:01:02.830 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-08-18T16:01:02.830 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-08-18T16:01:02.830 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-08-18T16:01:02.831 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-08-18T16:01:02.837 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:01:02.843 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:01:02.896 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:01:02.945 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:01:02.951 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:01:02.962 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:01:03.011 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:01:03.019 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:01:03.075 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:01:03.131 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:01:03.137 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:01:03.193 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:01:03.245 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-08-18T16:01:03.251 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-08-18T16:01:03.302 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-08-18T16:01:03.310 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-08-18T16:01:03.359 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-08-18T16:01:03.414 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-08-18T16:01:03.422 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-08-18T16:01:03.428 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-08-18T16:01:03.482 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-08-18T16:01:03.489 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-08-18T16:01:03.543 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-08-18T16:01:03.598 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-08-18T16:01:03.606 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-63 zones={test-zone2, test-zone1} 2024-08-18T16:01:03.606 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-63 zones={test-zone3, test-zone1} 2024-08-18T16:01:03.610 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:01:03.612 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-08-18T16:01:03.612 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-08-18T16:01:03.612 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-08-18T16:01:03.612 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-08-18T16:01:03.613 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-08-18T16:01:03.613 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-08-18T16:01:03.613 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-08-18T16:01:03.613 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-08-18T16:01:03.613 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:01:03.615 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-08-18T16:01:03.615 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-08-18T16:01:03.615 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-08-18T16:01:03.616 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-08-18T16:01:03.616 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-08-18T16:01:03.616 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-08-18T16:01:03.616 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-08-18T16:01:03.616 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-08-18T16:01:03.622 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:01:03.628 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:01:03.684 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:01:03.691 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:01:03.738 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:01:03.749 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:01:03.806 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:01:03.812 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:01:03.822 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:01:03.871 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:01:03.877 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:01:03.930 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:01:03.937 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-08-18T16:01:03.987 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-08-18T16:01:04.039 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-08-18T16:01:04.089 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-08-18T16:01:04.098 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-08-18T16:01:04.152 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-08-18T16:01:04.159 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-08-18T16:01:04.207 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-08-18T16:01:04.267 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-08-18T16:01:04.362 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-08-18T16:01:04.371 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-08-18T16:01:04.426 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-08-18T16:01:04.434 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-63 zones={test-zone3, test-zone1} 2024-08-18T16:01:04.434 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-63 zones={test-zone3, test-zone2} 2024-08-18T16:01:04.438 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:01:04.441 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-08-18T16:01:04.441 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-08-18T16:01:04.441 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-08-18T16:01:04.441 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-08-18T16:01:04.441 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-08-18T16:01:04.442 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-08-18T16:01:04.442 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-08-18T16:01:04.442 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-08-18T16:01:04.442 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:01:04.445 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-08-18T16:01:04.445 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-08-18T16:01:04.445 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-08-18T16:01:04.445 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-08-18T16:01:04.445 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-08-18T16:01:04.445 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-08-18T16:01:04.445 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-08-18T16:01:04.445 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-08-18T16:01:04.450 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:01:04.503 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:01:04.558 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:01:04.611 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:01:04.619 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:01:04.671 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:01:04.720 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:01:04.727 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:01:04.783 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:01:04.832 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:01:04.838 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:01:04.891 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:01:04.944 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-08-18T16:01:04.950 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-08-18T16:01:05.049 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-08-18T16:01:05.146 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-08-18T16:01:05.153 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-08-18T16:01:05.166 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-08-18T16:01:05.218 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-08-18T16:01:05.268 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-08-18T16:01:05.325 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-08-18T16:01:05.378 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-08-18T16:01:05.385 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-08-18T16:01:05.441 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-08-18T16:01:05.490 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-63 zones={test-zone3, test-zone2} 2024-08-18T16:01:05.490 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:01:05.490 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:01:05.490 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-64 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T16:01:05.553 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:01:05.554 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:01:05.592 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:01:05.592+0000 7f4de2b27880 1 bucket sync caught up with source: 2024-08-18T16:01:05.593 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000004.106.6, 00000000002.96.1, 00000000004.67.6, 00000000002.104.1, 00000000004.63.6, 00000000002.83.1, 00000000002.66.1, 00000000004.53.6, 00000000002.61.1, 00000000002.89.1, 00000000002.105.1] 2024-08-18T16:01:05.593 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000004.106.6, 00000000002.96.1, 00000000004.67.6, 00000000002.104.1, 00000000004.63.6, 00000000002.83.1, 00000000002.66.1, 00000000004.53.6, 00000000002.61.1, 00000000002.89.1, 00000000002.105.1] 2024-08-18T16:01:05.593 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:01:05.592+0000 7f4de2b27880 0 bucket checkpoint complete 2024-08-18T16:01:05.608 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:01:05.608 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:01:05.608 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-64 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:01:05.662 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:01:05.662 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:01:05.695 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:01:05.695+0000 7f91a04bb880 1 bucket sync caught up with source: 2024-08-18T16:01:05.696 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000004.106.6, 00000000002.96.1, 00000000004.67.6, 00000000002.104.1, 00000000004.63.6, 00000000002.83.1, 00000000002.66.1, 00000000004.53.6, 00000000002.61.1, 00000000002.89.1, 00000000002.105.1] 2024-08-18T16:01:05.696 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000004.106.6, 00000000002.96.1, 00000000004.67.6, 00000000002.104.1, 00000000004.63.6, 00000000002.83.1, 00000000002.66.1, 00000000004.53.6, 00000000002.61.1, 00000000002.89.1, 00000000002.105.1] 2024-08-18T16:01:05.696 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:01:05.695+0000 7f91a04bb880 0 bucket checkpoint complete 2024-08-18T16:01:05.708 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-64', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:01:05.708 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-64', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:01:05.708 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-64 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:01:05.761 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:01:05.761 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:01:05.796 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:01:05.795+0000 7fa5a5010880 1 bucket sync caught up with source: 2024-08-18T16:01:05.796 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000002.441.6, , 00000000002.339.6, , 00000000002.329.6, , , 00000000002.320.6, , , ] 2024-08-18T16:01:05.796 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000002.441.6, , 00000000002.339.6, , 00000000002.329.6, , , 00000000002.320.6, , , ] 2024-08-18T16:01:05.796 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:01:05.795+0000 7fa5a5010880 0 bucket checkpoint complete 2024-08-18T16:01:05.809 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-64', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:01:05.810 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-64', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:01:05.810 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-64 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:01:05.862 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:01:05.863 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:01:05.896 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:01:05.896+0000 7fb225926880 1 bucket sync caught up with source: 2024-08-18T16:01:05.896 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000002.441.6, , 00000000002.339.6, , 00000000002.329.6, , , 00000000002.320.6, , , ] 2024-08-18T16:01:05.897 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000002.441.6, , 00000000002.339.6, , 00000000002.329.6, , , 00000000002.320.6, , , ] 2024-08-18T16:01:05.897 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:01:05.896+0000 7fb225926880 0 bucket checkpoint complete 2024-08-18T16:01:05.909 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-64', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:01:05.909 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-64', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:01:05.910 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-64 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:01:05.963 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:01:05.963 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:01:05.997 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:01:05.996+0000 7f2b93731880 1 bucket sync caught up with source: 2024-08-18T16:01:05.997 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000002.219.6, , 00000000002.165.6, , 00000000002.274.6, , , 00000000002.186.6, , , ] 2024-08-18T16:01:05.997 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000002.219.6, , 00000000002.165.6, , 00000000002.274.6, , , 00000000002.186.6, , , ] 2024-08-18T16:01:05.997 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:01:05.996+0000 7f2b93731880 0 bucket checkpoint complete 2024-08-18T16:01:06.011 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-64', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:01:06.011 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-64', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:01:06.011 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-64 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T16:01:06.066 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:01:06.066 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:01:06.102 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:01:06.101+0000 7fbbf252f880 1 bucket sync caught up with source: 2024-08-18T16:01:06.102 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000002.219.6, , 00000000002.165.6, , 00000000002.274.6, , , 00000000002.186.6, , , ] 2024-08-18T16:01:06.102 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000002.219.6, , 00000000002.165.6, , 00000000002.274.6, , , 00000000002.186.6, , , ] 2024-08-18T16:01:06.102 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:01:06.101+0000 7fbbf252f880 0 bucket checkpoint complete 2024-08-18T16:01:06.116 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-64 zones={test-zone2, test-zone1} 2024-08-18T16:01:06.121 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:01:06.124 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-08-18T16:01:06.124 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-08-18T16:01:06.124 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-08-18T16:01:06.124 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-08-18T16:01:06.124 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-08-18T16:01:06.124 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-08-18T16:01:06.124 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-08-18T16:01:06.124 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-08-18T16:01:06.124 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:01:06.127 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-08-18T16:01:06.127 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-08-18T16:01:06.127 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-08-18T16:01:06.127 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-08-18T16:01:06.127 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-08-18T16:01:06.127 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-08-18T16:01:06.127 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-08-18T16:01:06.127 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-08-18T16:01:06.134 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:01:06.184 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:01:06.195 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:01:06.247 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:01:06.298 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:01:06.352 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:01:06.360 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:01:06.410 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:01:06.422 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:01:06.430 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:01:06.438 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:01:06.450 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:01:06.499 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-08-18T16:01:06.507 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-08-18T16:01:06.563 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-08-18T16:01:06.611 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-08-18T16:01:06.618 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-08-18T16:01:06.671 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-08-18T16:01:06.726 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-08-18T16:01:06.776 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-08-18T16:01:06.832 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-08-18T16:01:06.881 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-08-18T16:01:06.932 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-08-18T16:01:06.984 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-08-18T16:01:07.078 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-64 zones={test-zone2, test-zone1} 2024-08-18T16:01:07.078 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-64 zones={test-zone3, test-zone1} 2024-08-18T16:01:07.083 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:01:07.086 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-08-18T16:01:07.086 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-08-18T16:01:07.087 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-08-18T16:01:07.087 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-08-18T16:01:07.087 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-08-18T16:01:07.087 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-08-18T16:01:07.087 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-08-18T16:01:07.087 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-08-18T16:01:07.088 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:01:07.091 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-08-18T16:01:07.091 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-08-18T16:01:07.091 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-08-18T16:01:07.092 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-08-18T16:01:07.092 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-08-18T16:01:07.092 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-08-18T16:01:07.092 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-08-18T16:01:07.092 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-08-18T16:01:07.098 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:01:07.105 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:01:07.115 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:01:07.163 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:01:07.170 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:01:07.223 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:01:07.271 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:01:07.278 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:01:07.331 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:01:07.379 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:01:07.385 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:01:07.442 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:01:07.535 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-08-18T16:01:07.542 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-08-18T16:01:07.553 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-08-18T16:01:07.605 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-08-18T16:01:07.611 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-08-18T16:01:07.668 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-08-18T16:01:07.717 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-08-18T16:01:07.724 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-08-18T16:01:07.777 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-08-18T16:01:07.827 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-08-18T16:01:07.833 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-08-18T16:01:07.843 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-08-18T16:01:07.891 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-64 zones={test-zone3, test-zone1} 2024-08-18T16:01:07.891 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-64 zones={test-zone3, test-zone2} 2024-08-18T16:01:07.896 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:01:07.898 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-08-18T16:01:07.898 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-08-18T16:01:07.898 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-08-18T16:01:07.898 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-08-18T16:01:07.898 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-08-18T16:01:07.898 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-08-18T16:01:07.899 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-08-18T16:01:07.899 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-08-18T16:01:07.899 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:01:07.902 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-08-18T16:01:07.902 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-08-18T16:01:07.902 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-08-18T16:01:07.902 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-08-18T16:01:07.902 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-08-18T16:01:07.902 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-08-18T16:01:07.902 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-08-18T16:01:07.902 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-08-18T16:01:07.909 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:01:07.917 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:01:07.973 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:01:08.026 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:01:08.033 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:01:08.089 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:01:08.142 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:01:08.150 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:01:08.161 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:01:08.212 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:01:08.218 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:01:08.275 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:01:08.324 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-08-18T16:01:08.330 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-08-18T16:01:08.386 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-08-18T16:01:08.439 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-08-18T16:01:08.446 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-08-18T16:01:08.458 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-08-18T16:01:08.551 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-08-18T16:01:08.558 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-08-18T16:01:08.569 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-08-18T16:01:08.621 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-08-18T16:01:08.628 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-08-18T16:01:08.681 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-08-18T16:01:08.730 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-64 zones={test-zone3, test-zone2} 2024-08-18T16:01:08.730 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:01:08.730 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:01:08.731 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:01:08.777 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:01:08.777 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:01:08.825 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T16:01:08.825 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:01:08.825 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000008276", 2024-08-18T16:01:08.825 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:00:34.259637Z" 2024-08-18T16:01:08.825 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:01:08.825 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:01:08.825 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000007415", 2024-08-18T16:01:08.825 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:00:34.352331Z" 2024-08-18T16:01:08.825 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:01:08.825 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:01:08.826 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000013190", 2024-08-18T16:01:08.826 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:00:25.222256Z" 2024-08-18T16:01:08.826 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:01:08.826 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:01:08.826 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000012344", 2024-08-18T16:01:08.826 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:00:22.401220Z" 2024-08-18T16:01:08.826 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:01:08.826 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T16:01:08.827 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-08-18T16:01:08.827 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:01:08.827 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:01:08.828 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:01:08.884 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:01:08.884 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:01:08.934 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T16:01:08.934 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T16:01:08.934 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T16:01:08.935 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T16:01:08.935 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T16:01:08.935 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 8935070562734594544 2024-08-18T16:01:08.935 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:01:08.935 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T16:01:08.935 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:01:08.935 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T16:01:08.935 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:01:08.935 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T16:01:08.935 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000008276", 2024-08-18T16:01:08.935 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:01:08.935 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:01:08.935 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:01:08.936 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:00:34.259637Z" 2024-08-18T16:01:08.936 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:01:08.936 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:01:08.936 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:01:08.936 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T16:01:08.936 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:01:08.936 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T16:01:08.936 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000007415", 2024-08-18T16:01:08.936 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:01:08.936 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:01:08.936 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:01:08.936 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:00:34.352331Z" 2024-08-18T16:01:08.937 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:01:08.937 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:01:08.937 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:01:08.937 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T16:01:08.937 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:01:08.937 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T16:01:08.937 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000013190", 2024-08-18T16:01:08.937 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:01:08.937 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:01:08.937 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:01:08.937 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:00:25.222256Z" 2024-08-18T16:01:08.937 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:01:08.937 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:01:08.938 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:01:08.938 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T16:01:08.938 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:01:08.938 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T16:01:08.938 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000012344", 2024-08-18T16:01:08.938 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:01:08.938 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:01:08.938 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:01:08.938 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:00:22.401220Z" 2024-08-18T16:01:08.938 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:01:08.938 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:01:08.938 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T16:01:08.938 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:01:08.939 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T16:01:08.939 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 0, 2024-08-18T16:01:08.939 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 0 2024-08-18T16:01:08.939 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:01:08.939 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T16:01:08Z" 2024-08-18T16:01:08.939 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T16:01:08.940 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-08-18T16:01:08.940 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:01:08.940 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:01:08.940 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:01:08.987 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:01:08.987 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:01:09.034 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T16:01:09.035 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:01:09.035 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000008276", 2024-08-18T16:01:09.035 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:00:34.259637Z" 2024-08-18T16:01:09.035 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:01:09.035 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:01:09.035 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000007415", 2024-08-18T16:01:09.035 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:00:34.352331Z" 2024-08-18T16:01:09.035 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:01:09.035 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:01:09.035 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000013190", 2024-08-18T16:01:09.036 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:00:25.222256Z" 2024-08-18T16:01:09.036 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:01:09.036 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:01:09.036 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000012344", 2024-08-18T16:01:09.036 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:00:22.401220Z" 2024-08-18T16:01:09.036 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:01:09.036 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T16:01:09.037 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2024-08-18T16:01:09.037 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:01:09.037 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:01:09.037 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:01:09.086 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:01:09.086 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:01:09.129 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T16:01:09.129 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T16:01:09.129 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T16:01:09.129 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T16:01:09.129 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T16:01:09.129 INFO:teuthology.orchestra.run.smithi134.stdout: "instance_id": 6908254841848037526 2024-08-18T16:01:09.129 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:01:09.129 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T16:01:09.129 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:01:09.129 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T16:01:09.130 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:01:09.130 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T16:01:09.130 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000008276", 2024-08-18T16:01:09.130 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:01:09.130 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:01:09.130 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:01:09.130 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:00:34.259637Z" 2024-08-18T16:01:09.130 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:01:09.130 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:01:09.130 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:01:09.130 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T16:01:09.130 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:01:09.130 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T16:01:09.131 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000007415", 2024-08-18T16:01:09.131 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:01:09.131 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:01:09.131 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:01:09.131 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:00:34.352331Z" 2024-08-18T16:01:09.131 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:01:09.131 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:01:09.131 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:01:09.131 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T16:01:09.131 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:01:09.131 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T16:01:09.131 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000013190", 2024-08-18T16:01:09.131 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:01:09.132 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:01:09.132 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:01:09.132 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:00:25.222256Z" 2024-08-18T16:01:09.132 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:01:09.132 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:01:09.132 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:01:09.132 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T16:01:09.132 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:01:09.132 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T16:01:09.132 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000012344", 2024-08-18T16:01:09.132 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:01:09.132 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:01:09.132 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:01:09.133 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:00:22.401220Z" 2024-08-18T16:01:09.133 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:01:09.133 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:01:09.133 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T16:01:09.133 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:01:09.133 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T16:01:09.133 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 0, 2024-08-18T16:01:09.133 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 0 2024-08-18T16:01:09.133 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:01:09.133 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T16:01:09Z" 2024-08-18T16:01:09.133 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T16:01:09.134 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2024-08-18T16:01:09.135 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:01:09.135 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:01:09.135 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:01:09.184 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:01:09.184 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:01:09.234 INFO:teuthology.orchestra.run.smithi193.stdout:[ 2024-08-18T16:01:09.234 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:01:09.234 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000005450", 2024-08-18T16:01:09.234 INFO:teuthology.orchestra.run.smithi193.stdout: "last_update": "2024-08-18T16:00:33.126651Z" 2024-08-18T16:01:09.234 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:01:09.235 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:01:09.235 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000005864", 2024-08-18T16:01:09.235 INFO:teuthology.orchestra.run.smithi193.stdout: "last_update": "2024-08-18T16:00:33.166932Z" 2024-08-18T16:01:09.235 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:01:09.235 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:01:09.235 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000010229", 2024-08-18T16:01:09.235 INFO:teuthology.orchestra.run.smithi193.stdout: "last_update": "2024-08-18T16:00:33.127666Z" 2024-08-18T16:01:09.235 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:01:09.235 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:01:09.235 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000008540", 2024-08-18T16:01:09.235 INFO:teuthology.orchestra.run.smithi193.stdout: "last_update": "2024-08-18T15:54:07.557744Z" 2024-08-18T16:01:09.235 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:01:09.235 INFO:teuthology.orchestra.run.smithi193.stdout:] 2024-08-18T16:01:09.236 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2024-08-18T16:01:09.237 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:01:09.237 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:01:09.237 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:01:09.286 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:01:09.286 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:01:09.329 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T16:01:09.329 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T16:01:09.329 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T16:01:09.330 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T16:01:09.330 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T16:01:09.330 INFO:teuthology.orchestra.run.smithi134.stdout: "instance_id": 15510818030922206205 2024-08-18T16:01:09.330 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:01:09.330 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T16:01:09.330 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:01:09.330 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T16:01:09.330 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:01:09.330 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T16:01:09.330 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000005450", 2024-08-18T16:01:09.330 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:01:09.330 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:01:09.331 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:01:09.331 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:00:33.126651Z" 2024-08-18T16:01:09.331 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:01:09.331 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:01:09.331 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:01:09.331 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T16:01:09.331 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:01:09.331 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T16:01:09.331 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000005864", 2024-08-18T16:01:09.331 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:01:09.331 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:01:09.331 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:01:09.332 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:00:33.166932Z" 2024-08-18T16:01:09.332 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:01:09.332 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:01:09.332 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:01:09.332 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T16:01:09.332 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:01:09.332 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T16:01:09.332 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000010229", 2024-08-18T16:01:09.332 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:01:09.332 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:01:09.332 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:01:09.332 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:00:33.127666Z" 2024-08-18T16:01:09.333 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:01:09.333 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:01:09.333 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:01:09.333 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T16:01:09.333 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:01:09.333 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T16:01:09.333 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000008540", 2024-08-18T16:01:09.333 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:01:09.333 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:01:09.333 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:01:09.333 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:54:07.557744Z" 2024-08-18T16:01:09.333 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:01:09.334 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:01:09.334 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T16:01:09.334 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:01:09.334 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T16:01:09.334 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 0, 2024-08-18T16:01:09.334 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 0 2024-08-18T16:01:09.334 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:01:09.334 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T16:01:09Z" 2024-08-18T16:01:09.334 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T16:01:09.335 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2024-08-18T16:01:09.335 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:01:09.336 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:01:09.336 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:01:09.383 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:01:09.383 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:01:09.433 INFO:teuthology.orchestra.run.smithi193.stdout:[ 2024-08-18T16:01:09.433 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:01:09.433 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000005450", 2024-08-18T16:01:09.433 INFO:teuthology.orchestra.run.smithi193.stdout: "last_update": "2024-08-18T16:00:33.126651Z" 2024-08-18T16:01:09.433 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:01:09.433 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:01:09.433 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000005864", 2024-08-18T16:01:09.433 INFO:teuthology.orchestra.run.smithi193.stdout: "last_update": "2024-08-18T16:00:33.166932Z" 2024-08-18T16:01:09.433 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:01:09.433 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:01:09.433 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000010229", 2024-08-18T16:01:09.433 INFO:teuthology.orchestra.run.smithi193.stdout: "last_update": "2024-08-18T16:00:33.127666Z" 2024-08-18T16:01:09.434 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:01:09.434 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:01:09.434 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000008540", 2024-08-18T16:01:09.434 INFO:teuthology.orchestra.run.smithi193.stdout: "last_update": "2024-08-18T15:54:07.557744Z" 2024-08-18T16:01:09.434 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:01:09.434 INFO:teuthology.orchestra.run.smithi193.stdout:] 2024-08-18T16:01:09.435 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2024-08-18T16:01:09.435 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:01:09.435 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:01:09.435 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:01:09.485 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:01:09.485 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:01:09.527 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T16:01:09.527 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T16:01:09.527 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T16:01:09.527 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T16:01:09.528 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T16:01:09.528 INFO:teuthology.orchestra.run.smithi134.stdout: "instance_id": 9282142518545740118 2024-08-18T16:01:09.528 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:01:09.528 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T16:01:09.528 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:01:09.528 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T16:01:09.528 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:01:09.528 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T16:01:09.528 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000005450", 2024-08-18T16:01:09.528 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:01:09.528 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:01:09.528 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:01:09.529 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:00:33.126651Z" 2024-08-18T16:01:09.529 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:01:09.529 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:01:09.529 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:01:09.529 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T16:01:09.529 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:01:09.529 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T16:01:09.529 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000005864", 2024-08-18T16:01:09.529 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:01:09.529 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:01:09.529 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:01:09.529 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:00:33.166932Z" 2024-08-18T16:01:09.529 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:01:09.530 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:01:09.530 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:01:09.530 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T16:01:09.530 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:01:09.530 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T16:01:09.530 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000010229", 2024-08-18T16:01:09.530 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:01:09.530 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:01:09.530 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:01:09.530 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:00:33.127666Z" 2024-08-18T16:01:09.530 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:01:09.530 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:01:09.530 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:01:09.531 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T16:01:09.531 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:01:09.531 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T16:01:09.531 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000008540", 2024-08-18T16:01:09.531 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:01:09.531 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:01:09.531 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:01:09.531 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:54:07.557744Z" 2024-08-18T16:01:09.531 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:01:09.531 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:01:09.531 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T16:01:09.531 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:01:09.531 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T16:01:09.531 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 0, 2024-08-18T16:01:09.532 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 0 2024-08-18T16:01:09.532 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:01:09.532 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T16:01:09Z" 2024-08-18T16:01:09.532 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T16:01:09.533 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2024-08-18T16:01:09.533 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:01:09.533 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:01:09.533 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:01:09.579 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:01:09.579 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:01:09.625 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T16:01:09.625 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:01:09.625 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000005592", 2024-08-18T16:01:09.626 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:00:34.344738Z" 2024-08-18T16:01:09.626 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:01:09.626 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:01:09.626 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000006146", 2024-08-18T16:01:09.626 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:00:33.974055Z" 2024-08-18T16:01:09.626 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:01:09.626 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:01:09.626 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000010511", 2024-08-18T16:01:09.626 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:00:33.110340Z" 2024-08-18T16:01:09.626 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:01:09.626 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:01:09.626 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000009949", 2024-08-18T16:01:09.626 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:54:00.235535Z" 2024-08-18T16:01:09.627 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:01:09.627 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T16:01:09.628 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2024-08-18T16:01:09.628 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:01:09.628 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:01:09.628 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:01:09.677 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:01:09.677 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:01:09.720 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T16:01:09.720 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T16:01:09.721 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T16:01:09.721 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T16:01:09.721 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T16:01:09.721 INFO:teuthology.orchestra.run.smithi134.stdout: "instance_id": 16929744281786545893 2024-08-18T16:01:09.721 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:01:09.721 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T16:01:09.721 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:01:09.721 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T16:01:09.721 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:01:09.721 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T16:01:09.721 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000005592", 2024-08-18T16:01:09.721 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:01:09.722 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:01:09.722 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:01:09.722 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:00:34.344738Z" 2024-08-18T16:01:09.722 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:01:09.722 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:01:09.722 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:01:09.722 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T16:01:09.722 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:01:09.722 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T16:01:09.722 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000006146", 2024-08-18T16:01:09.722 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:01:09.722 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:01:09.723 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:01:09.723 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:00:33.974055Z" 2024-08-18T16:01:09.723 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:01:09.723 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:01:09.723 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:01:09.723 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T16:01:09.723 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:01:09.723 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T16:01:09.723 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000010511", 2024-08-18T16:01:09.723 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:01:09.723 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:01:09.723 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:01:09.723 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:00:33.110340Z" 2024-08-18T16:01:09.723 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:01:09.724 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:01:09.724 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:01:09.724 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T16:01:09.724 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:01:09.724 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T16:01:09.724 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000009949", 2024-08-18T16:01:09.724 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:01:09.724 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:01:09.724 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:01:09.724 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:54:00.235535Z" 2024-08-18T16:01:09.725 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:01:09.725 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:01:09.725 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T16:01:09.725 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:01:09.725 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T16:01:09.725 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 0, 2024-08-18T16:01:09.725 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 0 2024-08-18T16:01:09.725 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:01:09.725 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T16:01:09Z" 2024-08-18T16:01:09.725 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T16:01:09.726 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2024-08-18T16:01:09.726 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:01:09.726 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:01:09.727 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:01:09.773 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:01:09.773 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:01:09.820 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T16:01:09.820 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:01:09.820 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000005592", 2024-08-18T16:01:09.820 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:00:34.344738Z" 2024-08-18T16:01:09.820 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:01:09.820 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:01:09.820 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000006146", 2024-08-18T16:01:09.821 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:00:33.974055Z" 2024-08-18T16:01:09.821 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:01:09.821 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:01:09.821 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000010511", 2024-08-18T16:01:09.821 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:00:33.110340Z" 2024-08-18T16:01:09.821 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:01:09.821 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:01:09.821 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000009949", 2024-08-18T16:01:09.821 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T15:54:00.235535Z" 2024-08-18T16:01:09.821 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:01:09.821 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T16:01:09.822 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2024-08-18T16:01:09.822 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:01:09.822 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:01:09.823 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:01:09.873 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:01:09.873 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:01:09.919 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T16:01:09.919 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T16:01:09.919 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T16:01:09.919 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T16:01:09.919 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T16:01:09.919 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17055221296585213178 2024-08-18T16:01:09.919 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:01:09.919 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T16:01:09.919 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:01:09.919 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T16:01:09.920 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:01:09.920 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T16:01:09.920 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000005592", 2024-08-18T16:01:09.920 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:01:09.920 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:01:09.920 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:01:09.921 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:00:34.344738Z" 2024-08-18T16:01:09.921 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:01:09.921 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:01:09.921 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:01:09.921 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T16:01:09.921 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:01:09.921 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T16:01:09.921 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000006146", 2024-08-18T16:01:09.921 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:01:09.921 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:01:09.921 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:01:09.921 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:00:33.974055Z" 2024-08-18T16:01:09.921 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:01:09.922 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:01:09.922 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:01:09.922 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T16:01:09.922 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:01:09.922 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T16:01:09.922 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000010511", 2024-08-18T16:01:09.922 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:01:09.922 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:01:09.922 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:01:09.922 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:00:33.110340Z" 2024-08-18T16:01:09.922 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:01:09.922 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:01:09.922 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:01:09.923 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T16:01:09.923 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:01:09.923 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T16:01:09.923 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000009949", 2024-08-18T16:01:09.923 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:01:09.923 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:01:09.923 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:01:09.923 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:54:00.235535Z" 2024-08-18T16:01:09.923 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:01:09.923 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:01:09.923 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T16:01:09.923 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:01:09.923 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T16:01:09.923 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 0, 2024-08-18T16:01:09.923 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 0 2024-08-18T16:01:09.923 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:01:09.924 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T16:01:09Z" 2024-08-18T16:01:09.924 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T16:01:09.925 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2024-08-18T16:01:09.925 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_enable_right_after_disable ... ok 2024-08-18T16:01:09.925 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=kibgmb-65 2024-08-18T16:01:09.940 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=kibgmb-66 2024-08-18T16:01:09.965 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=kibgmb-67 2024-08-18T16:01:10.241 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:01:10.241 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:01:10.241 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:01:10.288 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:01:10.288 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:01:10.318 INFO:teuthology.orchestra.run.smithi134.stderr:No --period given, using current period=01050e30-179b-43be-bdbb-3bb23cf37ae8 2024-08-18T16:01:10.330 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T16:01:10.330 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:01:10.331 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T16:01:10.331 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T16:01:10.331 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:01:10.331 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:01:10.331 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T16:01:10.331 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T16:01:10.331 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:01:10.331 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:01:10.332 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T16:01:10.332 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T16:01:10.332 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:01:10.332 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:01:10.332 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723996869.980393_576.1", 2024-08-18T16:01:10.332 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:01:09.980393Z" 2024-08-18T16:01:10.332 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:01:10.332 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T16:01:10.333 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-08-18T16:01:10.333 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:01:10.333 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:01:10.334 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:01:10.381 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:01:10.382 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:01:10.411 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:01:10.410+0000 7ff240f2c880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:01:10.412 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:01:10.410+0000 7ff240f2c880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:01:10.425 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T16:01:10.425 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T16:01:10.425 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T16:01:10.425 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T16:01:10.425 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T16:01:10.425 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T16:01:10.425 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T16:01:10.426 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:01:10.426 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T16:01:10.426 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:01:10.426 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T16:01:10.426 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:01:10.426 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:01:10.426 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T16:01:10.426 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:01:10.426 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T16:01:10.426 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:01:10.426 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T16:01:10.426 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T16:01:10.426 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:01:10.427 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:01:10.427 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:01:10.427 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T16:01:10.427 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:01:10.427 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:01:10.427 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T16:01:10.427 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:01:10.427 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:01:10.427 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:01:10.427 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T16:01:10.427 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T16:01:10.427 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:01:10.427 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:01:10.428 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:01:10.428 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T16:01:10.428 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:01:10.428 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:01:10.428 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T16:01:10.428 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:01:10.428 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:01:10.428 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:01:10.428 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T16:01:10.428 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T16:01:10.428 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:01:10.429 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:01:10.429 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:01:10.429 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T16:01:10.429 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:01:10.429 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:01:10.429 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723996869.933528_565.1", 2024-08-18T16:01:10.429 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:01:10.429 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:01:10.429 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:01:10.430 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:01:09.933528Z", 2024-08-18T16:01:10.430 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T16:01:10.430 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:01:10.430 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:01:10.430 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T16:01:10.430 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:01:10.430 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T16:01:10.430 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T16:01:10.430 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T16:01:10.430 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:01:10.430 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T16:01:10Z" 2024-08-18T16:01:10.430 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T16:01:10.434 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1723996869.980393_576.1 target=1_1723996869.933528_565.1 2024-08-18T16:01:15.435 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:01:15.436 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:01:15.436 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:01:15.484 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:01:15.484 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:01:15.513 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:01:15.512+0000 7f5485f31880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:01:15.514 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:01:15.513+0000 7f5485f31880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:01:15.528 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T16:01:15.528 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T16:01:15.528 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T16:01:15.528 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T16:01:15.528 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T16:01:15.529 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T16:01:15.529 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T16:01:15.529 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:01:15.529 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T16:01:15.529 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:01:15.529 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T16:01:15.530 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:01:15.530 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:01:15.530 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T16:01:15.530 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:01:15.530 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T16:01:15.531 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:01:15.531 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T16:01:15.531 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T16:01:15.531 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:01:15.531 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:01:15.531 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:01:15.532 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T16:01:15.532 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:01:15.532 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:01:15.532 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T16:01:15.532 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:01:15.532 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:01:15.533 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:01:15.533 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T16:01:15.533 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T16:01:15.533 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:01:15.533 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:01:15.534 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:01:15.534 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T16:01:15.534 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:01:15.534 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:01:15.534 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T16:01:15.535 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:01:15.535 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:01:15.535 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:01:15.535 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T16:01:15.535 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T16:01:15.535 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:01:15.536 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:01:15.536 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:01:15.536 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T16:01:15.536 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:01:15.536 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:01:15.536 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723996869.980393_576.1", 2024-08-18T16:01:15.537 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:01:15.537 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:01:15.537 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:01:15.537 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:01:09.980393Z", 2024-08-18T16:01:15.537 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T16:01:15.537 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:01:15.538 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:01:15.538 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T16:01:15.538 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:01:15.538 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T16:01:15.538 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T16:01:15.538 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T16:01:15.539 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:01:15.539 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T16:01:15Z" 2024-08-18T16:01:15.539 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T16:01:15.541 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-08-18T16:01:15.541 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-08-18T16:01:15.541 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:01:15.541 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:01:15.542 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:01:15.596 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:01:15.596 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:01:15.625 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:01:15.624+0000 7fb333326880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:01:15.626 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:01:15.625+0000 7fb333326880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:01:15.638 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T16:01:15.638 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T16:01:15.638 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T16:01:15.638 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T16:01:15.638 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T16:01:15.638 INFO:teuthology.orchestra.run.smithi134.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T16:01:15.638 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T16:01:15.638 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:01:15.639 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T16:01:15.639 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:01:15.639 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T16:01:15.639 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:01:15.639 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T16:01:15.639 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T16:01:15.639 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:01:15.639 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 1, 2024-08-18T16:01:15.639 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:01:15.639 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T16:01:15.639 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T16:01:15.640 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:01:15.640 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:01:15.640 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:01:15.640 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T16:01:15.640 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:01:15.640 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T16:01:15.640 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T16:01:15.640 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:01:15.640 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:01:15.640 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:01:15.640 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T16:01:15.640 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T16:01:15.640 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:01:15.641 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:01:15.641 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:01:15.641 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T16:01:15.641 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:01:15.641 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T16:01:15.641 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T16:01:15.641 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:01:15.641 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:01:15.641 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:01:15.641 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T16:01:15.641 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T16:01:15.642 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:01:15.642 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:01:15.642 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:01:15.642 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T16:01:15.642 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:01:15.642 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T16:01:15.642 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723996869.980393_576.1", 2024-08-18T16:01:15.642 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:01:15.642 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:01:15.642 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:01:15.643 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:01:09.980393Z", 2024-08-18T16:01:15.643 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T16:01:15.643 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:01:15.643 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:01:15.643 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T16:01:15.643 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:01:15.643 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T16:01:15.643 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 1, 2024-08-18T16:01:15.643 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 1 2024-08-18T16:01:15.643 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:01:15.643 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T16:01:15Z" 2024-08-18T16:01:15.643 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T16:01:15.644 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-08-18T16:01:15.645 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-65', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:01:15.645 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-65', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:01:15.645 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-65 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T16:01:15.698 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:01:15.699 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:01:15.734 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:01:15.733+0000 7f31cb22f880 1 bucket sync caught up with source: 2024-08-18T16:01:15.734 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000001.56.6, , 00000000001.63.6, , 00000000001.63.6, , , 00000000001.57.6, , , ] 2024-08-18T16:01:15.734 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000001.56.6, , 00000000001.63.6, , 00000000001.63.6, , , 00000000001.57.6, , , ] 2024-08-18T16:01:15.735 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:01:15.733+0000 7f31cb22f880 0 bucket checkpoint complete 2024-08-18T16:01:15.748 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-65', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:01:15.749 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-65', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:01:15.749 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-65 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:01:15.804 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:01:15.804 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:01:15.838 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:01:15.837+0000 7ff93ce1f880 1 bucket sync caught up with source: 2024-08-18T16:01:15.838 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000001.56.6, , 00000000001.63.6, , 00000000001.63.6, , , 00000000001.57.6, , , ] 2024-08-18T16:01:15.839 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000001.56.6, , 00000000001.63.6, , 00000000001.63.6, , , 00000000001.57.6, , , ] 2024-08-18T16:01:15.839 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:01:15.837+0000 7ff93ce1f880 0 bucket checkpoint complete 2024-08-18T16:01:15.851 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-65', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:01:15.851 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-65', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:01:15.851 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-65 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:01:15.905 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:01:15.905 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:01:15.940 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:01:15.939+0000 7fa6ef720880 1 waiting to reach incremental sync.. 2024-08-18T16:01:20.941 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:01:20.940+0000 7fa6ef720880 1 waiting to reach incremental sync.. 2024-08-18T16:01:25.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:01:25.942+0000 7fa6ef720880 1 bucket sync caught up with source: 2024-08-18T16:01:25.943 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000001.113.6, , 00000000001.261.6, , 00000000001.323.6, , , 00000000001.212.6, , , ] 2024-08-18T16:01:25.943 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000001.113.6, , 00000000001.261.6, , 00000000001.323.6, , , 00000000001.212.6, , , ] 2024-08-18T16:01:25.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:01:25.942+0000 7fa6ef720880 0 bucket checkpoint complete 2024-08-18T16:01:25.960 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-65', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:01:25.960 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-65', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:01:25.960 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-65 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:01:26.019 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:01:26.019 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:01:26.055 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:01:26.055+0000 7f3e36136880 1 waiting to reach incremental sync.. 2024-08-18T16:01:31.056 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:01:31.055+0000 7f3e36136880 1 waiting to reach incremental sync.. 2024-08-18T16:01:36.057 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:01:36.056+0000 7f3e36136880 1 bucket sync caught up with source: 2024-08-18T16:01:36.057 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000001.113.6, , 00000000001.261.6, , 00000000001.323.6, , , 00000000001.212.6, , , ] 2024-08-18T16:01:36.057 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000001.113.6, , 00000000001.261.6, , 00000000001.323.6, , , 00000000001.212.6, , , ] 2024-08-18T16:01:36.057 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:01:36.056+0000 7f3e36136880 0 bucket checkpoint complete 2024-08-18T16:01:36.071 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-65', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:01:36.071 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-65', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:01:36.071 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-65 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:01:36.125 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:01:36.125 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:01:36.159 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:01:36.158+0000 7f5f5b342880 1 bucket sync caught up with source: 2024-08-18T16:01:36.159 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000001.140.6, , 00000000001.127.6, , 00000000001.168.6, , , 00000000001.199.6, , , ] 2024-08-18T16:01:36.159 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000001.140.6, , 00000000001.127.6, , 00000000001.168.6, , , 00000000001.199.6, , , ] 2024-08-18T16:01:36.159 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:01:36.158+0000 7f5f5b342880 0 bucket checkpoint complete 2024-08-18T16:01:36.173 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-65', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:01:36.173 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-65', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:01:36.173 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-65 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T16:01:36.233 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:01:36.234 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:01:36.272 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:01:36.272+0000 7fa739326880 1 bucket sync caught up with source: 2024-08-18T16:01:36.272 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000001.140.6, , 00000000001.127.6, , 00000000001.168.6, , , 00000000001.199.6, , , ] 2024-08-18T16:01:36.272 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000001.140.6, , 00000000001.127.6, , 00000000001.168.6, , , 00000000001.199.6, , , ] 2024-08-18T16:01:36.272 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:01:36.272+0000 7fa739326880 0 bucket checkpoint complete 2024-08-18T16:01:36.287 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-65 zones={test-zone2, test-zone1} 2024-08-18T16:01:36.293 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:01:36.296 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-08-18T16:01:36.296 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-08-18T16:01:36.296 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-08-18T16:01:36.296 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-08-18T16:01:36.296 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:01:36.299 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-08-18T16:01:36.299 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-08-18T16:01:36.299 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-08-18T16:01:36.299 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-08-18T16:01:36.305 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:01:36.355 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:01:36.366 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:01:36.374 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:01:36.424 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:01:36.435 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:01:36.487 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:01:36.494 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:01:36.547 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:01:36.599 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:01:36.605 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:01:36.658 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:01:36.707 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-65 zones={test-zone2, test-zone1} 2024-08-18T16:01:36.707 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-65 zones={test-zone3, test-zone1} 2024-08-18T16:01:36.711 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:01:36.714 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-08-18T16:01:36.714 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-08-18T16:01:36.714 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-08-18T16:01:36.714 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-08-18T16:01:36.714 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:01:36.717 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-08-18T16:01:36.717 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-08-18T16:01:36.717 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-08-18T16:01:36.717 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-08-18T16:01:36.722 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:01:36.774 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:01:36.784 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:01:36.837 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:01:36.843 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:01:36.895 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:01:36.904 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:01:36.910 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:01:36.920 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:01:36.972 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:01:36.979 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:01:37.031 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:01:37.083 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-65 zones={test-zone3, test-zone1} 2024-08-18T16:01:37.083 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-65 zones={test-zone3, test-zone2} 2024-08-18T16:01:37.088 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:01:37.091 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-08-18T16:01:37.091 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-08-18T16:01:37.091 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-08-18T16:01:37.091 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-08-18T16:01:37.091 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:01:37.094 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-08-18T16:01:37.094 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-08-18T16:01:37.094 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-08-18T16:01:37.094 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-08-18T16:01:37.099 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:01:37.150 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:01:37.162 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:01:37.213 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:01:37.220 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:01:37.275 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:01:37.328 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:01:37.335 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:01:37.387 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:01:37.435 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:01:37.443 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:01:37.503 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:01:37.556 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-65 zones={test-zone3, test-zone2} 2024-08-18T16:01:37.556 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-66', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:01:37.556 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-66', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:01:37.557 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-66 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T16:01:37.615 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:01:37.615 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:01:37.653 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:01:37.652+0000 7f19c9884880 1 bucket sync caught up with source: 2024-08-18T16:01:37.653 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000001.81.6, , 00000000001.54.6, , 00000000001.43.6, , , 00000000001.65.6, , , ] 2024-08-18T16:01:37.653 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000001.81.6, , 00000000001.54.6, , 00000000001.43.6, , , 00000000001.65.6, , , ] 2024-08-18T16:01:37.653 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:01:37.652+0000 7f19c9884880 0 bucket checkpoint complete 2024-08-18T16:01:37.668 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-66', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:01:37.668 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-66', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:01:37.668 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-66 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:01:37.722 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:01:37.722 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:01:37.756 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:01:37.755+0000 7f41a1538880 1 bucket sync caught up with source: 2024-08-18T16:01:37.756 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000001.81.6, , 00000000001.54.6, , 00000000001.43.6, , , 00000000001.65.6, , , ] 2024-08-18T16:01:37.756 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000001.81.6, , 00000000001.54.6, , 00000000001.43.6, , , 00000000001.65.6, , , ] 2024-08-18T16:01:37.756 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:01:37.755+0000 7f41a1538880 0 bucket checkpoint complete 2024-08-18T16:01:37.769 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-66', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:01:37.769 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-66', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:01:37.769 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-66 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:01:37.822 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:01:37.823 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:01:37.857 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:01:37.857+0000 7f0eead23880 1 bucket sync caught up with source: 2024-08-18T16:01:37.858 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000001.352.6, , 00000000001.163.6, , 00000000001.240.6, , , 00000000001.317.6, , , ] 2024-08-18T16:01:37.858 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000001.352.6, , 00000000001.163.6, , 00000000001.240.6, , , 00000000001.317.6, , , ] 2024-08-18T16:01:37.858 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:01:37.857+0000 7f0eead23880 0 bucket checkpoint complete 2024-08-18T16:01:37.871 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-66', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:01:37.871 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-66', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:01:37.871 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-66 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:01:37.925 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:01:37.925 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:01:37.959 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:01:37.958+0000 7f231ad27880 1 bucket sync caught up with source: 2024-08-18T16:01:37.959 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000001.352.6, , 00000000001.163.6, , 00000000001.240.6, , , 00000000001.317.6, , , ] 2024-08-18T16:01:37.959 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000001.352.6, , 00000000001.163.6, , 00000000001.240.6, , , 00000000001.317.6, , , ] 2024-08-18T16:01:37.959 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:01:37.958+0000 7f231ad27880 0 bucket checkpoint complete 2024-08-18T16:01:37.972 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-66', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:01:37.972 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-66', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:01:37.972 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-66 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:01:38.025 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:01:38.025 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:01:38.058 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:01:38.058+0000 7faa8c210880 1 bucket sync caught up with source: 2024-08-18T16:01:38.059 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000001.162.6, , 00000000001.144.6, , 00000000001.168.6, , , 00000000001.166.6, , , ] 2024-08-18T16:01:38.059 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000001.162.6, , 00000000001.144.6, , 00000000001.168.6, , , 00000000001.166.6, , , ] 2024-08-18T16:01:38.059 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:01:38.058+0000 7faa8c210880 0 bucket checkpoint complete 2024-08-18T16:01:38.072 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-66', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:01:38.072 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-66', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:01:38.072 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-66 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T16:01:38.133 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:01:38.133 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:01:38.172 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:01:38.172+0000 7fd28992d880 1 bucket sync caught up with source: 2024-08-18T16:01:38.172 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000001.162.6, , 00000000001.144.6, , 00000000001.168.6, , , 00000000001.166.6, , , ] 2024-08-18T16:01:38.173 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000001.162.6, , 00000000001.144.6, , 00000000001.168.6, , , 00000000001.166.6, , , ] 2024-08-18T16:01:38.173 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:01:38.172+0000 7fd28992d880 0 bucket checkpoint complete 2024-08-18T16:01:38.188 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-66 zones={test-zone2, test-zone1} 2024-08-18T16:01:38.192 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:01:38.195 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-08-18T16:01:38.195 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-08-18T16:01:38.195 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-08-18T16:01:38.195 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-08-18T16:01:38.195 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:01:38.198 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-08-18T16:01:38.198 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-08-18T16:01:38.198 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-08-18T16:01:38.198 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-08-18T16:01:38.203 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:01:38.252 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:01:38.263 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:01:38.270 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:01:38.318 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:01:38.329 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:01:38.381 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:01:38.388 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:01:38.440 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:01:38.489 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:01:38.495 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:01:38.548 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:01:38.597 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-66 zones={test-zone2, test-zone1} 2024-08-18T16:01:38.597 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-66 zones={test-zone3, test-zone1} 2024-08-18T16:01:38.601 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:01:38.604 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-08-18T16:01:38.604 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-08-18T16:01:38.604 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-08-18T16:01:38.604 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-08-18T16:01:38.604 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:01:38.607 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-08-18T16:01:38.607 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-08-18T16:01:38.607 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-08-18T16:01:38.607 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-08-18T16:01:38.612 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:01:38.661 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:01:38.671 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:01:38.678 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:01:38.684 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:01:38.735 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:01:38.791 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:01:38.797 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:01:38.848 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:01:38.857 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:01:38.863 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:01:38.916 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:01:38.965 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-66 zones={test-zone3, test-zone1} 2024-08-18T16:01:38.966 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-66 zones={test-zone3, test-zone2} 2024-08-18T16:01:38.970 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:01:38.973 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-08-18T16:01:38.973 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-08-18T16:01:38.973 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-08-18T16:01:38.973 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-08-18T16:01:38.973 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:01:38.976 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-08-18T16:01:38.976 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-08-18T16:01:38.976 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-08-18T16:01:38.976 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-08-18T16:01:38.981 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:01:39.032 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:01:39.089 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:01:39.141 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:01:39.149 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:01:39.205 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:01:39.257 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:01:39.264 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:01:39.317 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:01:39.370 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:01:39.377 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:01:39.433 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:01:39.442 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-66 zones={test-zone3, test-zone2} 2024-08-18T16:01:39.442 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-67', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:01:39.442 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-67', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:01:39.442 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-67 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T16:01:39.501 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:01:39.501 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:01:39.538 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:01:39.538+0000 7fc7e44bb880 1 bucket sync caught up with source: 2024-08-18T16:01:39.539 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000001.53.6, , 00000000001.58.6, , 00000000001.102.6, , , 00000000001.108.6, , , ] 2024-08-18T16:01:39.539 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000001.53.6, , 00000000001.58.6, , 00000000001.102.6, , , 00000000001.108.6, , , ] 2024-08-18T16:01:39.539 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:01:39.538+0000 7fc7e44bb880 0 bucket checkpoint complete 2024-08-18T16:01:39.553 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-67', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:01:39.553 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-67', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:01:39.553 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-67 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:01:39.607 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:01:39.607 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:01:39.641 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:01:39.640+0000 7f05e1010880 1 bucket sync caught up with source: 2024-08-18T16:01:39.641 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000001.53.6, , 00000000001.58.6, , 00000000001.102.6, , , 00000000001.108.6, , , ] 2024-08-18T16:01:39.641 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000001.53.6, , 00000000001.58.6, , 00000000001.102.6, , , 00000000001.108.6, , , ] 2024-08-18T16:01:39.641 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:01:39.640+0000 7f05e1010880 0 bucket checkpoint complete 2024-08-18T16:01:39.653 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-67', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:01:39.654 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-67', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:01:39.654 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-67 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:01:39.707 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:01:39.708 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:01:39.743 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:01:39.743+0000 7f987b726880 1 bucket sync caught up with source: 2024-08-18T16:01:39.743 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000001.168.6, , 00000000001.328.6, , 00000000001.211.6, , , 00000000001.245.6, , , ] 2024-08-18T16:01:39.743 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000001.168.6, , 00000000001.328.6, , 00000000001.211.6, , , 00000000001.245.6, , , ] 2024-08-18T16:01:39.744 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:01:39.743+0000 7f987b726880 0 bucket checkpoint complete 2024-08-18T16:01:39.756 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-67', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:01:39.757 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-67', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:01:39.757 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-67 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:01:39.809 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:01:39.809 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:01:39.844 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:01:39.843+0000 7fc82840e880 1 bucket sync caught up with source: 2024-08-18T16:01:39.844 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000001.168.6, , 00000000001.328.6, , 00000000001.211.6, , , 00000000001.245.6, , , ] 2024-08-18T16:01:39.844 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000001.168.6, , 00000000001.328.6, , 00000000001.211.6, , , 00000000001.245.6, , , ] 2024-08-18T16:01:39.844 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:01:39.843+0000 7fc82840e880 0 bucket checkpoint complete 2024-08-18T16:01:39.856 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-67', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:01:39.857 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-67', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:01:39.857 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-67 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:01:39.910 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:01:39.911 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:01:39.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:01:39.943+0000 7fe05292e880 1 bucket sync caught up with source: 2024-08-18T16:01:39.944 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000001.133.6, , 00000000001.200.6, , 00000000001.178.6, , , 00000000001.288.6, , , ] 2024-08-18T16:01:39.944 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000001.133.6, , 00000000001.200.6, , 00000000001.178.6, , , 00000000001.288.6, , , ] 2024-08-18T16:01:39.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:01:39.943+0000 7fe05292e880 0 bucket checkpoint complete 2024-08-18T16:01:39.957 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-67', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:01:39.957 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-67', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:01:39.957 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-67 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T16:01:40.012 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:01:40.012 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:01:40.050 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:01:40.050+0000 7f8561528880 1 bucket sync caught up with source: 2024-08-18T16:01:40.050 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000001.133.6, , 00000000001.200.6, , 00000000001.178.6, , , 00000000001.288.6, , , ] 2024-08-18T16:01:40.050 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000001.133.6, , 00000000001.200.6, , 00000000001.178.6, , , 00000000001.288.6, , , ] 2024-08-18T16:01:40.051 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:01:40.050+0000 7f8561528880 0 bucket checkpoint complete 2024-08-18T16:01:40.065 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-67 zones={test-zone2, test-zone1} 2024-08-18T16:01:40.069 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:01:40.071 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-08-18T16:01:40.071 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-08-18T16:01:40.072 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-08-18T16:01:40.072 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-08-18T16:01:40.072 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:01:40.074 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-08-18T16:01:40.075 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-08-18T16:01:40.075 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-08-18T16:01:40.075 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-08-18T16:01:40.079 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:01:40.132 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:01:40.186 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:01:40.235 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:01:40.285 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:01:40.338 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:01:40.387 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:01:40.438 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:01:40.490 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:01:40.587 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:01:40.642 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:01:40.739 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:01:40.831 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-67 zones={test-zone2, test-zone1} 2024-08-18T16:01:40.831 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-67 zones={test-zone3, test-zone1} 2024-08-18T16:01:40.835 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:01:40.838 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-08-18T16:01:40.838 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-08-18T16:01:40.838 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-08-18T16:01:40.838 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-08-18T16:01:40.838 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:01:40.840 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-08-18T16:01:40.841 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-08-18T16:01:40.841 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-08-18T16:01:40.841 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-08-18T16:01:40.845 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:01:40.900 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:01:40.998 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:01:41.047 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:01:41.098 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:01:41.150 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:01:41.199 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:01:41.250 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:01:41.302 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:01:41.395 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:01:41.446 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:01:41.546 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:01:41.595 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-67 zones={test-zone3, test-zone1} 2024-08-18T16:01:41.595 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-67 zones={test-zone3, test-zone2} 2024-08-18T16:01:41.599 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:01:41.602 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-08-18T16:01:41.602 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-08-18T16:01:41.602 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-08-18T16:01:41.602 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-08-18T16:01:41.602 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:01:41.605 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-08-18T16:01:41.605 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-08-18T16:01:41.605 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-08-18T16:01:41.605 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-08-18T16:01:41.611 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:01:41.662 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:01:41.673 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:01:41.729 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:01:41.737 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:01:41.797 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:01:41.850 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:01:41.857 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:01:41.912 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:01:41.921 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:01:41.928 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:01:41.981 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:01:42.034 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-67 zones={test-zone3, test-zone2} 2024-08-18T16:01:42.034 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'kibgmb-65', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:01:42.034 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'kibgmb-65', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:01:42.034 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket kibgmb-65 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:01:42.084 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:01:42.084 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:01:42.153 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'kibgmb-66', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:01:42.153 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'kibgmb-66', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:01:42.153 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket kibgmb-66 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:01:42.201 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:01:42.201 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:01:42.272 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'kibgmb-67', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:01:42.272 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'kibgmb-67', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:01:42.272 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket kibgmb-67 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:01:42.320 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:01:42.320 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:01:42.388 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:01:42.388 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:01:42.388 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:01:42.434 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:01:42.434 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:01:42.464 INFO:teuthology.orchestra.run.smithi134.stderr:No --period given, using current period=01050e30-179b-43be-bdbb-3bb23cf37ae8 2024-08-18T16:01:42.477 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T16:01:42.478 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:01:42.478 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T16:01:42.478 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T16:01:42.478 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:01:42.478 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:01:42.478 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T16:01:42.478 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T16:01:42.478 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:01:42.478 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:01:42.478 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T16:01:42.478 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T16:01:42.479 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:01:42.479 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:01:42.479 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723996902.374246_606.1", 2024-08-18T16:01:42.479 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:01:42.374246Z" 2024-08-18T16:01:42.479 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:01:42.479 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T16:01:42.480 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-08-18T16:01:42.480 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:01:42.480 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:01:42.480 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:01:42.535 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:01:42.535 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:01:42.569 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:01:42.568+0000 7f2bb5831880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:01:42.569 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:01:42.569+0000 7f2bb5831880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:01:42.584 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T16:01:42.584 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T16:01:42.584 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T16:01:42.584 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T16:01:42.584 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T16:01:42.584 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T16:01:42.584 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T16:01:42.585 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:01:42.585 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T16:01:42.585 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:01:42.585 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T16:01:42.585 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:01:42.585 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:01:42.585 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T16:01:42.585 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:01:42.585 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T16:01:42.585 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:01:42.585 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T16:01:42.585 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T16:01:42.586 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:01:42.586 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:01:42.586 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:01:42.586 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T16:01:42.586 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:01:42.586 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:01:42.586 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T16:01:42.586 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:01:42.586 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:01:42.586 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:01:42.586 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T16:01:42.587 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T16:01:42.587 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:01:42.587 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:01:42.587 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:01:42.587 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T16:01:42.587 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:01:42.587 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:01:42.587 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T16:01:42.587 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:01:42.587 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:01:42.587 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:01:42.587 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T16:01:42.587 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T16:01:42.588 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:01:42.588 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:01:42.588 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:01:42.588 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T16:01:42.588 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:01:42.588 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:01:42.588 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723996869.980393_576.1", 2024-08-18T16:01:42.588 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:01:42.588 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:01:42.588 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:01:42.588 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:01:09.980393Z", 2024-08-18T16:01:42.589 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T16:01:42.589 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:01:42.589 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:01:42.589 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T16:01:42.589 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:01:42.589 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T16:01:42.589 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T16:01:42.589 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T16:01:42.589 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:01:42.589 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T16:01:42Z" 2024-08-18T16:01:42.589 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T16:01:42.590 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1723996902.374246_606.1 target=1_1723996869.980393_576.1 2024-08-18T16:01:47.591 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:01:47.592 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:01:47.592 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:01:47.647 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:01:47.648 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:01:47.681 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:01:47.680+0000 7f9df6a15880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:01:47.685 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:01:47.681+0000 7f9df6a15880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:01:47.711 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T16:01:47.711 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T16:01:47.711 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T16:01:47.711 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T16:01:47.711 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T16:01:47.711 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T16:01:47.711 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T16:01:47.711 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:01:47.712 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T16:01:47.712 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:01:47.712 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T16:01:47.712 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:01:47.712 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:01:47.712 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T16:01:47.712 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:01:47.712 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T16:01:47.712 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:01:47.712 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T16:01:47.712 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T16:01:47.712 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:01:47.712 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:01:47.712 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:01:47.713 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T16:01:47.713 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:01:47.713 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:01:47.713 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T16:01:47.713 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:01:47.713 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:01:47.713 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:01:47.713 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T16:01:47.713 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T16:01:47.713 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:01:47.713 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:01:47.714 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:01:47.714 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T16:01:47.714 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:01:47.714 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:01:47.714 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T16:01:47.714 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:01:47.714 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:01:47.714 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:01:47.715 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T16:01:47.715 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T16:01:47.715 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:01:47.715 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:01:47.715 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:01:47.715 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T16:01:47.715 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:01:47.715 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:01:47.715 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723996869.980393_576.1", 2024-08-18T16:01:47.715 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:01:47.716 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:01:47.716 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:01:47.716 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:01:09.980393Z", 2024-08-18T16:01:47.716 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T16:01:47.716 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:01:47.716 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:01:47.716 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T16:01:47.716 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:01:47.717 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T16:01:47.717 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T16:01:47.717 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T16:01:47.717 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:01:47.717 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T16:01:47Z" 2024-08-18T16:01:47.717 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T16:01:47.718 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1723996902.374246_606.1 target=1_1723996869.980393_576.1 2024-08-18T16:01:52.718 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:01:52.718 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:01:52.719 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:01:52.774 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:01:52.774 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:01:52.807 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:01:52.807+0000 7ff799926880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:01:52.808 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:01:52.808+0000 7ff799926880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:01:52.823 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T16:01:52.823 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T16:01:52.823 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T16:01:52.823 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T16:01:52.823 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T16:01:52.823 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T16:01:52.824 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T16:01:52.824 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:01:52.824 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T16:01:52.824 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:01:52.824 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T16:01:52.824 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:01:52.824 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:01:52.824 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T16:01:52.824 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:01:52.824 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T16:01:52.824 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:01:52.824 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T16:01:52.825 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T16:01:52.825 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:01:52.825 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:01:52.825 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:01:52.825 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T16:01:52.825 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:01:52.825 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:01:52.825 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T16:01:52.825 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:01:52.825 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:01:52.825 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:01:52.825 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T16:01:52.825 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T16:01:52.825 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:01:52.826 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:01:52.826 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:01:52.826 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T16:01:52.826 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:01:52.826 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:01:52.826 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T16:01:52.826 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:01:52.826 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:01:52.826 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:01:52.826 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T16:01:52.826 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T16:01:52.826 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:01:52.827 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:01:52.827 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:01:52.827 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T16:01:52.827 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:01:52.827 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:01:52.827 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723996902.374246_606.1", 2024-08-18T16:01:52.827 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:01:52.827 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:01:52.827 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:01:52.827 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:01:42.374246Z", 2024-08-18T16:01:52.827 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T16:01:52.827 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:01:52.827 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:01:52.828 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T16:01:52.828 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:01:52.828 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T16:01:52.828 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T16:01:52.828 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T16:01:52.828 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:01:52.828 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T16:01:52Z" 2024-08-18T16:01:52.828 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T16:01:52.829 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-08-18T16:01:52.829 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-08-18T16:01:52.829 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:01:52.829 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:01:52.830 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:01:52.884 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:01:52.885 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:01:52.916 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:01:52.916+0000 7f48d9815880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:01:52.917 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:01:52.917+0000 7f48d9815880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:01:52.934 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T16:01:52.934 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T16:01:52.934 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T16:01:52.934 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T16:01:52.934 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T16:01:52.934 INFO:teuthology.orchestra.run.smithi134.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T16:01:52.934 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T16:01:52.934 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:01:52.934 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T16:01:52.934 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:01:52.935 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T16:01:52.935 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:01:52.935 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T16:01:52.935 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T16:01:52.935 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:01:52.935 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 1, 2024-08-18T16:01:52.935 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:01:52.935 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T16:01:52.935 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T16:01:52.935 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:01:52.935 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:01:52.935 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:01:52.935 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T16:01:52.936 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:01:52.936 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T16:01:52.936 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T16:01:52.936 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:01:52.936 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:01:52.936 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:01:52.936 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T16:01:52.936 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T16:01:52.936 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:01:52.936 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:01:52.936 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:01:52.936 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T16:01:52.936 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:01:52.936 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T16:01:52.937 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T16:01:52.937 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:01:52.937 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:01:52.937 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:01:52.937 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T16:01:52.937 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T16:01:52.937 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:01:52.937 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:01:52.937 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:01:52.937 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T16:01:52.937 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:01:52.937 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T16:01:52.937 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723996902.374246_606.1", 2024-08-18T16:01:52.938 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:01:52.938 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:01:52.938 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:01:52.938 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:01:42.374246Z", 2024-08-18T16:01:52.938 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T16:01:52.938 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:01:52.938 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:01:52.938 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T16:01:52.938 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:01:52.938 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T16:01:52.938 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 1, 2024-08-18T16:01:52.938 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 1 2024-08-18T16:01:52.938 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:01:52.939 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T16:01:52Z" 2024-08-18T16:01:52.939 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T16:01:52.940 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-08-18T16:01:53.137 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'kibgmb-65', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:01:53.137 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'kibgmb-65', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:01:53.138 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket kibgmb-65 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:01:53.191 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:01:53.192 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:01:53.265 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'kibgmb-66', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:01:53.265 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'kibgmb-66', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:01:53.266 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket kibgmb-66 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:01:53.313 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:01:53.313 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:01:53.380 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'kibgmb-67', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:01:53.381 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'kibgmb-67', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:01:53.381 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket kibgmb-67 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:01:53.428 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:01:53.429 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:01:53.497 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-65', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:01:53.497 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-65', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:01:53.497 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-65 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T16:01:53.552 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:01:53.552 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:01:53.588 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:01:53.587+0000 7ff226d26880 1 waiting for incremental sync to catch up: 2024-08-18T16:01:53.588 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000001.56.6, , 00000000001.63.6, , 00000000001.63.6, , , 00000000001.57.6, , , ] 2024-08-18T16:01:53.588 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000004.60.1, 00000000002.30.1, 00000000004.68.1, 00000000002.62.1, 00000000004.72.1, 00000000002.73.1, 00000000002.78.1, 00000000004.62.1, 00000000002.54.1, 00000000002.46.1, 00000000002.107.1] 2024-08-18T16:01:58.590 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:01:58.589+0000 7ff226d26880 1 bucket sync caught up with source: 2024-08-18T16:01:58.590 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000004.60.1, 00000000002.30.1, 00000000004.68.1, 00000000002.62.1, 00000000004.72.1, 00000000002.73.1, 00000000002.78.1, 00000000004.62.1, 00000000002.54.1, 00000000002.46.1, 00000000002.107.1] 2024-08-18T16:01:58.590 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000004.60.1, 00000000002.30.1, 00000000004.68.1, 00000000002.62.1, 00000000004.72.1, 00000000002.73.1, 00000000002.78.1, 00000000004.62.1, 00000000002.54.1, 00000000002.46.1, 00000000002.107.1] 2024-08-18T16:01:58.590 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:01:58.589+0000 7ff226d26880 0 bucket checkpoint complete 2024-08-18T16:01:58.609 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-65', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:01:58.609 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-65', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:01:58.609 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-65 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:01:58.671 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:01:58.671 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:01:58.709 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:01:58.708+0000 7f015a012880 1 bucket sync caught up with source: 2024-08-18T16:01:58.709 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000004.60.1, 00000000002.30.1, 00000000004.68.1, 00000000002.62.1, 00000000004.72.1, 00000000002.73.1, 00000000002.78.1, 00000000004.62.1, 00000000002.54.1, 00000000002.46.1, 00000000002.107.1] 2024-08-18T16:01:58.709 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000004.60.1, 00000000002.30.1, 00000000004.68.1, 00000000002.62.1, 00000000004.72.1, 00000000002.73.1, 00000000002.78.1, 00000000004.62.1, 00000000002.54.1, 00000000002.46.1, 00000000002.107.1] 2024-08-18T16:01:58.709 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:01:58.708+0000 7f015a012880 0 bucket checkpoint complete 2024-08-18T16:01:58.723 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-65', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:01:58.723 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-65', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:01:58.723 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-65 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:01:58.778 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:01:58.778 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:01:58.812 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:01:58.812+0000 7f834a4bb880 1 waiting to reach incremental sync.. 2024-08-18T16:02:03.814 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:02:03.812+0000 7f834a4bb880 1 waiting to reach incremental sync.. 2024-08-18T16:02:08.814 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:02:08.813+0000 7f834a4bb880 1 waiting to reach incremental sync.. 2024-08-18T16:02:13.814 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:02:13.814+0000 7f834a4bb880 1 waiting to reach incremental sync.. 2024-08-18T16:02:18.816 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:02:18.815+0000 7f834a4bb880 1 bucket sync caught up with source: 2024-08-18T16:02:18.816 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000004.118.1, 00000000002.134.1, 00000000004.269.1, 00000000002.337.1, 00000000004.338.1, 00000000002.341.1, 00000000002.214.1, 00000000004.220.1, 00000000002.242.1, 00000000002.250.1, 00000000002.223.1] 2024-08-18T16:02:18.817 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000002.114.1, 00000000001.132.1, 00000000002.263.1, 00000000001.331.1, 00000000002.326.1, 00000000001.329.1, 00000000001.212.1, 00000000002.214.1, 00000000001.240.1, 00000000001.244.1, 00000000001.215.1] 2024-08-18T16:02:18.817 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:02:18.815+0000 7f834a4bb880 0 bucket checkpoint complete 2024-08-18T16:02:18.835 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-65', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:02:18.835 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-65', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:02:18.835 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-65 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:02:18.894 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:02:18.894 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:02:18.931 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:02:18.931+0000 7f4f24d26880 1 bucket sync caught up with source: 2024-08-18T16:02:18.932 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000004.118.1, 00000000002.134.1, 00000000004.269.1, 00000000002.337.1, 00000000004.338.1, 00000000002.341.1, 00000000002.214.1, 00000000004.220.1, 00000000002.242.1, 00000000002.250.1, 00000000002.223.1] 2024-08-18T16:02:18.932 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000004.118.1, 00000000002.134.1, 00000000004.269.1, 00000000002.337.1, 00000000004.338.1, 00000000002.341.1, 00000000002.214.1, 00000000004.220.1, 00000000002.242.1, 00000000002.250.1, 00000000002.223.1] 2024-08-18T16:02:18.932 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:02:18.931+0000 7f4f24d26880 0 bucket checkpoint complete 2024-08-18T16:02:18.945 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-65', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:02:18.945 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-65', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:02:18.945 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-65 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:02:18.999 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:02:18.999 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:02:19.033 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:02:19.032+0000 7f5573942880 1 bucket sync caught up with source: 2024-08-18T16:02:19.033 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000004.145.1, 00000000002.96.1, 00000000004.135.1, 00000000002.220.1, 00000000004.179.1, 00000000002.180.1, 00000000002.209.1, 00000000004.206.1, 00000000002.159.1, 00000000002.172.1, 00000000002.202.1] 2024-08-18T16:02:19.033 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000004.145.1, 00000000002.96.1, 00000000004.135.1, 00000000002.220.1, 00000000004.179.1, 00000000002.180.1, 00000000002.209.1, 00000000004.206.1, 00000000002.159.1, 00000000002.172.1, 00000000002.202.1] 2024-08-18T16:02:19.033 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:02:19.032+0000 7f5573942880 0 bucket checkpoint complete 2024-08-18T16:02:19.047 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-65', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:02:19.047 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-65', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:02:19.047 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-65 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T16:02:19.109 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:02:19.109 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:02:19.148 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:02:19.148+0000 7fa7a23af880 1 bucket sync caught up with source: 2024-08-18T16:02:19.148 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000004.145.1, 00000000002.96.1, 00000000004.135.1, 00000000002.220.1, 00000000004.179.1, 00000000002.180.1, 00000000002.209.1, 00000000004.206.1, 00000000002.159.1, 00000000002.172.1, 00000000002.202.1] 2024-08-18T16:02:19.149 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000004.145.1, 00000000002.96.1, 00000000004.135.1, 00000000002.220.1, 00000000004.179.1, 00000000002.180.1, 00000000002.209.1, 00000000004.206.1, 00000000002.159.1, 00000000002.172.1, 00000000002.202.1] 2024-08-18T16:02:19.149 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:02:19.148+0000 7fa7a23af880 0 bucket checkpoint complete 2024-08-18T16:02:19.164 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-65 zones={test-zone2, test-zone1} 2024-08-18T16:02:19.170 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:02:19.174 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-08-18T16:02:19.175 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-08-18T16:02:19.175 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-08-18T16:02:19.175 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-08-18T16:02:19.175 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-08-18T16:02:19.175 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-08-18T16:02:19.175 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-08-18T16:02:19.175 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-08-18T16:02:19.175 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:02:19.178 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-08-18T16:02:19.178 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-08-18T16:02:19.178 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-08-18T16:02:19.179 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-08-18T16:02:19.179 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-08-18T16:02:19.179 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-08-18T16:02:19.179 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-08-18T16:02:19.179 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-08-18T16:02:19.185 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:02:19.236 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:02:19.294 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:02:19.302 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:02:19.309 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:02:19.369 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:02:19.422 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:02:19.429 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:02:19.484 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:02:19.493 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:02:19.545 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:02:19.601 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:02:19.653 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-08-18T16:02:19.662 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-08-18T16:02:19.716 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-08-18T16:02:19.725 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-08-18T16:02:19.734 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-08-18T16:02:19.790 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-08-18T16:02:19.841 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-08-18T16:02:19.891 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-08-18T16:02:19.903 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-08-18T16:02:19.955 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-08-18T16:02:19.962 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-08-18T16:02:20.014 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-08-18T16:02:20.067 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-65 zones={test-zone2, test-zone1} 2024-08-18T16:02:20.067 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-65 zones={test-zone3, test-zone1} 2024-08-18T16:02:20.071 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:02:20.073 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-08-18T16:02:20.073 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-08-18T16:02:20.074 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-08-18T16:02:20.074 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-08-18T16:02:20.074 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-08-18T16:02:20.074 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-08-18T16:02:20.074 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-08-18T16:02:20.074 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-08-18T16:02:20.074 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:02:20.077 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-08-18T16:02:20.077 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-08-18T16:02:20.077 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-08-18T16:02:20.077 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-08-18T16:02:20.077 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-08-18T16:02:20.077 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-08-18T16:02:20.077 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-08-18T16:02:20.077 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-08-18T16:02:20.082 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:02:20.131 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:02:20.142 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:02:20.191 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:02:20.197 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:02:20.208 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:02:20.257 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:02:20.263 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:02:20.316 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:02:20.371 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:02:20.377 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:02:20.392 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:02:20.445 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-08-18T16:02:20.451 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-08-18T16:02:20.504 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-08-18T16:02:20.557 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-08-18T16:02:20.563 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-08-18T16:02:20.616 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-08-18T16:02:20.667 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-08-18T16:02:20.674 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-08-18T16:02:20.684 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-08-18T16:02:20.733 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-08-18T16:02:20.738 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-08-18T16:02:20.800 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-08-18T16:02:20.849 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-65 zones={test-zone3, test-zone1} 2024-08-18T16:02:20.849 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-65 zones={test-zone3, test-zone2} 2024-08-18T16:02:20.854 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:02:20.856 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-08-18T16:02:20.856 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-08-18T16:02:20.856 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-08-18T16:02:20.857 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-08-18T16:02:20.857 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-08-18T16:02:20.857 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-08-18T16:02:20.857 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-08-18T16:02:20.857 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-08-18T16:02:20.857 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:02:20.860 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-08-18T16:02:20.860 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-08-18T16:02:20.860 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-08-18T16:02:20.860 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-08-18T16:02:20.860 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-08-18T16:02:20.861 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-08-18T16:02:20.861 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-08-18T16:02:20.861 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-08-18T16:02:20.866 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:02:20.916 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:02:20.928 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:02:20.979 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:02:20.986 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:02:21.002 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:02:21.054 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:02:21.060 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:02:21.113 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:02:21.165 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:02:21.172 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:02:21.225 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:02:21.278 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-08-18T16:02:21.284 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-08-18T16:02:21.295 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-08-18T16:02:21.303 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-08-18T16:02:21.350 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-08-18T16:02:21.362 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-08-18T16:02:21.415 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-08-18T16:02:21.422 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-08-18T16:02:21.433 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-08-18T16:02:21.485 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-08-18T16:02:21.492 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-08-18T16:02:21.545 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-08-18T16:02:21.598 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-65 zones={test-zone3, test-zone2} 2024-08-18T16:02:21.598 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-66', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:02:21.598 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-66', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:02:21.598 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-66 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T16:02:21.659 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:02:21.659 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:02:21.699 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:02:21.698+0000 7fd167c30880 1 bucket sync caught up with source: 2024-08-18T16:02:21.699 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000004.85.6, 00000000002.59.1, 00000000004.64.6, 00000000002.74.1, 00000000004.49.6, 00000000002.108.1, 00000000002.81.1, 00000000004.77.6, 00000000002.109.1, 00000000002.64.1, 00000000002.63.1] 2024-08-18T16:02:21.699 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000004.85.6, 00000000002.59.1, 00000000004.64.6, 00000000002.74.1, 00000000004.49.6, 00000000002.108.1, 00000000002.81.1, 00000000004.77.6, 00000000002.109.1, 00000000002.64.1, 00000000002.63.1] 2024-08-18T16:02:21.699 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:02:21.698+0000 7fd167c30880 0 bucket checkpoint complete 2024-08-18T16:02:21.714 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-66', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:02:21.714 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-66', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:02:21.715 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-66 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:02:21.769 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:02:21.769 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:02:21.802 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:02:21.802+0000 7fd88fd22880 1 bucket sync caught up with source: 2024-08-18T16:02:21.802 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000004.85.6, 00000000002.59.1, 00000000004.64.6, 00000000002.74.1, 00000000004.49.6, 00000000002.108.1, 00000000002.81.1, 00000000004.77.6, 00000000002.109.1, 00000000002.64.1, 00000000002.63.1] 2024-08-18T16:02:21.803 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000004.85.6, 00000000002.59.1, 00000000004.64.6, 00000000002.74.1, 00000000004.49.6, 00000000002.108.1, 00000000002.81.1, 00000000004.77.6, 00000000002.109.1, 00000000002.64.1, 00000000002.63.1] 2024-08-18T16:02:21.803 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:02:21.802+0000 7fd88fd22880 0 bucket checkpoint complete 2024-08-18T16:02:21.815 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-66', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:02:21.816 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-66', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:02:21.816 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-66 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:02:21.869 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:02:21.869 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:02:21.905 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:02:21.904+0000 7f8624c26880 1 bucket sync caught up with source: 2024-08-18T16:02:21.905 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000004.358.1, 00000000002.225.1, 00000000004.179.1, 00000000002.336.1, 00000000004.249.1, 00000000002.221.1, 00000000002.409.1, 00000000004.337.1, 00000000002.222.1, 00000000002.233.1, 00000000002.219.1] 2024-08-18T16:02:21.905 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000004.358.1, 00000000002.225.1, 00000000004.179.1, 00000000002.336.1, 00000000004.249.1, 00000000002.221.1, 00000000002.409.1, 00000000004.337.1, 00000000002.222.1, 00000000002.233.1, 00000000002.219.1] 2024-08-18T16:02:21.905 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:02:21.904+0000 7f8624c26880 0 bucket checkpoint complete 2024-08-18T16:02:21.918 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-66', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:02:21.919 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-66', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:02:21.919 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-66 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:02:21.973 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:02:21.973 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:02:22.008 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:02:22.008+0000 7f62d0298880 1 bucket sync caught up with source: 2024-08-18T16:02:22.009 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000004.358.1, 00000000002.225.1, 00000000004.179.1, 00000000002.336.1, 00000000004.249.1, 00000000002.221.1, 00000000002.409.1, 00000000004.337.1, 00000000002.222.1, 00000000002.233.1, 00000000002.219.1] 2024-08-18T16:02:22.009 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000004.358.1, 00000000002.225.1, 00000000004.179.1, 00000000002.336.1, 00000000004.249.1, 00000000002.221.1, 00000000002.409.1, 00000000004.337.1, 00000000002.222.1, 00000000002.233.1, 00000000002.219.1] 2024-08-18T16:02:22.009 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:02:22.008+0000 7f62d0298880 0 bucket checkpoint complete 2024-08-18T16:02:22.023 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-66', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:02:22.024 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-66', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:02:22.024 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-66 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:02:22.078 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:02:22.078 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:02:22.113 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:02:22.112+0000 7f5333baf880 1 waiting for incremental sync to catch up: 2024-08-18T16:02:22.113 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000003.164.1, 00000000002.145.1, 00000000003.155.1, 00000000002.181.1, 00000000003.174.1, 00000000002.204.1, 00000000002.191.1, 00000000003.184.1, 00000000002.206.1, 00000000002.107.1, 00000000002.207.1] 2024-08-18T16:02:22.113 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000004.168.6, 00000000002.145.1, 00000000004.162.6, 00000000002.181.1, 00000000004.179.6, 00000000002.204.1, 00000000002.191.1, 00000000004.190.6, 00000000002.206.1, 00000000002.107.1, 00000000002.207.1] 2024-08-18T16:02:27.114 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:02:27.114+0000 7f5333baf880 1 waiting for incremental sync to catch up: 2024-08-18T16:02:27.114 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000004.168.6, 00000000002.145.1, 00000000003.155.1, 00000000002.181.1, 00000000004.179.6, 00000000002.204.1, 00000000002.191.1, 00000000004.190.6, 00000000002.206.1, 00000000002.107.1, 00000000002.207.1] 2024-08-18T16:02:27.115 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000004.168.6, 00000000002.145.1, 00000000004.162.6, 00000000002.181.1, 00000000004.179.6, 00000000002.204.1, 00000000002.191.1, 00000000004.190.6, 00000000002.206.1, 00000000002.107.1, 00000000002.207.1] 2024-08-18T16:02:32.115 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:02:32.115+0000 7f5333baf880 1 waiting for incremental sync to catch up: 2024-08-18T16:02:32.116 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000004.168.6, 00000000002.145.1, 00000000003.155.1, 00000000002.181.1, 00000000004.179.6, 00000000002.204.1, 00000000002.191.1, 00000000004.190.6, 00000000002.206.1, 00000000002.107.1, 00000000002.207.1] 2024-08-18T16:02:32.116 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000004.168.6, 00000000002.145.1, 00000000004.162.6, 00000000002.181.1, 00000000004.179.6, 00000000002.204.1, 00000000002.191.1, 00000000004.190.6, 00000000002.206.1, 00000000002.107.1, 00000000002.207.1] 2024-08-18T16:02:37.117 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:02:37.116+0000 7f5333baf880 1 waiting for incremental sync to catch up: 2024-08-18T16:02:37.117 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000004.168.6, 00000000002.145.1, 00000000003.155.1, 00000000002.181.1, 00000000004.179.6, 00000000002.204.1, 00000000002.191.1, 00000000004.190.6, 00000000002.206.1, 00000000002.107.1, 00000000002.207.1] 2024-08-18T16:02:37.117 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000004.168.6, 00000000002.145.1, 00000000004.162.6, 00000000002.181.1, 00000000004.179.6, 00000000002.204.1, 00000000002.191.1, 00000000004.190.6, 00000000002.206.1, 00000000002.107.1, 00000000002.207.1] 2024-08-18T16:02:42.118 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:02:42.117+0000 7f5333baf880 1 waiting for incremental sync to catch up: 2024-08-18T16:02:42.118 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000004.168.6, 00000000002.145.1, 00000000003.155.1, 00000000002.181.1, 00000000004.179.6, 00000000002.204.1, 00000000002.191.1, 00000000004.190.6, 00000000002.206.1, 00000000002.107.1, 00000000002.207.1] 2024-08-18T16:02:42.118 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000004.168.6, 00000000002.145.1, 00000000004.162.6, 00000000002.181.1, 00000000004.179.6, 00000000002.204.1, 00000000002.191.1, 00000000004.190.6, 00000000002.206.1, 00000000002.107.1, 00000000002.207.1] 2024-08-18T16:02:47.119 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:02:47.119+0000 7f5333baf880 1 bucket sync caught up with source: 2024-08-18T16:02:47.120 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000004.168.6, 00000000002.145.1, 00000000004.162.6, 00000000002.181.1, 00000000004.179.6, 00000000002.204.1, 00000000002.191.1, 00000000004.190.6, 00000000002.206.1, 00000000002.107.1, 00000000002.207.1] 2024-08-18T16:02:47.120 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000004.168.6, 00000000002.145.1, 00000000004.162.6, 00000000002.181.1, 00000000004.179.6, 00000000002.204.1, 00000000002.191.1, 00000000004.190.6, 00000000002.206.1, 00000000002.107.1, 00000000002.207.1] 2024-08-18T16:02:47.120 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:02:47.119+0000 7f5333baf880 0 bucket checkpoint complete 2024-08-18T16:02:47.137 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-66', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:02:47.137 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-66', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:02:47.137 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-66 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T16:02:47.202 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:02:47.202 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:02:47.242 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:02:47.242+0000 7fb1488bb880 1 bucket sync caught up with source: 2024-08-18T16:02:47.243 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000004.168.6, 00000000002.145.1, 00000000004.162.6, 00000000002.181.1, 00000000004.179.6, 00000000002.204.1, 00000000002.191.1, 00000000004.190.6, 00000000002.206.1, 00000000002.107.1, 00000000002.207.1] 2024-08-18T16:02:47.243 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000004.168.6, 00000000002.145.1, 00000000004.162.6, 00000000002.181.1, 00000000004.179.6, 00000000002.204.1, 00000000002.191.1, 00000000004.190.6, 00000000002.206.1, 00000000002.107.1, 00000000002.207.1] 2024-08-18T16:02:47.243 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:02:47.242+0000 7fb1488bb880 0 bucket checkpoint complete 2024-08-18T16:02:47.258 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-66 zones={test-zone2, test-zone1} 2024-08-18T16:02:47.262 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:02:47.265 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-08-18T16:02:47.265 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-08-18T16:02:47.265 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-08-18T16:02:47.265 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-08-18T16:02:47.265 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-08-18T16:02:47.266 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-08-18T16:02:47.266 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-08-18T16:02:47.266 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-08-18T16:02:47.266 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:02:47.269 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-08-18T16:02:47.269 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-08-18T16:02:47.269 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-08-18T16:02:47.269 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-08-18T16:02:47.270 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-08-18T16:02:47.270 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-08-18T16:02:47.270 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-08-18T16:02:47.270 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-08-18T16:02:47.275 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:02:47.326 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:02:47.339 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:02:47.388 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:02:47.395 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:02:47.451 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:02:47.499 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:02:47.506 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:02:47.559 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:02:47.612 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:02:47.618 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:02:47.671 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:02:47.727 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-08-18T16:02:47.734 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-08-18T16:02:47.788 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-08-18T16:02:47.839 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-08-18T16:02:47.846 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-08-18T16:02:47.899 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-08-18T16:02:47.955 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-08-18T16:02:47.963 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-08-18T16:02:48.019 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-08-18T16:02:48.072 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-08-18T16:02:48.079 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-08-18T16:02:48.135 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-08-18T16:02:48.191 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-66 zones={test-zone2, test-zone1} 2024-08-18T16:02:48.191 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-66 zones={test-zone3, test-zone1} 2024-08-18T16:02:48.195 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:02:48.198 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-08-18T16:02:48.198 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-08-18T16:02:48.198 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-08-18T16:02:48.198 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-08-18T16:02:48.198 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-08-18T16:02:48.199 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-08-18T16:02:48.199 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-08-18T16:02:48.199 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-08-18T16:02:48.199 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:02:48.201 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-08-18T16:02:48.201 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-08-18T16:02:48.202 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-08-18T16:02:48.202 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-08-18T16:02:48.202 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-08-18T16:02:48.202 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-08-18T16:02:48.202 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-08-18T16:02:48.202 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-08-18T16:02:48.207 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:02:48.262 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:02:48.277 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:02:48.329 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:02:48.336 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:02:48.391 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:02:48.441 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:02:48.449 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:02:48.505 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:02:48.554 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:02:48.561 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:02:48.617 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:02:48.670 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-08-18T16:02:48.676 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-08-18T16:02:48.688 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-08-18T16:02:48.696 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-08-18T16:02:48.702 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-08-18T16:02:48.759 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-08-18T16:02:48.766 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-08-18T16:02:48.814 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-08-18T16:02:48.824 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-08-18T16:02:48.873 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-08-18T16:02:48.879 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-08-18T16:02:48.934 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-08-18T16:02:48.941 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-66 zones={test-zone3, test-zone1} 2024-08-18T16:02:48.941 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-66 zones={test-zone3, test-zone2} 2024-08-18T16:02:48.945 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:02:48.948 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-08-18T16:02:48.948 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-08-18T16:02:48.948 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-08-18T16:02:48.948 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-08-18T16:02:48.948 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-08-18T16:02:48.948 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-08-18T16:02:48.948 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-08-18T16:02:48.948 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-08-18T16:02:48.949 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:02:48.952 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-08-18T16:02:48.952 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-08-18T16:02:48.952 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-08-18T16:02:48.952 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-08-18T16:02:48.952 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-08-18T16:02:48.952 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-08-18T16:02:48.952 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-08-18T16:02:48.952 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-08-18T16:02:48.958 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:02:49.008 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:02:49.021 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:02:49.073 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:02:49.080 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:02:49.132 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:02:49.181 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:02:49.188 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:02:49.245 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:02:49.309 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:02:49.359 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:02:49.423 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:02:49.505 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-08-18T16:02:49.561 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-08-18T16:02:49.589 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-08-18T16:02:49.646 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-08-18T16:02:49.654 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-08-18T16:02:49.668 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-08-18T16:02:49.680 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-08-18T16:02:49.734 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-08-18T16:02:49.791 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-08-18T16:02:49.843 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-08-18T16:02:49.852 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-08-18T16:02:49.866 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-08-18T16:02:49.875 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-66 zones={test-zone3, test-zone2} 2024-08-18T16:02:49.875 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-67', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:02:49.875 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-67', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:02:49.876 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-67 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T16:02:49.938 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:02:49.938 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:02:49.977 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:02:49.977+0000 7fd16f926880 1 bucket sync caught up with source: 2024-08-18T16:02:49.977 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000004.62.6, 00000000002.69.1, 00000000004.66.6, 00000000002.60.1, 00000000004.112.6, 00000000002.49.1, 00000000002.51.1, 00000000004.112.6, 00000000002.64.1, 00000000002.86.1, 00000000002.109.1] 2024-08-18T16:02:49.977 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000004.62.6, 00000000002.69.1, 00000000004.66.6, 00000000002.60.1, 00000000004.112.6, 00000000002.49.1, 00000000002.51.1, 00000000004.112.6, 00000000002.64.1, 00000000002.86.1, 00000000002.109.1] 2024-08-18T16:02:49.977 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:02:49.977+0000 7fd16f926880 0 bucket checkpoint complete 2024-08-18T16:02:49.992 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-67', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:02:49.993 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-67', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:02:49.993 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-67 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:02:50.054 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:02:50.055 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:02:50.092 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:02:50.092+0000 7f335f926880 1 bucket sync caught up with source: 2024-08-18T16:02:50.092 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000004.62.6, 00000000002.69.1, 00000000004.66.6, 00000000002.60.1, 00000000004.112.6, 00000000002.49.1, 00000000002.51.1, 00000000004.112.6, 00000000002.64.1, 00000000002.86.1, 00000000002.109.1] 2024-08-18T16:02:50.092 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000004.62.6, 00000000002.69.1, 00000000004.66.6, 00000000002.60.1, 00000000004.112.6, 00000000002.49.1, 00000000002.51.1, 00000000004.112.6, 00000000002.64.1, 00000000002.86.1, 00000000002.109.1] 2024-08-18T16:02:50.092 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:02:50.092+0000 7f335f926880 0 bucket checkpoint complete 2024-08-18T16:02:50.106 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-67', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:02:50.106 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-67', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:02:50.106 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-67 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:02:50.161 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:02:50.161 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:02:50.196 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:02:50.196+0000 7fdc5bd32880 1 bucket sync caught up with source: 2024-08-18T16:02:50.196 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000004.184.6, 00000000002.268.1, 00000000004.342.6, 00000000002.178.1, 00000000004.229.6, 00000000002.284.1, 00000000002.230.1, 00000000004.251.6, 00000000002.336.1, 00000000002.199.1, 00000000002.447.1] 2024-08-18T16:02:50.197 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000004.184.6, 00000000002.268.1, 00000000004.342.6, 00000000002.178.1, 00000000004.229.6, 00000000002.284.1, 00000000002.230.1, 00000000004.251.6, 00000000002.336.1, 00000000002.199.1, 00000000002.447.1] 2024-08-18T16:02:50.197 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:02:50.196+0000 7fdc5bd32880 0 bucket checkpoint complete 2024-08-18T16:02:50.210 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-67', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:02:50.210 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-67', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:02:50.210 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-67 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:02:50.265 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:02:50.265 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:02:50.299 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:02:50.299+0000 7f7297319880 1 bucket sync caught up with source: 2024-08-18T16:02:50.299 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000004.184.6, 00000000002.268.1, 00000000004.342.6, 00000000002.178.1, 00000000004.229.6, 00000000002.284.1, 00000000002.230.1, 00000000004.251.6, 00000000002.336.1, 00000000002.199.1, 00000000002.447.1] 2024-08-18T16:02:50.299 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000004.184.6, 00000000002.268.1, 00000000004.342.6, 00000000002.178.1, 00000000004.229.6, 00000000002.284.1, 00000000002.230.1, 00000000004.251.6, 00000000002.336.1, 00000000002.199.1, 00000000002.447.1] 2024-08-18T16:02:50.300 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:02:50.299+0000 7f7297319880 0 bucket checkpoint complete 2024-08-18T16:02:50.312 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-67', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:02:50.312 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-67', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:02:50.312 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-67 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:02:50.365 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:02:50.365 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:02:50.399 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:02:50.398+0000 7fbd3f540880 1 bucket sync caught up with source: 2024-08-18T16:02:50.399 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000004.153.1, 00000000002.134.1, 00000000004.218.1, 00000000002.154.1, 00000000004.201.1, 00000000002.154.1, 00000000002.95.1, 00000000004.296.1, 00000000002.219.1, 00000000002.254.1, 00000000002.235.1] 2024-08-18T16:02:50.399 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000004.153.1, 00000000002.134.1, 00000000004.218.1, 00000000002.154.1, 00000000004.201.1, 00000000002.154.1, 00000000002.95.1, 00000000004.296.1, 00000000002.219.1, 00000000002.254.1, 00000000002.235.1] 2024-08-18T16:02:50.399 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:02:50.398+0000 7fbd3f540880 0 bucket checkpoint complete 2024-08-18T16:02:50.412 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-67', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:02:50.412 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-67', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:02:50.413 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-67 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T16:02:50.470 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:02:50.470 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:02:50.507 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:02:50.506+0000 7f5fb723f880 1 bucket sync caught up with source: 2024-08-18T16:02:50.507 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000004.153.1, 00000000002.134.1, 00000000004.218.1, 00000000002.154.1, 00000000004.201.1, 00000000002.154.1, 00000000002.95.1, 00000000004.296.1, 00000000002.219.1, 00000000002.254.1, 00000000002.235.1] 2024-08-18T16:02:50.507 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000004.153.1, 00000000002.134.1, 00000000004.218.1, 00000000002.154.1, 00000000004.201.1, 00000000002.154.1, 00000000002.95.1, 00000000004.296.1, 00000000002.219.1, 00000000002.254.1, 00000000002.235.1] 2024-08-18T16:02:50.507 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:02:50.506+0000 7f5fb723f880 0 bucket checkpoint complete 2024-08-18T16:02:50.521 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-67 zones={test-zone2, test-zone1} 2024-08-18T16:02:50.525 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:02:50.528 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-08-18T16:02:50.528 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-08-18T16:02:50.529 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-08-18T16:02:50.529 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-08-18T16:02:50.529 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-08-18T16:02:50.529 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-08-18T16:02:50.529 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-08-18T16:02:50.529 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-08-18T16:02:50.529 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:02:50.532 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-08-18T16:02:50.532 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-08-18T16:02:50.532 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-08-18T16:02:50.532 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-08-18T16:02:50.532 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-08-18T16:02:50.532 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-08-18T16:02:50.532 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-08-18T16:02:50.533 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-08-18T16:02:50.538 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:02:50.588 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:02:50.598 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:02:50.606 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:02:50.612 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:02:50.670 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:02:50.678 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:02:50.728 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:02:50.739 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:02:50.788 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:02:50.795 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:02:50.851 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:02:50.903 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-08-18T16:02:50.954 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-08-18T16:02:50.965 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-08-18T16:02:50.973 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-08-18T16:02:51.028 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-08-18T16:02:51.045 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-08-18T16:02:51.101 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-08-18T16:02:51.107 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-08-18T16:02:51.162 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-08-18T16:02:51.169 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-08-18T16:02:51.225 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-08-18T16:02:51.238 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-08-18T16:02:51.289 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-67 zones={test-zone2, test-zone1} 2024-08-18T16:02:51.289 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-67 zones={test-zone3, test-zone1} 2024-08-18T16:02:51.293 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:02:51.296 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-08-18T16:02:51.296 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-08-18T16:02:51.296 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-08-18T16:02:51.296 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-08-18T16:02:51.296 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-08-18T16:02:51.296 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-08-18T16:02:51.296 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-08-18T16:02:51.296 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-08-18T16:02:51.296 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:02:51.299 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-08-18T16:02:51.299 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-08-18T16:02:51.299 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-08-18T16:02:51.299 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-08-18T16:02:51.299 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-08-18T16:02:51.299 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-08-18T16:02:51.299 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-08-18T16:02:51.299 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-08-18T16:02:51.304 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:02:51.351 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:02:51.403 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:02:51.455 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:02:51.461 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:02:51.471 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:02:51.521 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:02:51.570 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:02:51.580 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:02:51.628 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:02:51.634 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:02:51.686 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:02:51.694 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-08-18T16:02:51.700 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-08-18T16:02:51.754 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-08-18T16:02:51.761 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-08-18T16:02:51.811 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-08-18T16:02:51.862 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-08-18T16:02:51.870 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-08-18T16:02:51.877 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-08-18T16:02:51.928 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-08-18T16:02:51.935 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-08-18T16:02:51.941 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-08-18T16:02:51.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-08-18T16:02:52.043 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-67 zones={test-zone3, test-zone1} 2024-08-18T16:02:52.043 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-67 zones={test-zone3, test-zone2} 2024-08-18T16:02:52.048 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:02:52.051 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-08-18T16:02:52.051 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-08-18T16:02:52.051 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-08-18T16:02:52.051 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-08-18T16:02:52.051 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-08-18T16:02:52.051 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-08-18T16:02:52.051 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-08-18T16:02:52.051 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-08-18T16:02:52.051 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:02:52.055 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-08-18T16:02:52.055 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-08-18T16:02:52.055 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-08-18T16:02:52.055 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-08-18T16:02:52.055 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-08-18T16:02:52.055 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-08-18T16:02:52.055 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-08-18T16:02:52.055 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-08-18T16:02:52.062 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:02:52.110 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:02:52.122 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-08-18T16:02:52.173 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:02:52.181 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:02:52.239 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-08-18T16:02:52.292 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:02:52.300 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:02:52.312 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-08-18T16:02:52.363 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:02:52.370 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:02:52.382 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-08-18T16:02:52.431 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-08-18T16:02:52.438 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-08-18T16:02:52.497 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-08-18T16:02:52.550 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-08-18T16:02:52.558 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-08-18T16:02:52.617 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-08-18T16:02:52.669 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-08-18T16:02:52.680 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-08-18T16:02:52.691 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-08-18T16:02:52.740 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-08-18T16:02:52.747 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-08-18T16:02:52.803 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-08-18T16:02:52.851 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-67 zones={test-zone3, test-zone2} 2024-08-18T16:02:52.851 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:02:52.852 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:02:52.852 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:02:52.899 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:02:52.899 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:02:52.947 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T16:02:52.947 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:02:52.948 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000011666", 2024-08-18T16:02:52.948 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:02:14.274938Z" 2024-08-18T16:02:52.948 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:02:52.948 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:02:52.948 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000009953", 2024-08-18T16:02:52.948 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:02:15.302119Z" 2024-08-18T16:02:52.948 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:02:52.948 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:02:52.948 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000017420", 2024-08-18T16:02:52.948 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:02:14.276143Z" 2024-08-18T16:02:52.948 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:02:52.949 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:02:52.949 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000014882", 2024-08-18T16:02:52.949 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:01:53.478894Z" 2024-08-18T16:02:52.949 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:02:52.949 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T16:02:52.950 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-08-18T16:02:52.950 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:02:52.950 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:02:52.950 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:02:53.006 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:02:53.006 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:02:53.056 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T16:02:53.057 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T16:02:53.057 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T16:02:53.057 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T16:02:53.057 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T16:02:53.057 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 8935070562734594544 2024-08-18T16:02:53.057 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:02:53.057 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T16:02:53.057 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:02:53.057 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T16:02:53.057 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:02:53.057 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T16:02:53.057 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000011666", 2024-08-18T16:02:53.057 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:02:53.058 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:02:53.058 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:02:53.058 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:02:14.274938Z" 2024-08-18T16:02:53.058 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:02:53.058 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:02:53.058 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:02:53.058 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T16:02:53.058 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:02:53.058 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T16:02:53.058 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000009953", 2024-08-18T16:02:53.058 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:02:53.058 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:02:53.059 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:02:53.059 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:02:15.302119Z" 2024-08-18T16:02:53.059 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:02:53.059 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:02:53.059 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:02:53.059 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T16:02:53.059 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:02:53.059 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T16:02:53.059 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000017420", 2024-08-18T16:02:53.059 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:02:53.059 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:02:53.059 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:02:53.060 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:02:14.276143Z" 2024-08-18T16:02:53.060 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:02:53.060 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:02:53.060 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:02:53.060 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T16:02:53.060 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:02:53.060 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T16:02:53.060 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000014882", 2024-08-18T16:02:53.060 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:02:53.060 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:02:53.060 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:02:53.060 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:01:53.478894Z" 2024-08-18T16:02:53.060 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:02:53.061 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:02:53.061 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T16:02:53.061 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:02:53.061 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T16:02:53.061 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 0, 2024-08-18T16:02:53.061 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 0 2024-08-18T16:02:53.061 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:02:53.061 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T16:02:53Z" 2024-08-18T16:02:53.061 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T16:02:53.062 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-08-18T16:02:53.062 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:02:53.062 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:02:53.063 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:02:53.109 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:02:53.110 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:02:53.161 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T16:02:53.161 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:02:53.161 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000011666", 2024-08-18T16:02:53.161 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:02:14.274938Z" 2024-08-18T16:02:53.161 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:02:53.161 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:02:53.161 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000009953", 2024-08-18T16:02:53.162 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:02:15.302119Z" 2024-08-18T16:02:53.162 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:02:53.162 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:02:53.162 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000017420", 2024-08-18T16:02:53.162 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:02:14.276143Z" 2024-08-18T16:02:53.162 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:02:53.162 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:02:53.162 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000014882", 2024-08-18T16:02:53.162 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:01:53.478894Z" 2024-08-18T16:02:53.162 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:02:53.162 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T16:02:53.163 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2024-08-18T16:02:53.164 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:02:53.164 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:02:53.164 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:02:53.214 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:02:53.214 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:02:53.256 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T16:02:53.256 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T16:02:53.256 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T16:02:53.256 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T16:02:53.256 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T16:02:53.256 INFO:teuthology.orchestra.run.smithi134.stdout: "instance_id": 6908254841848037526 2024-08-18T16:02:53.256 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:02:53.257 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T16:02:53.257 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:02:53.257 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T16:02:53.257 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:02:53.257 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T16:02:53.257 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000011666", 2024-08-18T16:02:53.257 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:02:53.257 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:02:53.257 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:02:53.257 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:02:14.274938Z" 2024-08-18T16:02:53.257 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:02:53.257 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:02:53.257 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:02:53.258 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T16:02:53.258 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:02:53.258 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T16:02:53.258 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000009953", 2024-08-18T16:02:53.258 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:02:53.258 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:02:53.258 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:02:53.258 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:02:15.302119Z" 2024-08-18T16:02:53.258 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:02:53.258 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:02:53.258 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:02:53.258 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T16:02:53.258 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:02:53.259 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T16:02:53.259 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000017420", 2024-08-18T16:02:53.259 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:02:53.259 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:02:53.259 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:02:53.259 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:02:14.276143Z" 2024-08-18T16:02:53.259 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:02:53.259 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:02:53.259 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:02:53.259 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T16:02:53.259 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:02:53.259 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T16:02:53.259 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000014882", 2024-08-18T16:02:53.259 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:02:53.260 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:02:53.260 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:02:53.260 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:01:53.478894Z" 2024-08-18T16:02:53.260 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:02:53.260 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:02:53.260 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T16:02:53.260 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:02:53.260 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T16:02:53.260 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 0, 2024-08-18T16:02:53.260 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 0 2024-08-18T16:02:53.260 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:02:53.260 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T16:02:53Z" 2024-08-18T16:02:53.260 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T16:02:53.261 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2024-08-18T16:02:53.261 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:02:53.262 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:02:53.262 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:02:53.312 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:02:53.312 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:02:53.363 INFO:teuthology.orchestra.run.smithi193.stdout:[ 2024-08-18T16:02:53.363 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:02:53.363 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000008558", 2024-08-18T16:02:53.363 INFO:teuthology.orchestra.run.smithi193.stdout: "last_update": "2024-08-18T16:02:31.685488Z" 2024-08-18T16:02:53.363 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:02:53.363 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:02:53.363 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000008120", 2024-08-18T16:02:53.363 INFO:teuthology.orchestra.run.smithi193.stdout: "last_update": "2024-08-18T16:02:31.687143Z" 2024-08-18T16:02:53.364 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:02:53.364 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:02:53.364 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000013895", 2024-08-18T16:02:53.364 INFO:teuthology.orchestra.run.smithi193.stdout: "last_update": "2024-08-18T16:02:31.688767Z" 2024-08-18T16:02:53.364 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:02:53.364 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:02:53.364 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000011077", 2024-08-18T16:02:53.364 INFO:teuthology.orchestra.run.smithi193.stdout: "last_update": "2024-08-18T16:02:09.683687Z" 2024-08-18T16:02:53.364 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:02:53.364 INFO:teuthology.orchestra.run.smithi193.stdout:] 2024-08-18T16:02:53.365 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2024-08-18T16:02:53.365 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:02:53.365 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:02:53.366 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:02:53.414 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:02:53.414 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:02:53.458 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T16:02:53.458 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T16:02:53.458 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T16:02:53.458 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T16:02:53.458 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T16:02:53.458 INFO:teuthology.orchestra.run.smithi134.stdout: "instance_id": 15510818030922206205 2024-08-18T16:02:53.458 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:02:53.458 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T16:02:53.458 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:02:53.458 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T16:02:53.458 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:02:53.459 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T16:02:53.459 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000008558", 2024-08-18T16:02:53.459 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:02:53.459 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:02:53.459 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:02:53.459 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:02:31.685488Z" 2024-08-18T16:02:53.459 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:02:53.459 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:02:53.459 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:02:53.459 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T16:02:53.459 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:02:53.459 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T16:02:53.459 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000008120", 2024-08-18T16:02:53.460 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:02:53.460 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:02:53.460 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:02:53.460 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:02:31.687143Z" 2024-08-18T16:02:53.460 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:02:53.460 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:02:53.460 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:02:53.460 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T16:02:53.460 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:02:53.460 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T16:02:53.460 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000013895", 2024-08-18T16:02:53.461 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:02:53.461 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:02:53.461 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:02:53.461 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:02:31.688767Z" 2024-08-18T16:02:53.461 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:02:53.461 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:02:53.461 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:02:53.461 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T16:02:53.461 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:02:53.461 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T16:02:53.461 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000011077", 2024-08-18T16:02:53.461 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:02:53.461 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:02:53.462 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:02:53.462 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:02:09.683687Z" 2024-08-18T16:02:53.462 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:02:53.462 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:02:53.462 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T16:02:53.462 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:02:53.462 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T16:02:53.462 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 0, 2024-08-18T16:02:53.462 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 0 2024-08-18T16:02:53.462 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:02:53.462 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T16:02:53Z" 2024-08-18T16:02:53.462 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T16:02:53.463 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2024-08-18T16:02:53.464 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:02:53.464 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:02:53.464 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:02:53.510 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:02:53.511 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:02:53.561 INFO:teuthology.orchestra.run.smithi193.stdout:[ 2024-08-18T16:02:53.561 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:02:53.561 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000008558", 2024-08-18T16:02:53.561 INFO:teuthology.orchestra.run.smithi193.stdout: "last_update": "2024-08-18T16:02:31.685488Z" 2024-08-18T16:02:53.561 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:02:53.561 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:02:53.561 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000008120", 2024-08-18T16:02:53.561 INFO:teuthology.orchestra.run.smithi193.stdout: "last_update": "2024-08-18T16:02:31.687143Z" 2024-08-18T16:02:53.561 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:02:53.561 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:02:53.561 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000013895", 2024-08-18T16:02:53.561 INFO:teuthology.orchestra.run.smithi193.stdout: "last_update": "2024-08-18T16:02:31.688767Z" 2024-08-18T16:02:53.562 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:02:53.562 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:02:53.562 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000011077", 2024-08-18T16:02:53.562 INFO:teuthology.orchestra.run.smithi193.stdout: "last_update": "2024-08-18T16:02:09.683687Z" 2024-08-18T16:02:53.562 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:02:53.562 INFO:teuthology.orchestra.run.smithi193.stdout:] 2024-08-18T16:02:53.563 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2024-08-18T16:02:53.563 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:02:53.563 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:02:53.563 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:02:53.612 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:02:53.613 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:02:53.655 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T16:02:53.655 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T16:02:53.655 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T16:02:53.655 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T16:02:53.655 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T16:02:53.655 INFO:teuthology.orchestra.run.smithi134.stdout: "instance_id": 9282142518545740118 2024-08-18T16:02:53.655 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:02:53.655 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T16:02:53.656 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:02:53.656 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T16:02:53.656 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:02:53.656 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T16:02:53.656 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000008558", 2024-08-18T16:02:53.656 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:02:53.656 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:02:53.656 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:02:53.656 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:02:31.685488Z" 2024-08-18T16:02:53.656 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:02:53.656 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:02:53.656 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:02:53.656 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T16:02:53.656 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:02:53.657 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T16:02:53.657 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000008120", 2024-08-18T16:02:53.657 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:02:53.657 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:02:53.657 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:02:53.657 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:02:31.687143Z" 2024-08-18T16:02:53.657 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:02:53.657 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:02:53.657 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:02:53.657 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T16:02:53.657 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:02:53.657 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T16:02:53.657 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000013895", 2024-08-18T16:02:53.657 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:02:53.658 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:02:53.658 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:02:53.658 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:02:31.688767Z" 2024-08-18T16:02:53.658 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:02:53.658 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:02:53.658 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:02:53.658 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T16:02:53.658 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:02:53.658 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T16:02:53.659 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000011077", 2024-08-18T16:02:53.659 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:02:53.659 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:02:53.659 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:02:53.659 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:02:09.683687Z" 2024-08-18T16:02:53.659 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:02:53.659 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:02:53.659 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T16:02:53.659 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:02:53.659 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T16:02:53.659 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 0, 2024-08-18T16:02:53.659 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 0 2024-08-18T16:02:53.659 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:02:53.659 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T16:02:53Z" 2024-08-18T16:02:53.659 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T16:02:53.660 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2024-08-18T16:02:53.661 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:02:53.661 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:02:53.661 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:02:53.707 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:02:53.707 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:02:53.754 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T16:02:53.754 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:02:53.754 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000008700", 2024-08-18T16:02:53.754 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:02:31.773392Z" 2024-08-18T16:02:53.754 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:02:53.754 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:02:53.754 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000008402", 2024-08-18T16:02:53.755 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:02:31.775134Z" 2024-08-18T16:02:53.755 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:02:53.755 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:02:53.755 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000014177", 2024-08-18T16:02:53.755 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:02:31.776725Z" 2024-08-18T16:02:53.755 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:02:53.755 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:02:53.755 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000012486", 2024-08-18T16:02:53.755 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:02:09.771525Z" 2024-08-18T16:02:53.755 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:02:53.755 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T16:02:53.756 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2024-08-18T16:02:53.756 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:02:53.756 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:02:53.757 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:02:53.805 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:02:53.806 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:02:53.849 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T16:02:53.849 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T16:02:53.849 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T16:02:53.849 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T16:02:53.849 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T16:02:53.849 INFO:teuthology.orchestra.run.smithi134.stdout: "instance_id": 16929744281786545893 2024-08-18T16:02:53.849 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:02:53.849 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T16:02:53.849 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:02:53.849 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T16:02:53.849 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:02:53.849 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T16:02:53.850 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000008700", 2024-08-18T16:02:53.850 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:02:53.850 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:02:53.850 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:02:53.850 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:02:31.773392Z" 2024-08-18T16:02:53.850 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:02:53.850 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:02:53.850 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:02:53.850 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T16:02:53.850 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:02:53.850 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T16:02:53.850 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000008402", 2024-08-18T16:02:53.850 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:02:53.851 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:02:53.851 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:02:53.851 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:02:31.775134Z" 2024-08-18T16:02:53.851 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:02:53.851 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:02:53.851 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:02:53.851 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T16:02:53.851 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:02:53.851 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T16:02:53.851 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000014177", 2024-08-18T16:02:53.851 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:02:53.851 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:02:53.851 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:02:53.851 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:02:31.776725Z" 2024-08-18T16:02:53.852 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:02:53.852 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:02:53.852 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:02:53.852 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T16:02:53.852 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:02:53.852 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T16:02:53.852 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000012486", 2024-08-18T16:02:53.852 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:02:53.852 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:02:53.852 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:02:53.852 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:02:09.771525Z" 2024-08-18T16:02:53.852 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:02:53.852 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:02:53.852 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T16:02:53.853 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:02:53.853 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T16:02:53.853 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 0, 2024-08-18T16:02:53.853 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 0 2024-08-18T16:02:53.853 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:02:53.853 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T16:02:53Z" 2024-08-18T16:02:53.853 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T16:02:53.854 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2024-08-18T16:02:53.854 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:02:53.854 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:02:53.855 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:02:53.902 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:02:53.902 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:02:53.948 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T16:02:53.948 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:02:53.948 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000008700", 2024-08-18T16:02:53.948 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:02:31.773392Z" 2024-08-18T16:02:53.948 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:02:53.948 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:02:53.948 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000008402", 2024-08-18T16:02:53.949 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:02:31.775134Z" 2024-08-18T16:02:53.949 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:02:53.949 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:02:53.949 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000014177", 2024-08-18T16:02:53.949 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:02:31.776725Z" 2024-08-18T16:02:53.949 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:02:53.949 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:02:53.949 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000012486", 2024-08-18T16:02:53.949 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:02:09.771525Z" 2024-08-18T16:02:53.949 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:02:53.949 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T16:02:53.950 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2024-08-18T16:02:53.951 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:02:53.951 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:02:53.951 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:02:54.005 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:02:54.005 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:02:54.052 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T16:02:54.053 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T16:02:54.053 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T16:02:54.053 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T16:02:54.053 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T16:02:54.053 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17055221296585213178 2024-08-18T16:02:54.053 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:02:54.053 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T16:02:54.053 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:02:54.053 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T16:02:54.053 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:02:54.053 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T16:02:54.053 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000008700", 2024-08-18T16:02:54.054 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:02:54.054 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:02:54.054 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:02:54.054 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:02:31.773392Z" 2024-08-18T16:02:54.054 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:02:54.054 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:02:54.054 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:02:54.054 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T16:02:54.054 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:02:54.054 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T16:02:54.054 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000008402", 2024-08-18T16:02:54.055 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:02:54.055 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:02:54.055 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:02:54.055 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:02:31.775134Z" 2024-08-18T16:02:54.055 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:02:54.055 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:02:54.055 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:02:54.055 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T16:02:54.055 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:02:54.055 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T16:02:54.055 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000014177", 2024-08-18T16:02:54.055 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:02:54.056 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:02:54.056 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:02:54.056 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:02:31.776725Z" 2024-08-18T16:02:54.056 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:02:54.056 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:02:54.056 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:02:54.056 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T16:02:54.056 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:02:54.056 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T16:02:54.056 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000012486", 2024-08-18T16:02:54.056 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:02:54.056 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:02:54.056 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:02:54.057 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:02:09.771525Z" 2024-08-18T16:02:54.057 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:02:54.057 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:02:54.057 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T16:02:54.057 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:02:54.057 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T16:02:54.057 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 0, 2024-08-18T16:02:54.057 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 0 2024-08-18T16:02:54.057 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:02:54.057 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T16:02:54Z" 2024-08-18T16:02:54.057 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T16:02:54.058 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2024-08-18T16:02:54.059 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_disable_enable ... ok 2024-08-18T16:02:54.059 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=kibgmb-68 2024-08-18T16:02:54.074 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=kibgmb-69 2024-08-18T16:02:54.100 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=kibgmb-70 2024-08-18T16:02:55.684 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:02:55.684 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:02:55.684 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:02:55.730 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:02:55.730 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:02:55.761 INFO:teuthology.orchestra.run.smithi134.stderr:No --period given, using current period=01050e30-179b-43be-bdbb-3bb23cf37ae8 2024-08-18T16:02:55.773 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T16:02:55.773 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:02:55.773 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T16:02:55.774 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T16:02:55.774 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:02:55.774 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:02:55.774 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T16:02:55.774 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T16:02:55.774 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:02:55.774 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:02:55.774 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T16:02:55.774 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T16:02:55.774 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:02:55.774 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:02:55.774 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723996974.115504_640.1", 2024-08-18T16:02:55.774 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:02:54.115504Z" 2024-08-18T16:02:55.775 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:02:55.775 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T16:02:55.775 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-08-18T16:02:55.776 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:02:55.776 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:02:55.776 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:02:55.830 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:02:55.830 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:02:55.862 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:02:55.862+0000 7f2720931880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:02:55.863 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:02:55.863+0000 7f2720931880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:02:55.877 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T16:02:55.877 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T16:02:55.877 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T16:02:55.877 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T16:02:55.877 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T16:02:55.877 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T16:02:55.877 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T16:02:55.877 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:02:55.877 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T16:02:55.877 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:02:55.878 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T16:02:55.878 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:02:55.878 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:02:55.878 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T16:02:55.878 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:02:55.878 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T16:02:55.878 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:02:55.878 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T16:02:55.878 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T16:02:55.878 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:02:55.878 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:02:55.878 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:02:55.878 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T16:02:55.879 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:02:55.879 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:02:55.879 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T16:02:55.879 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:02:55.879 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:02:55.879 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:02:55.879 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T16:02:55.879 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T16:02:55.879 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:02:55.879 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:02:55.879 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:02:55.879 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T16:02:55.879 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:02:55.879 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:02:55.880 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T16:02:55.880 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:02:55.880 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:02:55.880 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:02:55.880 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T16:02:55.880 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T16:02:55.880 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:02:55.880 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:02:55.880 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:02:55.880 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T16:02:55.880 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:02:55.880 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:02:55.880 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723996974.086391_633.1", 2024-08-18T16:02:55.881 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:02:55.881 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:02:55.881 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:02:55.881 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:02:54.086391Z", 2024-08-18T16:02:55.881 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T16:02:55.881 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:02:55.881 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:02:55.881 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T16:02:55.881 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:02:55.881 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T16:02:55.881 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T16:02:55.881 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T16:02:55.881 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:02:55.881 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T16:02:55Z" 2024-08-18T16:02:55.882 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T16:02:55.882 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1723996974.115504_640.1 target=1_1723996974.086391_633.1 2024-08-18T16:03:00.884 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:03:00.884 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:03:00.884 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:03:00.939 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:03:00.940 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:03:00.973 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:03:00.973+0000 7f0fe8c15880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:03:00.974 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:03:00.974+0000 7f0fe8c15880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:03:00.988 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T16:03:00.989 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T16:03:00.989 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T16:03:00.989 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T16:03:00.989 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T16:03:00.989 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T16:03:00.989 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T16:03:00.989 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:03:00.989 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T16:03:00.989 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:03:00.989 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T16:03:00.989 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:03:00.990 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:03:00.990 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T16:03:00.990 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:03:00.990 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T16:03:00.990 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:03:00.990 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T16:03:00.990 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T16:03:00.990 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:03:00.990 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:03:00.990 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:03:00.990 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T16:03:00.990 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:03:00.990 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:03:00.991 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T16:03:00.991 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:03:00.991 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:03:00.991 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:03:00.991 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T16:03:00.991 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T16:03:00.991 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:03:00.991 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:03:00.991 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:03:00.991 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T16:03:00.991 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:03:00.991 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:03:00.991 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T16:03:00.992 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:03:00.992 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:03:00.992 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:03:00.992 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T16:03:00.992 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T16:03:00.992 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:03:00.992 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:03:00.992 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:03:00.992 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T16:03:00.992 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:03:00.992 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:03:00.992 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723996974.115504_640.1", 2024-08-18T16:03:00.992 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:03:00.993 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:03:00.993 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:03:00.993 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:02:54.115504Z", 2024-08-18T16:03:00.993 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T16:03:00.993 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:03:00.993 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:03:00.993 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T16:03:00.993 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:03:00.993 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T16:03:00.993 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T16:03:00.993 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T16:03:00.993 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:03:00.994 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T16:03:00Z" 2024-08-18T16:03:00.994 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T16:03:00.994 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-08-18T16:03:00.995 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-08-18T16:03:00.995 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:03:00.995 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:03:00.995 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:03:01.050 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:03:01.050 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:03:01.082 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:03:01.082+0000 7f93c2d24880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:03:01.083 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:03:01.083+0000 7f93c2d24880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:03:01.097 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T16:03:01.097 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T16:03:01.097 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T16:03:01.097 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T16:03:01.097 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T16:03:01.097 INFO:teuthology.orchestra.run.smithi134.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T16:03:01.097 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T16:03:01.097 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:03:01.097 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T16:03:01.097 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:03:01.097 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T16:03:01.098 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:03:01.098 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T16:03:01.098 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T16:03:01.098 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:03:01.098 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 1, 2024-08-18T16:03:01.098 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:03:01.098 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T16:03:01.098 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T16:03:01.098 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:03:01.098 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:03:01.098 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:03:01.098 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T16:03:01.098 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:03:01.099 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T16:03:01.099 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T16:03:01.099 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:03:01.099 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:03:01.099 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:03:01.099 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T16:03:01.099 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T16:03:01.099 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:03:01.099 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:03:01.099 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:03:01.099 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T16:03:01.099 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:03:01.099 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T16:03:01.099 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T16:03:01.100 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:03:01.100 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:03:01.100 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:03:01.100 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T16:03:01.100 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T16:03:01.100 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:03:01.100 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:03:01.100 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:03:01.100 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T16:03:01.100 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:03:01.100 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T16:03:01.100 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723996974.115504_640.1", 2024-08-18T16:03:01.100 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:03:01.101 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:03:01.101 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:03:01.101 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:02:54.115504Z", 2024-08-18T16:03:01.101 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T16:03:01.101 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:03:01.101 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:03:01.101 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T16:03:01.101 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:03:01.101 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T16:03:01.101 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 1, 2024-08-18T16:03:01.101 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 1 2024-08-18T16:03:01.101 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:03:01.101 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T16:03:01Z" 2024-08-18T16:03:01.101 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T16:03:01.102 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-08-18T16:03:01.103 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-68', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:03:01.103 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-68', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:03:01.103 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-68 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T16:03:01.160 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:03:01.160 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:03:01.195 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:03:01.194+0000 7f993a124880 1 waiting to reach incremental sync.. 2024-08-18T16:03:06.196 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:03:06.195+0000 7f993a124880 1 waiting to reach incremental sync.. 2024-08-18T16:03:11.196 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:03:11.195+0000 7f993a124880 1 waiting to reach incremental sync.. 2024-08-18T16:03:16.198 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:03:16.196+0000 7f993a124880 1 bucket sync caught up with source: 2024-08-18T16:03:16.198 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000006.94.6, , , , , ] 2024-08-18T16:03:16.198 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000006.94.6, , , , , ] 2024-08-18T16:03:16.198 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:03:16.196+0000 7f993a124880 0 bucket checkpoint complete 2024-08-18T16:03:16.216 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-68', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:03:16.216 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-68', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:03:16.217 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-68 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:03:16.277 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:03:16.277 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:03:16.311 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:03:16.310+0000 7f722c410880 1 bucket sync caught up with source: 2024-08-18T16:03:16.311 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , 00000000006.94.6, , , , , ] 2024-08-18T16:03:16.311 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , 00000000006.94.6, , , , , ] 2024-08-18T16:03:16.311 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:03:16.310+0000 7f722c410880 0 bucket checkpoint complete 2024-08-18T16:03:16.324 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-68', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:03:16.324 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-68', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:03:16.324 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-68 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:03:16.385 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:03:16.385 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:03:16.419 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:03:16.418+0000 7f1c14e17880 1 waiting to reach incremental sync.. 2024-08-18T16:03:21.420 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:03:21.419+0000 7f1c14e17880 1 waiting to reach incremental sync.. 2024-08-18T16:03:26.420 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:03:26.420+0000 7f1c14e17880 1 waiting to reach incremental sync.. 2024-08-18T16:03:31.420 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:03:31.420+0000 7f1c14e17880 1 waiting to reach incremental sync.. 2024-08-18T16:03:36.422 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:03:36.422+0000 7f1c14e17880 1 bucket sync caught up with source: 2024-08-18T16:03:36.422 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , 00000000001.242.6, , , , , ] 2024-08-18T16:03:36.422 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , 00000000001.242.6, , , , , ] 2024-08-18T16:03:36.423 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:03:36.422+0000 7f1c14e17880 0 bucket checkpoint complete 2024-08-18T16:03:36.446 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-68', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:03:36.446 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-68', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:03:36.447 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-68 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:03:36.517 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:03:36.518 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:03:36.553 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:03:36.553+0000 7f7d83e62880 1 bucket sync caught up with source: 2024-08-18T16:03:36.554 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , 00000000001.242.6, , , , , ] 2024-08-18T16:03:36.554 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , 00000000001.242.6, , , , , ] 2024-08-18T16:03:36.554 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:03:36.553+0000 7f7d83e62880 0 bucket checkpoint complete 2024-08-18T16:03:36.570 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-68', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:03:36.570 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-68', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:03:36.570 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-68 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:03:36.627 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:03:36.627 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:03:36.662 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:03:36.661+0000 7fda6941f880 1 bucket sync caught up with source: 2024-08-18T16:03:36.662 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , 00000000001.230.6, , , , , ] 2024-08-18T16:03:36.662 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , 00000000001.230.6, , , , , ] 2024-08-18T16:03:36.662 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:03:36.661+0000 7fda6941f880 0 bucket checkpoint complete 2024-08-18T16:03:36.676 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-68', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:03:36.676 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-68', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:03:36.677 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-68 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T16:03:36.731 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:03:36.731 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:03:36.767 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:03:36.766+0000 7f60d7338880 1 bucket sync caught up with source: 2024-08-18T16:03:36.767 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000001.230.6, , , , , ] 2024-08-18T16:03:36.767 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000001.230.6, , , , , ] 2024-08-18T16:03:36.767 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:03:36.766+0000 7f60d7338880 0 bucket checkpoint complete 2024-08-18T16:03:36.782 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-68 zones={test-zone2, test-zone1} 2024-08-18T16:03:36.788 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:03:36.793 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2024-08-18T16:03:36.793 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:03:36.797 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2024-08-18T16:03:36.843 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2024-08-18T16:03:37.081 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2024-08-18T16:03:37.284 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2024-08-18T16:03:37.487 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-68 zones={test-zone2, test-zone1} 2024-08-18T16:03:37.487 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-68 zones={test-zone3, test-zone1} 2024-08-18T16:03:37.491 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:03:37.493 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2024-08-18T16:03:37.493 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:03:37.543 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2024-08-18T16:03:37.548 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2024-08-18T16:03:37.750 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2024-08-18T16:03:38.002 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2024-08-18T16:03:38.204 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-68 zones={test-zone3, test-zone1} 2024-08-18T16:03:38.204 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-68 zones={test-zone3, test-zone2} 2024-08-18T16:03:38.208 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:03:38.211 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2024-08-18T16:03:38.211 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:03:38.213 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2024-08-18T16:03:38.218 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2024-08-18T16:03:38.430 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2024-08-18T16:03:38.617 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2024-08-18T16:03:38.835 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-68 zones={test-zone3, test-zone2} 2024-08-18T16:03:38.835 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multipart_object_sync ... ok 2024-08-18T16:03:38.836 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=kibgmb-71 2024-08-18T16:03:38.953 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:03:38.954 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:03:38.954 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:03:39.000 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:03:39.000 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:03:39.030 INFO:teuthology.orchestra.run.smithi134.stderr:No --period given, using current period=01050e30-179b-43be-bdbb-3bb23cf37ae8 2024-08-18T16:03:39.043 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T16:03:39.043 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:03:39.043 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T16:03:39.043 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T16:03:39.043 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:03:39.043 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:03:39.043 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T16:03:39.043 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T16:03:39.043 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:03:39.043 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:03:39.043 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T16:03:39.044 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T16:03:39.044 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:03:39.044 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:03:39.044 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723997018.849359_648.1", 2024-08-18T16:03:39.044 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:03:38.849359Z" 2024-08-18T16:03:39.044 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:03:39.044 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T16:03:39.045 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-08-18T16:03:39.045 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:03:39.045 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:03:39.045 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:03:39.095 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:03:39.095 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:03:39.126 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:03:39.125+0000 7f4435226880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:03:39.126 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:03:39.126+0000 7f4435226880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:03:39.143 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T16:03:39.143 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T16:03:39.143 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T16:03:39.143 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T16:03:39.143 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T16:03:39.143 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T16:03:39.144 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T16:03:39.144 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:03:39.144 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T16:03:39.144 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:03:39.144 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T16:03:39.144 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:03:39.144 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:03:39.144 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T16:03:39.144 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:03:39.144 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T16:03:39.144 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:03:39.144 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T16:03:39.144 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T16:03:39.144 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:03:39.145 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:03:39.145 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:03:39.145 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T16:03:39.145 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:03:39.145 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:03:39.145 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T16:03:39.145 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:03:39.145 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:03:39.145 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:03:39.145 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T16:03:39.145 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T16:03:39.145 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:03:39.145 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:03:39.146 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:03:39.146 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T16:03:39.146 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:03:39.146 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:03:39.146 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T16:03:39.146 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:03:39.146 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:03:39.146 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:03:39.146 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T16:03:39.146 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T16:03:39.146 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:03:39.146 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:03:39.146 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:03:39.146 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T16:03:39.147 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:03:39.147 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:03:39.147 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723997018.843566_645.1", 2024-08-18T16:03:39.147 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:03:39.147 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:03:39.147 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:03:39.147 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:03:38.843566Z", 2024-08-18T16:03:39.147 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T16:03:39.147 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:03:39.147 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:03:39.147 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T16:03:39.147 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:03:39.147 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T16:03:39.147 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T16:03:39.147 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T16:03:39.148 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:03:39.148 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T16:03:39Z" 2024-08-18T16:03:39.148 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T16:03:39.149 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1723997018.849359_648.1 target=1_1723997018.843566_645.1 2024-08-18T16:03:44.150 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:03:44.150 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:03:44.150 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:03:44.206 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:03:44.206 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:03:44.240 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:03:44.239+0000 7ffaf932c880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:03:44.240 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:03:44.240+0000 7ffaf932c880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:03:44.255 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T16:03:44.255 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T16:03:44.255 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T16:03:44.256 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T16:03:44.256 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T16:03:44.256 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T16:03:44.256 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T16:03:44.256 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:03:44.256 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T16:03:44.256 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:03:44.256 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T16:03:44.256 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:03:44.256 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:03:44.256 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T16:03:44.256 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:03:44.257 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T16:03:44.257 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:03:44.257 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T16:03:44.257 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T16:03:44.257 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:03:44.257 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:03:44.257 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:03:44.257 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T16:03:44.257 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:03:44.257 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:03:44.257 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T16:03:44.257 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:03:44.257 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:03:44.257 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:03:44.258 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T16:03:44.258 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T16:03:44.258 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:03:44.258 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:03:44.258 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:03:44.258 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T16:03:44.258 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:03:44.258 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:03:44.258 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T16:03:44.258 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:03:44.258 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:03:44.258 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:03:44.258 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T16:03:44.259 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T16:03:44.259 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:03:44.259 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:03:44.259 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:03:44.259 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T16:03:44.259 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:03:44.259 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:03:44.259 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723997018.849359_648.1", 2024-08-18T16:03:44.259 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:03:44.259 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:03:44.259 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:03:44.259 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:03:38.849359Z", 2024-08-18T16:03:44.259 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T16:03:44.260 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:03:44.260 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:03:44.260 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T16:03:44.260 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:03:44.260 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T16:03:44.260 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T16:03:44.260 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T16:03:44.260 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:03:44.260 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T16:03:44Z" 2024-08-18T16:03:44.260 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T16:03:44.261 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-08-18T16:03:44.261 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-08-18T16:03:44.261 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:03:44.262 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:03:44.262 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:03:44.316 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:03:44.316 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:03:44.348 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:03:44.348+0000 7f9c8d519880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:03:44.349 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:03:44.349+0000 7f9c8d519880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:03:44.362 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T16:03:44.363 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T16:03:44.363 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T16:03:44.363 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T16:03:44.363 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T16:03:44.363 INFO:teuthology.orchestra.run.smithi134.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T16:03:44.363 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T16:03:44.363 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:03:44.363 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T16:03:44.363 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:03:44.363 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T16:03:44.363 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:03:44.363 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T16:03:44.364 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T16:03:44.364 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:03:44.364 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 1, 2024-08-18T16:03:44.364 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:03:44.364 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T16:03:44.364 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T16:03:44.364 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:03:44.364 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:03:44.364 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:03:44.364 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T16:03:44.364 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:03:44.364 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T16:03:44.365 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T16:03:44.365 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:03:44.365 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:03:44.365 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:03:44.365 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T16:03:44.365 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T16:03:44.365 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:03:44.365 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:03:44.365 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:03:44.365 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T16:03:44.365 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:03:44.365 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T16:03:44.365 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T16:03:44.365 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:03:44.366 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:03:44.366 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:03:44.366 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T16:03:44.366 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T16:03:44.366 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:03:44.366 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:03:44.366 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:03:44.366 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T16:03:44.366 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:03:44.366 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T16:03:44.366 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723997018.849359_648.1", 2024-08-18T16:03:44.366 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:03:44.367 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:03:44.367 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:03:44.367 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:03:38.849359Z", 2024-08-18T16:03:44.367 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T16:03:44.367 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:03:44.367 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:03:44.367 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T16:03:44.367 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:03:44.367 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T16:03:44.367 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 1, 2024-08-18T16:03:44.367 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 1 2024-08-18T16:03:44.367 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:03:44.367 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T16:03:44Z" 2024-08-18T16:03:44.367 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T16:03:44.368 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-08-18T16:03:44.369 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-71', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:03:44.369 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-71', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:03:44.369 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-71 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T16:03:44.424 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:03:44.424 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:03:44.459 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:03:44.459+0000 7f2a3f435880 1 waiting to reach incremental sync.. 2024-08-18T16:03:49.460 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:03:49.459+0000 7f2a3f435880 1 waiting to reach incremental sync.. 2024-08-18T16:03:54.460 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:03:54.460+0000 7f2a3f435880 1 waiting to reach incremental sync.. 2024-08-18T16:03:59.462 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:03:59.462+0000 7f2a3f435880 1 bucket sync caught up with source: 2024-08-18T16:03:59.463 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, 00000000001.116.6, , , , 00000000001.67.6, , , , , ] 2024-08-18T16:03:59.463 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, 00000000001.116.6, , , , 00000000001.67.6, , , , , ] 2024-08-18T16:03:59.463 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:03:59.462+0000 7f2a3f435880 0 bucket checkpoint complete 2024-08-18T16:03:59.535 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_encrypted_object_sync ... ok 2024-08-18T16:03:59.535 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=kibgmb-72 2024-08-18T16:03:59.622 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:03:59.622 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:03:59.623 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:03:59.672 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:03:59.672 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:03:59.706 INFO:teuthology.orchestra.run.smithi134.stderr:No --period given, using current period=01050e30-179b-43be-bdbb-3bb23cf37ae8 2024-08-18T16:03:59.720 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T16:03:59.720 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:03:59.720 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T16:03:59.720 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T16:03:59.720 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:03:59.720 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:03:59.720 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T16:03:59.720 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T16:03:59.720 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:03:59.721 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:03:59.721 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T16:03:59.721 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T16:03:59.721 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:03:59.721 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:03:59.721 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723997039.553361_653.1", 2024-08-18T16:03:59.721 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:03:59.553361Z" 2024-08-18T16:03:59.721 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:03:59.721 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T16:03:59.722 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-08-18T16:03:59.722 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:03:59.722 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:03:59.723 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:03:59.777 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:03:59.777 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:03:59.808 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:03:59.808+0000 7f9f9ed27880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:03:59.809 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:03:59.809+0000 7f9f9ed27880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:03:59.823 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T16:03:59.823 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T16:03:59.823 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T16:03:59.823 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T16:03:59.824 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T16:03:59.824 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T16:03:59.824 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T16:03:59.824 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:03:59.824 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T16:03:59.824 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:03:59.824 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T16:03:59.824 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:03:59.824 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:03:59.824 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T16:03:59.824 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:03:59.824 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T16:03:59.824 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:03:59.825 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T16:03:59.825 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T16:03:59.825 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:03:59.825 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:03:59.825 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:03:59.825 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T16:03:59.825 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:03:59.825 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:03:59.825 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T16:03:59.825 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:03:59.825 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:03:59.825 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:03:59.825 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T16:03:59.826 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T16:03:59.826 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:03:59.826 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:03:59.826 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:03:59.826 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T16:03:59.826 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:03:59.826 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:03:59.826 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T16:03:59.826 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:03:59.826 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:03:59.826 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:03:59.827 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T16:03:59.827 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T16:03:59.827 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:03:59.827 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:03:59.827 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:03:59.827 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T16:03:59.827 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:03:59.827 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:03:59.827 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723997039.553361_653.1", 2024-08-18T16:03:59.827 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:03:59.827 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:03:59.827 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:03:59.827 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:03:59.553361Z", 2024-08-18T16:03:59.828 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T16:03:59.828 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:03:59.828 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:03:59.828 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T16:03:59.828 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:03:59.828 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T16:03:59.828 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T16:03:59.828 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T16:03:59.828 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:03:59.828 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T16:03:59Z" 2024-08-18T16:03:59.828 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T16:03:59.829 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-08-18T16:03:59.829 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-08-18T16:03:59.829 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:03:59.830 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:03:59.830 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:03:59.877 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:03:59.877 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:03:59.905 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:03:59.904+0000 7f9fb3012880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:03:59.905 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:03:59.905+0000 7f9fb3012880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:03:59.918 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T16:03:59.918 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T16:03:59.918 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T16:03:59.918 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T16:03:59.918 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T16:03:59.918 INFO:teuthology.orchestra.run.smithi134.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T16:03:59.918 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T16:03:59.918 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:03:59.918 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T16:03:59.919 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:03:59.919 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T16:03:59.919 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:03:59.919 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T16:03:59.919 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T16:03:59.919 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:03:59.919 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 1, 2024-08-18T16:03:59.919 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:03:59.919 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T16:03:59.919 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T16:03:59.919 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:03:59.919 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:03:59.919 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:03:59.919 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T16:03:59.920 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:03:59.920 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T16:03:59.920 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T16:03:59.920 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:03:59.920 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:03:59.920 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:03:59.920 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T16:03:59.920 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T16:03:59.920 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:03:59.920 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:03:59.920 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:03:59.920 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T16:03:59.920 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:03:59.921 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T16:03:59.921 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T16:03:59.921 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:03:59.921 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:03:59.921 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:03:59.921 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T16:03:59.921 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T16:03:59.921 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:03:59.921 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:03:59.921 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:03:59.921 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T16:03:59.921 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:03:59.921 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T16:03:59.921 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723997039.553361_653.1", 2024-08-18T16:03:59.921 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:03:59.921 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:03:59.922 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:03:59.922 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:03:59.553361Z", 2024-08-18T16:03:59.922 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T16:03:59.922 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:03:59.922 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:03:59.922 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T16:03:59.922 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:03:59.922 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T16:03:59.922 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 1, 2024-08-18T16:03:59.922 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 1 2024-08-18T16:03:59.922 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:03:59.922 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T16:03:59Z" 2024-08-18T16:03:59.922 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T16:03:59.923 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-08-18T16:03:59.924 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-72', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:03:59.924 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-72', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:03:59.924 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-72 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T16:03:59.978 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:03:59.978 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:04:00.014 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:04:00.013+0000 7ff00e732880 1 waiting to reach incremental sync.. 2024-08-18T16:04:05.015 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:04:05.014+0000 7ff00e732880 1 waiting to reach incremental sync.. 2024-08-18T16:04:10.015 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:04:10.015+0000 7ff00e732880 1 waiting to reach incremental sync.. 2024-08-18T16:04:15.016 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:04:15.015+0000 7ff00e732880 1 waiting to reach incremental sync.. 2024-08-18T16:04:20.019 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:04:20.016+0000 7ff00e732880 1 bucket sync caught up with source: 2024-08-18T16:04:20.019 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, 00000000001.64.6, , , , 00000000001.68.6, , , 00000000001.55.6, , 00000000001.63.6] 2024-08-18T16:04:20.019 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, 00000000001.64.6, , , , 00000000001.68.6, , , 00000000001.55.6, , 00000000001.63.6] 2024-08-18T16:04:20.019 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:04:20.016+0000 7ff00e732880 0 bucket checkpoint complete 2024-08-18T16:04:20.038 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-72', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:04:20.038 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-72', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:04:20.038 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-72 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:04:20.101 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:04:20.101 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:04:20.138 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:04:20.137+0000 7f2e90f42880 1 bucket sync caught up with source: 2024-08-18T16:04:20.138 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000001.64.6, , , , 00000000001.68.6, , , 00000000001.55.6, , 00000000001.63.6] 2024-08-18T16:04:20.138 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000001.64.6, , , , 00000000001.68.6, , , 00000000001.55.6, , 00000000001.63.6] 2024-08-18T16:04:20.138 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:04:20.137+0000 7f2e90f42880 0 bucket checkpoint complete 2024-08-18T16:04:20.152 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-72', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:04:20.152 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-72', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:04:20.152 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-72 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:04:20.206 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:04:20.207 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:04:20.243 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:04:20.243+0000 7f30a52bb880 1 bucket sync caught up with source: 2024-08-18T16:04:20.244 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000001.242.6, , , , 00000000001.305.6, , , 00000000001.242.6, , 00000000001.122.6] 2024-08-18T16:04:20.244 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000001.242.6, , , , 00000000001.305.6, , , 00000000001.242.6, , 00000000001.122.6] 2024-08-18T16:04:20.244 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:04:20.243+0000 7f30a52bb880 0 bucket checkpoint complete 2024-08-18T16:04:20.258 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-72', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:04:20.258 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-72', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:04:20.258 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-72 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:04:20.311 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:04:20.311 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:04:20.346 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:04:20.346+0000 7f850a612880 1 bucket sync caught up with source: 2024-08-18T16:04:20.346 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000001.242.6, , , , 00000000001.305.6, , , 00000000001.242.6, , 00000000001.122.6] 2024-08-18T16:04:20.346 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000001.242.6, , , , 00000000001.305.6, , , 00000000001.242.6, , 00000000001.122.6] 2024-08-18T16:04:20.347 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:04:20.346+0000 7f850a612880 0 bucket checkpoint complete 2024-08-18T16:04:20.359 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-72', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:04:20.359 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-72', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:04:20.359 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-72 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:04:20.413 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:04:20.413 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:04:20.447 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:04:20.446+0000 7f56a3211880 1 waiting to reach incremental sync.. 2024-08-18T16:04:25.447 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:04:25.446+0000 7f56a3211880 1 waiting to reach incremental sync.. 2024-08-18T16:04:30.449 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:04:30.448+0000 7f56a3211880 1 bucket sync caught up with source: 2024-08-18T16:04:30.449 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000001.158.6, , , , 00000000001.142.6, , , 00000000001.105.6, , 00000000001.150.6] 2024-08-18T16:04:30.449 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000001.158.6, , , , 00000000001.142.6, , , 00000000001.105.6, , 00000000001.150.6] 2024-08-18T16:04:30.449 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:04:30.448+0000 7f56a3211880 0 bucket checkpoint complete 2024-08-18T16:04:30.467 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-72', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:04:30.467 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-72', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:04:30.467 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-72 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T16:04:30.528 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:04:30.529 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:04:30.570 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:04:30.569+0000 7f1cfd132880 1 bucket sync caught up with source: 2024-08-18T16:04:30.570 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, 00000000001.158.6, , , , 00000000001.142.6, , , 00000000001.105.6, , 00000000001.150.6] 2024-08-18T16:04:30.570 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, 00000000001.158.6, , , , 00000000001.142.6, , , 00000000001.105.6, , 00000000001.150.6] 2024-08-18T16:04:30.570 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:04:30.569+0000 7f1cfd132880 0 bucket checkpoint complete 2024-08-18T16:04:30.585 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-72 zones={test-zone2, test-zone1} 2024-08-18T16:04:30.590 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:04:30.593 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-08-18T16:04:30.594 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-08-18T16:04:30.594 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-08-18T16:04:30.594 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-08-18T16:04:30.594 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:04:30.596 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-08-18T16:04:30.596 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-08-18T16:04:30.597 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-08-18T16:04:30.597 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-08-18T16:04:30.602 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-08-18T16:04:30.608 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-08-18T16:04:30.661 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-08-18T16:04:30.713 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-08-18T16:04:30.720 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-08-18T16:04:30.773 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-08-18T16:04:30.825 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-08-18T16:04:30.833 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-08-18T16:04:30.890 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-08-18T16:04:30.939 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-08-18T16:04:30.947 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-08-18T16:04:31.003 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-08-18T16:04:31.057 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-72 zones={test-zone2, test-zone1} 2024-08-18T16:04:31.058 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-72 zones={test-zone3, test-zone1} 2024-08-18T16:04:31.062 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:04:31.065 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-08-18T16:04:31.065 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-08-18T16:04:31.065 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-08-18T16:04:31.065 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-08-18T16:04:31.065 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:04:31.068 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-08-18T16:04:31.068 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-08-18T16:04:31.068 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-08-18T16:04:31.068 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-08-18T16:04:31.074 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-08-18T16:04:31.126 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-08-18T16:04:31.138 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-08-18T16:04:31.188 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-08-18T16:04:31.236 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-08-18T16:04:31.288 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-08-18T16:04:31.337 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-08-18T16:04:31.388 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-08-18T16:04:31.441 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-08-18T16:04:31.538 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-08-18T16:04:31.588 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-08-18T16:04:31.646 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-08-18T16:04:31.741 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-72 zones={test-zone3, test-zone1} 2024-08-18T16:04:31.742 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-72 zones={test-zone3, test-zone2} 2024-08-18T16:04:31.746 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:04:31.749 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-08-18T16:04:31.749 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-08-18T16:04:31.749 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-08-18T16:04:31.749 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-08-18T16:04:31.749 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:04:31.752 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-08-18T16:04:31.752 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-08-18T16:04:31.752 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-08-18T16:04:31.752 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-08-18T16:04:31.757 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-08-18T16:04:31.806 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-08-18T16:04:31.905 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-08-18T16:04:32.001 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-08-18T16:04:32.056 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-08-18T16:04:32.114 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-08-18T16:04:32.209 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-08-18T16:04:32.260 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-08-18T16:04:32.357 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-08-18T16:04:32.453 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-08-18T16:04:32.505 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-08-18T16:04:32.562 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-08-18T16:04:32.662 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-72 zones={test-zone3, test-zone2} 2024-08-18T16:04:32.662 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:04:32.662 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:04:32.662 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --rgw-sync-log-trim-max-buckets 0 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:04:32.714 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:04:32.714 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:04:32.765 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=kibgmb-73 2024-08-18T16:04:32.887 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:04:32.887 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:04:32.887 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:04:32.936 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:04:32.936 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:04:32.969 INFO:teuthology.orchestra.run.smithi134.stderr:No --period given, using current period=01050e30-179b-43be-bdbb-3bb23cf37ae8 2024-08-18T16:04:32.982 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T16:04:32.982 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:04:32.982 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T16:04:32.982 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T16:04:32.982 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:04:32.982 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:04:32.983 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T16:04:32.983 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T16:04:32.983 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:04:32.983 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:04:32.983 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T16:04:32.983 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T16:04:32.983 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:04:32.983 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:04:32.983 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723997072.779739_665.1", 2024-08-18T16:04:32.983 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:04:32.779739Z" 2024-08-18T16:04:32.983 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:04:32.983 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T16:04:32.984 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-08-18T16:04:32.984 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:04:32.984 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:04:32.985 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:04:33.038 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:04:33.038 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:04:33.071 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:04:33.070+0000 7fcde4234880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:04:33.073 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:04:33.071+0000 7fcde4234880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:04:33.086 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T16:04:33.086 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T16:04:33.086 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T16:04:33.086 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T16:04:33.086 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T16:04:33.086 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T16:04:33.086 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T16:04:33.086 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:04:33.086 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T16:04:33.086 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:04:33.086 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T16:04:33.087 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:04:33.087 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:04:33.087 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T16:04:33.087 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:04:33.087 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T16:04:33.087 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:04:33.087 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T16:04:33.087 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T16:04:33.087 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:04:33.087 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:04:33.087 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:04:33.087 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T16:04:33.087 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:04:33.088 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:04:33.088 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T16:04:33.088 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:04:33.088 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:04:33.088 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:04:33.088 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T16:04:33.088 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T16:04:33.088 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:04:33.088 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:04:33.088 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:04:33.088 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T16:04:33.088 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:04:33.088 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:04:33.089 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T16:04:33.089 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:04:33.089 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:04:33.089 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:04:33.089 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T16:04:33.089 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T16:04:33.089 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:04:33.089 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:04:33.089 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:04:33.089 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T16:04:33.089 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:04:33.089 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:04:33.089 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723997072.779739_665.1", 2024-08-18T16:04:33.090 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:04:33.090 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:04:33.090 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:04:33.090 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:04:32.779739Z", 2024-08-18T16:04:33.090 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T16:04:33.090 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:04:33.090 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:04:33.090 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T16:04:33.090 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:04:33.090 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T16:04:33.090 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T16:04:33.090 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T16:04:33.090 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:04:33.091 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T16:04:33Z" 2024-08-18T16:04:33.091 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T16:04:33.092 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-08-18T16:04:33.092 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-08-18T16:04:33.092 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:04:33.092 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:04:33.092 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:04:33.139 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:04:33.139 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:04:33.169 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:04:33.169+0000 7ff5e4213880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:04:33.170 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:04:33.170+0000 7ff5e4213880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:04:33.183 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T16:04:33.183 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T16:04:33.183 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T16:04:33.183 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T16:04:33.183 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T16:04:33.183 INFO:teuthology.orchestra.run.smithi134.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T16:04:33.183 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T16:04:33.183 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:04:33.183 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T16:04:33.183 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:04:33.183 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T16:04:33.183 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:04:33.184 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T16:04:33.184 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T16:04:33.184 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:04:33.184 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 1, 2024-08-18T16:04:33.184 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:04:33.184 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T16:04:33.184 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T16:04:33.184 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:04:33.184 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:04:33.184 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:04:33.184 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T16:04:33.184 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:04:33.184 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T16:04:33.185 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T16:04:33.185 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:04:33.185 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:04:33.185 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:04:33.185 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T16:04:33.185 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T16:04:33.185 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:04:33.185 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:04:33.185 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:04:33.185 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T16:04:33.185 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:04:33.185 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T16:04:33.185 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T16:04:33.185 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:04:33.186 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:04:33.186 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:04:33.186 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T16:04:33.186 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T16:04:33.186 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:04:33.186 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:04:33.186 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:04:33.186 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T16:04:33.186 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:04:33.186 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T16:04:33.186 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723997072.779739_665.1", 2024-08-18T16:04:33.186 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:04:33.186 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:04:33.187 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:04:33.187 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:04:32.779739Z", 2024-08-18T16:04:33.187 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T16:04:33.187 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:04:33.187 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:04:33.187 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T16:04:33.187 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:04:33.187 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T16:04:33.187 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 1, 2024-08-18T16:04:33.187 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 1 2024-08-18T16:04:33.187 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:04:33.187 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T16:04:33Z" 2024-08-18T16:04:33.187 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T16:04:33.188 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-08-18T16:04:33.188 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-73', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:04:33.189 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-73', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:04:33.189 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-73 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T16:04:33.243 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:04:33.243 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:04:33.278 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:04:33.278+0000 7ff96b33a880 1 waiting to reach incremental sync.. 2024-08-18T16:04:38.280 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:04:38.279+0000 7ff96b33a880 1 bucket sync caught up with source: 2024-08-18T16:04:38.280 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, 00000000001.89.6, , , , 00000000001.60.6, , , 00000000001.34.6, , 00000000001.71.6] 2024-08-18T16:04:38.280 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, 00000000001.89.6, , , , 00000000001.60.6, , , 00000000001.34.6, , 00000000001.71.6] 2024-08-18T16:04:38.280 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:04:38.279+0000 7ff96b33a880 0 bucket checkpoint complete 2024-08-18T16:04:38.297 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-73', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:04:38.297 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-73', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:04:38.298 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-73 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:04:38.355 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:04:38.355 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:04:38.390 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:04:38.389+0000 7fd7e2342880 1 bucket sync caught up with source: 2024-08-18T16:04:38.390 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000001.89.6, , , , 00000000001.60.6, , , 00000000001.34.6, , 00000000001.71.6] 2024-08-18T16:04:38.391 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000001.89.6, , , , 00000000001.60.6, , , 00000000001.34.6, , 00000000001.71.6] 2024-08-18T16:04:38.391 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:04:38.389+0000 7fd7e2342880 0 bucket checkpoint complete 2024-08-18T16:04:38.403 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-73', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:04:38.403 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-73', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:04:38.404 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-73 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:04:38.457 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:04:38.457 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:04:38.493 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:04:38.493+0000 7fc29deb9880 1 bucket sync caught up with source: 2024-08-18T16:04:38.494 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000001.369.6, , , , 00000000001.261.6, , , 00000000001.140.6, , 00000000001.353.6] 2024-08-18T16:04:38.494 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000001.369.6, , , , 00000000001.261.6, , , 00000000001.140.6, , 00000000001.353.6] 2024-08-18T16:04:38.494 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:04:38.493+0000 7fc29deb9880 0 bucket checkpoint complete 2024-08-18T16:04:38.507 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-73', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:04:38.507 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-73', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:04:38.508 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-73 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:04:38.561 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:04:38.561 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:04:38.595 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:04:38.594+0000 7f3ec5030880 1 bucket sync caught up with source: 2024-08-18T16:04:38.595 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000001.369.6, , , , 00000000001.261.6, , , 00000000001.140.6, , 00000000001.353.6] 2024-08-18T16:04:38.595 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000001.369.6, , , , 00000000001.261.6, , , 00000000001.140.6, , 00000000001.353.6] 2024-08-18T16:04:38.595 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:04:38.594+0000 7f3ec5030880 0 bucket checkpoint complete 2024-08-18T16:04:38.608 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-73', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:04:38.608 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-73', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:04:38.608 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-73 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:04:38.661 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:04:38.662 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:04:38.695 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:04:38.694+0000 7f64f100f880 1 waiting to reach incremental sync.. 2024-08-18T16:04:43.695 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:04:43.695+0000 7f64f100f880 1 waiting to reach incremental sync.. 2024-08-18T16:04:48.697 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:04:48.696+0000 7f64f100f880 1 bucket sync caught up with source: 2024-08-18T16:04:48.697 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000001.175.6, , , , 00000000001.174.6, , , 00000000001.103.6, , 00000000001.234.6] 2024-08-18T16:04:48.697 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000001.175.6, , , , 00000000001.174.6, , , 00000000001.103.6, , 00000000001.234.6] 2024-08-18T16:04:48.698 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:04:48.696+0000 7f64f100f880 0 bucket checkpoint complete 2024-08-18T16:04:48.715 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-73', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:04:48.715 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-73', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:04:48.715 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-73 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T16:04:48.777 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:04:48.777 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:04:48.816 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:04:48.816+0000 7f251c146880 1 bucket sync caught up with source: 2024-08-18T16:04:48.816 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, 00000000001.175.6, , , , 00000000001.174.6, , , 00000000001.103.6, , 00000000001.234.6] 2024-08-18T16:04:48.817 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, 00000000001.175.6, , , , 00000000001.174.6, , , 00000000001.103.6, , 00000000001.234.6] 2024-08-18T16:04:48.817 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:04:48.816+0000 7f251c146880 0 bucket checkpoint complete 2024-08-18T16:04:48.834 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-73 zones={test-zone2, test-zone1} 2024-08-18T16:04:48.840 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:04:48.842 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-08-18T16:04:48.842 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-08-18T16:04:48.843 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-08-18T16:04:48.843 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-08-18T16:04:48.843 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:04:48.846 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-08-18T16:04:48.846 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-08-18T16:04:48.846 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-08-18T16:04:48.846 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-08-18T16:04:48.852 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-08-18T16:04:48.860 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-08-18T16:04:48.922 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-08-18T16:04:48.932 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-08-18T16:04:48.986 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-08-18T16:04:48.999 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-08-18T16:04:49.056 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-08-18T16:04:49.063 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-08-18T16:04:49.120 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-08-18T16:04:49.130 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-08-18T16:04:49.184 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-08-18T16:04:49.196 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-08-18T16:04:49.205 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-73 zones={test-zone2, test-zone1} 2024-08-18T16:04:49.205 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-73 zones={test-zone3, test-zone1} 2024-08-18T16:04:49.210 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:04:49.213 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-08-18T16:04:49.213 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-08-18T16:04:49.213 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-08-18T16:04:49.213 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-08-18T16:04:49.213 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:04:49.216 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-08-18T16:04:49.216 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-08-18T16:04:49.217 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-08-18T16:04:49.217 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-08-18T16:04:49.222 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-08-18T16:04:49.229 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-08-18T16:04:49.284 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-08-18T16:04:49.379 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-08-18T16:04:49.428 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-08-18T16:04:49.533 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-08-18T16:04:49.590 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-08-18T16:04:49.645 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-08-18T16:04:49.749 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-08-18T16:04:49.846 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-08-18T16:04:49.896 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-08-18T16:04:49.954 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-08-18T16:04:50.060 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-73 zones={test-zone3, test-zone1} 2024-08-18T16:04:50.060 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-73 zones={test-zone3, test-zone2} 2024-08-18T16:04:50.065 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:04:50.069 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-08-18T16:04:50.069 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-08-18T16:04:50.069 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-08-18T16:04:50.069 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-08-18T16:04:50.069 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:04:50.073 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-08-18T16:04:50.073 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-08-18T16:04:50.073 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-08-18T16:04:50.073 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-08-18T16:04:50.079 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-08-18T16:04:50.129 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-08-18T16:04:50.229 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-08-18T16:04:50.282 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-08-18T16:04:50.333 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-08-18T16:04:50.390 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-08-18T16:04:50.442 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-08-18T16:04:50.497 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-08-18T16:04:50.553 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-08-18T16:04:50.650 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-08-18T16:04:50.701 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-08-18T16:04:50.757 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-08-18T16:04:50.858 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-73 zones={test-zone3, test-zone2} 2024-08-18T16:04:50.858 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '1', '--rgw-sync-log-trim-min-cold-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:04:50.859 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '1', '--rgw-sync-log-trim-min-cold-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:04:50.859 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --rgw-sync-log-trim-max-buckets 1 --rgw-sync-log-trim-min-cold-buckets 0 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:04:50.912 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:04:50.912 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:04:50.971 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'kibgmb-73', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:04:50.971 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'kibgmb-73', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:04:50.972 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket kibgmb-73 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:04:51.017 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:04:51.017 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:04:51.061 INFO:teuthology.orchestra.run.smithi134.stdout:[] 2024-08-18T16:04:51.063 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'kibgmb-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:04:51.063 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'kibgmb-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:04:51.063 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket kibgmb-72 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:04:51.109 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:04:51.109 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:04:51.153 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T16:04:51.153 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:04:51.153 INFO:teuthology.orchestra.run.smithi134.stdout: "op_id": "1#00000000001.64.6", 2024-08-18T16:04:51.153 INFO:teuthology.orchestra.run.smithi134.stdout: "op_tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.505681445125890248", 2024-08-18T16:04:51.153 INFO:teuthology.orchestra.run.smithi134.stdout: "op": "write", 2024-08-18T16:04:51.153 INFO:teuthology.orchestra.run.smithi134.stdout: "object": "a", 2024-08-18T16:04:51.153 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:04:51.153 INFO:teuthology.orchestra.run.smithi134.stdout: "state": "complete", 2024-08-18T16:04:51.154 INFO:teuthology.orchestra.run.smithi134.stdout: "index_ver": 1, 2024-08-18T16:04:51.154 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:03:59.559734824Z", 2024-08-18T16:04:51.154 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:04:51.154 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:04:51.154 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 6 2024-08-18T16:04:51.154 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:04:51.154 INFO:teuthology.orchestra.run.smithi134.stdout: "bilog_flags": 0, 2024-08-18T16:04:51.154 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned": false, 2024-08-18T16:04:51.154 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "", 2024-08-18T16:04:51.154 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "", 2024-08-18T16:04:51.154 INFO:teuthology.orchestra.run.smithi134.stdout: "zones_trace": [ 2024-08-18T16:04:51.154 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:04:51.154 INFO:teuthology.orchestra.run.smithi134.stdout: "entry": "96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-72:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.20" 2024-08-18T16:04:51.155 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:04:51.155 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T16:04:51.155 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:04:51.155 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:04:51.155 INFO:teuthology.orchestra.run.smithi134.stdout: "op_id": "5#00000000001.68.6", 2024-08-18T16:04:51.155 INFO:teuthology.orchestra.run.smithi134.stdout: "op_tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.1816320002821086996", 2024-08-18T16:04:51.155 INFO:teuthology.orchestra.run.smithi134.stdout: "op": "write", 2024-08-18T16:04:51.155 INFO:teuthology.orchestra.run.smithi134.stdout: "object": "d", 2024-08-18T16:04:51.155 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:04:51.155 INFO:teuthology.orchestra.run.smithi134.stdout: "state": "complete", 2024-08-18T16:04:51.155 INFO:teuthology.orchestra.run.smithi134.stdout: "index_ver": 1, 2024-08-18T16:04:51.155 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:03:59.619186082Z", 2024-08-18T16:04:51.155 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:04:51.156 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:04:51.156 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 4 2024-08-18T16:04:51.156 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:04:51.156 INFO:teuthology.orchestra.run.smithi134.stdout: "bilog_flags": 0, 2024-08-18T16:04:51.156 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned": false, 2024-08-18T16:04:51.156 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "", 2024-08-18T16:04:51.156 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "", 2024-08-18T16:04:51.156 INFO:teuthology.orchestra.run.smithi134.stdout: "zones_trace": [ 2024-08-18T16:04:51.156 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:04:51.156 INFO:teuthology.orchestra.run.smithi134.stdout: "entry": "96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-72:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.20" 2024-08-18T16:04:51.156 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:04:51.156 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T16:04:51.156 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:04:51.157 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:04:51.157 INFO:teuthology.orchestra.run.smithi134.stdout: "op_id": "8#00000000001.55.6", 2024-08-18T16:04:51.157 INFO:teuthology.orchestra.run.smithi134.stdout: "op_tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.14641648917476400624", 2024-08-18T16:04:51.157 INFO:teuthology.orchestra.run.smithi134.stdout: "op": "write", 2024-08-18T16:04:51.157 INFO:teuthology.orchestra.run.smithi134.stdout: "object": "c", 2024-08-18T16:04:51.157 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:04:51.157 INFO:teuthology.orchestra.run.smithi134.stdout: "state": "complete", 2024-08-18T16:04:51.157 INFO:teuthology.orchestra.run.smithi134.stdout: "index_ver": 1, 2024-08-18T16:04:51.157 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:03:59.575560353Z", 2024-08-18T16:04:51.157 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:04:51.157 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:04:51.157 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 7 2024-08-18T16:04:51.157 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:04:51.157 INFO:teuthology.orchestra.run.smithi134.stdout: "bilog_flags": 0, 2024-08-18T16:04:51.158 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned": false, 2024-08-18T16:04:51.158 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "", 2024-08-18T16:04:51.158 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "", 2024-08-18T16:04:51.158 INFO:teuthology.orchestra.run.smithi134.stdout: "zones_trace": [ 2024-08-18T16:04:51.158 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:04:51.158 INFO:teuthology.orchestra.run.smithi134.stdout: "entry": "96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-72:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.20" 2024-08-18T16:04:51.158 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:04:51.158 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T16:04:51.158 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:04:51.158 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:04:51.158 INFO:teuthology.orchestra.run.smithi134.stdout: "op_id": "10#00000000001.63.6", 2024-08-18T16:04:51.158 INFO:teuthology.orchestra.run.smithi134.stdout: "op_tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.15636784428861639007", 2024-08-18T16:04:51.158 INFO:teuthology.orchestra.run.smithi134.stdout: "op": "write", 2024-08-18T16:04:51.159 INFO:teuthology.orchestra.run.smithi134.stdout: "object": "b", 2024-08-18T16:04:51.159 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:04:51.159 INFO:teuthology.orchestra.run.smithi134.stdout: "state": "complete", 2024-08-18T16:04:51.159 INFO:teuthology.orchestra.run.smithi134.stdout: "index_ver": 1, 2024-08-18T16:04:51.159 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:03:59.567772487Z", 2024-08-18T16:04:51.159 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:04:51.159 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:04:51.159 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 9 2024-08-18T16:04:51.159 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:04:51.159 INFO:teuthology.orchestra.run.smithi134.stdout: "bilog_flags": 0, 2024-08-18T16:04:51.159 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned": false, 2024-08-18T16:04:51.159 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "", 2024-08-18T16:04:51.159 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "", 2024-08-18T16:04:51.159 INFO:teuthology.orchestra.run.smithi134.stdout: "zones_trace": [ 2024-08-18T16:04:51.160 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:04:51.160 INFO:teuthology.orchestra.run.smithi134.stdout: "entry": "96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-72:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.20" 2024-08-18T16:04:51.160 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:04:51.160 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T16:04:51.160 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:04:51.160 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T16:04:51.161 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '999', '--rgw-sync-log-trim-min-cold-buckets', '999', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:04:51.161 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '999', '--rgw-sync-log-trim-min-cold-buckets', '999', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:04:51.161 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --rgw-sync-log-trim-max-buckets 999 --rgw-sync-log-trim-min-cold-buckets 999 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:04:51.206 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:04:51.206 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:04:51.254 INFO:tasks.rgw.c2.client.0.smithi193.stdout:2024-08-18T16:04:51.254+0000 7f43d95cd640 -1 req 13669979098445012237 0.002999952s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=kibgmb-37:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.37,z=96bf8502-2f39-4d53-8196-3378c28da3b4,az=0},d={b=kibgmb-37:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.37,z=10090ccf-aad6-416d-9775-df8b7b27345a,az=0}} returned ret=-2 2024-08-18T16:04:51.255 INFO:tasks.rgw.c1.client.1.smithi134.stdout:2024-08-18T16:04:51.253+0000 7fc08ac9c640 -1 req 7251236743458646258 0.001999954s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=kibgmb-37:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.37,z=96bf8502-2f39-4d53-8196-3378c28da3b4,az=0},d={b=kibgmb-37:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.37,z=018ea73b-7528-48da-aafa-0118123784f7,az=0}} returned ret=-2 2024-08-18T16:04:51.255 INFO:tasks.rgw.c1.client.1.smithi134.stdout:2024-08-18T16:04:51.254+0000 7fbfc6af6640 -1 req 14455052489286278535 0.001999953s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=kibgmb-5:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.5,z=96bf8502-2f39-4d53-8196-3378c28da3b4,az=0},d={b=kibgmb-5:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.5,z=018ea73b-7528-48da-aafa-0118123784f7,az=0}} returned ret=-2 2024-08-18T16:04:51.255 INFO:tasks.rgw.c2.client.0.smithi193.stdout:2024-08-18T16:04:51.254+0000 7f4376507640 -1 req 14881404696438371351 0.001999968s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=kibgmb-5:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.5,z=96bf8502-2f39-4d53-8196-3378c28da3b4,az=0},d={b=kibgmb-5:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.5,z=10090ccf-aad6-416d-9775-df8b7b27345a,az=0}} returned ret=-2 2024-08-18T16:04:51.299 INFO:tasks.rgw.c1.client.1.smithi134.stdout:2024-08-18T16:04:51.298+0000 7fc067455640 -1 req 323528601250896478 0.001999953s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=kibgmb-9:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.9,z=96bf8502-2f39-4d53-8196-3378c28da3b4,az=0},d={b=kibgmb-9:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.9,z=018ea73b-7528-48da-aafa-0118123784f7,az=0}} returned ret=-2 2024-08-18T16:04:51.299 INFO:tasks.rgw.c1.client.1.smithi134.stdout:2024-08-18T16:04:51.298+0000 7fc067c56640 -1 req 1884321768614745410 0.001999953s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=kibgmb-4:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.4,z=96bf8502-2f39-4d53-8196-3378c28da3b4,az=0},d={b=kibgmb-4:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.4,z=018ea73b-7528-48da-aafa-0118123784f7,az=0}} returned ret=-2 2024-08-18T16:04:51.299 INFO:tasks.rgw.c2.client.0.smithi193.stdout:2024-08-18T16:04:51.299+0000 7f438e537640 -1 req 15024441317779688765 0.002999952s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=kibgmb-9:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.9,z=96bf8502-2f39-4d53-8196-3378c28da3b4,az=0},d={b=kibgmb-9:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.9,z=10090ccf-aad6-416d-9775-df8b7b27345a,az=0}} returned ret=-2 2024-08-18T16:04:51.299 INFO:tasks.rgw.c2.client.0.smithi193.stdout:2024-08-18T16:04:51.299+0000 7f430e437640 -1 req 16927300864360973051 0.002999952s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=kibgmb-4:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.4,z=96bf8502-2f39-4d53-8196-3378c28da3b4,az=0},d={b=kibgmb-4:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.4,z=10090ccf-aad6-416d-9775-df8b7b27345a,az=0}} returned ret=-2 2024-08-18T16:04:51.354 INFO:tasks.rgw.c1.client.1.smithi134.stdout:2024-08-18T16:04:51.353+0000 7fc050a24640 -1 req 15694531970186602819 0.000999976s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=kibgmb-38:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.38,z=96bf8502-2f39-4d53-8196-3378c28da3b4,az=0},d={b=kibgmb-38:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.38,z=018ea73b-7528-48da-aafa-0118123784f7,az=0}} returned ret=-2 2024-08-18T16:04:51.354 INFO:tasks.rgw.c2.client.0.smithi193.stdout:2024-08-18T16:04:51.354+0000 7f431b451640 -1 req 7082411448458311151 0.001999968s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=kibgmb-38:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.38,z=96bf8502-2f39-4d53-8196-3378c28da3b4,az=0},d={b=kibgmb-38:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.38,z=10090ccf-aad6-416d-9775-df8b7b27345a,az=0}} returned ret=-2 2024-08-18T16:04:51.391 INFO:tasks.rgw.c1.client.1.smithi134.stdout:2024-08-18T16:04:51.391+0000 7fc02f0df640 -1 req 2217301040467248467 0.002999930s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=kibgmb-36:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.36,z=96bf8502-2f39-4d53-8196-3378c28da3b4,az=0},d={b=kibgmb-36:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.36,z=018ea73b-7528-48da-aafa-0118123784f7,az=0}} returned ret=-2 2024-08-18T16:04:51.392 INFO:tasks.rgw.c2.client.0.smithi193.stdout:2024-08-18T16:04:51.392+0000 7f4345ca6640 -1 req 15679811103690449649 0.002999952s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=kibgmb-36:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.36,z=96bf8502-2f39-4d53-8196-3378c28da3b4,az=0},d={b=kibgmb-36:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.36,z=10090ccf-aad6-416d-9775-df8b7b27345a,az=0}} returned ret=-2 2024-08-18T16:04:51.399 INFO:tasks.rgw.c1.client.1.smithi134.stdout:2024-08-18T16:04:51.399+0000 7fc076473640 -1 req 18337749923759088296 0.001999953s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=kibgmb-43:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.43,z=96bf8502-2f39-4d53-8196-3378c28da3b4,az=0},d={b=kibgmb-43:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.43,z=018ea73b-7528-48da-aafa-0118123784f7,az=0}} returned ret=-2 2024-08-18T16:04:51.400 INFO:tasks.rgw.c2.client.0.smithi193.stdout:2024-08-18T16:04:51.400+0000 7f4330c7c640 -1 req 9920582435241706305 0.002999952s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=kibgmb-43:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.43,z=96bf8502-2f39-4d53-8196-3378c28da3b4,az=0},d={b=kibgmb-43:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.43,z=10090ccf-aad6-416d-9775-df8b7b27345a,az=0}} returned ret=-2 2024-08-18T16:04:51.447 INFO:tasks.rgw.c1.client.1.smithi134.stdout:2024-08-18T16:04:51.447+0000 7fc066453640 -1 req 9724950139047477305 0.001999954s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=kibgmb-3:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.3,z=96bf8502-2f39-4d53-8196-3378c28da3b4,az=0},d={b=kibgmb-3:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.3,z=018ea73b-7528-48da-aafa-0118123784f7,az=0}} returned ret=-2 2024-08-18T16:04:51.448 INFO:tasks.rgw.c2.client.0.smithi193.stdout:2024-08-18T16:04:51.447+0000 7f43464a7640 -1 req 132750987531014040 0.000999984s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=kibgmb-3:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.3,z=96bf8502-2f39-4d53-8196-3378c28da3b4,az=0},d={b=kibgmb-3:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.3,z=10090ccf-aad6-416d-9775-df8b7b27345a,az=0}} returned ret=-2 2024-08-18T16:04:51.485 INFO:tasks.rgw.c1.client.1.smithi134.stdout:2024-08-18T16:04:51.484+0000 7fc07346d640 -1 req 1785342386094679226 0.001999953s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=kibgmb-35:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.35,z=96bf8502-2f39-4d53-8196-3378c28da3b4,az=0},d={b=kibgmb-35:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.35,z=018ea73b-7528-48da-aafa-0118123784f7,az=0}} returned ret=-2 2024-08-18T16:04:51.486 INFO:tasks.rgw.c2.client.0.smithi193.stdout:2024-08-18T16:04:51.485+0000 7f4310c3c640 -1 req 4008292725006443280 0.001999968s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=kibgmb-35:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.35,z=96bf8502-2f39-4d53-8196-3378c28da3b4,az=0},d={b=kibgmb-35:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.35,z=10090ccf-aad6-416d-9775-df8b7b27345a,az=0}} returned ret=-2 2024-08-18T16:04:51.491 INFO:tasks.rgw.c1.client.1.smithi134.stdout:2024-08-18T16:04:51.491+0000 7fbfc62f5640 -1 req 6534325558169898811 0.001999953s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=kibgmb-31:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.31,z=96bf8502-2f39-4d53-8196-3378c28da3b4,az=0},d={b=kibgmb-31:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.31,z=018ea73b-7528-48da-aafa-0118123784f7,az=0}} returned ret=-2 2024-08-18T16:04:51.492 INFO:tasks.rgw.c2.client.0.smithi193.stdout:2024-08-18T16:04:51.491+0000 7f439cd54640 -1 req 7124760546091641268 0.001999968s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=kibgmb-31:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.31,z=96bf8502-2f39-4d53-8196-3378c28da3b4,az=0},d={b=kibgmb-31:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.31,z=10090ccf-aad6-416d-9775-df8b7b27345a,az=0}} returned ret=-2 2024-08-18T16:04:51.533 INFO:tasks.rgw.c1.client.1.smithi134.stdout:2024-08-18T16:04:51.533+0000 7fc0138a8640 -1 req 15518899549724932692 0.001999953s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=kibgmb-55:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.3,z=96bf8502-2f39-4d53-8196-3378c28da3b4,az=0},d={b=kibgmb-55:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.3,z=018ea73b-7528-48da-aafa-0118123784f7,az=0}} returned ret=-2 2024-08-18T16:04:51.534 INFO:tasks.rgw.c2.client.0.smithi193.stdout:2024-08-18T16:04:51.533+0000 7f439c553640 -1 req 5821143107982936890 0.001999968s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=kibgmb-55:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.3,z=96bf8502-2f39-4d53-8196-3378c28da3b4,az=0},d={b=kibgmb-55:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.3,z=10090ccf-aad6-416d-9775-df8b7b27345a,az=0}} returned ret=-2 2024-08-18T16:04:51.538 INFO:tasks.rgw.c1.client.1.smithi134.stdout:2024-08-18T16:04:51.538+0000 7fc065c52640 -1 req 14531226172914249277 0.000999977s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=kibgmb-53:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.1,z=96bf8502-2f39-4d53-8196-3378c28da3b4,az=0},d={b=kibgmb-53:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.1,z=018ea73b-7528-48da-aafa-0118123784f7,az=0}} returned ret=-2 2024-08-18T16:04:51.538 INFO:tasks.rgw.c2.client.0.smithi193.stdout:2024-08-18T16:04:51.538+0000 7f42ff419640 -1 req 14118976809545408508 0.000999984s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=kibgmb-53:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.1,z=96bf8502-2f39-4d53-8196-3378c28da3b4,az=0},d={b=kibgmb-53:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.1,z=10090ccf-aad6-416d-9775-df8b7b27345a,az=0}} returned ret=-2 2024-08-18T16:04:51.580 INFO:tasks.rgw.c1.client.1.smithi134.stdout:2024-08-18T16:04:51.579+0000 7fbfd3828640 -1 req 4743860466511818646 0.001999953s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=kibgmb-48:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.2,z=96bf8502-2f39-4d53-8196-3378c28da3b4,az=0},d={b=kibgmb-48:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.2,z=018ea73b-7528-48da-aafa-0118123784f7,az=0}} returned ret=-2 2024-08-18T16:04:51.580 INFO:tasks.rgw.c2.client.0.smithi193.stdout:2024-08-18T16:04:51.580+0000 7f4346ca8640 -1 req 9266460314881732462 0.002999951s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=kibgmb-48:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.2,z=96bf8502-2f39-4d53-8196-3378c28da3b4,az=0},d={b=kibgmb-48:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.2,z=10090ccf-aad6-416d-9775-df8b7b27345a,az=0}} returned ret=-2 2024-08-18T16:04:51.624 INFO:tasks.rgw.c1.client.1.smithi134.stdout:2024-08-18T16:04:51.623+0000 7fbfe4049640 -1 req 3387554991703202798 0.000999976s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=kibgmb-58:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.6,z=96bf8502-2f39-4d53-8196-3378c28da3b4,az=0},d={b=kibgmb-58:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.6,z=018ea73b-7528-48da-aafa-0118123784f7,az=0}} returned ret=-2 2024-08-18T16:04:51.624 INFO:tasks.rgw.c2.client.0.smithi193.stdout:2024-08-18T16:04:51.624+0000 7f43584cb640 -1 req 15343889283767035260 0.000999984s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=kibgmb-58:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.6,z=96bf8502-2f39-4d53-8196-3378c28da3b4,az=0},d={b=kibgmb-58:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.6,z=10090ccf-aad6-416d-9775-df8b7b27345a,az=0}} returned ret=-2 2024-08-18T16:04:51.670 INFO:tasks.rgw.c2.client.0.smithi193.stdout:2024-08-18T16:04:51.669+0000 7f435ecd8640 -1 req 13863879261576276015 0.001999968s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=kibgmb-7:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.7,z=96bf8502-2f39-4d53-8196-3378c28da3b4,az=0},d={b=kibgmb-7:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.7,z=10090ccf-aad6-416d-9775-df8b7b27345a,az=0}} returned ret=-2 2024-08-18T16:04:51.670 INFO:tasks.rgw.c2.client.0.smithi193.stdout:2024-08-18T16:04:51.670+0000 7f437ed18640 -1 req 3056101733672685571 0.001999968s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=kibgmb-46:10090ccf-aad6-416d-9775-df8b7b27345a.5181.2,z=96bf8502-2f39-4d53-8196-3378c28da3b4,az=0},d={b=kibgmb-46:10090ccf-aad6-416d-9775-df8b7b27345a.5181.2,z=10090ccf-aad6-416d-9775-df8b7b27345a,az=0}} returned ret=-2 2024-08-18T16:04:51.671 INFO:tasks.rgw.c1.client.1.smithi134.stdout:2024-08-18T16:04:51.671+0000 7fc002085640 -1 req 13170779144696750001 0.004999884s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=kibgmb-7:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.7,z=96bf8502-2f39-4d53-8196-3378c28da3b4,az=0},d={b=kibgmb-7:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.7,z=018ea73b-7528-48da-aafa-0118123784f7,az=0}} returned ret=-2 2024-08-18T16:04:51.671 INFO:tasks.rgw.c1.client.1.smithi134.stdout:2024-08-18T16:04:51.671+0000 7fc05c2bc640 -1 req 16295032760887276852 0.002999930s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=kibgmb-46:10090ccf-aad6-416d-9775-df8b7b27345a.5181.2,z=96bf8502-2f39-4d53-8196-3378c28da3b4,az=0},d={b=kibgmb-46:10090ccf-aad6-416d-9775-df8b7b27345a.5181.2,z=018ea73b-7528-48da-aafa-0118123784f7,az=0}} returned ret=-2 2024-08-18T16:04:51.676 INFO:tasks.rgw.c1.client.1.smithi134.stdout:2024-08-18T16:04:51.676+0000 7fc041904640 -1 req 707513179988233434 0.000999977s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=kibgmb-56:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.4,z=96bf8502-2f39-4d53-8196-3378c28da3b4,az=0},d={b=kibgmb-56:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.4,z=018ea73b-7528-48da-aafa-0118123784f7,az=0}} returned ret=-2 2024-08-18T16:04:51.677 INFO:tasks.rgw.c1.client.1.smithi134.stdout:2024-08-18T16:04:51.676+0000 7fc0248ca640 -1 req 6519327226025199423 0.001999954s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=kibgmb-30:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.30,z=96bf8502-2f39-4d53-8196-3378c28da3b4,az=0},d={b=kibgmb-30:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.30,z=018ea73b-7528-48da-aafa-0118123784f7,az=0}} returned ret=-2 2024-08-18T16:04:51.677 INFO:tasks.rgw.c2.client.0.smithi193.stdout:2024-08-18T16:04:51.676+0000 7f4314443640 -1 req 12116173371418034470 0.000999984s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=kibgmb-56:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.4,z=96bf8502-2f39-4d53-8196-3378c28da3b4,az=0},d={b=kibgmb-56:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.4,z=10090ccf-aad6-416d-9775-df8b7b27345a,az=0}} returned ret=-2 2024-08-18T16:04:51.677 INFO:tasks.rgw.c2.client.0.smithi193.stdout:2024-08-18T16:04:51.677+0000 7f439253f640 -1 req 6465906118570618262 0.001999968s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=kibgmb-30:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.30,z=96bf8502-2f39-4d53-8196-3378c28da3b4,az=0},d={b=kibgmb-30:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.30,z=10090ccf-aad6-416d-9775-df8b7b27345a,az=0}} returned ret=-2 2024-08-18T16:04:51.717 INFO:tasks.rgw.c1.client.1.smithi134.stdout:2024-08-18T16:04:51.716+0000 7fc04d19c640 -1 req 5008624487403130106 0.001999954s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=kibgmb-69:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.17,z=96bf8502-2f39-4d53-8196-3378c28da3b4,az=0},d={b=kibgmb-69:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.17,z=018ea73b-7528-48da-aafa-0118123784f7,az=0}} returned ret=-2 2024-08-18T16:04:51.717 INFO:tasks.rgw.c2.client.0.smithi193.stdout:2024-08-18T16:04:51.716+0000 7f42f0bfc640 -1 req 2087554384595161763 0.001999968s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=kibgmb-69:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.17,z=96bf8502-2f39-4d53-8196-3378c28da3b4,az=0},d={b=kibgmb-69:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.17,z=10090ccf-aad6-416d-9775-df8b7b27345a,az=0}} returned ret=-2 2024-08-18T16:04:51.761 INFO:tasks.rgw.c1.client.1.smithi134.stdout:2024-08-18T16:04:51.760+0000 7fc06344d640 -1 req 15444647438764938533 0.001999954s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=kibgmb-33:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.33,z=96bf8502-2f39-4d53-8196-3378c28da3b4,az=0},d={b=kibgmb-33:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.33,z=018ea73b-7528-48da-aafa-0118123784f7,az=0}} returned ret=-2 2024-08-18T16:04:51.761 INFO:tasks.rgw.c2.client.0.smithi193.stdout:2024-08-18T16:04:51.761+0000 7f4340c9c640 -1 req 7201335028466674471 0.001999968s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=kibgmb-33:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.33,z=96bf8502-2f39-4d53-8196-3378c28da3b4,az=0},d={b=kibgmb-33:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.33,z=10090ccf-aad6-416d-9775-df8b7b27345a,az=0}} returned ret=-2 2024-08-18T16:04:51.765 INFO:tasks.rgw.c1.client.1.smithi134.stdout:2024-08-18T16:04:51.765+0000 7fbfc6af6640 -1 req 8467939132320409513 0.001999954s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=kibgmb-34:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.34,z=96bf8502-2f39-4d53-8196-3378c28da3b4,az=0},d={b=kibgmb-34:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.34,z=018ea73b-7528-48da-aafa-0118123784f7,az=0}} returned ret=-2 2024-08-18T16:04:51.766 INFO:tasks.rgw.c2.client.0.smithi193.stdout:2024-08-18T16:04:51.765+0000 7f434fcba640 -1 req 6149505958241369635 0.000999984s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=kibgmb-34:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.34,z=96bf8502-2f39-4d53-8196-3378c28da3b4,az=0},d={b=kibgmb-34:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.34,z=10090ccf-aad6-416d-9775-df8b7b27345a,az=0}} returned ret=-2 2024-08-18T16:04:51.767 INFO:tasks.rgw.c1.client.1.smithi134.stdout:2024-08-18T16:04:51.767+0000 7fbfec85a640 -1 req 3422247442760963825 0.001999953s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=kibgmb-32:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.32,z=96bf8502-2f39-4d53-8196-3378c28da3b4,az=0},d={b=kibgmb-32:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.32,z=018ea73b-7528-48da-aafa-0118123784f7,az=0}} returned ret=-2 2024-08-18T16:04:51.768 INFO:tasks.rgw.c2.client.0.smithi193.stdout:2024-08-18T16:04:51.767+0000 7f42f4403640 -1 req 2911089435536995498 0.001999968s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=kibgmb-32:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.32,z=96bf8502-2f39-4d53-8196-3378c28da3b4,az=0},d={b=kibgmb-32:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.32,z=10090ccf-aad6-416d-9775-df8b7b27345a,az=0}} returned ret=-2 2024-08-18T16:04:51.805 INFO:tasks.rgw.c1.client.1.smithi134.stdout:2024-08-18T16:04:51.804+0000 7fc05fac3640 -1 req 4343717593997495780 0.000999977s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=kibgmb-42:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.42,z=96bf8502-2f39-4d53-8196-3378c28da3b4,az=0},d={b=kibgmb-42:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.42,z=018ea73b-7528-48da-aafa-0118123784f7,az=0}} returned ret=-2 2024-08-18T16:04:51.806 INFO:tasks.rgw.c2.client.0.smithi193.stdout:2024-08-18T16:04:51.805+0000 7f42e53e5640 -1 req 13640021142851261980 0.001999967s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=kibgmb-42:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.42,z=96bf8502-2f39-4d53-8196-3378c28da3b4,az=0},d={b=kibgmb-42:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.42,z=10090ccf-aad6-416d-9775-df8b7b27345a,az=0}} returned ret=-2 2024-08-18T16:04:51.810 INFO:tasks.rgw.c1.client.1.smithi134.stdout:2024-08-18T16:04:51.809+0000 7fc067c56640 -1 req 15263928901852568875 0.001999953s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=kibgmb-6:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.6,z=96bf8502-2f39-4d53-8196-3378c28da3b4,az=0},d={b=kibgmb-6:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.6,z=018ea73b-7528-48da-aafa-0118123784f7,az=0}} returned ret=-2 2024-08-18T16:04:51.810 INFO:tasks.rgw.c2.client.0.smithi193.stdout:2024-08-18T16:04:51.810+0000 7f43524bf640 -1 req 8152077365387264769 0.002999952s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=kibgmb-6:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.6,z=96bf8502-2f39-4d53-8196-3378c28da3b4,az=0},d={b=kibgmb-6:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.6,z=10090ccf-aad6-416d-9775-df8b7b27345a,az=0}} returned ret=-2 2024-08-18T16:04:51.849 INFO:tasks.rgw.c1.client.1.smithi134.stdout:2024-08-18T16:04:51.848+0000 7fc03e8fe640 -1 req 15777117946285087748 0.000999977s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=kibgmb-70:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.18,z=96bf8502-2f39-4d53-8196-3378c28da3b4,az=0},d={b=kibgmb-70:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.18,z=018ea73b-7528-48da-aafa-0118123784f7,az=0}} returned ret=-2 2024-08-18T16:04:51.849 INFO:tasks.rgw.c2.client.0.smithi193.stdout:2024-08-18T16:04:51.849+0000 7f43dfdda640 -1 req 13254674161908935099 0.000999984s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=kibgmb-70:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.18,z=96bf8502-2f39-4d53-8196-3378c28da3b4,az=0},d={b=kibgmb-70:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.18,z=10090ccf-aad6-416d-9775-df8b7b27345a,az=0}} returned ret=-2 2024-08-18T16:04:51.858 INFO:tasks.rgw.c2.client.0.smithi193.stdout:2024-08-18T16:04:51.858+0000 7f43bed98640 -1 req 16657169194004574666 0.001999968s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=kibgmb-47:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.1,z=96bf8502-2f39-4d53-8196-3378c28da3b4,az=0},d={b=kibgmb-47:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.1,z=10090ccf-aad6-416d-9775-df8b7b27345a,az=0}} returned ret=-2 2024-08-18T16:04:51.859 INFO:tasks.rgw.c1.client.1.smithi134.stdout:2024-08-18T16:04:51.858+0000 7fbf91108640 -1 req 2026155864864831455 0.001999953s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=kibgmb-47:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.1,z=96bf8502-2f39-4d53-8196-3378c28da3b4,az=0},d={b=kibgmb-47:96bf8502-2f39-4d53-8196-3378c28da3b4.5491.1,z=018ea73b-7528-48da-aafa-0118123784f7,az=0}} returned ret=-2 2024-08-18T16:04:51.893 INFO:tasks.rgw.c1.client.1.smithi134.stdout:2024-08-18T16:04:51.892+0000 7fc058233640 -1 req 12086087888657366455 0.000999976s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=kibgmb-57:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.5,z=96bf8502-2f39-4d53-8196-3378c28da3b4,az=0},d={b=kibgmb-57:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.5,z=018ea73b-7528-48da-aafa-0118123784f7,az=0}} returned ret=-2 2024-08-18T16:04:51.894 INFO:tasks.rgw.c2.client.0.smithi193.stdout:2024-08-18T16:04:51.893+0000 7f437cd14640 -1 req 7580466638435026359 0.001999968s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=kibgmb-57:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.5,z=96bf8502-2f39-4d53-8196-3378c28da3b4,az=0},d={b=kibgmb-57:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.5,z=10090ccf-aad6-416d-9775-df8b7b27345a,az=0}} returned ret=-2 2024-08-18T16:04:51.908 INFO:tasks.rgw.c1.client.1.smithi134.stdout:2024-08-18T16:04:51.907+0000 7fbfa292b640 -1 req 16642220585085347234 0.001999954s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=kibgmb-45:10090ccf-aad6-416d-9775-df8b7b27345a.5181.1,z=96bf8502-2f39-4d53-8196-3378c28da3b4,az=0},d={b=kibgmb-45:10090ccf-aad6-416d-9775-df8b7b27345a.5181.1,z=018ea73b-7528-48da-aafa-0118123784f7,az=0}} returned ret=-2 2024-08-18T16:04:51.908 INFO:tasks.rgw.c2.client.0.smithi193.stdout:2024-08-18T16:04:51.908+0000 7f430ec38640 -1 req 18013439984533340498 0.002999952s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=kibgmb-45:10090ccf-aad6-416d-9775-df8b7b27345a.5181.1,z=96bf8502-2f39-4d53-8196-3378c28da3b4,az=0},d={b=kibgmb-45:10090ccf-aad6-416d-9775-df8b7b27345a.5181.1,z=10090ccf-aad6-416d-9775-df8b7b27345a,az=0}} returned ret=-2 2024-08-18T16:04:51.947 INFO:tasks.rgw.c1.client.1.smithi134.stdout:2024-08-18T16:04:51.946+0000 7fbfbc15e640 -1 req 4523869482850284628 0.000999977s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=kibgmb-44:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.44,z=96bf8502-2f39-4d53-8196-3378c28da3b4,az=0},d={b=kibgmb-44:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.44,z=018ea73b-7528-48da-aafa-0118123784f7,az=0}} returned ret=-2 2024-08-18T16:04:51.947 INFO:tasks.rgw.c2.client.0.smithi193.stdout:2024-08-18T16:04:51.947+0000 7f43cbdb2640 -1 req 16220760628780542568 0.001999968s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=kibgmb-44:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.44,z=96bf8502-2f39-4d53-8196-3378c28da3b4,az=0},d={b=kibgmb-44:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.44,z=10090ccf-aad6-416d-9775-df8b7b27345a,az=0}} returned ret=-2 2024-08-18T16:04:51.953 INFO:tasks.rgw.c1.client.1.smithi134.stdout:2024-08-18T16:04:51.952+0000 7fbf96913640 -1 req 132579636322368717 0.000999977s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=kibgmb-54:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.2,z=96bf8502-2f39-4d53-8196-3378c28da3b4,az=0},d={b=kibgmb-54:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.2,z=018ea73b-7528-48da-aafa-0118123784f7,az=0}} returned ret=-2 2024-08-18T16:04:51.954 INFO:tasks.rgw.c2.client.0.smithi193.stdout:2024-08-18T16:04:51.953+0000 7f438b531640 -1 req 17669444391841350399 0.001999968s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=kibgmb-54:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.2,z=96bf8502-2f39-4d53-8196-3378c28da3b4,az=0},d={b=kibgmb-54:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.2,z=10090ccf-aad6-416d-9775-df8b7b27345a,az=0}} returned ret=-2 2024-08-18T16:04:51.991 INFO:tasks.rgw.c1.client.1.smithi134.stdout:2024-08-18T16:04:51.990+0000 7fbffd07b640 -1 req 11741002750872977317 0.000999977s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=kibgmb-8:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.8,z=96bf8502-2f39-4d53-8196-3378c28da3b4,az=0},d={b=kibgmb-8:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.8,z=018ea73b-7528-48da-aafa-0118123784f7,az=0}} returned ret=-2 2024-08-18T16:04:51.992 INFO:tasks.rgw.c2.client.0.smithi193.stdout:2024-08-18T16:04:51.991+0000 7f432b471640 -1 req 4795577659059370198 0.001999968s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=kibgmb-8:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.8,z=96bf8502-2f39-4d53-8196-3378c28da3b4,az=0},d={b=kibgmb-8:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.8,z=10090ccf-aad6-416d-9775-df8b7b27345a,az=0}} returned ret=-2 2024-08-18T16:04:51.997 INFO:tasks.rgw.c1.client.1.smithi134.stdout:2024-08-18T16:04:51.996+0000 7fbf9f925640 -1 req 4050035437231843887 0.000999976s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=kibgmb-2:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.2,z=96bf8502-2f39-4d53-8196-3378c28da3b4,az=0},d={b=kibgmb-2:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.2,z=018ea73b-7528-48da-aafa-0118123784f7,az=0}} returned ret=-2 2024-08-18T16:04:51.997 INFO:tasks.rgw.c2.client.0.smithi193.stdout:2024-08-18T16:04:51.997+0000 7f4311c3e640 -1 req 3872982827475494058 0.001999968s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=kibgmb-2:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.2,z=96bf8502-2f39-4d53-8196-3378c28da3b4,az=0},d={b=kibgmb-2:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.2,z=10090ccf-aad6-416d-9775-df8b7b27345a,az=0}} returned ret=-2 2024-08-18T16:04:52.000 INFO:tasks.rgw.c1.client.1.smithi134.stdout:2024-08-18T16:04:52.000+0000 7fc08f4a5640 -1 req 7986979559938452448 0.000999977s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=kibgmb-1:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.1,z=96bf8502-2f39-4d53-8196-3378c28da3b4,az=0},d={b=kibgmb-1:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.1,z=018ea73b-7528-48da-aafa-0118123784f7,az=0}} returned ret=-2 2024-08-18T16:04:52.001 INFO:tasks.rgw.c2.client.0.smithi193.stdout:2024-08-18T16:04:52.001+0000 7f432c473640 -1 req 12362582261403835157 0.001999968s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=kibgmb-1:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.1,z=96bf8502-2f39-4d53-8196-3378c28da3b4,az=0},d={b=kibgmb-1:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.1,z=10090ccf-aad6-416d-9775-df8b7b27345a,az=0}} returned ret=-2 2024-08-18T16:04:52.093 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'kibgmb-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:04:52.094 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'kibgmb-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:04:52.094 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket kibgmb-72 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:04:52.139 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:04:52.139 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:04:52.182 INFO:teuthology.orchestra.run.smithi134.stdout:[] 2024-08-18T16:04:52.184 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_index_log_trim ... ok 2024-08-18T16:04:52.185 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=kibgmb-74 2024-08-18T16:04:52.302 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:04:52.302 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:04:52.302 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:04:52.350 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:04:52.351 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:04:52.384 INFO:teuthology.orchestra.run.smithi134.stderr:No --period given, using current period=01050e30-179b-43be-bdbb-3bb23cf37ae8 2024-08-18T16:04:52.397 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T16:04:52.397 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:04:52.397 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T16:04:52.397 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T16:04:52.397 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:04:52.397 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:04:52.397 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T16:04:52.397 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T16:04:52.397 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:04:52.398 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:04:52.398 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T16:04:52.398 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T16:04:52.398 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:04:52.398 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:04:52.398 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723997092.198529_690.1", 2024-08-18T16:04:52.398 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:04:52.198529Z" 2024-08-18T16:04:52.398 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:04:52.398 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T16:04:52.399 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-08-18T16:04:52.399 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:04:52.399 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:04:52.400 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:04:52.451 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:04:52.452 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:04:52.484 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:04:52.483+0000 7fcaf7010880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:04:52.484 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:04:52.484+0000 7fcaf7010880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:04:52.498 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T16:04:52.498 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T16:04:52.499 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T16:04:52.499 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T16:04:52.499 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T16:04:52.499 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T16:04:52.499 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T16:04:52.499 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:04:52.499 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T16:04:52.499 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:04:52.499 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T16:04:52.499 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:04:52.499 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:04:52.499 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T16:04:52.499 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:04:52.500 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T16:04:52.500 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:04:52.500 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T16:04:52.500 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T16:04:52.500 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:04:52.500 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:04:52.500 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:04:52.500 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T16:04:52.500 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:04:52.500 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:04:52.500 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T16:04:52.500 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:04:52.500 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:04:52.501 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:04:52.501 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T16:04:52.501 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T16:04:52.501 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:04:52.501 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:04:52.501 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:04:52.501 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T16:04:52.501 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:04:52.501 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:04:52.502 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T16:04:52.502 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:04:52.502 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:04:52.502 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:04:52.502 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T16:04:52.502 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T16:04:52.502 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:04:52.502 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:04:52.502 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:04:52.502 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T16:04:52.502 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:04:52.502 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:04:52.503 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723997092.198529_690.1", 2024-08-18T16:04:52.503 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:04:52.503 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:04:52.503 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:04:52.503 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:04:52.198529Z", 2024-08-18T16:04:52.503 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T16:04:52.503 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:04:52.503 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:04:52.503 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T16:04:52.503 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:04:52.503 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T16:04:52.503 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T16:04:52.503 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T16:04:52.503 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:04:52.503 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T16:04:52Z" 2024-08-18T16:04:52.504 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T16:04:52.505 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-08-18T16:04:52.505 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-08-18T16:04:52.505 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:04:52.505 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:04:52.505 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:04:52.552 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:04:52.553 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:04:52.581 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:04:52.580+0000 7f6a71e11880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:04:52.583 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:04:52.583+0000 7f6a71e11880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:04:52.596 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T16:04:52.596 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T16:04:52.596 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T16:04:52.596 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T16:04:52.596 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T16:04:52.596 INFO:teuthology.orchestra.run.smithi134.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T16:04:52.596 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T16:04:52.597 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:04:52.597 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T16:04:52.597 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:04:52.597 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T16:04:52.597 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:04:52.597 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T16:04:52.597 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T16:04:52.597 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:04:52.597 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 1, 2024-08-18T16:04:52.597 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:04:52.597 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T16:04:52.597 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T16:04:52.598 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:04:52.598 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:04:52.598 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:04:52.598 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T16:04:52.598 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:04:52.598 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T16:04:52.598 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T16:04:52.598 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:04:52.598 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:04:52.598 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:04:52.599 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T16:04:52.599 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T16:04:52.599 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:04:52.599 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:04:52.599 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:04:52.599 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T16:04:52.599 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:04:52.599 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T16:04:52.599 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T16:04:52.599 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:04:52.599 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:04:52.599 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:04:52.600 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T16:04:52.600 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T16:04:52.600 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:04:52.600 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:04:52.600 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:04:52.600 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T16:04:52.600 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:04:52.600 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T16:04:52.600 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723997092.198529_690.1", 2024-08-18T16:04:52.600 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:04:52.600 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:04:52.600 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:04:52.601 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:04:52.198529Z", 2024-08-18T16:04:52.601 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T16:04:52.601 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:04:52.601 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:04:52.601 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T16:04:52.601 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:04:52.601 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T16:04:52.601 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 1, 2024-08-18T16:04:52.601 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 1 2024-08-18T16:04:52.601 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:04:52.602 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T16:04:52Z" 2024-08-18T16:04:52.602 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T16:04:52.603 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-08-18T16:04:52.603 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-74', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:04:52.603 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-74', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:04:52.603 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-74 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T16:04:52.657 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:04:52.657 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:04:52.693 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:04:52.692+0000 7fbf37f26880 1 waiting to reach incremental sync.. 2024-08-18T16:04:57.695 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:04:57.694+0000 7fbf37f26880 1 bucket sync caught up with source: 2024-08-18T16:04:57.695 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, 00000000001.75.6, , , , 00000000001.71.6, , , 00000000001.93.6, , 00000000001.70.6] 2024-08-18T16:04:57.695 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, 00000000001.75.6, , , , 00000000001.71.6, , , 00000000001.93.6, , 00000000001.70.6] 2024-08-18T16:04:57.695 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:04:57.694+0000 7fbf37f26880 0 bucket checkpoint complete 2024-08-18T16:04:57.710 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-74', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:04:57.710 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-74', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:04:57.710 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-74 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:04:57.765 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:04:57.765 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:04:57.803 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:04:57.802+0000 7f9e8fa15880 1 waiting to reach incremental sync.. 2024-08-18T16:05:02.805 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:05:02.804+0000 7f9e8fa15880 1 bucket sync caught up with source: 2024-08-18T16:05:02.806 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000001.75.6, , , , 00000000001.71.6, , , 00000000001.93.6, , 00000000001.70.6] 2024-08-18T16:05:02.806 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000001.75.6, , , , 00000000001.71.6, , , 00000000001.93.6, , 00000000001.70.6] 2024-08-18T16:05:02.806 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:05:02.804+0000 7f9e8fa15880 0 bucket checkpoint complete 2024-08-18T16:05:02.823 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-74', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:05:02.823 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-74', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:05:02.823 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-74 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:05:02.883 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:05:02.883 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:05:02.922 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:05:02.921+0000 7f1f74830880 1 bucket sync caught up with source: 2024-08-18T16:05:02.922 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000001.279.6, , , , 00000000001.341.6, , , 00000000001.435.6, , 00000000001.125.6] 2024-08-18T16:05:02.922 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000001.279.6, , , , 00000000001.341.6, , , 00000000001.435.6, , 00000000001.125.6] 2024-08-18T16:05:02.922 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:05:02.921+0000 7f1f74830880 0 bucket checkpoint complete 2024-08-18T16:05:02.936 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-74', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:05:02.936 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-74', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:05:02.937 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-74 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:05:02.990 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:05:02.991 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:05:03.025 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:05:03.024+0000 7f19da327880 1 bucket sync caught up with source: 2024-08-18T16:05:03.025 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000001.279.6, , , , 00000000001.341.6, , , 00000000001.435.6, , 00000000001.125.6] 2024-08-18T16:05:03.025 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000001.279.6, , , , 00000000001.341.6, , , 00000000001.435.6, , 00000000001.125.6] 2024-08-18T16:05:03.025 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:05:03.024+0000 7f19da327880 0 bucket checkpoint complete 2024-08-18T16:05:03.038 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-74', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:05:03.038 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-74', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:05:03.039 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-74 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:05:03.094 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:05:03.116 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:05:03.127 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:05:03.127+0000 7f82b7931880 1 waiting to reach incremental sync.. 2024-08-18T16:05:08.129 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:05:08.128+0000 7f82b7931880 1 bucket sync caught up with source: 2024-08-18T16:05:08.129 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000001.182.6, , , , 00000000001.288.6, , , 00000000001.210.6, , 00000000001.153.6] 2024-08-18T16:05:08.129 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000001.182.6, , , , 00000000001.288.6, , , 00000000001.210.6, , 00000000001.153.6] 2024-08-18T16:05:08.130 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:05:08.128+0000 7f82b7931880 0 bucket checkpoint complete 2024-08-18T16:05:08.147 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-74', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:05:08.147 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-74', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:05:08.148 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-74 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T16:05:08.209 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:05:08.210 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:05:08.249 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:05:08.248+0000 7f50d6826880 1 bucket sync caught up with source: 2024-08-18T16:05:08.249 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, 00000000001.182.6, , , , 00000000001.288.6, , , 00000000001.210.6, , 00000000001.153.6] 2024-08-18T16:05:08.249 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, 00000000001.182.6, , , , 00000000001.288.6, , , 00000000001.210.6, , 00000000001.153.6] 2024-08-18T16:05:08.249 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:05:08.248+0000 7f50d6826880 0 bucket checkpoint complete 2024-08-18T16:05:08.265 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-74 zones={test-zone2, test-zone1} 2024-08-18T16:05:08.270 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:05:08.273 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-08-18T16:05:08.273 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-08-18T16:05:08.273 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-08-18T16:05:08.273 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-08-18T16:05:08.273 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:05:08.276 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-08-18T16:05:08.276 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-08-18T16:05:08.276 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-08-18T16:05:08.276 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-08-18T16:05:08.282 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-08-18T16:05:08.332 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-08-18T16:05:08.345 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-08-18T16:05:08.398 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-08-18T16:05:08.405 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-08-18T16:05:08.461 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-08-18T16:05:08.476 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-08-18T16:05:08.528 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-08-18T16:05:08.541 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-08-18T16:05:08.593 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-08-18T16:05:08.600 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-08-18T16:05:08.653 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-08-18T16:05:08.705 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-74 zones={test-zone2, test-zone1} 2024-08-18T16:05:08.706 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-74 zones={test-zone3, test-zone1} 2024-08-18T16:05:08.710 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:05:08.713 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-08-18T16:05:08.713 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-08-18T16:05:08.713 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-08-18T16:05:08.713 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-08-18T16:05:08.713 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:05:08.716 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-08-18T16:05:08.716 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-08-18T16:05:08.716 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-08-18T16:05:08.716 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-08-18T16:05:08.721 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-08-18T16:05:08.770 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-08-18T16:05:08.826 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-08-18T16:05:08.877 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-08-18T16:05:08.932 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-08-18T16:05:08.985 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-08-18T16:05:09.037 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-08-18T16:05:09.088 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-08-18T16:05:09.142 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-08-18T16:05:09.193 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-08-18T16:05:09.244 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-08-18T16:05:09.341 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-08-18T16:05:09.437 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-74 zones={test-zone3, test-zone1} 2024-08-18T16:05:09.437 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-74 zones={test-zone3, test-zone2} 2024-08-18T16:05:09.442 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:05:09.445 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-08-18T16:05:09.445 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-08-18T16:05:09.445 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-08-18T16:05:09.445 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-08-18T16:05:09.446 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:05:09.449 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-08-18T16:05:09.449 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-08-18T16:05:09.449 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-08-18T16:05:09.449 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-08-18T16:05:09.455 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-08-18T16:05:09.504 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-08-18T16:05:09.558 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-08-18T16:05:09.611 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-08-18T16:05:09.660 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-08-18T16:05:09.714 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-08-18T16:05:09.809 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-08-18T16:05:09.860 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-08-18T16:05:09.961 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-08-18T16:05:10.057 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-08-18T16:05:10.112 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-08-18T16:05:10.165 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-08-18T16:05:10.262 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-74 zones={test-zone3, test-zone2} 2024-08-18T16:05:10.262 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'kibgmb-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:05:10.262 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'kibgmb-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:05:10.262 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket kibgmb-74 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:05:10.314 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:05:10.314 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:05:10.362 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T16:05:10.362 INFO:teuthology.orchestra.run.smithi134.stdout: "layout": { 2024-08-18T16:05:10.362 INFO:teuthology.orchestra.run.smithi134.stdout: "resharding": "None", 2024-08-18T16:05:10.362 INFO:teuthology.orchestra.run.smithi134.stdout: "current_index": { 2024-08-18T16:05:10.362 INFO:teuthology.orchestra.run.smithi134.stdout: "gen": 0, 2024-08-18T16:05:10.362 INFO:teuthology.orchestra.run.smithi134.stdout: "layout": { 2024-08-18T16:05:10.362 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "Normal", 2024-08-18T16:05:10.362 INFO:teuthology.orchestra.run.smithi134.stdout: "normal": { 2024-08-18T16:05:10.363 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 11, 2024-08-18T16:05:10.363 INFO:teuthology.orchestra.run.smithi134.stdout: "hash_type": "Mod" 2024-08-18T16:05:10.363 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:05:10.363 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:05:10.363 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:05:10.363 INFO:teuthology.orchestra.run.smithi134.stdout: "logs": [ 2024-08-18T16:05:10.363 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:05:10.363 INFO:teuthology.orchestra.run.smithi134.stdout: "gen": 0, 2024-08-18T16:05:10.363 INFO:teuthology.orchestra.run.smithi134.stdout: "layout": { 2024-08-18T16:05:10.363 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "InIndex", 2024-08-18T16:05:10.363 INFO:teuthology.orchestra.run.smithi134.stdout: "in_index": { 2024-08-18T16:05:10.363 INFO:teuthology.orchestra.run.smithi134.stdout: "gen": 0, 2024-08-18T16:05:10.363 INFO:teuthology.orchestra.run.smithi134.stdout: "layout": { 2024-08-18T16:05:10.364 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 11, 2024-08-18T16:05:10.364 INFO:teuthology.orchestra.run.smithi134.stdout: "hash_type": "Mod" 2024-08-18T16:05:10.364 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:05:10.364 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:05:10.364 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:05:10.364 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:05:10.364 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T16:05:10.364 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:05:10.364 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T16:05:10.365 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'kibgmb-74', '--gen', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:05:10.365 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'kibgmb-74', '--gen', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:05:10.366 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket kibgmb-74 --gen 0 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:05:10.414 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:05:10.414 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:05:10.457 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T16:05:10.457 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:05:10.457 INFO:teuthology.orchestra.run.smithi134.stdout: "op_id": "1#00000000001.75.6", 2024-08-18T16:05:10.457 INFO:teuthology.orchestra.run.smithi134.stdout: "op_tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.3975927000494656787", 2024-08-18T16:05:10.458 INFO:teuthology.orchestra.run.smithi134.stdout: "op": "write", 2024-08-18T16:05:10.458 INFO:teuthology.orchestra.run.smithi134.stdout: "object": "a", 2024-08-18T16:05:10.458 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:05:10.458 INFO:teuthology.orchestra.run.smithi134.stdout: "state": "complete", 2024-08-18T16:05:10.458 INFO:teuthology.orchestra.run.smithi134.stdout: "index_ver": 1, 2024-08-18T16:05:10.458 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:04:52.203652566Z", 2024-08-18T16:05:10.458 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:05:10.458 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:05:10.458 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 8 2024-08-18T16:05:10.458 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:05:10.458 INFO:teuthology.orchestra.run.smithi134.stdout: "bilog_flags": 0, 2024-08-18T16:05:10.459 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned": false, 2024-08-18T16:05:10.459 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "", 2024-08-18T16:05:10.459 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "", 2024-08-18T16:05:10.459 INFO:teuthology.orchestra.run.smithi134.stdout: "zones_trace": [ 2024-08-18T16:05:10.459 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:05:10.459 INFO:teuthology.orchestra.run.smithi134.stdout: "entry": "96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-74:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.22" 2024-08-18T16:05:10.459 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:05:10.459 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T16:05:10.459 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:05:10.459 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:05:10.459 INFO:teuthology.orchestra.run.smithi134.stdout: "op_id": "5#00000000001.71.6", 2024-08-18T16:05:10.459 INFO:teuthology.orchestra.run.smithi134.stdout: "op_tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.7280967450554460055", 2024-08-18T16:05:10.460 INFO:teuthology.orchestra.run.smithi134.stdout: "op": "write", 2024-08-18T16:05:10.460 INFO:teuthology.orchestra.run.smithi134.stdout: "object": "d", 2024-08-18T16:05:10.460 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:05:10.460 INFO:teuthology.orchestra.run.smithi134.stdout: "state": "complete", 2024-08-18T16:05:10.460 INFO:teuthology.orchestra.run.smithi134.stdout: "index_ver": 1, 2024-08-18T16:05:10.460 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:04:52.299374876Z", 2024-08-18T16:05:10.460 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:05:10.460 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:05:10.460 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 5 2024-08-18T16:05:10.460 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:05:10.460 INFO:teuthology.orchestra.run.smithi134.stdout: "bilog_flags": 0, 2024-08-18T16:05:10.460 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned": false, 2024-08-18T16:05:10.460 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "", 2024-08-18T16:05:10.461 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "", 2024-08-18T16:05:10.461 INFO:teuthology.orchestra.run.smithi134.stdout: "zones_trace": [ 2024-08-18T16:05:10.461 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:05:10.461 INFO:teuthology.orchestra.run.smithi134.stdout: "entry": "96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-74:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.22" 2024-08-18T16:05:10.461 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:05:10.461 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T16:05:10.461 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:05:10.461 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:05:10.461 INFO:teuthology.orchestra.run.smithi134.stdout: "op_id": "8#00000000001.93.6", 2024-08-18T16:05:10.461 INFO:teuthology.orchestra.run.smithi134.stdout: "op_tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.17445894353531357357", 2024-08-18T16:05:10.461 INFO:teuthology.orchestra.run.smithi134.stdout: "op": "write", 2024-08-18T16:05:10.462 INFO:teuthology.orchestra.run.smithi134.stdout: "object": "c", 2024-08-18T16:05:10.462 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:05:10.462 INFO:teuthology.orchestra.run.smithi134.stdout: "state": "complete", 2024-08-18T16:05:10.462 INFO:teuthology.orchestra.run.smithi134.stdout: "index_ver": 1, 2024-08-18T16:05:10.462 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:04:52.253512459Z", 2024-08-18T16:05:10.462 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:05:10.462 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:05:10.462 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 24 2024-08-18T16:05:10.462 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:05:10.462 INFO:teuthology.orchestra.run.smithi134.stdout: "bilog_flags": 0, 2024-08-18T16:05:10.462 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned": false, 2024-08-18T16:05:10.463 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "", 2024-08-18T16:05:10.463 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "", 2024-08-18T16:05:10.463 INFO:teuthology.orchestra.run.smithi134.stdout: "zones_trace": [ 2024-08-18T16:05:10.463 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:05:10.463 INFO:teuthology.orchestra.run.smithi134.stdout: "entry": "96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-74:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.22" 2024-08-18T16:05:10.463 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:05:10.463 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T16:05:10.463 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:05:10.463 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:05:10.463 INFO:teuthology.orchestra.run.smithi134.stdout: "op_id": "10#00000000001.70.6", 2024-08-18T16:05:10.464 INFO:teuthology.orchestra.run.smithi134.stdout: "op_tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.11077698423237993719", 2024-08-18T16:05:10.464 INFO:teuthology.orchestra.run.smithi134.stdout: "op": "write", 2024-08-18T16:05:10.464 INFO:teuthology.orchestra.run.smithi134.stdout: "object": "b", 2024-08-18T16:05:10.464 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:05:10.464 INFO:teuthology.orchestra.run.smithi134.stdout: "state": "complete", 2024-08-18T16:05:10.464 INFO:teuthology.orchestra.run.smithi134.stdout: "index_ver": 1, 2024-08-18T16:05:10.464 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:04:52.246893634Z", 2024-08-18T16:05:10.464 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:05:10.464 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:05:10.464 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 9 2024-08-18T16:05:10.464 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:05:10.464 INFO:teuthology.orchestra.run.smithi134.stdout: "bilog_flags": 0, 2024-08-18T16:05:10.465 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned": false, 2024-08-18T16:05:10.465 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "", 2024-08-18T16:05:10.465 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "", 2024-08-18T16:05:10.465 INFO:teuthology.orchestra.run.smithi134.stdout: "zones_trace": [ 2024-08-18T16:05:10.465 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:05:10.465 INFO:teuthology.orchestra.run.smithi134.stdout: "entry": "96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-74:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.22" 2024-08-18T16:05:10.465 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:05:10.465 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T16:05:10.465 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:05:10.465 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T16:05:10.466 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'kibgmb-74', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:05:10.466 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'kibgmb-74', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:05:10.467 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket kibgmb-74 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:05:10.514 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:05:10.514 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:05:10.554 INFO:teuthology.orchestra.run.smithi134.stdout:tenant: 2024-08-18T16:05:10.554 INFO:teuthology.orchestra.run.smithi134.stdout:bucket name: kibgmb-74 2024-08-18T16:05:10.558 INFO:teuthology.orchestra.run.smithi134.stdout:total entries: 4 2024-08-18T16:05:10.595 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'kibgmb-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:05:10.595 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'kibgmb-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:05:10.596 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket kibgmb-74 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:05:10.640 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:05:10.640 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:05:10.682 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T16:05:10.682 INFO:teuthology.orchestra.run.smithi134.stdout: "layout": { 2024-08-18T16:05:10.682 INFO:teuthology.orchestra.run.smithi134.stdout: "resharding": "None", 2024-08-18T16:05:10.683 INFO:teuthology.orchestra.run.smithi134.stdout: "current_index": { 2024-08-18T16:05:10.683 INFO:teuthology.orchestra.run.smithi134.stdout: "gen": 1, 2024-08-18T16:05:10.683 INFO:teuthology.orchestra.run.smithi134.stdout: "layout": { 2024-08-18T16:05:10.683 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "Normal", 2024-08-18T16:05:10.683 INFO:teuthology.orchestra.run.smithi134.stdout: "normal": { 2024-08-18T16:05:10.683 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 3, 2024-08-18T16:05:10.683 INFO:teuthology.orchestra.run.smithi134.stdout: "hash_type": "Mod" 2024-08-18T16:05:10.683 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:05:10.683 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:05:10.683 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:05:10.683 INFO:teuthology.orchestra.run.smithi134.stdout: "logs": [ 2024-08-18T16:05:10.683 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:05:10.683 INFO:teuthology.orchestra.run.smithi134.stdout: "gen": 0, 2024-08-18T16:05:10.684 INFO:teuthology.orchestra.run.smithi134.stdout: "layout": { 2024-08-18T16:05:10.684 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "InIndex", 2024-08-18T16:05:10.684 INFO:teuthology.orchestra.run.smithi134.stdout: "in_index": { 2024-08-18T16:05:10.684 INFO:teuthology.orchestra.run.smithi134.stdout: "gen": 0, 2024-08-18T16:05:10.684 INFO:teuthology.orchestra.run.smithi134.stdout: "layout": { 2024-08-18T16:05:10.684 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 11, 2024-08-18T16:05:10.684 INFO:teuthology.orchestra.run.smithi134.stdout: "hash_type": "Mod" 2024-08-18T16:05:10.684 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:05:10.684 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:05:10.684 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:05:10.684 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:05:10.684 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:05:10.684 INFO:teuthology.orchestra.run.smithi134.stdout: "gen": 1, 2024-08-18T16:05:10.684 INFO:teuthology.orchestra.run.smithi134.stdout: "layout": { 2024-08-18T16:05:10.685 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "InIndex", 2024-08-18T16:05:10.685 INFO:teuthology.orchestra.run.smithi134.stdout: "in_index": { 2024-08-18T16:05:10.685 INFO:teuthology.orchestra.run.smithi134.stdout: "gen": 1, 2024-08-18T16:05:10.685 INFO:teuthology.orchestra.run.smithi134.stdout: "layout": { 2024-08-18T16:05:10.685 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 3, 2024-08-18T16:05:10.685 INFO:teuthology.orchestra.run.smithi134.stdout: "hash_type": "Mod" 2024-08-18T16:05:10.685 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:05:10.685 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:05:10.685 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:05:10.685 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:05:10.685 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T16:05:10.685 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:05:10.685 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T16:05:10.686 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'kibgmb-74', '--gen', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:05:10.686 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'kibgmb-74', '--gen', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:05:10.687 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket kibgmb-74 --gen 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:05:10.774 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:05:10.774 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:05:10.817 INFO:teuthology.orchestra.run.smithi134.stdout:[] 2024-08-18T16:05:10.880 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-74', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:05:10.880 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-74', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:05:10.881 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-74 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T16:05:10.942 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:05:10.943 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:05:10.980 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:05:10.979+0000 7f3bc9012880 1 waiting to reach latest gen 1, on 0.. 2024-08-18T16:05:15.981 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:05:15.980+0000 7f3bc9012880 1 waiting to reach latest gen 1, on 0.. 2024-08-18T16:05:20.982 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:05:20.981+0000 7f3bc9012880 1 waiting for incremental sync to catch up: 2024-08-18T16:05:20.982 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000003.77.6, , 00000000002.76.6] 2024-08-18T16:05:20.983 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000003.77.6, 00000000002.65.6, 00000000002.76.6] 2024-08-18T16:05:25.983 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:05:25.982+0000 7f3bc9012880 1 waiting for incremental sync to catch up: 2024-08-18T16:05:25.983 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000003.77.6, , 00000000002.76.6] 2024-08-18T16:05:25.983 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000003.77.6, 00000000002.65.6, 00000000002.76.6] 2024-08-18T16:05:30.983 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:05:30.983+0000 7f3bc9012880 1 waiting for incremental sync to catch up: 2024-08-18T16:05:30.984 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000003.77.6, , 00000000002.76.6] 2024-08-18T16:05:30.984 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000003.77.6, 00000000002.65.6, 00000000002.76.6] 2024-08-18T16:05:35.984 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:05:35.984+0000 7f3bc9012880 1 waiting for incremental sync to catch up: 2024-08-18T16:05:35.984 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000003.77.6, , 00000000002.76.6] 2024-08-18T16:05:35.984 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000003.77.6, 00000000002.65.6, 00000000002.76.6] 2024-08-18T16:05:40.985 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:05:40.984+0000 7f3bc9012880 1 waiting for incremental sync to catch up: 2024-08-18T16:05:40.985 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000003.77.6, , 00000000002.76.6] 2024-08-18T16:05:40.985 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000003.77.6, 00000000002.65.6, 00000000002.76.6] 2024-08-18T16:05:45.986 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:05:45.985+0000 7f3bc9012880 1 waiting for incremental sync to catch up: 2024-08-18T16:05:45.986 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000003.77.6, , 00000000002.76.6] 2024-08-18T16:05:45.986 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000003.77.6, 00000000002.65.6, 00000000002.76.6] 2024-08-18T16:05:50.986 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:05:50.986+0000 7f3bc9012880 1 waiting for incremental sync to catch up: 2024-08-18T16:05:50.987 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000003.77.6, , 00000000002.76.6] 2024-08-18T16:05:50.987 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000003.77.6, 00000000002.65.6, 00000000002.76.6] 2024-08-18T16:05:55.987 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:05:55.987+0000 7f3bc9012880 1 waiting for incremental sync to catch up: 2024-08-18T16:05:55.987 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000003.77.6, , 00000000002.76.6] 2024-08-18T16:05:55.988 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000003.77.6, 00000000002.65.6, 00000000002.76.6] 2024-08-18T16:06:00.988 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:06:00.987+0000 7f3bc9012880 1 bucket sync caught up with source: 2024-08-18T16:06:00.988 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000003.77.6, 00000000002.65.6, 00000000002.76.6] 2024-08-18T16:06:00.988 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000003.77.6, 00000000002.65.6, 00000000002.76.6] 2024-08-18T16:06:00.988 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:06:00.987+0000 7f3bc9012880 0 bucket checkpoint complete 2024-08-18T16:06:01.007 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-74', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:06:01.007 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-74', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:06:01.008 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-74 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:06:01.069 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:06:01.069 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:06:01.106 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:06:01.106+0000 7fa0d061f880 1 bucket sync caught up with source: 2024-08-18T16:06:01.106 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000003.77.6, 00000000002.65.6, 00000000002.76.6] 2024-08-18T16:06:01.106 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000003.77.6, 00000000002.65.6, 00000000002.76.6] 2024-08-18T16:06:01.107 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:06:01.106+0000 7fa0d061f880 0 bucket checkpoint complete 2024-08-18T16:06:01.121 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-74', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:06:01.121 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-74', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:06:01.121 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-74 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:06:01.176 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:06:01.176 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:06:01.211 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:06:01.211+0000 7f3333611880 1 waiting for incremental sync to catch up: 2024-08-18T16:06:01.211 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000001.279.6, , , 00000000001.247.6, 00000000001.341.6, , , 00000000002.441.6, , 00000000002.129.6] 2024-08-18T16:06:01.211 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000002.283.6, , , 00000000001.247.6, 00000000001.341.6, , , 00000000002.441.6, , 00000000002.129.6] 2024-08-18T16:06:06.215 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:06:06.214+0000 7f3333611880 1 waiting for incremental sync to catch up: 2024-08-18T16:06:06.215 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000001.279.6, , , 00000000001.247.6, 00000000001.341.6, , , 00000000002.441.6, , 00000000002.129.6] 2024-08-18T16:06:06.215 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000002.283.6, , , 00000000001.247.6, 00000000001.341.6, , , 00000000002.441.6, , 00000000002.129.6] 2024-08-18T16:06:11.216 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:06:11.215+0000 7f3333611880 1 waiting for incremental sync to catch up: 2024-08-18T16:06:11.216 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000001.279.6, , , 00000000001.247.6, 00000000001.341.6, , , 00000000002.441.6, , 00000000002.129.6] 2024-08-18T16:06:11.216 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000002.283.6, , , 00000000001.247.6, 00000000001.341.6, , , 00000000002.441.6, , 00000000002.129.6] 2024-08-18T16:06:16.217 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:06:16.217+0000 7f3333611880 1 waiting for incremental sync to catch up: 2024-08-18T16:06:16.217 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000001.279.6, , , 00000000001.247.6, 00000000001.341.6, , , 00000000002.441.6, , 00000000002.129.6] 2024-08-18T16:06:16.218 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000002.283.6, , , 00000000001.247.6, 00000000001.341.6, , , 00000000002.441.6, , 00000000002.129.6] 2024-08-18T16:06:21.217 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:06:21.217+0000 7f3333611880 1 bucket sync caught up with source: 2024-08-18T16:06:21.218 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000002.283.6, , , 00000000001.247.6, 00000000001.341.6, , , 00000000002.441.6, , 00000000002.129.6] 2024-08-18T16:06:21.218 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000002.283.6, , , 00000000001.247.6, 00000000001.341.6, , , 00000000002.441.6, , 00000000002.129.6] 2024-08-18T16:06:21.218 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:06:21.217+0000 7f3333611880 0 bucket checkpoint complete 2024-08-18T16:06:21.235 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-74', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:06:21.235 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-74', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:06:21.235 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-74 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:06:21.294 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:06:21.295 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:06:21.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:06:21.331+0000 7f81c332a880 1 bucket sync caught up with source: 2024-08-18T16:06:21.332 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000002.283.6, , , 00000000001.247.6, 00000000001.341.6, , , 00000000002.441.6, , 00000000002.129.6] 2024-08-18T16:06:21.332 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000002.283.6, , , 00000000001.247.6, 00000000001.341.6, , , 00000000002.441.6, , 00000000002.129.6] 2024-08-18T16:06:21.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:06:21.331+0000 7f81c332a880 0 bucket checkpoint complete 2024-08-18T16:06:21.346 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-74', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:06:21.346 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-74', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:06:21.346 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-74 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:06:21.400 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:06:21.400 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:06:21.435 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:06:21.434+0000 7fde73e26880 1 bucket sync caught up with source: 2024-08-18T16:06:21.435 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000002.186.6, , , 00000000001.235.6, 00000000001.288.6, , , 00000000002.216.6, , 00000000002.157.6] 2024-08-18T16:06:21.435 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000002.186.6, , , 00000000001.235.6, 00000000001.288.6, , , 00000000002.216.6, , 00000000002.157.6] 2024-08-18T16:06:21.435 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:06:21.434+0000 7fde73e26880 0 bucket checkpoint complete 2024-08-18T16:06:21.449 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-74', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:06:21.449 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-74', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:06:21.449 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-74 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T16:06:21.511 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:06:21.511 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:06:21.551 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:06:21.551+0000 7f7e0de26880 1 bucket sync caught up with source: 2024-08-18T16:06:21.551 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, 00000000002.186.6, , , 00000000001.235.6, 00000000001.288.6, , , 00000000002.216.6, , 00000000002.157.6] 2024-08-18T16:06:21.551 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, 00000000002.186.6, , , 00000000001.235.6, 00000000001.288.6, , , 00000000002.216.6, , 00000000002.157.6] 2024-08-18T16:06:21.551 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:06:21.551+0000 7f7e0de26880 0 bucket checkpoint complete 2024-08-18T16:06:21.567 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-74 zones={test-zone2, test-zone1} 2024-08-18T16:06:21.576 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:06:21.579 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-08-18T16:06:21.579 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-08-18T16:06:21.579 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-08-18T16:06:21.580 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-08-18T16:06:21.580 DEBUG:tasks.rgw_multi.zone_rados:o=e 2024-08-18T16:06:21.580 DEBUG:tasks.rgw_multi.zone_rados:o=f 2024-08-18T16:06:21.580 DEBUG:tasks.rgw_multi.zone_rados:o=g 2024-08-18T16:06:21.580 DEBUG:tasks.rgw_multi.zone_rados:o=h 2024-08-18T16:06:21.580 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:06:21.582 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-08-18T16:06:21.582 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-08-18T16:06:21.582 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-08-18T16:06:21.582 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-08-18T16:06:21.582 DEBUG:tasks.rgw_multi.zone_rados:o=e 2024-08-18T16:06:21.582 DEBUG:tasks.rgw_multi.zone_rados:o=f 2024-08-18T16:06:21.582 DEBUG:tasks.rgw_multi.zone_rados:o=g 2024-08-18T16:06:21.583 DEBUG:tasks.rgw_multi.zone_rados:o=h 2024-08-18T16:06:21.587 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-08-18T16:06:21.593 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-08-18T16:06:21.603 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-08-18T16:06:21.656 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-08-18T16:06:21.663 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-08-18T16:06:21.719 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-08-18T16:06:21.767 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-08-18T16:06:21.774 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-08-18T16:06:21.826 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-08-18T16:06:21.832 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-08-18T16:06:21.838 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-08-18T16:06:21.889 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-08-18T16:06:21.941 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2024-08-18T16:06:21.947 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2024-08-18T16:06:22.002 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2024-08-18T16:06:22.010 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2024-08-18T16:06:22.018 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2024-08-18T16:06:22.028 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2024-08-18T16:06:22.076 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2024-08-18T16:06:22.082 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2024-08-18T16:06:22.139 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2024-08-18T16:06:22.187 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2024-08-18T16:06:22.194 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2024-08-18T16:06:22.204 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2024-08-18T16:06:22.211 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-74 zones={test-zone2, test-zone1} 2024-08-18T16:06:22.211 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-74 zones={test-zone3, test-zone1} 2024-08-18T16:06:22.217 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:06:22.220 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-08-18T16:06:22.220 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-08-18T16:06:22.220 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-08-18T16:06:22.220 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-08-18T16:06:22.220 DEBUG:tasks.rgw_multi.zone_rados:o=e 2024-08-18T16:06:22.220 DEBUG:tasks.rgw_multi.zone_rados:o=f 2024-08-18T16:06:22.220 DEBUG:tasks.rgw_multi.zone_rados:o=g 2024-08-18T16:06:22.220 DEBUG:tasks.rgw_multi.zone_rados:o=h 2024-08-18T16:06:22.220 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:06:22.222 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-08-18T16:06:22.223 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-08-18T16:06:22.223 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-08-18T16:06:22.223 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-08-18T16:06:22.223 DEBUG:tasks.rgw_multi.zone_rados:o=e 2024-08-18T16:06:22.223 DEBUG:tasks.rgw_multi.zone_rados:o=f 2024-08-18T16:06:22.223 DEBUG:tasks.rgw_multi.zone_rados:o=g 2024-08-18T16:06:22.223 DEBUG:tasks.rgw_multi.zone_rados:o=h 2024-08-18T16:06:22.228 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-08-18T16:06:22.278 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-08-18T16:06:22.288 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-08-18T16:06:22.336 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-08-18T16:06:22.387 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-08-18T16:06:22.439 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-08-18T16:06:22.533 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-08-18T16:06:22.588 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-08-18T16:06:22.688 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-08-18T16:06:22.736 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-08-18T16:06:22.792 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-08-18T16:06:22.851 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-08-18T16:06:22.904 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2024-08-18T16:06:22.951 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2024-08-18T16:06:23.044 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2024-08-18T16:06:23.092 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2024-08-18T16:06:23.139 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2024-08-18T16:06:23.192 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2024-08-18T16:06:23.240 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2024-08-18T16:06:23.295 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2024-08-18T16:06:23.352 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2024-08-18T16:06:23.441 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2024-08-18T16:06:23.495 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2024-08-18T16:06:23.588 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2024-08-18T16:06:23.684 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-74 zones={test-zone3, test-zone1} 2024-08-18T16:06:23.685 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-74 zones={test-zone3, test-zone2} 2024-08-18T16:06:23.689 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:06:23.692 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-08-18T16:06:23.692 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-08-18T16:06:23.692 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-08-18T16:06:23.692 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-08-18T16:06:23.692 DEBUG:tasks.rgw_multi.zone_rados:o=e 2024-08-18T16:06:23.692 DEBUG:tasks.rgw_multi.zone_rados:o=f 2024-08-18T16:06:23.693 DEBUG:tasks.rgw_multi.zone_rados:o=g 2024-08-18T16:06:23.693 DEBUG:tasks.rgw_multi.zone_rados:o=h 2024-08-18T16:06:23.693 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:06:23.696 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-08-18T16:06:23.696 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-08-18T16:06:23.696 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-08-18T16:06:23.696 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-08-18T16:06:23.696 DEBUG:tasks.rgw_multi.zone_rados:o=e 2024-08-18T16:06:23.696 DEBUG:tasks.rgw_multi.zone_rados:o=f 2024-08-18T16:06:23.696 DEBUG:tasks.rgw_multi.zone_rados:o=g 2024-08-18T16:06:23.696 DEBUG:tasks.rgw_multi.zone_rados:o=h 2024-08-18T16:06:23.702 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-08-18T16:06:23.752 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-08-18T16:06:23.805 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-08-18T16:06:23.862 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-08-18T16:06:23.912 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-08-18T16:06:23.964 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-08-18T16:06:24.013 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-08-18T16:06:24.064 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-08-18T16:06:24.121 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-08-18T16:06:24.174 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-08-18T16:06:24.224 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-08-18T16:06:24.277 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-08-18T16:06:24.329 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2024-08-18T16:06:24.385 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2024-08-18T16:06:24.441 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2024-08-18T16:06:24.498 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2024-08-18T16:06:24.548 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2024-08-18T16:06:24.606 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2024-08-18T16:06:24.702 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2024-08-18T16:06:24.753 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2024-08-18T16:06:24.809 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2024-08-18T16:06:24.866 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2024-08-18T16:06:24.916 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2024-08-18T16:06:24.970 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2024-08-18T16:06:25.070 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-74 zones={test-zone3, test-zone2} 2024-08-18T16:06:25.070 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'kibgmb-74', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:06:25.070 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'kibgmb-74', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:06:25.070 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket kibgmb-74 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:06:25.125 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:06:25.125 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:06:25.166 INFO:teuthology.orchestra.run.smithi134.stdout:tenant: 2024-08-18T16:06:25.166 INFO:teuthology.orchestra.run.smithi134.stdout:bucket name: kibgmb-74 2024-08-18T16:06:25.167 INFO:teuthology.orchestra.run.smithi134.stdout:total entries: 8 2024-08-18T16:06:25.197 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'kibgmb-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:06:25.197 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'kibgmb-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:06:25.197 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket kibgmb-74 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:06:25.242 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:06:25.242 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:06:25.284 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T16:06:25.285 INFO:teuthology.orchestra.run.smithi134.stdout: "layout": { 2024-08-18T16:06:25.285 INFO:teuthology.orchestra.run.smithi134.stdout: "resharding": "None", 2024-08-18T16:06:25.285 INFO:teuthology.orchestra.run.smithi134.stdout: "current_index": { 2024-08-18T16:06:25.285 INFO:teuthology.orchestra.run.smithi134.stdout: "gen": 2, 2024-08-18T16:06:25.285 INFO:teuthology.orchestra.run.smithi134.stdout: "layout": { 2024-08-18T16:06:25.285 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "Normal", 2024-08-18T16:06:25.285 INFO:teuthology.orchestra.run.smithi134.stdout: "normal": { 2024-08-18T16:06:25.285 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 3, 2024-08-18T16:06:25.285 INFO:teuthology.orchestra.run.smithi134.stdout: "hash_type": "Mod" 2024-08-18T16:06:25.285 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:06:25.285 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:06:25.285 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:06:25.285 INFO:teuthology.orchestra.run.smithi134.stdout: "logs": [ 2024-08-18T16:06:25.286 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:06:25.286 INFO:teuthology.orchestra.run.smithi134.stdout: "gen": 0, 2024-08-18T16:06:25.286 INFO:teuthology.orchestra.run.smithi134.stdout: "layout": { 2024-08-18T16:06:25.286 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "InIndex", 2024-08-18T16:06:25.286 INFO:teuthology.orchestra.run.smithi134.stdout: "in_index": { 2024-08-18T16:06:25.286 INFO:teuthology.orchestra.run.smithi134.stdout: "gen": 0, 2024-08-18T16:06:25.286 INFO:teuthology.orchestra.run.smithi134.stdout: "layout": { 2024-08-18T16:06:25.286 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 11, 2024-08-18T16:06:25.286 INFO:teuthology.orchestra.run.smithi134.stdout: "hash_type": "Mod" 2024-08-18T16:06:25.286 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:06:25.286 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:06:25.286 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:06:25.287 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:06:25.287 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:06:25.287 INFO:teuthology.orchestra.run.smithi134.stdout: "gen": 1, 2024-08-18T16:06:25.287 INFO:teuthology.orchestra.run.smithi134.stdout: "layout": { 2024-08-18T16:06:25.287 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "InIndex", 2024-08-18T16:06:25.287 INFO:teuthology.orchestra.run.smithi134.stdout: "in_index": { 2024-08-18T16:06:25.287 INFO:teuthology.orchestra.run.smithi134.stdout: "gen": 1, 2024-08-18T16:06:25.287 INFO:teuthology.orchestra.run.smithi134.stdout: "layout": { 2024-08-18T16:06:25.287 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 3, 2024-08-18T16:06:25.287 INFO:teuthology.orchestra.run.smithi134.stdout: "hash_type": "Mod" 2024-08-18T16:06:25.287 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:06:25.287 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:06:25.287 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:06:25.287 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:06:25.288 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:06:25.288 INFO:teuthology.orchestra.run.smithi134.stdout: "gen": 2, 2024-08-18T16:06:25.288 INFO:teuthology.orchestra.run.smithi134.stdout: "layout": { 2024-08-18T16:06:25.288 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "InIndex", 2024-08-18T16:06:25.288 INFO:teuthology.orchestra.run.smithi134.stdout: "in_index": { 2024-08-18T16:06:25.288 INFO:teuthology.orchestra.run.smithi134.stdout: "gen": 2, 2024-08-18T16:06:25.288 INFO:teuthology.orchestra.run.smithi134.stdout: "layout": { 2024-08-18T16:06:25.288 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 3, 2024-08-18T16:06:25.288 INFO:teuthology.orchestra.run.smithi134.stdout: "hash_type": "Mod" 2024-08-18T16:06:25.288 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:06:25.288 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:06:25.288 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:06:25.288 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:06:25.289 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T16:06:25.289 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:06:25.289 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T16:06:25.290 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-74', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:06:25.290 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-74', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:06:25.290 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-74 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T16:06:25.353 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:06:25.353 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:06:25.393 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:06:25.392+0000 7f719f917880 1 waiting to reach latest gen 2, on 1.. 2024-08-18T16:06:30.394 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:06:30.393+0000 7f719f917880 1 waiting to reach latest gen 2, on 1.. 2024-08-18T16:06:35.395 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:06:35.394+0000 7f719f917880 1 waiting to reach latest gen 2, on 1.. 2024-08-18T16:06:40.397 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:06:40.395+0000 7f719f917880 1 bucket sync caught up with empty source 2024-08-18T16:06:40.397 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:06:40.395+0000 7f719f917880 0 bucket checkpoint complete 2024-08-18T16:06:40.414 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-74', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:06:40.414 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-74', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:06:40.414 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-74 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:06:40.475 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:06:40.475 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:06:40.512 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:06:40.511+0000 7fbd2d22f880 1 bucket sync caught up with empty source 2024-08-18T16:06:40.512 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:06:40.511+0000 7fbd2d22f880 0 bucket checkpoint complete 2024-08-18T16:06:40.526 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-74', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:06:40.526 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-74', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:06:40.526 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-74 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:06:40.582 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:06:40.582 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:06:40.617 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:06:40.617+0000 7fe9c63af880 1 bucket sync caught up with source: 2024-08-18T16:06:40.618 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000002.283.6, , , 00000000001.247.6, 00000000001.341.6, , , 00000000002.441.6, , 00000000002.129.6] 2024-08-18T16:06:40.618 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000002.283.6, , , 00000000001.247.6, 00000000001.341.6, , , 00000000002.441.6, , 00000000002.129.6] 2024-08-18T16:06:40.618 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:06:40.617+0000 7fe9c63af880 0 bucket checkpoint complete 2024-08-18T16:06:40.631 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-74', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:06:40.632 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-74', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:06:40.632 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-74 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:06:40.685 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:06:40.685 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:06:40.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:06:40.719+0000 7f641f60f880 1 bucket sync caught up with source: 2024-08-18T16:06:40.720 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000002.283.6, , , 00000000001.247.6, 00000000001.341.6, , , 00000000002.441.6, , 00000000002.129.6] 2024-08-18T16:06:40.720 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000002.283.6, , , 00000000001.247.6, 00000000001.341.6, , , 00000000002.441.6, , 00000000002.129.6] 2024-08-18T16:06:40.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:06:40.719+0000 7f641f60f880 0 bucket checkpoint complete 2024-08-18T16:06:40.734 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-74', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:06:40.734 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-74', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:06:40.734 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-74 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:06:40.788 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:06:40.788 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:06:40.823 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:06:40.823+0000 7f1986531880 1 bucket sync caught up with source: 2024-08-18T16:06:40.823 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000002.186.6, , , 00000000001.235.6, 00000000001.288.6, , , 00000000002.216.6, , 00000000002.157.6] 2024-08-18T16:06:40.823 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000002.186.6, , , 00000000001.235.6, 00000000001.288.6, , , 00000000002.216.6, , 00000000002.157.6] 2024-08-18T16:06:40.823 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:06:40.823+0000 7f1986531880 0 bucket checkpoint complete 2024-08-18T16:06:40.837 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-74', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:06:40.837 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-74', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:06:40.838 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-74 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T16:06:40.899 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:06:40.899 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:06:40.938 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:06:40.938+0000 7fbcdc320880 1 bucket sync caught up with source: 2024-08-18T16:06:40.938 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, 00000000002.186.6, , , 00000000001.235.6, 00000000001.288.6, , , 00000000002.216.6, , 00000000002.157.6] 2024-08-18T16:06:40.939 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, 00000000002.186.6, , , 00000000001.235.6, 00000000001.288.6, , , 00000000002.216.6, , 00000000002.157.6] 2024-08-18T16:06:40.939 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:06:40.938+0000 7fbcdc320880 0 bucket checkpoint complete 2024-08-18T16:06:40.953 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-74 zones={test-zone2, test-zone1} 2024-08-18T16:06:40.959 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:06:40.962 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-08-18T16:06:40.962 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-08-18T16:06:40.962 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-08-18T16:06:40.962 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-08-18T16:06:40.962 DEBUG:tasks.rgw_multi.zone_rados:o=e 2024-08-18T16:06:40.962 DEBUG:tasks.rgw_multi.zone_rados:o=f 2024-08-18T16:06:40.962 DEBUG:tasks.rgw_multi.zone_rados:o=g 2024-08-18T16:06:40.962 DEBUG:tasks.rgw_multi.zone_rados:o=h 2024-08-18T16:06:40.962 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:06:40.964 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-08-18T16:06:40.965 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-08-18T16:06:40.965 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-08-18T16:06:40.965 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-08-18T16:06:40.965 DEBUG:tasks.rgw_multi.zone_rados:o=e 2024-08-18T16:06:40.965 DEBUG:tasks.rgw_multi.zone_rados:o=f 2024-08-18T16:06:40.965 DEBUG:tasks.rgw_multi.zone_rados:o=g 2024-08-18T16:06:40.965 DEBUG:tasks.rgw_multi.zone_rados:o=h 2024-08-18T16:06:40.970 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-08-18T16:06:40.975 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-08-18T16:06:41.028 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-08-18T16:06:41.079 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-08-18T16:06:41.085 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-08-18T16:06:41.144 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-08-18T16:06:41.151 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-08-18T16:06:41.202 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-08-18T16:06:41.213 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-08-18T16:06:41.264 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-08-18T16:06:41.270 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-08-18T16:06:41.280 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-08-18T16:06:41.328 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2024-08-18T16:06:41.334 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2024-08-18T16:06:41.386 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2024-08-18T16:06:41.393 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2024-08-18T16:06:41.399 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2024-08-18T16:06:41.456 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2024-08-18T16:06:41.507 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2024-08-18T16:06:41.514 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2024-08-18T16:06:41.524 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2024-08-18T16:06:41.572 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2024-08-18T16:06:41.578 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2024-08-18T16:06:41.634 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2024-08-18T16:06:41.642 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-74 zones={test-zone2, test-zone1} 2024-08-18T16:06:41.642 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-74 zones={test-zone3, test-zone1} 2024-08-18T16:06:41.646 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:06:41.649 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-08-18T16:06:41.649 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-08-18T16:06:41.649 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-08-18T16:06:41.649 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-08-18T16:06:41.649 DEBUG:tasks.rgw_multi.zone_rados:o=e 2024-08-18T16:06:41.649 DEBUG:tasks.rgw_multi.zone_rados:o=f 2024-08-18T16:06:41.649 DEBUG:tasks.rgw_multi.zone_rados:o=g 2024-08-18T16:06:41.650 DEBUG:tasks.rgw_multi.zone_rados:o=h 2024-08-18T16:06:41.650 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:06:41.652 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-08-18T16:06:41.652 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-08-18T16:06:41.652 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-08-18T16:06:41.652 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-08-18T16:06:41.652 DEBUG:tasks.rgw_multi.zone_rados:o=e 2024-08-18T16:06:41.652 DEBUG:tasks.rgw_multi.zone_rados:o=f 2024-08-18T16:06:41.652 DEBUG:tasks.rgw_multi.zone_rados:o=g 2024-08-18T16:06:41.652 DEBUG:tasks.rgw_multi.zone_rados:o=h 2024-08-18T16:06:41.657 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-08-18T16:06:41.706 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-08-18T16:06:41.715 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-08-18T16:06:41.804 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-08-18T16:06:41.856 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-08-18T16:06:41.912 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-08-18T16:06:42.005 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-08-18T16:06:42.055 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-08-18T16:06:42.148 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-08-18T16:06:42.200 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-08-18T16:06:42.248 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-08-18T16:06:42.299 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-08-18T16:06:42.348 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2024-08-18T16:06:42.395 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2024-08-18T16:06:42.447 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2024-08-18T16:06:42.541 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2024-08-18T16:06:42.591 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2024-08-18T16:06:42.688 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2024-08-18T16:06:42.785 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2024-08-18T16:06:42.836 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2024-08-18T16:06:42.893 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2024-08-18T16:06:42.945 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2024-08-18T16:06:42.996 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2024-08-18T16:06:43.053 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2024-08-18T16:06:43.106 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-74 zones={test-zone3, test-zone1} 2024-08-18T16:06:43.106 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-74 zones={test-zone3, test-zone2} 2024-08-18T16:06:43.110 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:06:43.113 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-08-18T16:06:43.113 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-08-18T16:06:43.113 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-08-18T16:06:43.113 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-08-18T16:06:43.113 DEBUG:tasks.rgw_multi.zone_rados:o=e 2024-08-18T16:06:43.113 DEBUG:tasks.rgw_multi.zone_rados:o=f 2024-08-18T16:06:43.114 DEBUG:tasks.rgw_multi.zone_rados:o=g 2024-08-18T16:06:43.114 DEBUG:tasks.rgw_multi.zone_rados:o=h 2024-08-18T16:06:43.114 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:06:43.116 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-08-18T16:06:43.117 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-08-18T16:06:43.117 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-08-18T16:06:43.117 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-08-18T16:06:43.117 DEBUG:tasks.rgw_multi.zone_rados:o=e 2024-08-18T16:06:43.117 DEBUG:tasks.rgw_multi.zone_rados:o=f 2024-08-18T16:06:43.117 DEBUG:tasks.rgw_multi.zone_rados:o=g 2024-08-18T16:06:43.117 DEBUG:tasks.rgw_multi.zone_rados:o=h 2024-08-18T16:06:43.123 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-08-18T16:06:43.172 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-08-18T16:06:43.269 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-08-18T16:06:43.369 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-08-18T16:06:43.421 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-08-18T16:06:43.477 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-08-18T16:06:43.529 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-08-18T16:06:43.580 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-08-18T16:06:43.634 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-08-18T16:06:43.733 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-08-18T16:06:43.784 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-08-18T16:06:43.881 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-08-18T16:06:43.977 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2024-08-18T16:06:44.029 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2024-08-18T16:06:44.086 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2024-08-18T16:06:44.182 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2024-08-18T16:06:44.233 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2024-08-18T16:06:44.289 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2024-08-18T16:06:44.341 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2024-08-18T16:06:44.393 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2024-08-18T16:06:44.450 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2024-08-18T16:06:44.546 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2024-08-18T16:06:44.596 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2024-08-18T16:06:44.654 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2024-08-18T16:06:44.754 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-74 zones={test-zone3, test-zone2} 2024-08-18T16:06:44.754 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:06:44.754 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:06:44.754 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:06:44.804 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:06:44.804 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:06:44.859 INFO:tasks.rgw.c2.client.0.smithi193.stdout:2024-08-18T16:06:44.859+0000 7f4346ca8640 -1 req 3245371967431996054 0.000999984s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=kibgmb-37:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.37,z=96bf8502-2f39-4d53-8196-3378c28da3b4,az=0},d={b=kibgmb-37:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.37,z=10090ccf-aad6-416d-9775-df8b7b27345a,az=0}} returned ret=-2 2024-08-18T16:06:44.860 INFO:tasks.rgw.c1.client.1.smithi134.stdout:2024-08-18T16:06:44.859+0000 7fc042906640 -1 req 5097548851891539680 0.000999977s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=kibgmb-37:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.37,z=96bf8502-2f39-4d53-8196-3378c28da3b4,az=0},d={b=kibgmb-37:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.37,z=018ea73b-7528-48da-aafa-0118123784f7,az=0}} returned ret=-2 2024-08-18T16:06:44.860 INFO:tasks.rgw.c2.client.0.smithi193.stdout:2024-08-18T16:06:44.860+0000 7f43434a1640 -1 req 17213875584906924692 0.000999984s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=kibgmb-5:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.5,z=96bf8502-2f39-4d53-8196-3378c28da3b4,az=0},d={b=kibgmb-5:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.5,z=10090ccf-aad6-416d-9775-df8b7b27345a,az=0}} returned ret=-2 2024-08-18T16:06:44.860 INFO:tasks.rgw.c1.client.1.smithi134.stdout:2024-08-18T16:06:44.860+0000 7fbfcd81c640 -1 req 7774645529945085912 0.000999977s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=kibgmb-5:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.5,z=96bf8502-2f39-4d53-8196-3378c28da3b4,az=0},d={b=kibgmb-5:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.5,z=018ea73b-7528-48da-aafa-0118123784f7,az=0}} returned ret=-2 2024-08-18T16:06:44.903 INFO:tasks.rgw.c1.client.1.smithi134.stdout:2024-08-18T16:06:44.902+0000 7fbfb0947640 -1 req 2665732339733130563 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=kibgmb-9:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.9,z=96bf8502-2f39-4d53-8196-3378c28da3b4,az=0},d={b=kibgmb-9:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.9,z=018ea73b-7528-48da-aafa-0118123784f7,az=0}} returned ret=-2 2024-08-18T16:06:44.903 INFO:tasks.rgw.c1.client.1.smithi134.stdout:2024-08-18T16:06:44.902+0000 7fc070c68640 -1 req 574561605177390123 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=kibgmb-4:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.4,z=96bf8502-2f39-4d53-8196-3378c28da3b4,az=0},d={b=kibgmb-4:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.4,z=018ea73b-7528-48da-aafa-0118123784f7,az=0}} returned ret=-2 2024-08-18T16:06:44.903 INFO:tasks.rgw.c2.client.0.smithi193.stdout:2024-08-18T16:06:44.903+0000 7f43ad575640 -1 req 9372724656746446143 0.000999984s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=kibgmb-9:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.9,z=96bf8502-2f39-4d53-8196-3378c28da3b4,az=0},d={b=kibgmb-9:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.9,z=10090ccf-aad6-416d-9775-df8b7b27345a,az=0}} returned ret=-2 2024-08-18T16:06:44.903 INFO:tasks.rgw.c2.client.0.smithi193.stdout:2024-08-18T16:06:44.903+0000 7f437ad10640 -1 req 16518998003377371079 0.000999984s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=kibgmb-4:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.4,z=96bf8502-2f39-4d53-8196-3378c28da3b4,az=0},d={b=kibgmb-4:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.4,z=10090ccf-aad6-416d-9775-df8b7b27345a,az=0}} returned ret=-2 2024-08-18T16:06:45.018 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'kibgmb-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:06:45.018 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'kibgmb-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:06:45.018 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket kibgmb-74 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:06:45.063 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:06:45.063 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:06:45.106 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T16:06:45.106 INFO:teuthology.orchestra.run.smithi134.stdout: "layout": { 2024-08-18T16:06:45.107 INFO:teuthology.orchestra.run.smithi134.stdout: "resharding": "None", 2024-08-18T16:06:45.107 INFO:teuthology.orchestra.run.smithi134.stdout: "current_index": { 2024-08-18T16:06:45.107 INFO:teuthology.orchestra.run.smithi134.stdout: "gen": 2, 2024-08-18T16:06:45.107 INFO:teuthology.orchestra.run.smithi134.stdout: "layout": { 2024-08-18T16:06:45.107 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "Normal", 2024-08-18T16:06:45.107 INFO:teuthology.orchestra.run.smithi134.stdout: "normal": { 2024-08-18T16:06:45.107 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 3, 2024-08-18T16:06:45.107 INFO:teuthology.orchestra.run.smithi134.stdout: "hash_type": "Mod" 2024-08-18T16:06:45.107 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:06:45.107 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:06:45.107 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:06:45.107 INFO:teuthology.orchestra.run.smithi134.stdout: "logs": [ 2024-08-18T16:06:45.107 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:06:45.108 INFO:teuthology.orchestra.run.smithi134.stdout: "gen": 1, 2024-08-18T16:06:45.108 INFO:teuthology.orchestra.run.smithi134.stdout: "layout": { 2024-08-18T16:06:45.108 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "InIndex", 2024-08-18T16:06:45.108 INFO:teuthology.orchestra.run.smithi134.stdout: "in_index": { 2024-08-18T16:06:45.108 INFO:teuthology.orchestra.run.smithi134.stdout: "gen": 1, 2024-08-18T16:06:45.108 INFO:teuthology.orchestra.run.smithi134.stdout: "layout": { 2024-08-18T16:06:45.108 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 3, 2024-08-18T16:06:45.108 INFO:teuthology.orchestra.run.smithi134.stdout: "hash_type": "Mod" 2024-08-18T16:06:45.108 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:06:45.108 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:06:45.108 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:06:45.108 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:06:45.108 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:06:45.108 INFO:teuthology.orchestra.run.smithi134.stdout: "gen": 2, 2024-08-18T16:06:45.109 INFO:teuthology.orchestra.run.smithi134.stdout: "layout": { 2024-08-18T16:06:45.109 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "InIndex", 2024-08-18T16:06:45.109 INFO:teuthology.orchestra.run.smithi134.stdout: "in_index": { 2024-08-18T16:06:45.109 INFO:teuthology.orchestra.run.smithi134.stdout: "gen": 2, 2024-08-18T16:06:45.109 INFO:teuthology.orchestra.run.smithi134.stdout: "layout": { 2024-08-18T16:06:45.109 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 3, 2024-08-18T16:06:45.109 INFO:teuthology.orchestra.run.smithi134.stdout: "hash_type": "Mod" 2024-08-18T16:06:45.109 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:06:45.109 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:06:45.109 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:06:45.109 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:06:45.109 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T16:06:45.109 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:06:45.109 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T16:06:45.110 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:06:45.110 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:06:45.111 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:06:45.155 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:06:45.155 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:06:45.194 INFO:tasks.rgw.c1.client.1.smithi134.stdout:2024-08-18T16:06:45.194+0000 7fc0370ef640 -1 req 325629788722908 0.000999977s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=kibgmb-36:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.36,z=96bf8502-2f39-4d53-8196-3378c28da3b4,az=0},d={b=kibgmb-36:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.36,z=018ea73b-7528-48da-aafa-0118123784f7,az=0}} returned ret=-2 2024-08-18T16:06:45.194 INFO:tasks.rgw.c1.client.1.smithi134.stdout:2024-08-18T16:06:45.194+0000 7fbfae943640 -1 req 11371742691530064084 0.000999977s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=kibgmb-38:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.38,z=96bf8502-2f39-4d53-8196-3378c28da3b4,az=0},d={b=kibgmb-38:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.38,z=018ea73b-7528-48da-aafa-0118123784f7,az=0}} returned ret=-2 2024-08-18T16:06:45.195 INFO:tasks.rgw.c1.client.1.smithi134.stdout:2024-08-18T16:06:45.194+0000 7fbfff880640 -1 req 7255765199139177590 0.000999977s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=kibgmb-43:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.43,z=96bf8502-2f39-4d53-8196-3378c28da3b4,az=0},d={b=kibgmb-43:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.43,z=018ea73b-7528-48da-aafa-0118123784f7,az=0}} returned ret=-2 2024-08-18T16:06:45.195 INFO:tasks.rgw.c2.client.0.smithi193.stdout:2024-08-18T16:06:45.194+0000 7f43dedd8640 -1 req 17326545927065842076 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=kibgmb-43:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.43,z=96bf8502-2f39-4d53-8196-3378c28da3b4,az=0},d={b=kibgmb-43:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.43,z=10090ccf-aad6-416d-9775-df8b7b27345a,az=0}} returned ret=-2 2024-08-18T16:06:45.195 INFO:tasks.rgw.c2.client.0.smithi193.stdout:2024-08-18T16:06:45.194+0000 7f431ac50640 -1 req 4410094999038405970 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=kibgmb-36:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.36,z=96bf8502-2f39-4d53-8196-3378c28da3b4,az=0},d={b=kibgmb-36:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.36,z=10090ccf-aad6-416d-9775-df8b7b27345a,az=0}} returned ret=-2 2024-08-18T16:06:45.195 INFO:tasks.rgw.c2.client.0.smithi193.stdout:2024-08-18T16:06:45.195+0000 7f4306c28640 -1 req 1550119675644212875 0.000999984s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=kibgmb-38:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.38,z=96bf8502-2f39-4d53-8196-3378c28da3b4,az=0},d={b=kibgmb-38:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.38,z=10090ccf-aad6-416d-9775-df8b7b27345a,az=0}} returned ret=-2 2024-08-18T16:06:45.247 INFO:tasks.rgw.c1.client.1.smithi134.stdout:2024-08-18T16:06:45.247+0000 7fc088497640 -1 req 9980883958053082470 0.000999976s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=kibgmb-3:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.3,z=96bf8502-2f39-4d53-8196-3378c28da3b4,az=0},d={b=kibgmb-3:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.3,z=018ea73b-7528-48da-aafa-0118123784f7,az=0}} returned ret=-2 2024-08-18T16:06:45.247 INFO:tasks.rgw.c2.client.0.smithi193.stdout:2024-08-18T16:06:45.247+0000 7f437dd16640 -1 req 15497921253458432524 0.000999984s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=kibgmb-3:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.3,z=96bf8502-2f39-4d53-8196-3378c28da3b4,az=0},d={b=kibgmb-3:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.3,z=10090ccf-aad6-416d-9775-df8b7b27345a,az=0}} returned ret=-2 2024-08-18T16:06:45.248 INFO:tasks.rgw.c1.client.1.smithi134.stdout:2024-08-18T16:06:45.247+0000 7fc052a28640 -1 req 3654560734630024081 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=kibgmb-35:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.35,z=96bf8502-2f39-4d53-8196-3378c28da3b4,az=0},d={b=kibgmb-35:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.35,z=018ea73b-7528-48da-aafa-0118123784f7,az=0}} returned ret=-2 2024-08-18T16:06:45.248 INFO:tasks.rgw.c2.client.0.smithi193.stdout:2024-08-18T16:06:45.248+0000 7f430fc3a640 -1 req 2864924092158420105 0.000999984s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=kibgmb-35:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.35,z=96bf8502-2f39-4d53-8196-3378c28da3b4,az=0},d={b=kibgmb-35:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.35,z=10090ccf-aad6-416d-9775-df8b7b27345a,az=0}} returned ret=-2 2024-08-18T16:06:45.248 INFO:tasks.rgw.c1.client.1.smithi134.stdout:2024-08-18T16:06:45.248+0000 7fbfcc019640 -1 req 7868413244547951567 0.000999977s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=kibgmb-31:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.31,z=96bf8502-2f39-4d53-8196-3378c28da3b4,az=0},d={b=kibgmb-31:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.31,z=018ea73b-7528-48da-aafa-0118123784f7,az=0}} returned ret=-2 2024-08-18T16:06:45.249 INFO:tasks.rgw.c2.client.0.smithi193.stdout:2024-08-18T16:06:45.249+0000 7f43e35e1640 -1 req 16434811271851117302 0.000999984s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=kibgmb-31:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.31,z=96bf8502-2f39-4d53-8196-3378c28da3b4,az=0},d={b=kibgmb-31:96bf8502-2f39-4d53-8196-3378c28da3b4.4364.31,z=10090ccf-aad6-416d-9775-df8b7b27345a,az=0}} returned ret=-2 2024-08-18T16:06:45.294 INFO:tasks.rgw.c1.client.1.smithi134.stdout:2024-08-18T16:06:45.294+0000 7fbfe684e640 -1 req 1535117918236274146 0.000999977s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=kibgmb-55:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.3,z=96bf8502-2f39-4d53-8196-3378c28da3b4,az=0},d={b=kibgmb-55:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.3,z=018ea73b-7528-48da-aafa-0118123784f7,az=0}} returned ret=-2 2024-08-18T16:06:45.295 INFO:tasks.rgw.c2.client.0.smithi193.stdout:2024-08-18T16:06:45.294+0000 7f4386d28640 -1 req 10435836397997439511 0.000999984s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=kibgmb-55:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.3,z=96bf8502-2f39-4d53-8196-3378c28da3b4,az=0},d={b=kibgmb-55:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.3,z=10090ccf-aad6-416d-9775-df8b7b27345a,az=0}} returned ret=-2 2024-08-18T16:06:45.356 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'kibgmb-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:06:45.357 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'kibgmb-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:06:45.357 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket kibgmb-74 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:06:45.401 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:06:45.402 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:06:45.444 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T16:06:45.445 INFO:teuthology.orchestra.run.smithi134.stdout: "layout": { 2024-08-18T16:06:45.445 INFO:teuthology.orchestra.run.smithi134.stdout: "resharding": "None", 2024-08-18T16:06:45.445 INFO:teuthology.orchestra.run.smithi134.stdout: "current_index": { 2024-08-18T16:06:45.445 INFO:teuthology.orchestra.run.smithi134.stdout: "gen": 2, 2024-08-18T16:06:45.445 INFO:teuthology.orchestra.run.smithi134.stdout: "layout": { 2024-08-18T16:06:45.445 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "Normal", 2024-08-18T16:06:45.445 INFO:teuthology.orchestra.run.smithi134.stdout: "normal": { 2024-08-18T16:06:45.445 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 3, 2024-08-18T16:06:45.445 INFO:teuthology.orchestra.run.smithi134.stdout: "hash_type": "Mod" 2024-08-18T16:06:45.445 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:06:45.445 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:06:45.445 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:06:45.445 INFO:teuthology.orchestra.run.smithi134.stdout: "logs": [ 2024-08-18T16:06:45.446 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:06:45.446 INFO:teuthology.orchestra.run.smithi134.stdout: "gen": 1, 2024-08-18T16:06:45.446 INFO:teuthology.orchestra.run.smithi134.stdout: "layout": { 2024-08-18T16:06:45.446 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "InIndex", 2024-08-18T16:06:45.446 INFO:teuthology.orchestra.run.smithi134.stdout: "in_index": { 2024-08-18T16:06:45.446 INFO:teuthology.orchestra.run.smithi134.stdout: "gen": 1, 2024-08-18T16:06:45.446 INFO:teuthology.orchestra.run.smithi134.stdout: "layout": { 2024-08-18T16:06:45.446 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 3, 2024-08-18T16:06:45.446 INFO:teuthology.orchestra.run.smithi134.stdout: "hash_type": "Mod" 2024-08-18T16:06:45.446 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:06:45.446 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:06:45.446 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:06:45.446 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:06:45.447 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:06:45.447 INFO:teuthology.orchestra.run.smithi134.stdout: "gen": 2, 2024-08-18T16:06:45.447 INFO:teuthology.orchestra.run.smithi134.stdout: "layout": { 2024-08-18T16:06:45.447 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "InIndex", 2024-08-18T16:06:45.447 INFO:teuthology.orchestra.run.smithi134.stdout: "in_index": { 2024-08-18T16:06:45.447 INFO:teuthology.orchestra.run.smithi134.stdout: "gen": 2, 2024-08-18T16:06:45.447 INFO:teuthology.orchestra.run.smithi134.stdout: "layout": { 2024-08-18T16:06:45.447 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 3, 2024-08-18T16:06:45.447 INFO:teuthology.orchestra.run.smithi134.stdout: "hash_type": "Mod" 2024-08-18T16:06:45.447 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:06:45.447 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:06:45.447 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:06:45.447 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:06:45.447 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T16:06:45.447 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:06:45.448 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T16:06:45.449 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_reshard_index_log_trim ... FAIL 2024-08-18T16:06:45.465 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:06:45.465 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:06:45.465 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:06:45.511 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:06:45.512 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:06:45.542 INFO:teuthology.orchestra.run.smithi134.stderr:No --period given, using current period=01050e30-179b-43be-bdbb-3bb23cf37ae8 2024-08-18T16:06:45.555 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T16:06:45.555 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:06:45.555 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T16:06:45.555 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T16:06:45.556 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:06:45.556 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:06:45.556 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T16:06:45.556 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T16:06:45.556 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:06:45.556 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:06:45.556 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T16:06:45.556 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T16:06:45.556 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:06:45.556 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:06:45.556 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723997205.464162_722.1", 2024-08-18T16:06:45.556 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:06:45.464162Z" 2024-08-18T16:06:45.556 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:06:45.557 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T16:06:45.557 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-08-18T16:06:45.558 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:06:45.558 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:06:45.558 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:06:45.614 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:06:45.614 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:06:45.650 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:06:45.649+0000 7fc24f20e880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:06:45.650 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:06:45.650+0000 7fc24f20e880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:06:45.665 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T16:06:45.665 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T16:06:45.665 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T16:06:45.665 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T16:06:45.665 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T16:06:45.665 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T16:06:45.665 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T16:06:45.665 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:06:45.665 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T16:06:45.666 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:06:45.666 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T16:06:45.666 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:06:45.666 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:06:45.666 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T16:06:45.666 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:06:45.666 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T16:06:45.666 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:06:45.666 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T16:06:45.666 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T16:06:45.666 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:06:45.666 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:06:45.666 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:06:45.666 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T16:06:45.667 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:06:45.667 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:06:45.667 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T16:06:45.667 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:06:45.667 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:06:45.667 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:06:45.667 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T16:06:45.667 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T16:06:45.667 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:06:45.667 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:06:45.667 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:06:45.667 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T16:06:45.667 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:06:45.667 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:06:45.668 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T16:06:45.668 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:06:45.668 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:06:45.668 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:06:45.668 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T16:06:45.668 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T16:06:45.668 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:06:45.668 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:06:45.668 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:06:45.668 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T16:06:45.668 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:06:45.668 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:06:45.668 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723997204.859220_711.1", 2024-08-18T16:06:45.669 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:06:45.669 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:06:45.669 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:06:45.669 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:06:44.859220Z", 2024-08-18T16:06:45.669 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T16:06:45.669 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:06:45.669 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:06:45.669 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T16:06:45.669 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:06:45.669 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T16:06:45.669 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T16:06:45.669 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T16:06:45.669 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:06:45.670 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T16:06:45Z" 2024-08-18T16:06:45.670 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T16:06:45.671 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1723997205.464162_722.1 target=1_1723997204.859220_711.1 2024-08-18T16:06:50.672 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:06:50.672 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:06:50.672 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:06:50.727 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:06:50.727 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:06:50.761 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:06:50.760+0000 7f010bb19880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:06:50.762 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:06:50.762+0000 7f010bb19880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:06:50.776 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T16:06:50.776 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T16:06:50.776 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T16:06:50.777 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T16:06:50.777 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T16:06:50.777 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T16:06:50.777 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T16:06:50.777 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:06:50.777 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T16:06:50.777 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:06:50.777 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T16:06:50.777 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:06:50.777 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:06:50.777 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T16:06:50.777 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:06:50.777 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T16:06:50.778 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:06:50.778 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T16:06:50.778 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T16:06:50.778 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:06:50.778 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:06:50.778 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:06:50.778 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T16:06:50.778 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:06:50.778 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:06:50.778 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T16:06:50.778 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:06:50.778 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:06:50.778 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:06:50.779 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T16:06:50.779 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T16:06:50.779 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:06:50.779 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:06:50.779 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:06:50.779 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T16:06:50.779 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:06:50.779 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:06:50.779 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T16:06:50.779 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:06:50.779 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:06:50.779 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:06:50.779 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T16:06:50.780 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T16:06:50.780 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:06:50.780 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:06:50.780 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:06:50.780 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T16:06:50.780 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:06:50.780 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:06:50.780 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723997205.464162_722.1", 2024-08-18T16:06:50.780 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:06:50.780 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:06:50.780 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:06:50.780 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:06:45.464162Z", 2024-08-18T16:06:50.780 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T16:06:50.780 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:06:50.781 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:06:50.781 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T16:06:50.781 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:06:50.781 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T16:06:50.781 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T16:06:50.781 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T16:06:50.781 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:06:50.781 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T16:06:50Z" 2024-08-18T16:06:50.781 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T16:06:50.782 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-08-18T16:06:50.782 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-08-18T16:06:50.782 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:06:50.782 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:06:50.783 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:06:50.837 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:06:50.837 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:06:50.870 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:06:50.869+0000 7fdb01523880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:06:50.870 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:06:50.870+0000 7fdb01523880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:06:50.884 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T16:06:50.884 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T16:06:50.884 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T16:06:50.885 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T16:06:50.885 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T16:06:50.885 INFO:teuthology.orchestra.run.smithi134.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T16:06:50.885 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T16:06:50.885 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:06:50.885 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T16:06:50.885 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:06:50.885 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T16:06:50.885 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:06:50.885 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T16:06:50.885 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T16:06:50.885 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:06:50.885 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 1, 2024-08-18T16:06:50.886 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:06:50.886 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T16:06:50.886 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T16:06:50.886 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:06:50.886 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:06:50.886 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:06:50.886 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T16:06:50.886 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:06:50.886 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T16:06:50.886 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T16:06:50.886 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:06:50.886 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:06:50.886 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:06:50.886 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T16:06:50.886 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T16:06:50.887 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:06:50.887 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:06:50.887 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:06:50.887 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T16:06:50.887 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:06:50.887 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T16:06:50.887 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T16:06:50.887 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:06:50.887 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:06:50.887 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:06:50.887 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T16:06:50.887 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T16:06:50.887 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:06:50.887 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:06:50.887 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:06:50.888 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T16:06:50.888 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:06:50.888 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T16:06:50.888 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723997205.464162_722.1", 2024-08-18T16:06:50.888 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:06:50.888 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:06:50.888 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:06:50.888 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:06:45.464162Z", 2024-08-18T16:06:50.888 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T16:06:50.888 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:06:50.888 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:06:50.888 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T16:06:50.888 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:06:50.888 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T16:06:50.889 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 1, 2024-08-18T16:06:50.889 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 1 2024-08-18T16:06:50.889 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:06:50.889 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T16:06:50Z" 2024-08-18T16:06:50.889 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T16:06:50.890 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-08-18T16:06:50.958 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-75', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:06:50.959 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-75', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:06:50.959 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-75 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T16:06:51.014 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:06:51.014 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:06:51.049 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:06:51.048+0000 7f8e7580f880 1 waiting to reach incremental sync.. 2024-08-18T16:06:56.049 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:06:56.049+0000 7f8e7580f880 1 waiting to reach incremental sync.. 2024-08-18T16:07:01.054 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:07:01.050+0000 7f8e7580f880 1 bucket sync caught up with source: 2024-08-18T16:07:01.054 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, 00000000001.74.6, , , , 00000000001.78.6, , , 00000000001.126.6, , 00000000001.69.6] 2024-08-18T16:07:01.054 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, 00000000001.74.6, , , , 00000000001.78.6, , , 00000000001.126.6, , 00000000001.69.6] 2024-08-18T16:07:01.054 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:07:01.050+0000 7f8e7580f880 0 bucket checkpoint complete 2024-08-18T16:07:01.069 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-75', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:07:01.070 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-75', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:07:01.070 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-75 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:07:01.133 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:07:01.134 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:07:01.174 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:07:01.174+0000 7f32a72bb880 1 bucket sync caught up with source: 2024-08-18T16:07:01.175 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000001.74.6, , , , 00000000001.78.6, , , 00000000001.126.6, , 00000000001.69.6] 2024-08-18T16:07:01.175 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000001.74.6, , , , 00000000001.78.6, , , 00000000001.126.6, , 00000000001.69.6] 2024-08-18T16:07:01.175 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:07:01.174+0000 7f32a72bb880 0 bucket checkpoint complete 2024-08-18T16:07:01.188 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-75', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:07:01.188 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-75', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:07:01.189 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-75 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:07:01.243 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:07:01.244 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:07:01.279 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:07:01.279+0000 7efe8ef31880 1 bucket sync caught up with source: 2024-08-18T16:07:01.279 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000001.248.6, , , , 00000000001.240.6, , , 00000000001.465.6, , 00000000001.265.6] 2024-08-18T16:07:01.280 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000001.248.6, , , , 00000000001.240.6, , , 00000000001.465.6, , 00000000001.265.6] 2024-08-18T16:07:01.280 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:07:01.279+0000 7efe8ef31880 0 bucket checkpoint complete 2024-08-18T16:07:01.293 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-75', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:07:01.294 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-75', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:07:01.294 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-75 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:07:01.347 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:07:01.347 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:07:01.383 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:07:01.382+0000 7f403b6bb880 1 waiting for incremental sync to catch up: 2024-08-18T16:07:01.383 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000001.248.6, , , , , , , 00000000001.465.6, , 00000000001.265.6] 2024-08-18T16:07:01.383 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000001.248.6, , , , 00000000001.240.6, , , 00000000001.465.6, , 00000000001.265.6] 2024-08-18T16:07:06.384 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:07:06.383+0000 7f403b6bb880 1 waiting for incremental sync to catch up: 2024-08-18T16:07:06.384 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000001.248.6, , , , , , , 00000000001.465.6, , 00000000001.265.6] 2024-08-18T16:07:06.384 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000001.248.6, , , , 00000000001.240.6, , , 00000000001.465.6, , 00000000001.265.6] 2024-08-18T16:07:11.385 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:07:11.385+0000 7f403b6bb880 1 waiting for incremental sync to catch up: 2024-08-18T16:07:11.386 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000001.248.6, , , , , , , 00000000001.465.6, , 00000000001.265.6] 2024-08-18T16:07:11.386 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000001.248.6, , , , 00000000001.240.6, , , 00000000001.465.6, , 00000000001.265.6] 2024-08-18T16:07:16.387 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:07:16.386+0000 7f403b6bb880 1 waiting for incremental sync to catch up: 2024-08-18T16:07:16.387 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000001.248.6, , , , , , , 00000000001.465.6, , 00000000001.265.6] 2024-08-18T16:07:16.387 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000001.248.6, , , , 00000000001.240.6, , , 00000000001.465.6, , 00000000001.265.6] 2024-08-18T16:07:21.388 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:07:21.387+0000 7f403b6bb880 1 bucket sync caught up with source: 2024-08-18T16:07:21.406 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000001.248.6, , , , 00000000001.240.6, , , 00000000001.465.6, , 00000000001.265.6] 2024-08-18T16:07:21.406 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000001.248.6, , , , 00000000001.240.6, , , 00000000001.465.6, , 00000000001.265.6] 2024-08-18T16:07:21.406 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:07:21.387+0000 7f403b6bb880 0 bucket checkpoint complete 2024-08-18T16:07:21.408 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-75', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:07:21.408 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-75', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:07:21.408 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-75 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:07:21.468 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:07:21.468 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:07:21.505 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:07:21.504+0000 7fbe932bb880 1 bucket sync caught up with source: 2024-08-18T16:07:21.505 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000001.109.6, , , , 00000000001.222.6, , , 00000000001.249.6, , 00000000001.177.6] 2024-08-18T16:07:21.505 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000001.109.6, , , , 00000000001.222.6, , , 00000000001.249.6, , 00000000001.177.6] 2024-08-18T16:07:21.505 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:07:21.504+0000 7fbe932bb880 0 bucket checkpoint complete 2024-08-18T16:07:21.520 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-75', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:07:21.520 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-75', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:07:21.520 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-75 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T16:07:21.582 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:07:21.582 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:07:21.622 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:07:21.622+0000 7fed7c719880 1 bucket sync caught up with source: 2024-08-18T16:07:21.623 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, 00000000001.109.6, , , , 00000000001.222.6, , , 00000000001.249.6, , 00000000001.177.6] 2024-08-18T16:07:21.623 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, 00000000001.109.6, , , , 00000000001.222.6, , , 00000000001.249.6, , 00000000001.177.6] 2024-08-18T16:07:21.623 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:07:21.622+0000 7fed7c719880 0 bucket checkpoint complete 2024-08-18T16:07:21.638 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-75 zones={test-zone2, test-zone1} 2024-08-18T16:07:21.644 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:07:21.648 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-08-18T16:07:21.648 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-08-18T16:07:21.648 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-08-18T16:07:21.648 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-08-18T16:07:21.648 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:07:21.650 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-08-18T16:07:21.651 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-08-18T16:07:21.651 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-08-18T16:07:21.651 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-08-18T16:07:21.655 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-08-18T16:07:21.706 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-08-18T16:07:21.721 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-08-18T16:07:21.729 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-08-18T16:07:21.780 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-08-18T16:07:21.834 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-08-18T16:07:21.842 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-08-18T16:07:21.849 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-08-18T16:07:21.860 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-08-18T16:07:21.913 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-08-18T16:07:21.921 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-08-18T16:07:21.981 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-08-18T16:07:22.037 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-75 zones={test-zone2, test-zone1} 2024-08-18T16:07:22.037 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-75 zones={test-zone3, test-zone1} 2024-08-18T16:07:22.041 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:07:22.044 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-08-18T16:07:22.044 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-08-18T16:07:22.044 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-08-18T16:07:22.044 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-08-18T16:07:22.044 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:07:22.047 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-08-18T16:07:22.047 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-08-18T16:07:22.047 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-08-18T16:07:22.047 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-08-18T16:07:22.052 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-08-18T16:07:22.102 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-08-18T16:07:22.113 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-08-18T16:07:22.165 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-08-18T16:07:22.216 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-08-18T16:07:22.268 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-08-18T16:07:22.317 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-08-18T16:07:22.368 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-08-18T16:07:22.420 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-08-18T16:07:22.473 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-08-18T16:07:22.479 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-08-18T16:07:22.534 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-08-18T16:07:22.542 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-75 zones={test-zone3, test-zone1} 2024-08-18T16:07:22.542 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-75 zones={test-zone3, test-zone2} 2024-08-18T16:07:22.546 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:07:22.549 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-08-18T16:07:22.549 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-08-18T16:07:22.549 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-08-18T16:07:22.549 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-08-18T16:07:22.550 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:07:22.552 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-08-18T16:07:22.552 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-08-18T16:07:22.552 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-08-18T16:07:22.552 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-08-18T16:07:22.558 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-08-18T16:07:22.608 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-08-18T16:07:22.705 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-08-18T16:07:22.758 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-08-18T16:07:22.812 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-08-18T16:07:22.918 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-08-18T16:07:23.014 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-08-18T16:07:23.064 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-08-18T16:07:23.158 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-08-18T16:07:23.214 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-08-18T16:07:23.266 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-08-18T16:07:23.278 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-08-18T16:07:23.287 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-75 zones={test-zone3, test-zone2} 2024-08-18T16:07:23.287 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'kibgmb-75', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:07:23.287 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'kibgmb-75', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:07:23.288 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket kibgmb-75 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:07:23.335 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:07:23.335 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:07:23.374 INFO:teuthology.orchestra.run.smithi134.stdout:tenant: 2024-08-18T16:07:23.374 INFO:teuthology.orchestra.run.smithi134.stdout:bucket name: kibgmb-75 2024-08-18T16:07:23.378 INFO:teuthology.orchestra.run.smithi134.stdout:total entries: 4 2024-08-18T16:07:23.417 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'reshard', '--bucket', 'kibgmb-75', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:07:23.417 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'reshard', '--bucket', 'kibgmb-75', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:07:23.417 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket reshard --bucket kibgmb-75 --num-shards 3 --yes-i-really-mean-it --cluster c2 --debug-rgw 0 --debug-ms 0 2024-08-18T16:07:23.472 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:07:23.472 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:07:23.514 INFO:teuthology.orchestra.run.smithi193.stdout:tenant: 2024-08-18T16:07:23.514 INFO:teuthology.orchestra.run.smithi193.stdout:bucket name: kibgmb-75 2024-08-18T16:07:23.518 INFO:teuthology.orchestra.run.smithi193.stdout:total entries: 4 2024-08-18T16:07:23.556 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'kibgmb-75', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:07:23.556 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'kibgmb-75', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:07:23.556 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket kibgmb-75 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:07:23.606 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:07:23.607 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:07:23.644 INFO:teuthology.orchestra.run.smithi134.stdout:tenant: 2024-08-18T16:07:23.645 INFO:teuthology.orchestra.run.smithi134.stdout:bucket name: kibgmb-75 2024-08-18T16:07:23.646 INFO:teuthology.orchestra.run.smithi134.stdout:total entries: 4 2024-08-18T16:07:23.737 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-75', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:07:23.738 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-75', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:07:23.738 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-75 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T16:07:23.793 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:07:23.793 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:07:23.827 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:07:23.826+0000 7fe471862880 1 waiting to reach latest gen 2, on 0.. 2024-08-18T16:07:28.828 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:07:28.827+0000 7fe471862880 1 waiting to reach latest gen 2, on 0.. 2024-08-18T16:07:33.828 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:07:33.828+0000 7fe471862880 1 waiting to reach latest gen 2, on 0.. 2024-08-18T16:07:38.829 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:07:38.828+0000 7fe471862880 1 waiting to reach latest gen 2, on 1.. 2024-08-18T16:07:43.829 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:07:43.829+0000 7fe471862880 1 waiting to reach latest gen 2, on 1.. 2024-08-18T16:07:48.830 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:07:48.829+0000 7fe471862880 1 waiting to reach latest gen 2, on 1.. 2024-08-18T16:07:53.830 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:07:53.829+0000 7fe471862880 1 waiting to reach latest gen 2, on 1.. 2024-08-18T16:07:58.830 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:07:58.829+0000 7fe471862880 1 waiting to reach latest gen 2, on 1.. 2024-08-18T16:08:03.830 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:08:03.829+0000 7fe471862880 1 waiting to reach latest gen 2, on 1.. 2024-08-18T16:08:08.830 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:08:08.830+0000 7fe471862880 1 waiting to reach latest gen 2, on 1.. 2024-08-18T16:08:13.831 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:08:13.830+0000 7fe471862880 1 waiting to reach latest gen 2, on 1.. 2024-08-18T16:08:18.832 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:08:18.831+0000 7fe471862880 1 waiting to reach latest gen 2, on 1.. 2024-08-18T16:08:23.832 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:08:23.832+0000 7fe471862880 1 waiting to reach latest gen 2, on 1.. 2024-08-18T16:08:28.833 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:08:28.832+0000 7fe471862880 1 waiting to reach latest gen 2, on 1.. 2024-08-18T16:08:33.833 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:08:33.833+0000 7fe471862880 1 waiting to reach latest gen 2, on 1.. 2024-08-18T16:08:38.834 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:08:38.834+0000 7fe471862880 1 waiting for incremental sync to catch up: 2024-08-18T16:08:38.834 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , ] 2024-08-18T16:08:38.835 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000003.53.6, 00000000002.127.6, 00000000002.62.6] 2024-08-18T16:08:43.835 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:08:43.835+0000 7fe471862880 1 waiting for incremental sync to catch up: 2024-08-18T16:08:43.835 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , ] 2024-08-18T16:08:43.835 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000003.53.6, 00000000002.127.6, 00000000002.62.6] 2024-08-18T16:08:48.836 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:08:48.835+0000 7fe471862880 1 waiting for incremental sync to catch up: 2024-08-18T16:08:48.836 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , ] 2024-08-18T16:08:48.836 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000003.53.6, 00000000002.127.6, 00000000002.62.6] 2024-08-18T16:08:53.837 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:08:53.836+0000 7fe471862880 1 waiting for incremental sync to catch up: 2024-08-18T16:08:53.837 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , ] 2024-08-18T16:08:53.837 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000003.53.6, 00000000002.127.6, 00000000002.62.6] 2024-08-18T16:08:58.837 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:08:58.837+0000 7fe471862880 1 waiting for incremental sync to catch up: 2024-08-18T16:08:58.838 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , ] 2024-08-18T16:08:58.838 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000003.53.6, 00000000002.127.6, 00000000002.62.6] 2024-08-18T16:09:03.838 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:09:03.838+0000 7fe471862880 1 waiting for incremental sync to catch up: 2024-08-18T16:09:03.839 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , ] 2024-08-18T16:09:03.839 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000003.53.6, 00000000002.127.6, 00000000002.62.6] 2024-08-18T16:09:08.839 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:09:08.839+0000 7fe471862880 1 waiting for incremental sync to catch up: 2024-08-18T16:09:08.839 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , ] 2024-08-18T16:09:08.839 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000003.53.6, 00000000002.127.6, 00000000002.62.6] 2024-08-18T16:09:13.840 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:09:13.839+0000 7fe471862880 1 waiting for incremental sync to catch up: 2024-08-18T16:09:13.840 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , ] 2024-08-18T16:09:13.840 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000003.53.6, 00000000002.127.6, 00000000002.62.6] 2024-08-18T16:09:18.841 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:09:18.840+0000 7fe471862880 1 waiting for incremental sync to catch up: 2024-08-18T16:09:18.841 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , 00000000002.62.6] 2024-08-18T16:09:18.841 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000003.53.6, 00000000002.127.6, 00000000002.62.6] 2024-08-18T16:09:23.842 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:09:23.841+0000 7fe471862880 1 waiting for incremental sync to catch up: 2024-08-18T16:09:23.842 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , 00000000002.62.6] 2024-08-18T16:09:23.842 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000003.53.6, 00000000002.127.6, 00000000002.62.6] 2024-08-18T16:09:28.842 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:09:28.842+0000 7fe471862880 1 waiting for incremental sync to catch up: 2024-08-18T16:09:28.843 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , 00000000002.62.6] 2024-08-18T16:09:28.843 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000003.53.6, 00000000002.127.6, 00000000002.62.6] 2024-08-18T16:09:33.843 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:09:33.841+0000 7fe471862880 1 waiting for incremental sync to catch up: 2024-08-18T16:09:33.843 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , 00000000002.62.6] 2024-08-18T16:09:33.843 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000003.53.6, 00000000002.127.6, 00000000002.62.6] 2024-08-18T16:09:38.843 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:09:38.842+0000 7fe471862880 1 waiting for incremental sync to catch up: 2024-08-18T16:09:38.843 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000003.53.6, , 00000000002.62.6] 2024-08-18T16:09:38.843 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000003.53.6, 00000000002.127.6, 00000000002.62.6] 2024-08-18T16:09:43.844 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:09:43.843+0000 7fe471862880 1 waiting for incremental sync to catch up: 2024-08-18T16:09:43.844 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000003.53.6, , 00000000002.62.6] 2024-08-18T16:09:43.844 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000003.53.6, 00000000002.127.6, 00000000002.62.6] 2024-08-18T16:09:48.844 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:09:48.844+0000 7fe471862880 1 waiting for incremental sync to catch up: 2024-08-18T16:09:48.845 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000003.53.6, , 00000000002.62.6] 2024-08-18T16:09:48.845 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000003.53.6, 00000000002.127.6, 00000000002.62.6] 2024-08-18T16:09:53.845 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:09:53.844+0000 7fe471862880 1 waiting for incremental sync to catch up: 2024-08-18T16:09:53.845 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000003.53.6, , 00000000002.62.6] 2024-08-18T16:09:53.845 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000003.53.6, 00000000002.127.6, 00000000002.62.6] 2024-08-18T16:09:58.845 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:09:58.845+0000 7fe471862880 1 waiting for incremental sync to catch up: 2024-08-18T16:09:58.846 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000003.53.6, , 00000000002.62.6] 2024-08-18T16:09:58.846 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000003.53.6, 00000000002.127.6, 00000000002.62.6] 2024-08-18T16:10:03.846 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:10:03.846+0000 7fe471862880 1 waiting for incremental sync to catch up: 2024-08-18T16:10:03.847 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000003.53.6, , 00000000002.62.6] 2024-08-18T16:10:03.847 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000003.53.6, 00000000002.127.6, 00000000002.62.6] 2024-08-18T16:10:08.847 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:10:08.847+0000 7fe471862880 1 waiting for incremental sync to catch up: 2024-08-18T16:10:08.847 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000003.53.6, , 00000000002.62.6] 2024-08-18T16:10:08.848 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000003.53.6, 00000000002.127.6, 00000000002.62.6] 2024-08-18T16:10:13.848 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:10:13.848+0000 7fe471862880 1 waiting for incremental sync to catch up: 2024-08-18T16:10:13.849 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000003.53.6, , 00000000002.62.6] 2024-08-18T16:10:13.849 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000003.53.6, 00000000002.127.6, 00000000002.62.6] 2024-08-18T16:10:18.849 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:10:18.849+0000 7fe471862880 1 waiting for incremental sync to catch up: 2024-08-18T16:10:18.849 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000003.53.6, , 00000000002.62.6] 2024-08-18T16:10:18.849 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000003.53.6, 00000000002.127.6, 00000000002.62.6] 2024-08-18T16:10:23.850 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:10:23.849+0000 7fe471862880 1 waiting for incremental sync to catch up: 2024-08-18T16:10:23.850 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000003.53.6, , 00000000002.62.6] 2024-08-18T16:10:23.850 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000003.53.6, 00000000002.127.6, 00000000002.62.6] 2024-08-18T16:10:28.851 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:10:28.850+0000 7fe471862880 1 waiting for incremental sync to catch up: 2024-08-18T16:10:28.851 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000003.53.6, , 00000000002.62.6] 2024-08-18T16:10:28.851 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000003.53.6, 00000000002.127.6, 00000000002.62.6] 2024-08-18T16:10:33.852 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:10:33.850+0000 7fe471862880 1 waiting for incremental sync to catch up: 2024-08-18T16:10:33.852 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000003.53.6, , 00000000002.62.6] 2024-08-18T16:10:33.852 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000003.53.6, 00000000002.127.6, 00000000002.62.6] 2024-08-18T16:10:38.852 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:10:38.852+0000 7fe471862880 1 waiting for incremental sync to catch up: 2024-08-18T16:10:38.853 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000003.53.6, , 00000000002.62.6] 2024-08-18T16:10:38.853 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000003.53.6, 00000000002.127.6, 00000000002.62.6] 2024-08-18T16:10:43.853 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:10:43.853+0000 7fe471862880 1 waiting for incremental sync to catch up: 2024-08-18T16:10:43.853 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000003.53.6, , 00000000002.62.6] 2024-08-18T16:10:43.854 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000003.53.6, 00000000002.127.6, 00000000002.62.6] 2024-08-18T16:10:48.854 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:10:48.853+0000 7fe471862880 1 waiting for incremental sync to catch up: 2024-08-18T16:10:48.855 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000003.53.6, , 00000000002.62.6] 2024-08-18T16:10:48.855 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000003.53.6, 00000000002.127.6, 00000000002.62.6] 2024-08-18T16:10:53.855 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:10:53.854+0000 7fe471862880 1 waiting for incremental sync to catch up: 2024-08-18T16:10:53.855 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000003.53.6, , 00000000002.62.6] 2024-08-18T16:10:53.855 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000003.53.6, 00000000002.127.6, 00000000002.62.6] 2024-08-18T16:10:58.856 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:10:58.855+0000 7fe471862880 1 waiting for incremental sync to catch up: 2024-08-18T16:10:58.856 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000003.53.6, , 00000000002.62.6] 2024-08-18T16:10:58.856 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000003.53.6, 00000000002.127.6, 00000000002.62.6] 2024-08-18T16:11:03.856 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:11:03.856+0000 7fe471862880 1 waiting for incremental sync to catch up: 2024-08-18T16:11:03.857 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000003.53.6, , 00000000002.62.6] 2024-08-18T16:11:03.857 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000003.53.6, 00000000002.127.6, 00000000002.62.6] 2024-08-18T16:11:08.857 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:11:08.857+0000 7fe471862880 1 waiting for incremental sync to catch up: 2024-08-18T16:11:08.857 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000003.53.6, , 00000000002.62.6] 2024-08-18T16:11:08.858 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000003.53.6, 00000000002.127.6, 00000000002.62.6] 2024-08-18T16:11:13.858 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:11:13.858+0000 7fe471862880 1 waiting for incremental sync to catch up: 2024-08-18T16:11:13.858 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000003.53.6, , 00000000002.62.6] 2024-08-18T16:11:13.859 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000003.53.6, 00000000002.127.6, 00000000002.62.6] 2024-08-18T16:11:18.860 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:11:18.858+0000 7fe471862880 1 waiting for incremental sync to catch up: 2024-08-18T16:11:18.860 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000003.53.6, , 00000000002.62.6] 2024-08-18T16:11:18.860 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000003.53.6, 00000000002.127.6, 00000000002.62.6] 2024-08-18T16:11:23.860 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:11:23.859+0000 7fe471862880 1 waiting for incremental sync to catch up: 2024-08-18T16:11:23.860 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000003.53.6, , 00000000002.62.6] 2024-08-18T16:11:23.861 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000003.53.6, 00000000002.127.6, 00000000002.62.6] 2024-08-18T16:11:28.861 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:11:28.860+0000 7fe471862880 1 waiting for incremental sync to catch up: 2024-08-18T16:11:28.861 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000003.53.6, , 00000000002.62.6] 2024-08-18T16:11:28.861 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000003.53.6, 00000000002.127.6, 00000000002.62.6] 2024-08-18T16:11:33.861 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:11:33.861+0000 7fe471862880 1 waiting for incremental sync to catch up: 2024-08-18T16:11:33.862 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000003.53.6, , 00000000002.62.6] 2024-08-18T16:11:33.862 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000003.53.6, 00000000002.127.6, 00000000002.62.6] 2024-08-18T16:11:38.862 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:11:38.861+0000 7fe471862880 1 bucket sync caught up with source: 2024-08-18T16:11:38.862 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000003.53.6, 00000000002.127.6, 00000000002.62.6] 2024-08-18T16:11:38.863 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000003.53.6, 00000000002.127.6, 00000000002.62.6] 2024-08-18T16:11:38.863 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:11:38.861+0000 7fe471862880 0 bucket checkpoint complete 2024-08-18T16:11:38.881 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-75', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:11:38.881 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-75', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:11:38.881 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-75 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:11:38.957 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:11:38.957 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:11:38.991 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:11:38.990+0000 7f913e010880 1 bucket sync caught up with source: 2024-08-18T16:11:38.991 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000003.53.6, 00000000002.127.6, 00000000002.62.6] 2024-08-18T16:11:38.991 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000003.53.6, 00000000002.127.6, 00000000002.62.6] 2024-08-18T16:11:38.991 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:11:38.990+0000 7f913e010880 0 bucket checkpoint complete 2024-08-18T16:11:39.004 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-75', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:11:39.004 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-75', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:11:39.004 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-75 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:11:39.058 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:11:39.058 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:11:39.092 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:11:39.092+0000 7fc758496880 1 bucket sync caught up with source: 2024-08-18T16:11:39.093 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000003.303.6, 00000000002.262.6, 00000000002.254.6] 2024-08-18T16:11:39.093 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000003.303.6, 00000000002.262.6, 00000000002.254.6] 2024-08-18T16:11:39.093 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:11:39.092+0000 7fc758496880 0 bucket checkpoint complete 2024-08-18T16:11:39.107 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-75', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:11:39.107 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-75', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:11:39.107 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-75 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:11:39.160 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:11:39.160 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:11:39.195 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:11:39.194+0000 7f8a1bf22880 1 bucket sync caught up with source: 2024-08-18T16:11:39.195 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000003.303.6, 00000000002.262.6, 00000000002.254.6] 2024-08-18T16:11:39.195 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000003.303.6, 00000000002.262.6, 00000000002.254.6] 2024-08-18T16:11:39.195 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:11:39.194+0000 7f8a1bf22880 0 bucket checkpoint complete 2024-08-18T16:11:39.208 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-75', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:11:39.209 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-75', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:11:39.209 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-75 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:11:39.262 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:11:39.262 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:11:39.296 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:11:39.296+0000 7f72d4215880 1 bucket sync caught up with source: 2024-08-18T16:11:39.296 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000002.125.6, , , 00000000001.200.6, 00000000001.222.6, , , 00000000002.258.6, , 00000000002.186.6] 2024-08-18T16:11:39.297 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000002.125.6, , , 00000000001.200.6, 00000000001.222.6, , , 00000000002.258.6, , 00000000002.186.6] 2024-08-18T16:11:39.297 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:11:39.296+0000 7f72d4215880 0 bucket checkpoint complete 2024-08-18T16:11:39.311 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-75', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:11:39.311 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-75', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:11:39.311 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-75 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T16:11:39.372 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:11:39.372 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:11:39.412 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:11:39.411+0000 7fc443235880 1 bucket sync caught up with source: 2024-08-18T16:11:39.412 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, 00000000002.125.6, , , 00000000001.200.6, 00000000001.222.6, , , 00000000002.258.6, , 00000000002.186.6] 2024-08-18T16:11:39.412 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, 00000000002.125.6, , , 00000000001.200.6, 00000000001.222.6, , , 00000000002.258.6, , 00000000002.186.6] 2024-08-18T16:11:39.412 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:11:39.411+0000 7fc443235880 0 bucket checkpoint complete 2024-08-18T16:11:39.428 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-75 zones={test-zone2, test-zone1} 2024-08-18T16:11:39.436 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:11:39.439 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-08-18T16:11:39.439 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-08-18T16:11:39.439 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-08-18T16:11:39.439 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-08-18T16:11:39.439 DEBUG:tasks.rgw_multi.zone_rados:o=e 2024-08-18T16:11:39.439 DEBUG:tasks.rgw_multi.zone_rados:o=f 2024-08-18T16:11:39.439 DEBUG:tasks.rgw_multi.zone_rados:o=g 2024-08-18T16:11:39.439 DEBUG:tasks.rgw_multi.zone_rados:o=h 2024-08-18T16:11:39.439 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:11:39.442 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-08-18T16:11:39.442 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-08-18T16:11:39.442 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-08-18T16:11:39.442 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-08-18T16:11:39.442 DEBUG:tasks.rgw_multi.zone_rados:o=e 2024-08-18T16:11:39.442 DEBUG:tasks.rgw_multi.zone_rados:o=f 2024-08-18T16:11:39.442 DEBUG:tasks.rgw_multi.zone_rados:o=g 2024-08-18T16:11:39.442 DEBUG:tasks.rgw_multi.zone_rados:o=h 2024-08-18T16:11:39.447 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-08-18T16:11:39.454 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-08-18T16:11:39.463 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-08-18T16:11:39.470 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-08-18T16:11:39.476 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-08-18T16:11:39.486 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-08-18T16:11:39.539 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-08-18T16:11:39.546 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-08-18T16:11:39.597 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-08-18T16:11:39.649 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-08-18T16:11:39.656 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-08-18T16:11:39.666 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-08-18T16:11:39.714 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2024-08-18T16:11:39.721 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2024-08-18T16:11:39.824 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2024-08-18T16:11:39.917 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2024-08-18T16:11:39.969 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2024-08-18T16:11:40.023 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2024-08-18T16:11:40.030 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2024-08-18T16:11:40.037 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2024-08-18T16:11:40.092 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2024-08-18T16:11:40.143 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2024-08-18T16:11:40.149 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2024-08-18T16:11:40.159 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2024-08-18T16:11:40.211 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-75 zones={test-zone2, test-zone1} 2024-08-18T16:11:40.211 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-75 zones={test-zone3, test-zone1} 2024-08-18T16:11:40.217 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:11:40.219 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-08-18T16:11:40.220 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-08-18T16:11:40.220 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-08-18T16:11:40.220 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-08-18T16:11:40.220 DEBUG:tasks.rgw_multi.zone_rados:o=e 2024-08-18T16:11:40.220 DEBUG:tasks.rgw_multi.zone_rados:o=f 2024-08-18T16:11:40.220 DEBUG:tasks.rgw_multi.zone_rados:o=g 2024-08-18T16:11:40.220 DEBUG:tasks.rgw_multi.zone_rados:o=h 2024-08-18T16:11:40.220 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:11:40.222 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-08-18T16:11:40.222 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-08-18T16:11:40.222 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-08-18T16:11:40.222 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-08-18T16:11:40.222 DEBUG:tasks.rgw_multi.zone_rados:o=e 2024-08-18T16:11:40.222 DEBUG:tasks.rgw_multi.zone_rados:o=f 2024-08-18T16:11:40.222 DEBUG:tasks.rgw_multi.zone_rados:o=g 2024-08-18T16:11:40.223 DEBUG:tasks.rgw_multi.zone_rados:o=h 2024-08-18T16:11:40.227 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-08-18T16:11:40.279 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-08-18T16:11:40.335 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-08-18T16:11:40.385 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-08-18T16:11:40.436 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-08-18T16:11:40.487 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-08-18T16:11:40.536 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-08-18T16:11:40.583 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-08-18T16:11:40.636 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-08-18T16:11:40.733 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-08-18T16:11:40.740 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-08-18T16:11:40.751 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-08-18T16:11:40.757 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2024-08-18T16:11:40.763 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2024-08-18T16:11:40.815 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2024-08-18T16:11:40.864 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2024-08-18T16:11:40.870 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2024-08-18T16:11:40.921 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2024-08-18T16:11:40.930 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2024-08-18T16:11:40.984 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2024-08-18T16:11:40.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2024-08-18T16:11:41.043 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2024-08-18T16:11:41.049 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2024-08-18T16:11:41.058 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2024-08-18T16:11:41.065 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-75 zones={test-zone3, test-zone1} 2024-08-18T16:11:41.065 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-75 zones={test-zone3, test-zone2} 2024-08-18T16:11:41.069 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:11:41.071 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-08-18T16:11:41.071 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-08-18T16:11:41.072 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-08-18T16:11:41.072 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-08-18T16:11:41.072 DEBUG:tasks.rgw_multi.zone_rados:o=e 2024-08-18T16:11:41.072 DEBUG:tasks.rgw_multi.zone_rados:o=f 2024-08-18T16:11:41.072 DEBUG:tasks.rgw_multi.zone_rados:o=g 2024-08-18T16:11:41.072 DEBUG:tasks.rgw_multi.zone_rados:o=h 2024-08-18T16:11:41.072 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:11:41.074 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-08-18T16:11:41.074 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-08-18T16:11:41.075 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-08-18T16:11:41.075 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-08-18T16:11:41.075 DEBUG:tasks.rgw_multi.zone_rados:o=e 2024-08-18T16:11:41.075 DEBUG:tasks.rgw_multi.zone_rados:o=f 2024-08-18T16:11:41.075 DEBUG:tasks.rgw_multi.zone_rados:o=g 2024-08-18T16:11:41.075 DEBUG:tasks.rgw_multi.zone_rados:o=h 2024-08-18T16:11:41.080 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-08-18T16:11:41.128 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-08-18T16:11:41.181 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-08-18T16:11:41.232 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-08-18T16:11:41.280 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-08-18T16:11:41.333 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-08-18T16:11:41.385 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-08-18T16:11:41.437 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-08-18T16:11:41.488 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-08-18T16:11:41.589 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-08-18T16:11:41.596 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-08-18T16:11:41.650 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-08-18T16:11:41.658 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2024-08-18T16:11:41.713 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2024-08-18T16:11:41.769 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2024-08-18T16:11:41.821 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2024-08-18T16:11:41.828 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2024-08-18T16:11:41.840 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2024-08-18T16:11:41.891 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2024-08-18T16:11:41.942 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2024-08-18T16:11:41.999 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2024-08-18T16:11:42.051 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2024-08-18T16:11:42.098 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2024-08-18T16:11:42.152 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2024-08-18T16:11:42.204 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-75 zones={test-zone3, test-zone2} 2024-08-18T16:11:42.204 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_reshard_incremental ... ok 2024-08-18T16:11:42.219 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:11:42.219 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:11:42.220 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:11:42.266 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:11:42.267 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:11:42.297 INFO:teuthology.orchestra.run.smithi134.stderr:No --period given, using current period=01050e30-179b-43be-bdbb-3bb23cf37ae8 2024-08-18T16:11:42.310 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T16:11:42.310 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:11:42.311 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T16:11:42.311 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T16:11:42.311 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:11:42.311 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:11:42.311 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T16:11:42.311 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T16:11:42.311 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:11:42.311 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:11:42.311 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T16:11:42.311 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T16:11:42.311 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:11:42.311 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:11:42.312 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723997502.218461_770.1", 2024-08-18T16:11:42.312 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:11:42.218461Z" 2024-08-18T16:11:42.312 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:11:42.312 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T16:11:42.313 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-08-18T16:11:42.313 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:11:42.313 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:11:42.313 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:11:42.369 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:11:42.369 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:11:42.406 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:11:42.405+0000 7f125d120880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:11:42.406 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:11:42.406+0000 7f125d120880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:11:42.421 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T16:11:42.421 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T16:11:42.422 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T16:11:42.422 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T16:11:42.422 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T16:11:42.422 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T16:11:42.422 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T16:11:42.422 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:11:42.422 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T16:11:42.422 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:11:42.422 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T16:11:42.422 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:11:42.422 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:11:42.423 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T16:11:42.423 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:11:42.423 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T16:11:42.423 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:11:42.423 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T16:11:42.423 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T16:11:42.423 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:11:42.423 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:11:42.423 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:11:42.423 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T16:11:42.423 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:11:42.423 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:11:42.423 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T16:11:42.423 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:11:42.424 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:11:42.424 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:11:42.424 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T16:11:42.424 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T16:11:42.424 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:11:42.424 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:11:42.424 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:11:42.424 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T16:11:42.424 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:11:42.424 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:11:42.424 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T16:11:42.424 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:11:42.424 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:11:42.424 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:11:42.425 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T16:11:42.425 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T16:11:42.425 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:11:42.425 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:11:42.425 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:11:42.425 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T16:11:42.425 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:11:42.425 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:11:42.425 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723997502.218461_770.1", 2024-08-18T16:11:42.425 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:11:42.425 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:11:42.425 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:11:42.425 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:11:42.218461Z", 2024-08-18T16:11:42.426 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T16:11:42.426 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:11:42.426 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:11:42.426 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T16:11:42.426 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:11:42.426 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T16:11:42.426 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T16:11:42.426 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T16:11:42.426 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:11:42.426 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T16:11:42Z" 2024-08-18T16:11:42.426 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T16:11:42.427 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-08-18T16:11:42.427 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-08-18T16:11:42.427 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:11:42.428 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:11:42.428 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:11:42.475 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:11:42.475 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:11:42.503 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:11:42.503+0000 7f9e3b0b9880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:11:42.504 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:11:42.504+0000 7f9e3b0b9880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:11:42.517 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T16:11:42.517 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T16:11:42.517 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T16:11:42.517 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T16:11:42.517 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T16:11:42.517 INFO:teuthology.orchestra.run.smithi134.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T16:11:42.517 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T16:11:42.517 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:11:42.517 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T16:11:42.517 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:11:42.518 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T16:11:42.518 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:11:42.518 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T16:11:42.518 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T16:11:42.518 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:11:42.518 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 1, 2024-08-18T16:11:42.518 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:11:42.518 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T16:11:42.518 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T16:11:42.518 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:11:42.518 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:11:42.518 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:11:42.518 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T16:11:42.519 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:11:42.519 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T16:11:42.519 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T16:11:42.519 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:11:42.519 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:11:42.519 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:11:42.519 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T16:11:42.519 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T16:11:42.519 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:11:42.519 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:11:42.519 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:11:42.519 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T16:11:42.519 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:11:42.519 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T16:11:42.520 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T16:11:42.520 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:11:42.520 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:11:42.520 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:11:42.520 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T16:11:42.520 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T16:11:42.520 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:11:42.520 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:11:42.520 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:11:42.520 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T16:11:42.520 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:11:42.520 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T16:11:42.520 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723997502.218461_770.1", 2024-08-18T16:11:42.520 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:11:42.520 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:11:42.521 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:11:42.521 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:11:42.218461Z", 2024-08-18T16:11:42.521 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T16:11:42.521 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:11:42.521 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:11:42.521 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T16:11:42.521 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:11:42.521 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T16:11:42.521 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 1, 2024-08-18T16:11:42.521 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 1 2024-08-18T16:11:42.521 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:11:42.521 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T16:11:42Z" 2024-08-18T16:11:42.521 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T16:11:42.522 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-08-18T16:11:42.523 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2024-08-18T16:11:42.523 INFO:teuthology.orchestra.run:waiting for 300 2024-08-18T16:11:42.523 INFO:tasks.rgw.c2.client.0.smithi193.stdout:2024-08-18T16:11:42.523+0000 7f440eeb5640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 39167) UID: 0 2024-08-18T16:11:42.523 INFO:tasks.rgw.c2.client.0.smithi193.stdout:2024-08-18T16:11:42.523+0000 7f4412714880 -1 shutting down 2024-08-18T16:11:46.205 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2024-08-18T16:11:48.624 INFO:tasks.rgw.c2.client.0:Stopped 2024-08-18T16:11:48.624 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2024-08-18T16:11:48.625 INFO:teuthology.orchestra.run:waiting for 300 2024-08-18T16:11:48.625 INFO:tasks.rgw.c1.client.1.smithi134.stdout:2024-08-18T16:11:48.625+0000 7fc0bcdfe640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 56147) UID: 0 2024-08-18T16:11:48.626 INFO:tasks.rgw.c1.client.1.smithi134.stdout:2024-08-18T16:11:48.625+0000 7fc0c2a13880 -1 shutting down 2024-08-18T16:11:51.811 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2024-08-18T16:11:54.255 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2024-08-18T16:11:54.726 INFO:tasks.rgw.c1.client.1:Stopped 2024-08-18T16:11:54.801 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'kibgmb-76', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:11:54.801 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'kibgmb-76', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:11:54.801 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket kibgmb-76 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:11:54.855 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:11:54.855 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:11:54.898 INFO:teuthology.orchestra.run.smithi134.stdout:tenant: 2024-08-18T16:11:54.899 INFO:teuthology.orchestra.run.smithi134.stdout:bucket name: kibgmb-76 2024-08-18T16:11:54.902 INFO:teuthology.orchestra.run.smithi134.stdout:total entries: 4 2024-08-18T16:11:55.044 INFO:tasks.rgw.c2.client.0:Restarting daemon 2024-08-18T16:11:55.044 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2024-08-18T16:11:55.047 INFO:tasks.rgw.c2.client.0:Started 2024-08-18T16:11:55.047 DEBUG:teuthology.orchestra.run.smithi193:> curl http://smithi193.front.sepia.ceph.com:8000 2024-08-18T16:11:55.071 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-08-18T16:11:55.071 INFO:teuthology.orchestra.run.smithi193.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-08-18T16:11:55.071 INFO:teuthology.orchestra.run.smithi193.stderr: Dload Upload Total Spent Left Speed 2024-08-18T16:11:55.072 INFO:teuthology.orchestra.run.smithi193.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2024-08-18T16:11:55.072 INFO:teuthology.orchestra.run.smithi193.stderr:curl: (7) Failed to connect to smithi193.front.sepia.ceph.com port 8000: Connection refused 2024-08-18T16:11:56.073 DEBUG:teuthology.orchestra.run.smithi193:> curl http://smithi193.front.sepia.ceph.com:8000 2024-08-18T16:11:56.095 INFO:teuthology.orchestra.run.smithi193.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-08-18T16:11:56.095 INFO:teuthology.orchestra.run.smithi193.stderr: Dload Upload Total Spent Left Speed 2024-08-18T16:11:56.095 INFO:teuthology.orchestra.run.smithi193.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-08-18T16:11:56.096 INFO:teuthology.orchestra.run.smithi193.stdout:anonymous 2024-08-18T16:11:56.096 INFO:tasks.rgw.c1.client.1:Restarting daemon 2024-08-18T16:11:56.096 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2024-08-18T16:11:56.099 INFO:tasks.rgw.c1.client.1:Started 2024-08-18T16:11:56.099 DEBUG:teuthology.orchestra.run.smithi134:> curl http://smithi134.front.sepia.ceph.com:8001 2024-08-18T16:11:56.121 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-08-18T16:11:56.122 INFO:teuthology.orchestra.run.smithi134.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-08-18T16:11:56.122 INFO:teuthology.orchestra.run.smithi134.stderr: Dload Upload Total Spent Left Speed 2024-08-18T16:11:56.122 INFO:teuthology.orchestra.run.smithi134.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2024-08-18T16:11:56.122 INFO:teuthology.orchestra.run.smithi134.stderr:curl: (7) Failed to connect to smithi134.front.sepia.ceph.com port 8001: Connection refused 2024-08-18T16:11:57.123 DEBUG:teuthology.orchestra.run.smithi134:> curl http://smithi134.front.sepia.ceph.com:8001 2024-08-18T16:11:57.144 INFO:teuthology.orchestra.run.smithi134.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-08-18T16:11:57.144 INFO:teuthology.orchestra.run.smithi134.stderr: Dload Upload Total Spent Left Speed 2024-08-18T16:11:57.145 INFO:teuthology.orchestra.run.smithi134.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-08-18T16:11:57.145 INFO:teuthology.orchestra.run.smithi134.stdout:anonymous 2024-08-18T16:11:57.146 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-76', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:11:57.146 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-76', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:11:57.147 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-76 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T16:11:57.210 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:11:57.211 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:11:57.249 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:11:57.249+0000 7fd9b6332880 1 waiting to reach incremental sync.. 2024-08-18T16:11:59.961 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2024-08-18T16:12:02.250 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:12:02.249+0000 7fd9b6332880 1 waiting to reach incremental sync.. 2024-08-18T16:12:07.250 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:12:07.250+0000 7fd9b6332880 1 waiting to reach incremental sync.. 2024-08-18T16:12:12.251 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:12:12.250+0000 7fd9b6332880 1 waiting to reach incremental sync.. 2024-08-18T16:12:17.252 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:12:17.251+0000 7fd9b6332880 1 waiting to reach incremental sync.. 2024-08-18T16:12:22.252 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:12:22.252+0000 7fd9b6332880 1 waiting to reach incremental sync.. 2024-08-18T16:12:27.253 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:12:27.252+0000 7fd9b6332880 1 waiting to reach incremental sync.. 2024-08-18T16:12:32.253 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:12:32.253+0000 7fd9b6332880 1 waiting to reach incremental sync.. 2024-08-18T16:12:37.254 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:12:37.253+0000 7fd9b6332880 1 waiting to reach incremental sync.. 2024-08-18T16:12:42.255 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:12:42.254+0000 7fd9b6332880 1 waiting to reach incremental sync.. 2024-08-18T16:12:47.255 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:12:47.255+0000 7fd9b6332880 1 waiting to reach incremental sync.. 2024-08-18T16:12:52.256 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:12:52.255+0000 7fd9b6332880 1 waiting to reach incremental sync.. 2024-08-18T16:12:57.256 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:12:57.256+0000 7fd9b6332880 1 waiting to reach incremental sync.. 2024-08-18T16:13:02.257 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:13:02.256+0000 7fd9b6332880 1 waiting to reach incremental sync.. 2024-08-18T16:13:07.257 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:13:07.257+0000 7fd9b6332880 1 waiting to reach incremental sync.. 2024-08-18T16:13:12.258 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:13:12.257+0000 7fd9b6332880 1 waiting to reach incremental sync.. 2024-08-18T16:13:17.258 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:13:17.258+0000 7fd9b6332880 1 waiting to reach incremental sync.. 2024-08-18T16:13:22.259 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:13:22.258+0000 7fd9b6332880 1 waiting to reach incremental sync.. 2024-08-18T16:13:27.259 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:13:27.259+0000 7fd9b6332880 1 waiting to reach incremental sync.. 2024-08-18T16:13:32.260 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:13:32.259+0000 7fd9b6332880 1 waiting to reach incremental sync.. 2024-08-18T16:13:37.261 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:13:37.260+0000 7fd9b6332880 1 waiting to reach incremental sync.. 2024-08-18T16:13:42.261 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:13:42.259+0000 7fd9b6332880 1 waiting to reach incremental sync.. 2024-08-18T16:13:47.262 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:13:47.261+0000 7fd9b6332880 1 waiting to reach incremental sync.. 2024-08-18T16:13:52.263 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:13:52.262+0000 7fd9b6332880 1 waiting to reach incremental sync.. 2024-08-18T16:13:57.264 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:13:57.263+0000 7fd9b6332880 1 bucket sync caught up with source: 2024-08-18T16:13:57.264 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000003.83.6, 00000000002.86.6, 00000000002.89.6] 2024-08-18T16:13:57.264 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000003.83.6, 00000000002.86.6, 00000000002.89.6] 2024-08-18T16:13:57.265 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:13:57.263+0000 7fd9b6332880 0 bucket checkpoint complete 2024-08-18T16:13:57.284 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-76', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:13:57.284 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-76', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:13:57.284 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-76 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:13:57.381 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:13:57.381 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:13:57.476 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:13:57.475+0000 7f6c8a213880 1 waiting to reach incremental sync.. 2024-08-18T16:14:02.476 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:14:02.476+0000 7f6c8a213880 1 waiting to reach incremental sync.. 2024-08-18T16:14:07.477 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:14:07.477+0000 7f6c8a213880 1 waiting to reach incremental sync.. 2024-08-18T16:14:12.478 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:14:12.478+0000 7f6c8a213880 1 waiting to reach incremental sync.. 2024-08-18T16:14:17.479 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:14:17.478+0000 7f6c8a213880 1 waiting to reach incremental sync.. 2024-08-18T16:14:22.479 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:14:22.479+0000 7f6c8a213880 1 waiting to reach incremental sync.. 2024-08-18T16:14:27.480 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:14:27.479+0000 7f6c8a213880 1 waiting to reach incremental sync.. 2024-08-18T16:14:32.481 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:14:32.480+0000 7f6c8a213880 1 waiting to reach incremental sync.. 2024-08-18T16:14:37.481 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:14:37.481+0000 7f6c8a213880 1 waiting to reach incremental sync.. 2024-08-18T16:14:42.482 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:14:42.481+0000 7f6c8a213880 1 waiting to reach incremental sync.. 2024-08-18T16:14:47.483 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:14:47.482+0000 7f6c8a213880 1 waiting to reach incremental sync.. 2024-08-18T16:14:52.483 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:14:52.483+0000 7f6c8a213880 1 waiting to reach incremental sync.. 2024-08-18T16:14:57.484 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:14:57.484+0000 7f6c8a213880 1 waiting to reach incremental sync.. 2024-08-18T16:15:02.485 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:15:02.484+0000 7f6c8a213880 1 waiting to reach incremental sync.. 2024-08-18T16:15:07.485 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:15:07.485+0000 7f6c8a213880 1 waiting to reach incremental sync.. 2024-08-18T16:15:12.486 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:15:12.485+0000 7f6c8a213880 1 waiting to reach incremental sync.. 2024-08-18T16:15:17.489 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:15:17.486+0000 7f6c8a213880 1 waiting to reach incremental sync.. 2024-08-18T16:15:22.487 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:15:22.486+0000 7f6c8a213880 1 waiting to reach incremental sync.. 2024-08-18T16:15:27.487 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:15:27.487+0000 7f6c8a213880 1 waiting to reach incremental sync.. 2024-08-18T16:15:32.488 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:15:32.488+0000 7f6c8a213880 1 waiting to reach incremental sync.. 2024-08-18T16:15:37.489 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:15:37.489+0000 7f6c8a213880 1 waiting to reach incremental sync.. 2024-08-18T16:15:42.490 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:15:42.488+0000 7f6c8a213880 1 waiting to reach incremental sync.. 2024-08-18T16:15:47.490 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:15:47.490+0000 7f6c8a213880 1 waiting to reach incremental sync.. 2024-08-18T16:15:52.491 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:15:52.490+0000 7f6c8a213880 1 waiting to reach incremental sync.. 2024-08-18T16:15:57.492 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:15:57.491+0000 7f6c8a213880 1 waiting to reach incremental sync.. 2024-08-18T16:16:02.492 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:16:02.492+0000 7f6c8a213880 1 waiting to reach incremental sync.. 2024-08-18T16:16:07.493 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:16:07.492+0000 7f6c8a213880 1 waiting to reach incremental sync.. 2024-08-18T16:16:12.493 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:16:12.493+0000 7f6c8a213880 1 waiting to reach incremental sync.. 2024-08-18T16:16:17.494 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:16:17.494+0000 7f6c8a213880 1 waiting to reach incremental sync.. 2024-08-18T16:16:22.495 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:16:22.494+0000 7f6c8a213880 1 waiting to reach incremental sync.. 2024-08-18T16:16:27.496 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:16:27.495+0000 7f6c8a213880 1 bucket sync caught up with source: 2024-08-18T16:16:27.496 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000003.83.6, 00000000002.86.6, 00000000002.89.6] 2024-08-18T16:16:27.496 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000003.83.6, 00000000002.86.6, 00000000002.89.6] 2024-08-18T16:16:27.496 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:16:27.495+0000 7f6c8a213880 0 bucket checkpoint complete 2024-08-18T16:16:27.510 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-76', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:16:27.510 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-76', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:16:27.510 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-76 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:16:27.565 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:16:27.565 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:16:27.601 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:16:27.601+0000 7fc73d019880 1 bucket sync caught up with source: 2024-08-18T16:16:27.602 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000002.254.6, , , 00000000001.309.6, 00000000001.279.6, , , 00000000002.280.6, , 00000000002.259.6] 2024-08-18T16:16:27.602 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000002.254.6, , , 00000000001.309.6, 00000000001.279.6, , , 00000000002.280.6, , 00000000002.259.6] 2024-08-18T16:16:27.602 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:16:27.601+0000 7fc73d019880 0 bucket checkpoint complete 2024-08-18T16:16:27.615 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-76', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:16:27.615 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-76', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:16:27.615 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-76 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:16:27.669 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:16:27.670 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:16:27.704 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:16:27.704+0000 7f0275613880 1 bucket sync caught up with source: 2024-08-18T16:16:27.704 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000002.254.6, , , 00000000001.309.6, 00000000001.279.6, , , 00000000002.280.6, , 00000000002.259.6] 2024-08-18T16:16:27.705 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000002.254.6, , , 00000000001.309.6, 00000000001.279.6, , , 00000000002.280.6, , 00000000002.259.6] 2024-08-18T16:16:27.705 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:16:27.704+0000 7f0275613880 0 bucket checkpoint complete 2024-08-18T16:16:27.718 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-76', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:16:27.718 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-76', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:16:27.718 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-76 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:16:27.771 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:16:27.772 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:16:27.806 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:16:27.806+0000 7f0b4f084880 1 bucket sync caught up with source: 2024-08-18T16:16:27.807 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000002.151.6, , , 00000000001.147.6, 00000000001.196.6, , , 00000000002.199.6, , 00000000002.163.6] 2024-08-18T16:16:27.807 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000002.151.6, , , 00000000001.147.6, 00000000001.196.6, , , 00000000002.199.6, , 00000000002.163.6] 2024-08-18T16:16:27.807 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:16:27.806+0000 7f0b4f084880 0 bucket checkpoint complete 2024-08-18T16:16:27.821 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-76', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:16:27.821 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-76', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:16:27.821 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-76 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T16:16:27.882 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:16:27.883 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:16:27.922 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:16:27.922+0000 7f250d131880 1 bucket sync caught up with source: 2024-08-18T16:16:27.922 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, 00000000002.151.6, , , 00000000001.147.6, 00000000001.196.6, , , 00000000002.199.6, , 00000000002.163.6] 2024-08-18T16:16:27.922 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, 00000000002.151.6, , , 00000000001.147.6, 00000000001.196.6, , , 00000000002.199.6, , 00000000002.163.6] 2024-08-18T16:16:27.922 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:16:27.922+0000 7f250d131880 0 bucket checkpoint complete 2024-08-18T16:16:27.938 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-76 zones={test-zone2, test-zone1} 2024-08-18T16:16:27.948 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:16:27.951 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-08-18T16:16:27.951 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-08-18T16:16:27.951 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-08-18T16:16:27.951 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-08-18T16:16:27.951 DEBUG:tasks.rgw_multi.zone_rados:o=e 2024-08-18T16:16:27.951 DEBUG:tasks.rgw_multi.zone_rados:o=f 2024-08-18T16:16:27.952 DEBUG:tasks.rgw_multi.zone_rados:o=g 2024-08-18T16:16:27.952 DEBUG:tasks.rgw_multi.zone_rados:o=h 2024-08-18T16:16:27.952 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:16:27.954 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-08-18T16:16:27.954 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-08-18T16:16:27.954 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-08-18T16:16:27.954 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-08-18T16:16:27.954 DEBUG:tasks.rgw_multi.zone_rados:o=e 2024-08-18T16:16:27.954 DEBUG:tasks.rgw_multi.zone_rados:o=f 2024-08-18T16:16:27.954 DEBUG:tasks.rgw_multi.zone_rados:o=g 2024-08-18T16:16:27.954 DEBUG:tasks.rgw_multi.zone_rados:o=h 2024-08-18T16:16:27.959 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-08-18T16:16:27.966 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-08-18T16:16:27.975 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-08-18T16:16:28.023 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-08-18T16:16:28.029 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-08-18T16:16:28.136 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-08-18T16:16:28.188 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-08-18T16:16:28.238 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-08-18T16:16:28.250 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-08-18T16:16:28.299 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-08-18T16:16:28.305 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-08-18T16:16:28.358 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-08-18T16:16:28.365 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2024-08-18T16:16:28.416 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2024-08-18T16:16:28.474 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2024-08-18T16:16:28.481 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2024-08-18T16:16:28.488 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2024-08-18T16:16:28.543 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2024-08-18T16:16:28.551 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2024-08-18T16:16:28.606 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2024-08-18T16:16:28.616 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2024-08-18T16:16:28.668 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2024-08-18T16:16:28.675 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2024-08-18T16:16:28.685 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2024-08-18T16:16:28.692 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-76 zones={test-zone2, test-zone1} 2024-08-18T16:16:28.692 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-76 zones={test-zone3, test-zone1} 2024-08-18T16:16:28.698 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:16:28.701 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-08-18T16:16:28.701 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-08-18T16:16:28.701 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-08-18T16:16:28.701 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-08-18T16:16:28.701 DEBUG:tasks.rgw_multi.zone_rados:o=e 2024-08-18T16:16:28.701 DEBUG:tasks.rgw_multi.zone_rados:o=f 2024-08-18T16:16:28.701 DEBUG:tasks.rgw_multi.zone_rados:o=g 2024-08-18T16:16:28.701 DEBUG:tasks.rgw_multi.zone_rados:o=h 2024-08-18T16:16:28.701 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:16:28.708 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-08-18T16:16:28.708 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-08-18T16:16:28.708 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-08-18T16:16:28.709 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-08-18T16:16:28.709 DEBUG:tasks.rgw_multi.zone_rados:o=e 2024-08-18T16:16:28.709 DEBUG:tasks.rgw_multi.zone_rados:o=f 2024-08-18T16:16:28.709 DEBUG:tasks.rgw_multi.zone_rados:o=g 2024-08-18T16:16:28.709 DEBUG:tasks.rgw_multi.zone_rados:o=h 2024-08-18T16:16:28.725 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-08-18T16:16:28.774 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-08-18T16:16:28.784 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-08-18T16:16:28.790 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-08-18T16:16:28.796 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-08-18T16:16:28.854 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-08-18T16:16:28.861 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-08-18T16:16:28.907 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-08-18T16:16:28.921 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-08-18T16:16:28.927 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-08-18T16:16:28.980 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-08-18T16:16:29.031 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-08-18T16:16:29.129 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2024-08-18T16:16:29.179 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2024-08-18T16:16:29.280 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2024-08-18T16:16:29.328 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2024-08-18T16:16:29.375 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2024-08-18T16:16:29.427 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2024-08-18T16:16:29.476 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2024-08-18T16:16:29.527 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2024-08-18T16:16:29.624 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2024-08-18T16:16:29.719 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2024-08-18T16:16:29.769 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2024-08-18T16:16:29.827 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2024-08-18T16:16:29.925 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-76 zones={test-zone3, test-zone1} 2024-08-18T16:16:29.926 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-76 zones={test-zone3, test-zone2} 2024-08-18T16:16:29.932 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:16:29.935 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-08-18T16:16:29.935 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-08-18T16:16:29.935 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-08-18T16:16:29.935 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-08-18T16:16:29.935 DEBUG:tasks.rgw_multi.zone_rados:o=e 2024-08-18T16:16:29.935 DEBUG:tasks.rgw_multi.zone_rados:o=f 2024-08-18T16:16:29.935 DEBUG:tasks.rgw_multi.zone_rados:o=g 2024-08-18T16:16:29.935 DEBUG:tasks.rgw_multi.zone_rados:o=h 2024-08-18T16:16:29.935 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:16:29.939 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-08-18T16:16:29.939 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-08-18T16:16:29.939 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-08-18T16:16:29.939 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-08-18T16:16:29.939 DEBUG:tasks.rgw_multi.zone_rados:o=e 2024-08-18T16:16:29.939 DEBUG:tasks.rgw_multi.zone_rados:o=f 2024-08-18T16:16:29.940 DEBUG:tasks.rgw_multi.zone_rados:o=g 2024-08-18T16:16:29.940 DEBUG:tasks.rgw_multi.zone_rados:o=h 2024-08-18T16:16:29.945 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-08-18T16:16:29.996 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-08-18T16:16:30.124 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-08-18T16:16:30.280 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-08-18T16:16:30.314 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-08-18T16:16:30.434 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-08-18T16:16:30.492 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-08-18T16:16:30.504 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-08-18T16:16:30.562 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-08-18T16:16:30.571 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-08-18T16:16:30.621 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-08-18T16:16:30.679 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-08-18T16:16:30.774 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2024-08-18T16:16:30.824 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2024-08-18T16:16:30.921 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2024-08-18T16:16:31.018 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2024-08-18T16:16:31.070 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2024-08-18T16:16:31.127 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2024-08-18T16:16:31.223 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2024-08-18T16:16:31.272 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2024-08-18T16:16:31.370 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2024-08-18T16:16:31.421 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2024-08-18T16:16:31.472 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2024-08-18T16:16:31.526 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2024-08-18T16:16:31.578 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-76 zones={test-zone3, test-zone2} 2024-08-18T16:16:31.578 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_reshard_full ... ok 2024-08-18T16:16:31.579 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:16:31.579 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:16:31.579 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:16:31.626 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:16:31.626 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:16:31.658 INFO:teuthology.orchestra.run.smithi134.stderr:No --period given, using current period=01050e30-179b-43be-bdbb-3bb23cf37ae8 2024-08-18T16:16:31.671 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T16:16:31.671 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:16:31.671 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T16:16:31.671 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T16:16:31.671 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:16:31.671 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:16:31.671 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T16:16:31.671 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T16:16:31.672 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:16:31.672 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:16:31.672 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T16:16:31.672 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T16:16:31.672 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:16:31.672 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:16:31.672 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723997514.907261_774.1", 2024-08-18T16:16:31.672 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:11:54.907261Z" 2024-08-18T16:16:31.672 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:16:31.672 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T16:16:31.673 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-08-18T16:16:31.674 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:16:31.674 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:16:31.674 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:16:31.729 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:16:31.729 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:16:31.763 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:16:31.762+0000 7fab87810880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:16:31.764 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:16:31.764+0000 7fab87810880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:16:31.779 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T16:16:31.779 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T16:16:31.779 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T16:16:31.779 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T16:16:31.779 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T16:16:31.779 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T16:16:31.779 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T16:16:31.779 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:16:31.780 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T16:16:31.780 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:16:31.780 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T16:16:31.780 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:16:31.780 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:16:31.780 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T16:16:31.780 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:16:31.780 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T16:16:31.780 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:16:31.780 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T16:16:31.780 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T16:16:31.780 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:16:31.780 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:16:31.781 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:16:31.781 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T16:16:31.781 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:16:31.781 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:16:31.781 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T16:16:31.781 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:16:31.781 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:16:31.781 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:16:31.781 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T16:16:31.781 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T16:16:31.781 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:16:31.781 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:16:31.781 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:16:31.781 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T16:16:31.782 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:16:31.782 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:16:31.782 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T16:16:31.782 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:16:31.782 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:16:31.782 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:16:31.782 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T16:16:31.782 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T16:16:31.782 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:16:31.782 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:16:31.782 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:16:31.783 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T16:16:31.783 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:16:31.783 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:16:31.783 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723997514.907261_774.1", 2024-08-18T16:16:31.783 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:16:31.783 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:16:31.783 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:16:31.783 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:11:54.907261Z", 2024-08-18T16:16:31.783 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T16:16:31.783 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:16:31.783 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:16:31.783 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T16:16:31.784 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:16:31.784 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T16:16:31.784 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T16:16:31.784 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T16:16:31.784 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:16:31.784 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T16:16:31Z" 2024-08-18T16:16:31.784 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T16:16:31.785 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-08-18T16:16:31.785 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-08-18T16:16:31.785 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:16:31.785 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:16:31.786 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:16:31.832 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:16:31.833 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:16:31.861 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:16:31.861+0000 7fdf74d23880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:16:31.862 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:16:31.861+0000 7fdf74d23880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:16:31.874 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T16:16:31.874 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T16:16:31.874 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T16:16:31.874 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T16:16:31.875 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T16:16:31.875 INFO:teuthology.orchestra.run.smithi134.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T16:16:31.875 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T16:16:31.875 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:16:31.875 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T16:16:31.875 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:16:31.875 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T16:16:31.875 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:16:31.875 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T16:16:31.875 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T16:16:31.875 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:16:31.875 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 1, 2024-08-18T16:16:31.875 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:16:31.876 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T16:16:31.876 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T16:16:31.876 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:16:31.876 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:16:31.876 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:16:31.876 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T16:16:31.876 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:16:31.876 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T16:16:31.876 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T16:16:31.876 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:16:31.876 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:16:31.876 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:16:31.876 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T16:16:31.876 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T16:16:31.877 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:16:31.877 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:16:31.877 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:16:31.877 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T16:16:31.877 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:16:31.877 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T16:16:31.877 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T16:16:31.877 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:16:31.877 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:16:31.877 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:16:31.877 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T16:16:31.877 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T16:16:31.878 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:16:31.878 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:16:31.878 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:16:31.878 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T16:16:31.878 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:16:31.878 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T16:16:31.878 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723997514.907261_774.1", 2024-08-18T16:16:31.878 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:16:31.878 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:16:31.878 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:16:31.878 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:11:54.907261Z", 2024-08-18T16:16:31.879 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T16:16:31.879 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:16:31.879 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:16:31.879 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T16:16:31.879 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:16:31.879 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T16:16:31.879 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 1, 2024-08-18T16:16:31.879 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 1 2024-08-18T16:16:31.879 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:16:31.879 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T16:16:31Z" 2024-08-18T16:16:31.879 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T16:16:31.880 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-08-18T16:16:31.894 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_creation_time ... ok 2024-08-18T16:16:31.908 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:16:31.909 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:16:31.909 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:16:31.957 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:16:31.957 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:16:31.987 INFO:teuthology.orchestra.run.smithi134.stderr:No --period given, using current period=01050e30-179b-43be-bdbb-3bb23cf37ae8 2024-08-18T16:16:32.001 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T16:16:32.001 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:16:32.001 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T16:16:32.001 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T16:16:32.001 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:16:32.002 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:16:32.002 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T16:16:32.002 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T16:16:32.002 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:16:32.002 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:16:32.002 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T16:16:32.002 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T16:16:32.002 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:16:32.002 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:16:32.002 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723997791.907797_801.1", 2024-08-18T16:16:32.002 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:16:31.907797Z" 2024-08-18T16:16:32.003 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:16:32.003 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T16:16:32.003 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-08-18T16:16:32.004 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:16:32.004 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:16:32.004 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:16:32.055 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:16:32.055 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:16:32.085 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:16:32.084+0000 7f59cca3f880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:16:32.086 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:16:32.085+0000 7f59cca3f880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:16:32.099 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T16:16:32.099 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T16:16:32.100 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T16:16:32.100 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T16:16:32.100 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T16:16:32.100 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T16:16:32.100 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T16:16:32.100 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:16:32.101 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T16:16:32.101 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:16:32.101 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T16:16:32.101 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:16:32.101 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:16:32.101 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T16:16:32.102 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:16:32.102 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T16:16:32.102 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:16:32.102 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T16:16:32.102 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T16:16:32.102 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:16:32.102 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:16:32.102 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:16:32.102 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T16:16:32.102 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:16:32.102 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:16:32.102 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T16:16:32.102 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:16:32.103 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:16:32.103 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:16:32.103 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T16:16:32.103 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T16:16:32.103 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:16:32.103 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:16:32.103 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:16:32.103 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T16:16:32.103 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:16:32.103 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:16:32.103 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T16:16:32.103 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:16:32.103 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:16:32.103 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:16:32.104 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T16:16:32.104 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T16:16:32.104 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:16:32.104 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:16:32.104 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:16:32.104 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T16:16:32.104 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:16:32.104 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:16:32.104 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723997514.907261_774.1", 2024-08-18T16:16:32.104 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:16:32.104 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:16:32.105 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:16:32.105 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:11:54.907261Z", 2024-08-18T16:16:32.105 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T16:16:32.105 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:16:32.105 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:16:32.105 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T16:16:32.105 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:16:32.105 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T16:16:32.105 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T16:16:32.105 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T16:16:32.106 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:16:32.106 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T16:16:32Z" 2024-08-18T16:16:32.106 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T16:16:32.107 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1723997791.907797_801.1 target=1_1723997514.907261_774.1 2024-08-18T16:16:37.108 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:16:37.108 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:16:37.108 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:16:37.164 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:16:37.164 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:16:37.199 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:16:37.199+0000 7f3eeda13880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:16:37.200 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:16:37.200+0000 7f3eeda13880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:16:37.215 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T16:16:37.215 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T16:16:37.215 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T16:16:37.215 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T16:16:37.215 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T16:16:37.215 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T16:16:37.215 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T16:16:37.215 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:16:37.215 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T16:16:37.215 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:16:37.215 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T16:16:37.215 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:16:37.216 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:16:37.216 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T16:16:37.216 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:16:37.216 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T16:16:37.216 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:16:37.216 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T16:16:37.216 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T16:16:37.216 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:16:37.216 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:16:37.216 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:16:37.216 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T16:16:37.216 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:16:37.216 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:16:37.216 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T16:16:37.217 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:16:37.217 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:16:37.217 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:16:37.217 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T16:16:37.217 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T16:16:37.217 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:16:37.217 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:16:37.217 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:16:37.217 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T16:16:37.217 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:16:37.217 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:16:37.217 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T16:16:37.217 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:16:37.217 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:16:37.218 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:16:37.218 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T16:16:37.218 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T16:16:37.218 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:16:37.218 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:16:37.218 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:16:37.218 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T16:16:37.218 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:16:37.218 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:16:37.218 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723997791.907797_801.1", 2024-08-18T16:16:37.218 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:16:37.219 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:16:37.219 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:16:37.219 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:16:31.907797Z", 2024-08-18T16:16:37.219 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T16:16:37.219 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:16:37.219 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:16:37.219 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T16:16:37.219 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:16:37.219 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T16:16:37.219 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T16:16:37.219 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T16:16:37.219 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:16:37.219 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T16:16:37Z" 2024-08-18T16:16:37.219 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T16:16:37.220 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-08-18T16:16:37.221 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-08-18T16:16:37.221 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:16:37.221 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:16:37.221 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:16:37.276 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:16:37.276 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:16:37.311 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:16:37.310+0000 7fa808a13880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:16:37.318 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:16:37.318+0000 7fa808a13880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:16:37.350 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T16:16:37.350 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T16:16:37.350 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T16:16:37.350 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T16:16:37.350 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T16:16:37.351 INFO:teuthology.orchestra.run.smithi134.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T16:16:37.351 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T16:16:37.351 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:16:37.351 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T16:16:37.351 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:16:37.351 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T16:16:37.351 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:16:37.351 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T16:16:37.351 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T16:16:37.351 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:16:37.351 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 1, 2024-08-18T16:16:37.352 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:16:37.352 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T16:16:37.352 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T16:16:37.352 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:16:37.352 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:16:37.352 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:16:37.352 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T16:16:37.352 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:16:37.352 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T16:16:37.352 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T16:16:37.352 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:16:37.352 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:16:37.352 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:16:37.353 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T16:16:37.353 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T16:16:37.353 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:16:37.353 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:16:37.353 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:16:37.353 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T16:16:37.353 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:16:37.353 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T16:16:37.353 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T16:16:37.353 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:16:37.353 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:16:37.353 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:16:37.354 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T16:16:37.354 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T16:16:37.354 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:16:37.354 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:16:37.354 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:16:37.354 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T16:16:37.354 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:16:37.354 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T16:16:37.354 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723997791.907797_801.1", 2024-08-18T16:16:37.354 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:16:37.355 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:16:37.355 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:16:37.355 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:16:31.907797Z", 2024-08-18T16:16:37.355 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T16:16:37.355 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:16:37.355 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:16:37.355 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T16:16:37.355 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:16:37.355 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T16:16:37.355 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 1, 2024-08-18T16:16:37.355 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 1 2024-08-18T16:16:37.355 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:16:37.355 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T16:16:37Z" 2024-08-18T16:16:37.356 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T16:16:37.357 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-08-18T16:16:37.357 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--num-shards', '11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:16:37.357 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--num-shards', '11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:16:37.357 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --num-shards 11 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:16:37.437 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:16:37.437 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:16:37.503 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T16:16:37.504 INFO:teuthology.orchestra.run.smithi134.stdout: "objs": [ 2024-08-18T16:16:37.504 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000001", 2024-08-18T16:16:37.504 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000005", 2024-08-18T16:16:37.504 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000012", 2024-08-18T16:16:37.504 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000004", 2024-08-18T16:16:37.504 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000011", 2024-08-18T16:16:37.504 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000000", 2024-08-18T16:16:37.504 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000010", 2024-08-18T16:16:37.504 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000003", 2024-08-18T16:16:37.504 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000013", 2024-08-18T16:16:37.505 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000002", 2024-08-18T16:16:37.505 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000006" 2024-08-18T16:16:37.505 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T16:16:37.505 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T16:16:37.695 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-77', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:16:37.695 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-77', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:16:37.695 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-77 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T16:16:37.755 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:16:37.756 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:16:37.792 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:16:37.792+0000 7f16e4615880 1 waiting to reach incremental sync.. 2024-08-18T16:16:42.793 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:16:42.793+0000 7f16e4615880 1 waiting to reach incremental sync.. 2024-08-18T16:16:47.794 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:16:47.792+0000 7f16e4615880 1 bucket sync caught up with source: 2024-08-18T16:16:47.794 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000001.83.6, 00000000001.95.6, 00000000001.60.6, , 00000000001.85.6, 00000000001.76.6, 00000000001.84.6, 00000000001.74.6, 00000000001.93.6, 00000000001.43.6, 00000000001.61.6] 2024-08-18T16:16:47.794 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000001.83.6, 00000000001.95.6, 00000000001.60.6, , 00000000001.85.6, 00000000001.76.6, 00000000001.84.6, 00000000001.74.6, 00000000001.93.6, 00000000001.43.6, 00000000001.61.6] 2024-08-18T16:16:47.794 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:16:47.792+0000 7f16e4615880 0 bucket checkpoint complete 2024-08-18T16:16:47.812 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-77', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:16:47.813 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-77', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:16:47.813 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-77 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:16:47.873 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:16:47.873 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:16:47.909 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:16:47.909+0000 7fe79ce0f880 1 waiting to reach incremental sync.. 2024-08-18T16:16:52.910 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:16:52.910+0000 7fe79ce0f880 1 waiting to reach incremental sync.. 2024-08-18T16:16:57.912 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:16:57.911+0000 7fe79ce0f880 1 bucket sync caught up with source: 2024-08-18T16:16:57.913 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000001.83.6, 00000000001.95.6, 00000000001.60.6, , 00000000001.85.6, 00000000001.76.6, 00000000001.84.6, 00000000001.74.6, 00000000001.93.6, 00000000001.43.6, 00000000001.61.6] 2024-08-18T16:16:57.913 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000001.83.6, 00000000001.95.6, 00000000001.60.6, , 00000000001.85.6, 00000000001.76.6, 00000000001.84.6, 00000000001.74.6, 00000000001.93.6, 00000000001.43.6, 00000000001.61.6] 2024-08-18T16:16:57.913 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:16:57.911+0000 7fe79ce0f880 0 bucket checkpoint complete 2024-08-18T16:16:57.930 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-77', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:16:57.930 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-77', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:16:57.930 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-77 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:16:57.989 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:16:57.989 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:16:58.027 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:16:58.026+0000 7f310c612880 1 waiting to reach incremental sync.. 2024-08-18T16:17:03.029 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:17:03.028+0000 7f310c612880 1 bucket sync caught up with source: 2024-08-18T16:17:03.029 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000001.285.6, 00000000001.201.6, 00000000001.260.6, , 00000000001.286.6, 00000000001.271.6, 00000000001.248.6, 00000000001.272.6, 00000000001.200.6, 00000000001.143.6, 00000000001.312.6] 2024-08-18T16:17:03.029 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000001.285.6, 00000000001.201.6, 00000000001.260.6, , 00000000001.286.6, 00000000001.271.6, 00000000001.248.6, 00000000001.272.6, 00000000001.200.6, 00000000001.143.6, 00000000001.312.6] 2024-08-18T16:17:03.029 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:17:03.028+0000 7f310c612880 0 bucket checkpoint complete 2024-08-18T16:17:03.047 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-77', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:17:03.047 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-77', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:17:03.048 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-77 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:17:03.107 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:17:03.107 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:17:03.144 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:17:03.143+0000 7f899e031880 1 waiting to reach incremental sync.. 2024-08-18T16:17:08.145 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:17:08.144+0000 7f899e031880 1 waiting to reach incremental sync.. 2024-08-18T16:17:13.145 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:17:13.145+0000 7f899e031880 1 waiting to reach incremental sync.. 2024-08-18T16:17:18.146 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:17:18.145+0000 7f899e031880 1 waiting to reach incremental sync.. 2024-08-18T16:17:23.147 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:17:23.146+0000 7f899e031880 1 waiting to reach incremental sync.. 2024-08-18T16:17:28.147 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:17:28.146+0000 7f899e031880 1 waiting to reach incremental sync.. 2024-08-18T16:17:33.148 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:17:33.147+0000 7f899e031880 1 waiting to reach incremental sync.. 2024-08-18T16:17:38.148 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:17:38.148+0000 7f899e031880 1 waiting to reach incremental sync.. 2024-08-18T16:17:43.149 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:17:43.148+0000 7f899e031880 1 waiting to reach incremental sync.. 2024-08-18T16:17:48.149 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:17:48.149+0000 7f899e031880 1 waiting to reach incremental sync.. 2024-08-18T16:17:53.150 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:17:53.150+0000 7f899e031880 1 waiting to reach incremental sync.. 2024-08-18T16:17:58.151 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:17:58.150+0000 7f899e031880 1 waiting to reach incremental sync.. 2024-08-18T16:18:03.152 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:18:03.151+0000 7f899e031880 1 waiting to reach incremental sync.. 2024-08-18T16:18:08.152 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:18:08.151+0000 7f899e031880 1 waiting to reach incremental sync.. 2024-08-18T16:18:13.153 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:18:13.152+0000 7f899e031880 1 waiting to reach incremental sync.. 2024-08-18T16:18:18.153 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:18:18.153+0000 7f899e031880 1 waiting to reach incremental sync.. 2024-08-18T16:18:23.154 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:18:23.154+0000 7f899e031880 1 waiting to reach incremental sync.. 2024-08-18T16:18:28.155 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:18:28.155+0000 7f899e031880 1 waiting to reach incremental sync.. 2024-08-18T16:18:33.155 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:18:33.155+0000 7f899e031880 1 waiting to reach incremental sync.. 2024-08-18T16:18:38.156 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:18:38.155+0000 7f899e031880 1 waiting to reach incremental sync.. 2024-08-18T16:18:43.157 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:18:43.156+0000 7f899e031880 1 waiting to reach incremental sync.. 2024-08-18T16:18:48.157 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:18:48.157+0000 7f899e031880 1 waiting to reach incremental sync.. 2024-08-18T16:18:53.158 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:18:53.157+0000 7f899e031880 1 waiting to reach incremental sync.. 2024-08-18T16:18:58.160 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:18:58.159+0000 7f899e031880 1 bucket sync caught up with source: 2024-08-18T16:18:58.161 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000001.285.6, 00000000001.201.6, 00000000001.260.6, , 00000000001.286.6, 00000000001.271.6, 00000000001.248.6, 00000000001.272.6, 00000000001.200.6, 00000000001.143.6, 00000000001.312.6] 2024-08-18T16:18:58.161 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000001.285.6, 00000000001.201.6, 00000000001.260.6, , 00000000001.286.6, 00000000001.271.6, 00000000001.248.6, 00000000001.272.6, 00000000001.200.6, 00000000001.143.6, 00000000001.312.6] 2024-08-18T16:18:58.161 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:18:58.159+0000 7f899e031880 0 bucket checkpoint complete 2024-08-18T16:18:58.179 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-77', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:18:58.179 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-77', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:18:58.180 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-77 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:18:58.240 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:18:58.240 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:18:58.284 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:18:58.283+0000 7f975a235880 1 bucket sync caught up with source: 2024-08-18T16:18:58.284 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000001.153.6, 00000000001.184.6, 00000000001.156.6, , 00000000001.152.6, 00000000001.192.6, 00000000001.235.6, 00000000001.191.6, 00000000001.185.6, 00000000001.106.6, 00000000001.131.6] 2024-08-18T16:18:58.284 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000001.153.6, 00000000001.184.6, 00000000001.156.6, , 00000000001.152.6, 00000000001.192.6, 00000000001.235.6, 00000000001.191.6, 00000000001.185.6, 00000000001.106.6, 00000000001.131.6] 2024-08-18T16:18:58.284 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:18:58.283+0000 7f975a235880 0 bucket checkpoint complete 2024-08-18T16:18:58.299 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-77', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:18:58.299 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-77', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:18:58.299 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-77 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T16:18:58.362 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:18:58.362 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:18:58.403 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:18:58.401+0000 7f7ea7298880 1 bucket sync caught up with source: 2024-08-18T16:18:58.403 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000001.153.6, 00000000001.184.6, 00000000001.156.6, , 00000000001.152.6, 00000000001.192.6, 00000000001.235.6, 00000000001.191.6, 00000000001.185.6, 00000000001.106.6, 00000000001.131.6] 2024-08-18T16:18:58.403 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000001.153.6, 00000000001.184.6, 00000000001.156.6, , 00000000001.152.6, 00000000001.192.6, 00000000001.235.6, 00000000001.191.6, 00000000001.185.6, 00000000001.106.6, 00000000001.131.6] 2024-08-18T16:18:58.403 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:18:58.401+0000 7f7ea7298880 0 bucket checkpoint complete 2024-08-18T16:18:58.422 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-77 zones={test-zone2, test-zone1} 2024-08-18T16:18:58.431 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:18:58.434 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000000 2024-08-18T16:18:58.435 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000001 2024-08-18T16:18:58.435 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000002 2024-08-18T16:18:58.435 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000003 2024-08-18T16:18:58.435 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000005 2024-08-18T16:18:58.435 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000006 2024-08-18T16:18:58.435 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000010 2024-08-18T16:18:58.435 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000011 2024-08-18T16:18:58.435 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000012 2024-08-18T16:18:58.435 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000013 2024-08-18T16:18:58.435 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:18:58.439 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000000 2024-08-18T16:18:58.439 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000001 2024-08-18T16:18:58.439 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000002 2024-08-18T16:18:58.439 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000003 2024-08-18T16:18:58.439 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000005 2024-08-18T16:18:58.439 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000006 2024-08-18T16:18:58.439 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000010 2024-08-18T16:18:58.439 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000011 2024-08-18T16:18:58.439 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000012 2024-08-18T16:18:58.439 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000013 2024-08-18T16:18:58.445 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2024-08-18T16:18:58.451 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2024-08-18T16:18:58.462 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2024-08-18T16:18:58.512 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000001 2024-08-18T16:18:58.518 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000001 2024-08-18T16:18:58.574 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000001 2024-08-18T16:18:58.583 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2024-08-18T16:18:58.637 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2024-08-18T16:18:58.693 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2024-08-18T16:18:58.745 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2024-08-18T16:18:58.753 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2024-08-18T16:18:58.808 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2024-08-18T16:18:58.859 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2024-08-18T16:18:58.866 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2024-08-18T16:18:58.877 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2024-08-18T16:18:58.929 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2024-08-18T16:18:58.935 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2024-08-18T16:18:58.989 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2024-08-18T16:18:59.045 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2024-08-18T16:18:59.052 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2024-08-18T16:18:59.105 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2024-08-18T16:18:59.161 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2024-08-18T16:18:59.168 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2024-08-18T16:18:59.220 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2024-08-18T16:18:59.269 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2024-08-18T16:18:59.276 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2024-08-18T16:18:59.328 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2024-08-18T16:18:59.378 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2024-08-18T16:18:59.384 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2024-08-18T16:18:59.436 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2024-08-18T16:18:59.491 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-77 zones={test-zone2, test-zone1} 2024-08-18T16:18:59.491 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-77 zones={test-zone3, test-zone1} 2024-08-18T16:18:59.497 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:18:59.499 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000000 2024-08-18T16:18:59.500 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000001 2024-08-18T16:18:59.500 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000002 2024-08-18T16:18:59.500 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000003 2024-08-18T16:18:59.500 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000005 2024-08-18T16:18:59.500 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000006 2024-08-18T16:18:59.500 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000010 2024-08-18T16:18:59.500 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000011 2024-08-18T16:18:59.500 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000012 2024-08-18T16:18:59.500 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000013 2024-08-18T16:18:59.500 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:18:59.503 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000000 2024-08-18T16:18:59.503 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000001 2024-08-18T16:18:59.503 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000002 2024-08-18T16:18:59.503 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000003 2024-08-18T16:18:59.503 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000005 2024-08-18T16:18:59.503 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000006 2024-08-18T16:18:59.503 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000010 2024-08-18T16:18:59.503 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000011 2024-08-18T16:18:59.504 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000012 2024-08-18T16:18:59.504 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000013 2024-08-18T16:18:59.509 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2024-08-18T16:18:59.516 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2024-08-18T16:18:59.568 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2024-08-18T16:18:59.623 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000001 2024-08-18T16:18:59.630 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000001 2024-08-18T16:18:59.728 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000001 2024-08-18T16:18:59.821 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2024-08-18T16:18:59.876 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2024-08-18T16:18:59.930 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2024-08-18T16:18:59.939 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2024-08-18T16:18:59.945 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2024-08-18T16:18:59.996 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2024-08-18T16:19:00.004 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2024-08-18T16:19:00.057 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2024-08-18T16:19:00.112 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2024-08-18T16:19:00.119 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2024-08-18T16:19:00.171 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2024-08-18T16:19:00.227 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2024-08-18T16:19:00.317 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2024-08-18T16:19:00.367 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2024-08-18T16:19:00.468 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2024-08-18T16:19:00.557 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2024-08-18T16:19:00.608 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2024-08-18T16:19:00.704 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2024-08-18T16:19:00.801 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2024-08-18T16:19:00.851 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2024-08-18T16:19:00.952 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2024-08-18T16:19:01.045 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2024-08-18T16:19:01.096 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2024-08-18T16:19:01.197 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2024-08-18T16:19:01.250 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-77 zones={test-zone3, test-zone1} 2024-08-18T16:19:01.250 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-77 zones={test-zone3, test-zone2} 2024-08-18T16:19:01.255 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:19:01.258 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000000 2024-08-18T16:19:01.258 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000001 2024-08-18T16:19:01.258 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000002 2024-08-18T16:19:01.258 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000003 2024-08-18T16:19:01.259 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000005 2024-08-18T16:19:01.259 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000006 2024-08-18T16:19:01.259 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000010 2024-08-18T16:19:01.259 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000011 2024-08-18T16:19:01.259 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000012 2024-08-18T16:19:01.259 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000013 2024-08-18T16:19:01.259 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:19:01.262 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000000 2024-08-18T16:19:01.262 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000001 2024-08-18T16:19:01.262 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000002 2024-08-18T16:19:01.262 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000003 2024-08-18T16:19:01.263 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000005 2024-08-18T16:19:01.263 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000006 2024-08-18T16:19:01.263 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000010 2024-08-18T16:19:01.263 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000011 2024-08-18T16:19:01.263 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000012 2024-08-18T16:19:01.263 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000013 2024-08-18T16:19:01.269 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2024-08-18T16:19:01.321 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2024-08-18T16:19:01.381 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2024-08-18T16:19:01.434 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000001 2024-08-18T16:19:01.441 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000001 2024-08-18T16:19:01.546 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000001 2024-08-18T16:19:01.603 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2024-08-18T16:19:01.656 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2024-08-18T16:19:01.766 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2024-08-18T16:19:01.818 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2024-08-18T16:19:01.873 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2024-08-18T16:19:01.885 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2024-08-18T16:19:01.938 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2024-08-18T16:19:01.988 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2024-08-18T16:19:02.041 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2024-08-18T16:19:02.094 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2024-08-18T16:19:02.148 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2024-08-18T16:19:02.201 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2024-08-18T16:19:02.298 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2024-08-18T16:19:02.348 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2024-08-18T16:19:02.405 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2024-08-18T16:19:02.502 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2024-08-18T16:19:02.552 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2024-08-18T16:19:02.610 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2024-08-18T16:19:02.711 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2024-08-18T16:19:02.760 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2024-08-18T16:19:02.857 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2024-08-18T16:19:02.957 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2024-08-18T16:19:03.011 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2024-08-18T16:19:03.114 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2024-08-18T16:19:03.209 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-77 zones={test-zone3, test-zone2} 2024-08-18T16:19:03.210 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2024-08-18T16:19:03.210 INFO:teuthology.orchestra.run:waiting for 300 2024-08-18T16:19:03.212 INFO:tasks.rgw.c2.client.0.smithi193.stdout:2024-08-18T16:19:03.210+0000 7f375e9ec640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 72515) UID: 0 2024-08-18T16:19:03.212 INFO:tasks.rgw.c2.client.0.smithi193.stdout:2024-08-18T16:19:03.210+0000 7f3764e12880 -1 shutting down 2024-08-18T16:19:05.375 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2024-08-18T16:19:09.310 INFO:tasks.rgw.c2.client.0:Stopped 2024-08-18T16:19:09.311 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2024-08-18T16:19:09.311 INFO:teuthology.orchestra.run:waiting for 300 2024-08-18T16:19:09.312 INFO:tasks.rgw.c1.client.1.smithi134.stdout:2024-08-18T16:19:09.311+0000 7f3ff6e3a640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 79946) UID: 0 2024-08-18T16:19:09.312 INFO:tasks.rgw.c1.client.1.smithi134.stdout:2024-08-18T16:19:09.311+0000 7f3ffa692880 -1 shutting down 2024-08-18T16:19:10.981 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2024-08-18T16:19:13.517 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2024-08-18T16:19:15.415 INFO:tasks.rgw.c1.client.1:Stopped 2024-08-18T16:19:15.416 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-77', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:19:15.416 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-77', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:19:15.416 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-77 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:19:15.475 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:19:15.475 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:19:15.521 INFO:teuthology.orchestra.run.smithi134.stdout:tenant: 2024-08-18T16:19:15.522 INFO:teuthology.orchestra.run.smithi134.stdout:bucket name: kibgmb-77 2024-08-18T16:19:15.526 INFO:teuthology.orchestra.run.smithi134.stdout:total entries: 10 2024-08-18T16:19:15.566 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--num-shards', '17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:19:15.566 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--num-shards', '17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:19:15.567 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --num-shards 17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:19:15.618 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:19:15.619 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:19:15.659 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T16:19:15.659 INFO:teuthology.orchestra.run.smithi134.stdout: "objs": [ 2024-08-18T16:19:15.659 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000007", 2024-08-18T16:19:15.659 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000016", 2024-08-18T16:19:15.659 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000024", 2024-08-18T16:19:15.659 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000020", 2024-08-18T16:19:15.659 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000028", 2024-08-18T16:19:15.660 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000000", 2024-08-18T16:19:15.660 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000032", 2024-08-18T16:19:15.660 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000001", 2024-08-18T16:19:15.660 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000010", 2024-08-18T16:19:15.660 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000019", 2024-08-18T16:19:15.660 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000013", 2024-08-18T16:19:15.660 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000021", 2024-08-18T16:19:15.660 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000030", 2024-08-18T16:19:15.661 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000026", 2024-08-18T16:19:15.661 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000033", 2024-08-18T16:19:15.661 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000004", 2024-08-18T16:19:15.661 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000076" 2024-08-18T16:19:15.661 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T16:19:15.661 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T16:19:15.983 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-77', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:19:15.983 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-77', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:19:15.983 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-77 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:19:16.033 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:19:16.033 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:19:16.076 INFO:teuthology.orchestra.run.smithi134.stdout:tenant: 2024-08-18T16:19:16.077 INFO:teuthology.orchestra.run.smithi134.stdout:bucket name: kibgmb-77 2024-08-18T16:19:16.082 INFO:teuthology.orchestra.run.smithi134.stdout:total entries: 23 2024-08-18T16:19:16.132 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--num-shards', '19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:19:16.133 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--num-shards', '19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:19:16.133 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --num-shards 19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:19:16.181 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:19:16.181 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:19:16.223 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T16:19:16.223 INFO:teuthology.orchestra.run.smithi134.stdout: "objs": [ 2024-08-18T16:19:16.223 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000005", 2024-08-18T16:19:16.223 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000010", 2024-08-18T16:19:16.223 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000016", 2024-08-18T16:19:16.223 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000021", 2024-08-18T16:19:16.223 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000004", 2024-08-18T16:19:16.223 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000025", 2024-08-18T16:19:16.223 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000030", 2024-08-18T16:19:16.223 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000015", 2024-08-18T16:19:16.223 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000019", 2024-08-18T16:19:16.224 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000003", 2024-08-18T16:19:16.224 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000009", 2024-08-18T16:19:16.224 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000014", 2024-08-18T16:19:16.224 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000012", 2024-08-18T16:19:16.224 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000002", 2024-08-18T16:19:16.224 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000000", 2024-08-18T16:19:16.224 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000006", 2024-08-18T16:19:16.224 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000011", 2024-08-18T16:19:16.224 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000001", 2024-08-18T16:19:16.224 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000007" 2024-08-18T16:19:16.224 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T16:19:16.224 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T16:19:16.667 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-77', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:19:16.667 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-77', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:19:16.667 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-77 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:19:16.720 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:19:16.720 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:19:16.766 INFO:teuthology.orchestra.run.smithi134.stdout:tenant: 2024-08-18T16:19:16.767 INFO:teuthology.orchestra.run.smithi134.stdout:bucket name: kibgmb-77 2024-08-18T16:19:16.773 INFO:teuthology.orchestra.run.smithi134.stdout:total entries: 27 2024-08-18T16:19:16.821 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--num-shards', '23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:19:16.821 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--num-shards', '23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:19:16.822 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --num-shards 23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:19:16.870 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:19:16.870 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:19:16.912 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T16:19:16.913 INFO:teuthology.orchestra.run.smithi134.stdout: "objs": [ 2024-08-18T16:19:16.913 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000058", 2024-08-18T16:19:16.913 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000026", 2024-08-18T16:19:16.913 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000011", 2024-08-18T16:19:16.913 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000046", 2024-08-18T16:19:16.913 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000014", 2024-08-18T16:19:16.913 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000050", 2024-08-18T16:19:16.913 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000032", 2024-08-18T16:19:16.913 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000002", 2024-08-18T16:19:16.913 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000039", 2024-08-18T16:19:16.913 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000007", 2024-08-18T16:19:16.913 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000006", 2024-08-18T16:19:16.913 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000040", 2024-08-18T16:19:16.914 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000009", 2024-08-18T16:19:16.914 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000045", 2024-08-18T16:19:16.914 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000013", 2024-08-18T16:19:16.914 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000012", 2024-08-18T16:19:16.914 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000033", 2024-08-18T16:19:16.914 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000001", 2024-08-18T16:19:16.914 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000000", 2024-08-18T16:19:16.914 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000021", 2024-08-18T16:19:16.914 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000003", 2024-08-18T16:19:16.914 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000055", 2024-08-18T16:19:16.914 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000008" 2024-08-18T16:19:16.914 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T16:19:16.915 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T16:19:17.407 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-77', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:19:17.407 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-77', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:19:17.407 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-77 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:19:17.458 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:19:17.459 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:19:17.506 INFO:teuthology.orchestra.run.smithi134.stdout:tenant: 2024-08-18T16:19:17.506 INFO:teuthology.orchestra.run.smithi134.stdout:bucket name: kibgmb-77 2024-08-18T16:19:17.514 INFO:teuthology.orchestra.run.smithi134.stdout:total entries: 35 2024-08-18T16:19:17.568 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--num-shards', '29', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:19:17.569 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--num-shards', '29', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:19:17.569 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --num-shards 29 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:19:17.618 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:19:17.618 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:19:17.662 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T16:19:17.662 INFO:teuthology.orchestra.run.smithi134.stdout: "objs": [ 2024-08-18T16:19:17.662 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000006", 2024-08-18T16:19:17.662 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000059", 2024-08-18T16:19:17.662 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000027", 2024-08-18T16:19:17.663 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000004", 2024-08-18T16:19:17.663 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000011", 2024-08-18T16:19:17.663 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000002", 2024-08-18T16:19:17.663 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000019", 2024-08-18T16:19:17.663 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000049", 2024-08-18T16:19:17.663 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000017", 2024-08-18T16:19:17.663 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000008", 2024-08-18T16:19:17.663 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000015", 2024-08-18T16:19:17.663 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000031", 2024-08-18T16:19:17.663 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000022", 2024-08-18T16:19:17.663 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000013", 2024-08-18T16:19:17.663 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000020", 2024-08-18T16:19:17.663 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000037", 2024-08-18T16:19:17.664 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000005", 2024-08-18T16:19:17.664 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000012", 2024-08-18T16:19:17.664 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000003", 2024-08-18T16:19:17.664 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000042", 2024-08-18T16:19:17.664 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000010", 2024-08-18T16:19:17.664 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000001", 2024-08-18T16:19:17.664 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000009", 2024-08-18T16:19:17.664 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000025", 2024-08-18T16:19:17.665 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000016", 2024-08-18T16:19:17.665 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000000", 2024-08-18T16:19:17.665 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000014", 2024-08-18T16:19:17.665 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000030", 2024-08-18T16:19:17.665 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000021" 2024-08-18T16:19:17.665 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T16:19:17.665 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T16:19:18.283 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-77', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:19:18.283 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-77', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:19:18.283 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-77 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:19:18.333 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:19:18.334 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:19:18.382 INFO:teuthology.orchestra.run.smithi134.stdout:tenant: 2024-08-18T16:19:18.382 INFO:teuthology.orchestra.run.smithi134.stdout:bucket name: kibgmb-77 2024-08-18T16:19:18.392 INFO:teuthology.orchestra.run.smithi134.stdout:total entries: 43 2024-08-18T16:19:18.451 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--num-shards', '31', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:19:18.451 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--num-shards', '31', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:19:18.451 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --num-shards 31 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:19:18.499 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:19:18.500 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:19:18.541 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T16:19:18.541 INFO:teuthology.orchestra.run.smithi134.stdout: "objs": [ 2024-08-18T16:19:18.542 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000019", 2024-08-18T16:19:18.542 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000006", 2024-08-18T16:19:18.542 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000467", 2024-08-18T16:19:18.542 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000005", 2024-08-18T16:19:18.542 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000056", 2024-08-18T16:19:18.542 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000004", 2024-08-18T16:19:18.542 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000023", 2024-08-18T16:19:18.542 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000317", 2024-08-18T16:19:18.542 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000022", 2024-08-18T16:19:18.543 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000000", 2024-08-18T16:19:18.543 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000021", 2024-08-18T16:19:18.543 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000009", 2024-08-18T16:19:18.543 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000271", 2024-08-18T16:19:18.543 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000008", 2024-08-18T16:19:18.543 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000018", 2024-08-18T16:19:18.543 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000007", 2024-08-18T16:19:18.543 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000017", 2024-08-18T16:19:18.543 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000162", 2024-08-18T16:19:18.543 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000016", 2024-08-18T16:19:18.543 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000003", 2024-08-18T16:19:18.543 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000423", 2024-08-18T16:19:18.543 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000002", 2024-08-18T16:19:18.544 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000012", 2024-08-18T16:19:18.544 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000001", 2024-08-18T16:19:18.544 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000011", 2024-08-18T16:19:18.544 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000283", 2024-08-18T16:19:18.544 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000010", 2024-08-18T16:19:18.544 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000061", 2024-08-18T16:19:18.544 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000050", 2024-08-18T16:19:18.544 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000029", 2024-08-18T16:19:18.544 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000133" 2024-08-18T16:19:18.544 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T16:19:18.544 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T16:19:19.122 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2024-08-18T16:19:19.208 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-77', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:19:19.208 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-77', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:19:19.208 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-77 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:19:19.258 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:19:19.258 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:19:19.307 INFO:teuthology.orchestra.run.smithi134.stdout:tenant: 2024-08-18T16:19:19.307 INFO:teuthology.orchestra.run.smithi134.stdout:bucket name: kibgmb-77 2024-08-18T16:19:19.318 INFO:teuthology.orchestra.run.smithi134.stdout:total entries: 55 2024-08-18T16:19:19.381 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--num-shards', '37', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:19:19.382 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--num-shards', '37', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:19:19.382 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --num-shards 37 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:19:19.431 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:19:19.431 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:19:19.473 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T16:19:19.473 INFO:teuthology.orchestra.run.smithi134.stdout: "objs": [ 2024-08-18T16:19:19.473 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000060", 2024-08-18T16:19:19.473 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000000", 2024-08-18T16:19:19.473 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000011", 2024-08-18T16:19:19.473 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000001", 2024-08-18T16:19:19.473 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000033", 2024-08-18T16:19:19.473 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000065", 2024-08-18T16:19:19.473 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000005", 2024-08-18T16:19:19.474 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000016", 2024-08-18T16:19:19.474 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000048", 2024-08-18T16:19:19.474 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000015", 2024-08-18T16:19:19.474 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000026", 2024-08-18T16:19:19.474 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000030", 2024-08-18T16:19:19.474 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000062", 2024-08-18T16:19:19.474 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000009", 2024-08-18T16:19:19.474 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000040", 2024-08-18T16:19:19.474 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000024", 2024-08-18T16:19:19.474 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000012", 2024-08-18T16:19:19.474 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000002", 2024-08-18T16:19:19.474 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000013", 2024-08-18T16:19:19.474 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000045", 2024-08-18T16:19:19.475 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000006", 2024-08-18T16:19:19.475 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000017", 2024-08-18T16:19:19.475 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000007", 2024-08-18T16:19:19.475 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000039", 2024-08-18T16:19:19.475 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000027", 2024-08-18T16:19:19.475 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000010", 2024-08-18T16:19:19.475 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000021", 2024-08-18T16:19:19.475 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000053", 2024-08-18T16:19:19.475 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000020", 2024-08-18T16:19:19.475 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000004", 2024-08-18T16:19:19.475 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000036", 2024-08-18T16:19:19.475 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000003", 2024-08-18T16:19:19.475 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000014", 2024-08-18T16:19:19.475 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000019", 2024-08-18T16:19:19.476 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000057", 2024-08-18T16:19:19.476 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000018", 2024-08-18T16:19:19.476 INFO:teuthology.orchestra.run.smithi134.stdout: "obj00000000000000000008" 2024-08-18T16:19:19.476 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T16:19:19.476 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T16:19:20.400 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'run', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-77', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:19:20.401 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'run', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-77', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:19:20.401 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync run --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-77 --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-08-18T16:19:20.459 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:19:20.459 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:19:20.541 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.538+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.541 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.538+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.541 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.538+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: sync status for source bucket shard: 2 2024-08-18T16:19:20.542 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.538+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: listing bilog for incremental sync; position=00000000001.83.6 2024-08-18T16:19:20.542 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.538+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.542 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.538+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: sync status for source bucket shard: 2 2024-08-18T16:19:20.542 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.538+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: listing bilog for incremental sync; position=00000000001.95.6 2024-08-18T16:19:20.542 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.539+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.542 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.539+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.542 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.539+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: sync status for source bucket shard: 2 2024-08-18T16:19:20.542 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.539+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: listing bilog for incremental sync; position=00000000001.84.6 2024-08-18T16:19:20.542 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.539+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.542 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.539+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: sync status for source bucket shard: 2 2024-08-18T16:19:20.542 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.539+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: listing bilog for incremental sync; position=00000000001.60.6 2024-08-18T16:19:20.543 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.539+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.543 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.539+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: sync status for source bucket shard: 2 2024-08-18T16:19:20.543 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.539+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: listing bilog for incremental sync; position=00000000001.85.6 2024-08-18T16:19:20.543 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.539+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.543 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.539+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: sync status for source bucket shard: 2 2024-08-18T16:19:20.543 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.539+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: listing bilog for incremental sync; position=00000000001.76.6 2024-08-18T16:19:20.543 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.540+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.543 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.540+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.543 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.540+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: sync status for source bucket shard: 2 2024-08-18T16:19:20.543 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.540+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: listing bilog for incremental sync; position=00000000001.74.6 2024-08-18T16:19:20.543 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.540+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: sync status for source bucket shard: 2 2024-08-18T16:19:20.543 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.540+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: listing bilog for incremental sync; position=00000000001.93.6 2024-08-18T16:19:20.544 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.540+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.544 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.540+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: sync status for source bucket shard: 2 2024-08-18T16:19:20.544 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.540+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: listing bilog for incremental sync; position=00000000001.43.6 2024-08-18T16:19:20.544 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.540+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: sync status for source bucket shard: 2 2024-08-18T16:19:20.544 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.540+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.544 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.541+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: sync status for source bucket shard: 2 2024-08-18T16:19:20.544 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.541+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: listing bilog for incremental sync; position=00000000001.61.6 2024-08-18T16:19:20.582 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.574+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.582 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.574+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.582 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.574+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: sync status for source bucket shard: 0 2024-08-18T16:19:20.582 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.574+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.582 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.575+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.582 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.575+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: sync status for source bucket shard: 0 2024-08-18T16:19:20.582 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.575+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.582 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.575+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.583 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.575+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: sync status for source bucket shard: 0 2024-08-18T16:19:20.583 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.575+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.583 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.575+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.583 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.575+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: sync status for source bucket shard: 0 2024-08-18T16:19:20.583 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.575+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.583 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.575+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.583 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.575+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: sync status for source bucket shard: 0 2024-08-18T16:19:20.583 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.575+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.583 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.575+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.583 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.576+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: sync status for source bucket shard: 0 2024-08-18T16:19:20.584 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.576+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.584 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.576+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.584 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.576+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: sync status for source bucket shard: 0 2024-08-18T16:19:20.584 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.576+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.584 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.576+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.584 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.576+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: sync status for source bucket shard: 0 2024-08-18T16:19:20.584 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.576+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.584 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.576+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.584 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.576+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: sync status for source bucket shard: 0 2024-08-18T16:19:20.584 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.576+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.584 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.576+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.585 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.577+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: sync status for source bucket shard: 0 2024-08-18T16:19:20.585 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.577+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.585 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.577+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.585 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.577+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: sync status for source bucket shard: 0 2024-08-18T16:19:20.585 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.577+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.585 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.577+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.585 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.577+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: sync status for source bucket shard: 0 2024-08-18T16:19:20.585 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.577+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.586 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.577+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.586 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.577+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]: sync status for source bucket shard: 0 2024-08-18T16:19:20.586 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.577+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.586 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.577+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.586 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.577+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: sync status for source bucket shard: 0 2024-08-18T16:19:20.586 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.577+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.586 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.578+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.586 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.578+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]: sync status for source bucket shard: 0 2024-08-18T16:19:20.586 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.578+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.587 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.578+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.587 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.578+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: sync status for source bucket shard: 0 2024-08-18T16:19:20.587 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.578+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.587 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.578+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: sync status for source bucket shard: 0 2024-08-18T16:19:20.587 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.578+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.587 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.578+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: parsed entry: id=00000000002.69.6 iter->object=obj00000000000000000013 iter->instance= name=obj00000000000000000013 instance= ns= 2024-08-18T16:19:20.587 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.578+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10/obj00000000000000000013 2024-08-18T16:19:20.587 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.578+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: entry->timestamp=2024-08-18T16:19:15.734909+0000 2024-08-18T16:19:20.587 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.578+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:entry[obj00000000000000000013]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.69.6 op=0 op_state=1 2024-08-18T16:19:20.587 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.578+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:entry[obj00000000000000000013]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000013[0] 2024-08-18T16:19:20.588 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.578+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: listing bilog for incremental sync; position=00000000002.69.6 2024-08-18T16:19:20.588 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.579+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: parsed entry: id=00000000001.85.6 iter->object=obj00000000000000000030 iter->instance= name=obj00000000000000000030 instance= ns= 2024-08-18T16:19:20.588 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.579+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12/obj00000000000000000030 2024-08-18T16:19:20.588 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.579+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: entry->timestamp=2024-08-18T16:19:15.806045+0000 2024-08-18T16:19:20.588 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.579+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:entry[obj00000000000000000030]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.85.6 op=0 op_state=1 2024-08-18T16:19:20.588 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.579+0000 7f63ec126880 0 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:entry[obj00000000000000000030]: : injecting data sync error on key=obj00000000000000000030 2024-08-18T16:19:20.588 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.579+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: listing bilog for incremental sync; position=00000000001.85.6 2024-08-18T16:19:20.589 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.579+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:entry[obj00000000000000000030]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:19:20.589 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.579+0000 7f63ec126880 0 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:entry[obj00000000000000000030]: ERROR: failed to sync object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12/obj00000000000000000030 2024-08-18T16:19:20.589 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.579+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]: parsed entry: id=00000000001.87.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2024-08-18T16:19:20.589 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.579+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11/obj00000000000000000021 2024-08-18T16:19:20.589 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.579+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]: entry->timestamp=2024-08-18T16:19:15.911763+0000 2024-08-18T16:19:20.589 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.579+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.87.6 op=0 op_state=1 2024-08-18T16:19:20.589 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.579+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:entry[obj00000000000000000021]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000021[0] 2024-08-18T16:19:20.590 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.579+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]: listing bilog for incremental sync; position=00000000001.87.6 2024-08-18T16:19:20.590 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.579+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: parsed entry: id=00000000001.109.6 iter->object=obj00000000000000000033 iter->instance= name=obj00000000000000000033 instance= ns= 2024-08-18T16:19:20.590 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.579+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14/obj00000000000000000033 2024-08-18T16:19:20.590 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.579+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: entry->timestamp=2024-08-18T16:19:15.918835+0000 2024-08-18T16:19:20.590 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.579+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:entry[obj00000000000000000033]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.109.6 op=0 op_state=1 2024-08-18T16:19:20.590 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.579+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:entry[obj00000000000000000033]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000033[0] 2024-08-18T16:19:20.590 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.579+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: listing bilog for incremental sync; position=00000000001.109.6 2024-08-18T16:19:20.591 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.580+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]: parsed entry: id=00000000001.90.6 iter->object=obj00000000000000000026 iter->instance= name=obj00000000000000000026 instance= ns= 2024-08-18T16:19:20.591 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.580+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13/obj00000000000000000026 2024-08-18T16:19:20.591 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.580+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]: entry->timestamp=2024-08-18T16:19:15.798609+0000 2024-08-18T16:19:20.591 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.580+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:entry[obj00000000000000000026]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.90.6 op=0 op_state=1 2024-08-18T16:19:20.591 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.580+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:entry[obj00000000000000000026]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000026[0] 2024-08-18T16:19:20.591 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.580+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]: listing bilog for incremental sync; position=00000000001.90.6 2024-08-18T16:19:20.591 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.580+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: a sync operation returned error 2024-08-18T16:19:20.591 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.580+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: backing out with sync_status=-5 2024-08-18T16:19:20.591 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.580+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: incremental sync on bucket failed, retcode=-5 2024-08-18T16:19:20.592 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.580+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: ERROR: incremental sync failed. error: -5 2024-08-18T16:19:20.592 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.580+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.592 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.581+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: sync status for source bucket shard: 0 2024-08-18T16:19:20.592 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.581+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.593 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.590+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:entry[obj00000000000000000021]: success 2024-08-18T16:19:20.593 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.590+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11:1 marker=00000000001.87.6 timestamp=2024-08-18T16:19:15.911763+0000 2024-08-18T16:19:20.593 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.590+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:entry[obj00000000000000000033]: success 2024-08-18T16:19:20.593 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.590+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14:1 marker=00000000001.109.6 timestamp=2024-08-18T16:19:15.918835+0000 2024-08-18T16:19:20.593 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.591+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:entry[obj00000000000000000013]: success 2024-08-18T16:19:20.594 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.591+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10:1 marker=00000000002.69.6 timestamp=2024-08-18T16:19:15.734909+0000 2024-08-18T16:19:20.594 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.591+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:entry[obj00000000000000000026]: success 2024-08-18T16:19:20.594 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.591+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13:1 marker=00000000001.90.6 timestamp=2024-08-18T16:19:15.798609+0000 2024-08-18T16:19:20.619 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.617+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: parsed entry: id=00000000001.86.6 iter->object=obj00000000000000000028 iter->instance= name=obj00000000000000000028 instance= ns= 2024-08-18T16:19:20.619 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.617+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4/obj00000000000000000028 2024-08-18T16:19:20.619 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.617+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: entry->timestamp=2024-08-18T16:19:15.858769+0000 2024-08-18T16:19:20.619 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.617+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:entry[obj00000000000000000028]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.86.6 op=0 op_state=1 2024-08-18T16:19:20.619 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.617+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:entry[obj00000000000000000028]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000028[0] 2024-08-18T16:19:20.619 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.617+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: listing bilog for incremental sync; position=00000000001.86.6 2024-08-18T16:19:20.620 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.617+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: parsed entry: id=00000000001.87.6 iter->object=obj00000000000000000024 iter->instance= name=obj00000000000000000024 instance= ns= 2024-08-18T16:19:20.620 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.617+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2/obj00000000000000000024 2024-08-18T16:19:20.620 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.617+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: entry->timestamp=2024-08-18T16:19:15.851851+0000 2024-08-18T16:19:20.620 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.617+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:entry[obj00000000000000000024]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.87.6 op=0 op_state=1 2024-08-18T16:19:20.620 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.617+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: parsed entry: id=00000000001.105.6 iter->object=obj00000000000000000016 iter->instance= name=obj00000000000000000016 instance= ns= 2024-08-18T16:19:20.620 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.617+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1/obj00000000000000000016 2024-08-18T16:19:20.620 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.617+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: entry->timestamp=2024-08-18T16:19:15.677788+0000 2024-08-18T16:19:20.620 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.617+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:entry[obj00000000000000000016]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.105.6 op=0 op_state=1 2024-08-18T16:19:20.620 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.617+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: parsed entry: id=00000000001.103.6 iter->object=obj00000000000000000007 iter->instance= name=obj00000000000000000007 instance= ns= 2024-08-18T16:19:20.620 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.617+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0/obj00000000000000000007 2024-08-18T16:19:20.620 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.617+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: entry->timestamp=2024-08-18T16:19:15.668069+0000 2024-08-18T16:19:20.621 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.617+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:entry[obj00000000000000000007]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.103.6 op=0 op_state=1 2024-08-18T16:19:20.621 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.617+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: parsed entry: id=00000000001.117.6 iter->object=obj00000000000000000020 iter->instance= name=obj00000000000000000020 instance= ns= 2024-08-18T16:19:20.621 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.617+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3/obj00000000000000000020 2024-08-18T16:19:20.621 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.617+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: entry->timestamp=2024-08-18T16:19:15.791409+0000 2024-08-18T16:19:20.621 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.617+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:entry[obj00000000000000000020]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.117.6 op=0 op_state=1 2024-08-18T16:19:20.621 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.617+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:entry[obj00000000000000000024]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000024[0] 2024-08-18T16:19:20.621 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.617+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: listing bilog for incremental sync; position=00000000001.87.6 2024-08-18T16:19:20.621 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.617+0000 7f63ec126880 0 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:entry[obj00000000000000000016]: : injecting data sync error on key=obj00000000000000000016 2024-08-18T16:19:20.621 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.617+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: listing bilog for incremental sync; position=00000000001.105.6 2024-08-18T16:19:20.621 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.617+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:entry[obj00000000000000000007]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000007[0] 2024-08-18T16:19:20.621 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.617+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: listing bilog for incremental sync; position=00000000001.103.6 2024-08-18T16:19:20.622 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.617+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:entry[obj00000000000000000020]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000020[0] 2024-08-18T16:19:20.622 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.617+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: listing bilog for incremental sync; position=00000000001.117.6 2024-08-18T16:19:20.622 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.617+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:entry[obj00000000000000000016]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:19:20.622 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.617+0000 7f63ec126880 0 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:entry[obj00000000000000000016]: ERROR: failed to sync object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1/obj00000000000000000016 2024-08-18T16:19:20.622 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.618+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: parsed entry: id=00000000001.161.6 iter->object=obj00000000000000000032 iter->instance= name=obj00000000000000000032 instance= ns= 2024-08-18T16:19:20.622 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.618+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6/obj00000000000000000032 2024-08-18T16:19:20.622 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.618+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: entry->timestamp=2024-08-18T16:19:15.971787+0000 2024-08-18T16:19:20.622 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.618+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:entry[obj00000000000000000032]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.161.6 op=0 op_state=1 2024-08-18T16:19:20.622 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.618+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: parsed entry: id=00000000002.70.6 iter->object=obj00000000000000000000 iter->instance= name=obj00000000000000000000 instance= ns= 2024-08-18T16:19:20.622 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.618+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5/obj00000000000000000000 2024-08-18T16:19:20.622 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.618+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: entry->timestamp=2024-08-18T16:19:15.979352+0000 2024-08-18T16:19:20.623 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.618+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:entry[obj00000000000000000000]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.70.6 op=0 op_state=1 2024-08-18T16:19:20.623 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.618+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: parsed entry: id=00000000002.138.6 iter->object=obj00000000000000000010 iter->instance= name=obj00000000000000000010 instance= ns= 2024-08-18T16:19:20.623 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.618+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8/obj00000000000000000010 2024-08-18T16:19:20.623 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.618+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: entry->timestamp=2024-08-18T16:19:15.866337+0000 2024-08-18T16:19:20.623 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.618+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:entry[obj00000000000000000010]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.138.6 op=0 op_state=1 2024-08-18T16:19:20.623 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.618+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:entry[obj00000000000000000032]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000032[0] 2024-08-18T16:19:20.623 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.618+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: listing bilog for incremental sync; position=00000000001.161.6 2024-08-18T16:19:20.623 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.618+0000 7f63ec126880 0 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:entry[obj00000000000000000000]: : injecting data sync error on key=obj00000000000000000000 2024-08-18T16:19:20.624 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.618+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: listing bilog for incremental sync; position=00000000002.70.6 2024-08-18T16:19:20.624 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.618+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:entry[obj00000000000000000010]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000010[0] 2024-08-18T16:19:20.624 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.618+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: listing bilog for incremental sync; position=00000000002.138.6 2024-08-18T16:19:20.624 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.618+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:entry[obj00000000000000000000]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:19:20.624 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.618+0000 7f63ec126880 0 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:entry[obj00000000000000000000]: ERROR: failed to sync object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5/obj00000000000000000000 2024-08-18T16:19:20.626 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.620+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: parsed entry: id=00000000001.56.6 iter->object=obj00000000000000000019 iter->instance= name=obj00000000000000000019 instance= ns= 2024-08-18T16:19:20.626 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.620+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9/obj00000000000000000019 2024-08-18T16:19:20.626 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.620+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: entry->timestamp=2024-08-18T16:19:15.728112+0000 2024-08-18T16:19:20.626 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.620+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:entry[obj00000000000000000019]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.56.6 op=0 op_state=1 2024-08-18T16:19:20.626 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.620+0000 7f63ec126880 0 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:entry[obj00000000000000000019]: : injecting data sync error on key=obj00000000000000000019 2024-08-18T16:19:20.626 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.620+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: listing bilog for incremental sync; position=00000000001.56.6 2024-08-18T16:19:20.626 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.620+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: parsed entry: id=00000000001.88.6 iter->object=obj00000000000000000076 iter->instance= name=obj00000000000000000076 instance= ns= 2024-08-18T16:19:20.626 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.620+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16/obj00000000000000000076 2024-08-18T16:19:20.626 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.620+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: entry->timestamp=2024-08-18T16:19:15.743831+0000 2024-08-18T16:19:20.627 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.620+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:entry[obj00000000000000000076]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.88.6 op=0 op_state=1 2024-08-18T16:19:20.627 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.620+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: parsed entry: id=00000000002.131.6 iter->object=obj00000000000000000004 iter->instance= name=obj00000000000000000004 instance= ns= 2024-08-18T16:19:20.627 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.620+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15/obj00000000000000000004 2024-08-18T16:19:20.627 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.620+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: entry->timestamp=2024-08-18T16:19:15.925675+0000 2024-08-18T16:19:20.627 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.620+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:entry[obj00000000000000000004]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.131.6 op=0 op_state=1 2024-08-18T16:19:20.627 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.620+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:entry[obj00000000000000000019]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:19:20.627 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.620+0000 7f63ec126880 0 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:entry[obj00000000000000000019]: ERROR: failed to sync object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9/obj00000000000000000019 2024-08-18T16:19:20.627 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.620+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:entry[obj00000000000000000076]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000076[0] 2024-08-18T16:19:20.628 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.620+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: listing bilog for incremental sync; position=00000000001.88.6 2024-08-18T16:19:20.628 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.620+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:entry[obj00000000000000000004]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000004[0] 2024-08-18T16:19:20.628 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.620+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: listing bilog for incremental sync; position=00000000002.131.6 2024-08-18T16:19:20.628 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.623+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: parsed entry: id=00000000001.85.6 iter->object=obj00000000000000000030 iter->instance= name=obj00000000000000000030 instance= ns= 2024-08-18T16:19:20.628 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.623+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12/obj00000000000000000030 2024-08-18T16:19:20.628 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.623+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: entry->timestamp=2024-08-18T16:19:15.806045+0000 2024-08-18T16:19:20.628 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.623+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:entry[obj00000000000000000030]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.85.6 op=0 op_state=1 2024-08-18T16:19:20.628 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.623+0000 7f63ec126880 0 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:entry[obj00000000000000000030]: : injecting data sync error on key=obj00000000000000000030 2024-08-18T16:19:20.628 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.623+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: listing bilog for incremental sync; position=00000000001.85.6 2024-08-18T16:19:20.628 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.623+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:entry[obj00000000000000000030]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:19:20.628 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.623+0000 7f63ec126880 0 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:entry[obj00000000000000000030]: ERROR: failed to sync object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12/obj00000000000000000030 2024-08-18T16:19:20.629 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.626+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:entry[obj00000000000000000007]: success 2024-08-18T16:19:20.629 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.626+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0:1 marker=00000000001.103.6 timestamp=2024-08-18T16:19:15.668069+0000 2024-08-18T16:19:20.629 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.628+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:entry[obj00000000000000000028]: success 2024-08-18T16:19:20.629 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.628+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4:1 marker=00000000001.86.6 timestamp=2024-08-18T16:19:15.858769+0000 2024-08-18T16:19:20.631 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.630+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:entry[obj00000000000000000020]: success 2024-08-18T16:19:20.631 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.630+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3:1 marker=00000000001.117.6 timestamp=2024-08-18T16:19:15.791409+0000 2024-08-18T16:19:20.631 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.630+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:entry[obj00000000000000000010]: success 2024-08-18T16:19:20.631 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.630+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8:1 marker=00000000002.138.6 timestamp=2024-08-18T16:19:15.866337+0000 2024-08-18T16:19:20.631 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.630+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:entry[obj00000000000000000032]: success 2024-08-18T16:19:20.631 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.630+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6:1 marker=00000000001.161.6 timestamp=2024-08-18T16:19:15.971787+0000 2024-08-18T16:19:20.662 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.661+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: a sync operation returned error 2024-08-18T16:19:20.662 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.661+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: a sync operation returned error 2024-08-18T16:19:20.662 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.661+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: backing out with sync_status=-5 2024-08-18T16:19:20.662 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.661+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: backing out with sync_status=-5 2024-08-18T16:19:20.662 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.661+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: incremental sync on bucket failed, retcode=-5 2024-08-18T16:19:20.663 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.661+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: incremental sync on bucket failed, retcode=-5 2024-08-18T16:19:20.663 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.661+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: ERROR: incremental sync failed. error: -5 2024-08-18T16:19:20.663 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.661+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: ERROR: incremental sync failed. error: -5 2024-08-18T16:19:20.663 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.662+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: a sync operation returned error 2024-08-18T16:19:20.663 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.662+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: backing out with sync_status=-5 2024-08-18T16:19:20.663 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.662+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: incremental sync on bucket failed, retcode=-5 2024-08-18T16:19:20.663 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.662+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: ERROR: incremental sync failed. error: -5 2024-08-18T16:19:20.664 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.662+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.664 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.662+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.664 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.663+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: sync status for source bucket shard: 0 2024-08-18T16:19:20.664 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.663+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.664 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.663+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: sync status for source bucket shard: 0 2024-08-18T16:19:20.665 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.663+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.666 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.664+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.666 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.665+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: a sync operation returned error 2024-08-18T16:19:20.666 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.665+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: backing out with sync_status=-5 2024-08-18T16:19:20.666 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.665+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: incremental sync on bucket failed, retcode=-5 2024-08-18T16:19:20.666 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.665+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: ERROR: incremental sync failed. error: -5 2024-08-18T16:19:20.666 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.665+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:entry[obj00000000000000000024]: success 2024-08-18T16:19:20.666 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.665+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2:1 marker=00000000001.87.6 timestamp=2024-08-18T16:19:15.851851+0000 2024-08-18T16:19:20.666 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.665+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: sync status for source bucket shard: 0 2024-08-18T16:19:20.667 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.665+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.668 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.668+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:entry[obj00000000000000000004]: success 2024-08-18T16:19:20.668 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.668+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15:1 marker=00000000002.131.6 timestamp=2024-08-18T16:19:15.925675+0000 2024-08-18T16:19:20.669 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.668+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:entry[obj00000000000000000076]: success 2024-08-18T16:19:20.669 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.668+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16:1 marker=00000000001.88.6 timestamp=2024-08-18T16:19:15.743831+0000 2024-08-18T16:19:20.669 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.668+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.670 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.669+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: sync status for source bucket shard: 0 2024-08-18T16:19:20.670 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.669+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.705 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.705+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: parsed entry: id=00000000002.70.6 iter->object=obj00000000000000000000 iter->instance= name=obj00000000000000000000 instance= ns= 2024-08-18T16:19:20.706 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.705+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5/obj00000000000000000000 2024-08-18T16:19:20.706 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.705+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: entry->timestamp=2024-08-18T16:19:15.979352+0000 2024-08-18T16:19:20.706 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.705+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:entry[obj00000000000000000000]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.70.6 op=0 op_state=1 2024-08-18T16:19:20.706 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.705+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:entry[obj00000000000000000000]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000000[0] 2024-08-18T16:19:20.706 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.705+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: listing bilog for incremental sync; position=00000000002.70.6 2024-08-18T16:19:20.706 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.705+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: parsed entry: id=00000000001.105.6 iter->object=obj00000000000000000016 iter->instance= name=obj00000000000000000016 instance= ns= 2024-08-18T16:19:20.706 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.705+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1/obj00000000000000000016 2024-08-18T16:19:20.706 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.705+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: entry->timestamp=2024-08-18T16:19:15.677788+0000 2024-08-18T16:19:20.707 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.705+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:entry[obj00000000000000000016]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.105.6 op=0 op_state=1 2024-08-18T16:19:20.707 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.705+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:entry[obj00000000000000000016]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000016[0] 2024-08-18T16:19:20.707 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.705+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: listing bilog for incremental sync; position=00000000001.105.6 2024-08-18T16:19:20.708 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.708+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: parsed entry: id=00000000001.56.6 iter->object=obj00000000000000000019 iter->instance= name=obj00000000000000000019 instance= ns= 2024-08-18T16:19:20.708 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.708+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9/obj00000000000000000019 2024-08-18T16:19:20.708 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.708+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: entry->timestamp=2024-08-18T16:19:15.728112+0000 2024-08-18T16:19:20.709 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.708+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:entry[obj00000000000000000019]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.56.6 op=0 op_state=1 2024-08-18T16:19:20.709 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.708+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:entry[obj00000000000000000019]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000019[0] 2024-08-18T16:19:20.709 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.708+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: listing bilog for incremental sync; position=00000000001.56.6 2024-08-18T16:19:20.710 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.710+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:entry[obj00000000000000000000]: success 2024-08-18T16:19:20.711 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.710+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5:1 marker=00000000002.70.6 timestamp=2024-08-18T16:19:15.979352+0000 2024-08-18T16:19:20.712 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.711+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: parsed entry: id=00000000001.85.6 iter->object=obj00000000000000000030 iter->instance= name=obj00000000000000000030 instance= ns= 2024-08-18T16:19:20.712 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.711+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12/obj00000000000000000030 2024-08-18T16:19:20.712 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.711+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: entry->timestamp=2024-08-18T16:19:15.806045+0000 2024-08-18T16:19:20.712 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.711+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:entry[obj00000000000000000030]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.85.6 op=0 op_state=1 2024-08-18T16:19:20.712 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.711+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:entry[obj00000000000000000030]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000030[0] 2024-08-18T16:19:20.712 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.711+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: listing bilog for incremental sync; position=00000000001.85.6 2024-08-18T16:19:20.713 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.712+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:entry[obj00000000000000000016]: success 2024-08-18T16:19:20.713 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.712+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1:1 marker=00000000001.105.6 timestamp=2024-08-18T16:19:15.677788+0000 2024-08-18T16:19:20.713 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.713+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:entry[obj00000000000000000019]: success 2024-08-18T16:19:20.713 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.713+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9:1 marker=00000000001.56.6 timestamp=2024-08-18T16:19:15.728112+0000 2024-08-18T16:19:20.715 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.715+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:entry[obj00000000000000000030]: success 2024-08-18T16:19:20.715 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.715+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12:1 marker=00000000001.85.6 timestamp=2024-08-18T16:19:15.806045+0000 2024-08-18T16:19:20.767 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.756+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.768 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.756+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.768 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.756+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.768 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.756+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: sync status for source bucket shard: 0 2024-08-18T16:19:20.768 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.756+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.768 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.757+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: sync status for source bucket shard: 0 2024-08-18T16:19:20.768 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.757+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.769 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.757+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.769 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.757+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: sync status for source bucket shard: 0 2024-08-18T16:19:20.769 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.757+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.769 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.757+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.769 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.757+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.769 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.757+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: sync status for source bucket shard: 0 2024-08-18T16:19:20.769 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.757+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.769 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.757+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: sync status for source bucket shard: 0 2024-08-18T16:19:20.769 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.757+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.770 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.757+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.770 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.757+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: sync status for source bucket shard: 0 2024-08-18T16:19:20.770 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.757+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.770 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.757+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.770 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.757+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.770 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.758+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: sync status for source bucket shard: 0 2024-08-18T16:19:20.770 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.758+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.770 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.758+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: sync status for source bucket shard: 0 2024-08-18T16:19:20.770 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.758+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.770 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.758+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: sync status for source bucket shard: 0 2024-08-18T16:19:20.770 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.758+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.771 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.758+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.771 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.758+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.771 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.758+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.771 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.758+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: sync status for source bucket shard: 0 2024-08-18T16:19:20.771 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.758+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.771 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.758+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: sync status for source bucket shard: 0 2024-08-18T16:19:20.771 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.758+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.771 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.758+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.771 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.758+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]: sync status for source bucket shard: 0 2024-08-18T16:19:20.772 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.759+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.772 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.759+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.772 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.759+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: sync status for source bucket shard: 0 2024-08-18T16:19:20.772 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.759+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]: sync status for source bucket shard: 0 2024-08-18T16:19:20.772 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.759+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.772 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.759+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.772 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.759+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: parsed entry: id=00000000001.144.6 iter->object=obj00000000000000000025 iter->instance= name=obj00000000000000000025 instance= ns= 2024-08-18T16:19:20.772 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.759+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5/obj00000000000000000025 2024-08-18T16:19:20.772 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.759+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: entry->timestamp=2024-08-18T16:19:16.429591+0000 2024-08-18T16:19:20.772 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.759+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:entry[obj00000000000000000025]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.144.6 op=0 op_state=1 2024-08-18T16:19:20.773 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.759+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.773 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.759+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:entry[obj00000000000000000025]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000025[0] 2024-08-18T16:19:20.773 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.759+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: listing bilog for incremental sync; position=00000000001.144.6 2024-08-18T16:19:20.773 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.759+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.773 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.760+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.773 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.760+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: parsed entry: id=00000000002.146.6 iter->object=obj00000000000000000015 iter->instance= name=obj00000000000000000015 instance= ns= 2024-08-18T16:19:20.773 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.760+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7/obj00000000000000000015 2024-08-18T16:19:20.773 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.760+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: entry->timestamp=2024-08-18T16:19:16.555569+0000 2024-08-18T16:19:20.773 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.760+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:entry[obj00000000000000000015]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.146.6 op=0 op_state=1 2024-08-18T16:19:20.773 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.760+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.773 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.760+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.774 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.760+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:entry[obj00000000000000000015]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000015[0] 2024-08-18T16:19:20.774 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.760+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: listing bilog for incremental sync; position=00000000002.146.6 2024-08-18T16:19:20.774 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.760+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: parsed entry: id=00000000002.130.6 iter->object=obj00000000000000000030 iter->instance= name=obj00000000000000000030 instance= ns= 2024-08-18T16:19:20.774 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.760+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6/obj00000000000000000030 2024-08-18T16:19:20.775 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.760+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: entry->timestamp=2024-08-18T16:19:16.612682+0000 2024-08-18T16:19:20.775 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.760+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:entry[obj00000000000000000030]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.130.6 op=0 op_state=1 2024-08-18T16:19:20.775 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.760+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: parsed entry: id=00000000002.166.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2024-08-18T16:19:20.775 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.760+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3/obj00000000000000000021 2024-08-18T16:19:20.775 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.760+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: entry->timestamp=2024-08-18T16:19:16.547723+0000 2024-08-18T16:19:20.775 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.760+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.166.6 op=0 op_state=1 2024-08-18T16:19:20.775 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.760+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:entry[obj00000000000000000030]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000030[0] 2024-08-18T16:19:20.776 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.760+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: listing bilog for incremental sync; position=00000000002.130.6 2024-08-18T16:19:20.776 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.760+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:entry[obj00000000000000000021]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000021[0] 2024-08-18T16:19:20.776 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.760+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: listing bilog for incremental sync; position=00000000002.166.6 2024-08-18T16:19:20.776 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.760+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: parsed entry: id=00000000002.74.6 iter->object=obj00000000000000000019 iter->instance= name=obj00000000000000000019 instance= ns= 2024-08-18T16:19:20.776 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.760+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8/obj00000000000000000019 2024-08-18T16:19:20.776 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.760+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: entry->timestamp=2024-08-18T16:19:16.421104+0000 2024-08-18T16:19:20.776 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.760+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:entry[obj00000000000000000019]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.74.6 op=0 op_state=1 2024-08-18T16:19:20.776 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.760+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: sync status for source bucket shard: 0 2024-08-18T16:19:20.777 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.760+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:entry[obj00000000000000000019]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000019[0] 2024-08-18T16:19:20.777 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.760+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: listing bilog for incremental sync; position=00000000002.74.6 2024-08-18T16:19:20.777 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.760+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: parsed entry: id=00000000001.95.6 iter->object=obj00000000000000000009 iter->instance= name=obj00000000000000000009 instance= ns= 2024-08-18T16:19:20.777 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.760+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10/obj00000000000000000009 2024-08-18T16:19:20.777 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.760+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: entry->timestamp=2024-08-18T16:19:16.283741+0000 2024-08-18T16:19:20.777 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.760+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:entry[obj00000000000000000009]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.95.6 op=0 op_state=1 2024-08-18T16:19:20.777 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.760+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.777 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.760+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]: parsed entry: id=00000000001.81.6 iter->object=obj00000000000000000014 iter->instance= name=obj00000000000000000014 instance= ns= 2024-08-18T16:19:20.778 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.760+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11/obj00000000000000000014 2024-08-18T16:19:20.778 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.760+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]: entry->timestamp=2024-08-18T16:19:16.298314+0000 2024-08-18T16:19:20.778 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.760+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:entry[obj00000000000000000014]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.81.6 op=0 op_state=1 2024-08-18T16:19:20.778 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.760+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:entry[obj00000000000000000009]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000009[0] 2024-08-18T16:19:20.778 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.760+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: listing bilog for incremental sync; position=00000000001.95.6 2024-08-18T16:19:20.778 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.760+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:entry[obj00000000000000000014]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000014[0] 2024-08-18T16:19:20.779 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.760+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]: listing bilog for incremental sync; position=00000000001.81.6 2024-08-18T16:19:20.779 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.761+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: sync status for source bucket shard: 0 2024-08-18T16:19:20.779 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.761+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: sync status for source bucket shard: 0 2024-08-18T16:19:20.779 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.761+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.779 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.761+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: sync status for source bucket shard: 0 2024-08-18T16:19:20.779 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.761+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: sync status for source bucket shard: 0 2024-08-18T16:19:20.780 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.761+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.780 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.761+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.780 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.761+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.780 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.761+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: parsed entry: id=00000000002.61.6 iter->object=obj00000000000000000003 iter->instance= name=obj00000000000000000003 instance= ns= 2024-08-18T16:19:20.780 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.761+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9/obj00000000000000000003 2024-08-18T16:19:20.780 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.761+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: entry->timestamp=2024-08-18T16:19:16.604379+0000 2024-08-18T16:19:20.781 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.761+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:entry[obj00000000000000000003]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.61.6 op=0 op_state=1 2024-08-18T16:19:20.781 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.761+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:entry[obj00000000000000000003]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000003[0] 2024-08-18T16:19:20.781 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.761+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: listing bilog for incremental sync; position=00000000002.61.6 2024-08-18T16:19:20.781 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.763+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:entry[obj00000000000000000021]: success 2024-08-18T16:19:20.781 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.763+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3:2 marker=00000000002.166.6 timestamp=2024-08-18T16:19:16.547723+0000 2024-08-18T16:19:20.782 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.763+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:entry[obj00000000000000000030]: success 2024-08-18T16:19:20.782 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.763+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6:2 marker=00000000002.130.6 timestamp=2024-08-18T16:19:16.612682+0000 2024-08-18T16:19:20.782 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.764+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: parsed entry: id=00000000002.95.6 iter->object=obj00000000000000000011 iter->instance= name=obj00000000000000000011 instance= ns= 2024-08-18T16:19:20.782 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.764+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16/obj00000000000000000011 2024-08-18T16:19:20.782 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.764+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: entry->timestamp=2024-08-18T16:19:16.480839+0000 2024-08-18T16:19:20.782 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.764+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:entry[obj00000000000000000011]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.95.6 op=0 op_state=1 2024-08-18T16:19:20.782 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.764+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:entry[obj00000000000000000011]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000011[0] 2024-08-18T16:19:20.782 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.764+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: listing bilog for incremental sync; position=00000000002.95.6 2024-08-18T16:19:20.783 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.765+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:entry[obj00000000000000000019]: success 2024-08-18T16:19:20.783 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.765+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8:2 marker=00000000002.74.6 timestamp=2024-08-18T16:19:16.421104+0000 2024-08-18T16:19:20.783 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.765+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: parsed entry: id=00000000002.114.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2024-08-18T16:19:20.783 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.765+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17/obj00000000000000000001 2024-08-18T16:19:20.783 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.765+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: entry->timestamp=2024-08-18T16:19:16.621827+0000 2024-08-18T16:19:20.783 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.765+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.114.6 op=0 op_state=1 2024-08-18T16:19:20.783 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.765+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:entry[obj00000000000000000001]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000001[0] 2024-08-18T16:19:20.783 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.765+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: listing bilog for incremental sync; position=00000000002.114.6 2024-08-18T16:19:20.783 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.765+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: parsed entry: id=00000000002.93.6 iter->object=obj00000000000000000007 iter->instance= name=obj00000000000000000007 instance= ns= 2024-08-18T16:19:20.783 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.765+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18/obj00000000000000000007 2024-08-18T16:19:20.783 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.765+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: entry->timestamp=2024-08-18T16:19:16.290628+0000 2024-08-18T16:19:20.783 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.765+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:entry[obj00000000000000000007]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.93.6 op=0 op_state=1 2024-08-18T16:19:20.784 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.765+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:entry[obj00000000000000000007]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000007[0] 2024-08-18T16:19:20.784 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.765+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: listing bilog for incremental sync; position=00000000002.93.6 2024-08-18T16:19:20.784 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.765+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: parsed entry: id=00000000002.128.6 iter->object=obj00000000000000000006 iter->instance= name=obj00000000000000000006 instance= ns= 2024-08-18T16:19:20.784 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.765+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15/obj00000000000000000006 2024-08-18T16:19:20.784 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.765+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: entry->timestamp=2024-08-18T16:19:16.489682+0000 2024-08-18T16:19:20.784 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.765+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:entry[obj00000000000000000006]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.128.6 op=0 op_state=1 2024-08-18T16:19:20.784 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.765+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:entry[obj00000000000000000006]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000006[0] 2024-08-18T16:19:20.784 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.765+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: listing bilog for incremental sync; position=00000000002.128.6 2024-08-18T16:19:20.784 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.766+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:entry[obj00000000000000000025]: success 2024-08-18T16:19:20.784 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.766+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5:2 marker=00000000001.144.6 timestamp=2024-08-18T16:19:16.429591+0000 2024-08-18T16:19:20.786 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.769+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:entry[obj00000000000000000007]: success 2024-08-18T16:19:20.786 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.769+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18:2 marker=00000000002.93.6 timestamp=2024-08-18T16:19:16.290628+0000 2024-08-18T16:19:20.786 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.771+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:entry[obj00000000000000000015]: success 2024-08-18T16:19:20.786 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.771+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7:2 marker=00000000002.146.6 timestamp=2024-08-18T16:19:16.555569+0000 2024-08-18T16:19:20.786 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.772+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:entry[obj00000000000000000014]: success 2024-08-18T16:19:20.786 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.772+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11:2 marker=00000000001.81.6 timestamp=2024-08-18T16:19:16.298314+0000 2024-08-18T16:19:20.786 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.773+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:entry[obj00000000000000000009]: success 2024-08-18T16:19:20.786 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.773+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10:2 marker=00000000001.95.6 timestamp=2024-08-18T16:19:16.283741+0000 2024-08-18T16:19:20.787 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.773+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:entry[obj00000000000000000001]: success 2024-08-18T16:19:20.787 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.773+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17:2 marker=00000000002.114.6 timestamp=2024-08-18T16:19:16.621827+0000 2024-08-18T16:19:20.787 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.775+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:entry[obj00000000000000000011]: success 2024-08-18T16:19:20.787 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.775+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16:2 marker=00000000002.95.6 timestamp=2024-08-18T16:19:16.480839+0000 2024-08-18T16:19:20.787 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.776+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:entry[obj00000000000000000003]: success 2024-08-18T16:19:20.787 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.776+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9:2 marker=00000000002.61.6 timestamp=2024-08-18T16:19:16.604379+0000 2024-08-18T16:19:20.787 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.779+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:entry[obj00000000000000000006]: success 2024-08-18T16:19:20.787 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.779+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15:2 marker=00000000002.128.6 timestamp=2024-08-18T16:19:16.489682+0000 2024-08-18T16:19:20.800 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.799+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: parsed entry: id=00000000002.126.6 iter->object=obj00000000000000000010 iter->instance= name=obj00000000000000000010 instance= ns= 2024-08-18T16:19:20.800 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.799+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1/obj00000000000000000010 2024-08-18T16:19:20.801 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.799+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: entry->timestamp=2024-08-18T16:19:16.412010+0000 2024-08-18T16:19:20.801 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.799+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:entry[obj00000000000000000010]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.126.6 op=0 op_state=1 2024-08-18T16:19:20.801 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.799+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:entry[obj00000000000000000010]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000010[0] 2024-08-18T16:19:20.801 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.799+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: listing bilog for incremental sync; position=00000000002.126.6 2024-08-18T16:19:20.801 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.799+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: parsed entry: id=00000000002.65.6 iter->object=obj00000000000000000016 iter->instance= name=obj00000000000000000016 instance= ns= 2024-08-18T16:19:20.802 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.799+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2/obj00000000000000000016 2024-08-18T16:19:20.802 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.799+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: entry->timestamp=2024-08-18T16:19:16.360394+0000 2024-08-18T16:19:20.802 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.799+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:entry[obj00000000000000000016]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.65.6 op=0 op_state=1 2024-08-18T16:19:20.802 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.799+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: parsed entry: id=00000000002.67.6 iter->object=obj00000000000000000004 iter->instance= name=obj00000000000000000004 instance= ns= 2024-08-18T16:19:20.802 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.799+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4/obj00000000000000000004 2024-08-18T16:19:20.803 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.799+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: entry->timestamp=2024-08-18T16:19:16.233425+0000 2024-08-18T16:19:20.803 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.799+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:entry[obj00000000000000000004]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.67.6 op=0 op_state=1 2024-08-18T16:19:20.803 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.799+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:entry[obj00000000000000000016]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000016[0] 2024-08-18T16:19:20.803 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.799+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: listing bilog for incremental sync; position=00000000002.65.6 2024-08-18T16:19:20.803 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.799+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:entry[obj00000000000000000004]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000004[0] 2024-08-18T16:19:20.804 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.799+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: listing bilog for incremental sync; position=00000000002.67.6 2024-08-18T16:19:20.809 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.801+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: parsed entry: id=00000000002.111.6 iter->object=obj00000000000000000012 iter->instance= name=obj00000000000000000012 instance= ns= 2024-08-18T16:19:20.809 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.801+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12/obj00000000000000000012 2024-08-18T16:19:20.809 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.801+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: entry->timestamp=2024-08-18T16:19:16.351846+0000 2024-08-18T16:19:20.809 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.801+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:entry[obj00000000000000000012]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.111.6 op=0 op_state=1 2024-08-18T16:19:20.810 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.801+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:entry[obj00000000000000000012]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000012[0] 2024-08-18T16:19:20.810 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.801+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: listing bilog for incremental sync; position=00000000002.111.6 2024-08-18T16:19:20.810 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.801+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:entry[obj00000000000000000016]: success 2024-08-18T16:19:20.810 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.801+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2:2 marker=00000000002.65.6 timestamp=2024-08-18T16:19:16.360394+0000 2024-08-18T16:19:20.810 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.801+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:entry[obj00000000000000000004]: success 2024-08-18T16:19:20.810 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.801+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4:2 marker=00000000002.67.6 timestamp=2024-08-18T16:19:16.233425+0000 2024-08-18T16:19:20.811 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.801+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:entry[obj00000000000000000010]: success 2024-08-18T16:19:20.811 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.801+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1:2 marker=00000000002.126.6 timestamp=2024-08-18T16:19:16.412010+0000 2024-08-18T16:19:20.811 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.802+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]: parsed entry: id=00000000002.136.6 iter->object=obj00000000000000000002 iter->instance= name=obj00000000000000000002 instance= ns= 2024-08-18T16:19:20.811 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.802+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13/obj00000000000000000002 2024-08-18T16:19:20.811 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.802+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]: entry->timestamp=2024-08-18T16:19:16.343786+0000 2024-08-18T16:19:20.811 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.802+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:entry[obj00000000000000000002]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.136.6 op=0 op_state=1 2024-08-18T16:19:20.811 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.802+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:entry[obj00000000000000000002]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000002[0] 2024-08-18T16:19:20.812 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.802+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]: listing bilog for incremental sync; position=00000000002.136.6 2024-08-18T16:19:20.812 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.806+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: parsed entry: id=00000000002.93.6 iter->object=obj00000000000000000000 iter->instance= name=obj00000000000000000000 instance= ns= 2024-08-18T16:19:20.812 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.806+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14/obj00000000000000000000 2024-08-18T16:19:20.812 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.806+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: entry->timestamp=2024-08-18T16:19:16.539475+0000 2024-08-18T16:19:20.812 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.806+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:entry[obj00000000000000000000]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.93.6 op=0 op_state=1 2024-08-18T16:19:20.813 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.806+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:entry[obj00000000000000000000]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000000[0] 2024-08-18T16:19:20.813 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.806+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: listing bilog for incremental sync; position=00000000002.93.6 2024-08-18T16:19:20.813 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.807+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:entry[obj00000000000000000002]: success 2024-08-18T16:19:20.813 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.807+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13:2 marker=00000000002.136.6 timestamp=2024-08-18T16:19:16.343786+0000 2024-08-18T16:19:20.813 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.807+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:entry[obj00000000000000000000]: success 2024-08-18T16:19:20.813 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.807+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14:2 marker=00000000002.93.6 timestamp=2024-08-18T16:19:16.539475+0000 2024-08-18T16:19:20.848 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.848+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:entry[obj00000000000000000012]: success 2024-08-18T16:19:20.849 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.848+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12:2 marker=00000000002.111.6 timestamp=2024-08-18T16:19:16.351846+0000 2024-08-18T16:19:20.869 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.854+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.869 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.854+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.869 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.854+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.869 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.854+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: sync status for source bucket shard: 0 2024-08-18T16:19:20.869 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.854+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.869 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.854+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: sync status for source bucket shard: 0 2024-08-18T16:19:20.869 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.854+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.870 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.854+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.870 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.854+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: sync status for source bucket shard: 0 2024-08-18T16:19:20.870 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.854+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.870 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.854+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.870 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.855+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.870 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.855+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.870 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.855+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: sync status for source bucket shard: 0 2024-08-18T16:19:20.870 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.855+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.870 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.855+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.870 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.855+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: sync status for source bucket shard: 0 2024-08-18T16:19:20.870 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.855+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.870 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.855+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: sync status for source bucket shard: 0 2024-08-18T16:19:20.871 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.855+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.871 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.855+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: sync status for source bucket shard: 0 2024-08-18T16:19:20.871 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.855+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.871 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.855+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.871 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.855+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: sync status for source bucket shard: 0 2024-08-18T16:19:20.871 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.855+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.871 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.855+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.871 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.855+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.871 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.856+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: sync status for source bucket shard: 0 2024-08-18T16:19:20.871 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.856+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.871 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.856+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: sync status for source bucket shard: 0 2024-08-18T16:19:20.871 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.856+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: sync status for source bucket shard: 0 2024-08-18T16:19:20.871 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.856+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.872 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.856+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.872 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.856+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.872 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.856+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.872 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.856+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.872 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.856+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]: sync status for source bucket shard: 0 2024-08-18T16:19:20.872 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.856+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.872 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.856+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: sync status for source bucket shard: 0 2024-08-18T16:19:20.872 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.856+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.872 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.856+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]: sync status for source bucket shard: 0 2024-08-18T16:19:20.872 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.856+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.872 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.856+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.872 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.857+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.872 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.857+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: parsed entry: id=00000000001.71.6 iter->object=obj00000000000000000050 iter->instance= name=obj00000000000000000050 instance= ns= 2024-08-18T16:19:20.873 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.857+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5/obj00000000000000000050 2024-08-18T16:19:20.873 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.857+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: entry->timestamp=2024-08-18T16:19:17.125461+0000 2024-08-18T16:19:20.873 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.857+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:entry[obj00000000000000000050]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.71.6 op=0 op_state=1 2024-08-18T16:19:20.873 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.857+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:entry[obj00000000000000000050]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000050[0] 2024-08-18T16:19:20.873 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.857+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: listing bilog for incremental sync; position=00000000001.71.6 2024-08-18T16:19:20.873 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.857+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.873 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.857+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: sync status for source bucket shard: 0 2024-08-18T16:19:20.873 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.857+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.873 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.857+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.873 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.857+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: sync status for source bucket shard: 0 2024-08-18T16:19:20.873 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.857+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.873 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.857+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.874 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.858+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: sync status for source bucket shard: 0 2024-08-18T16:19:20.874 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.858+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: sync status for source bucket shard: 0 2024-08-18T16:19:20.874 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.858+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.874 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.858+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.874 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.858+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: parsed entry: id=00000000002.175.6 iter->object=obj00000000000000000006 iter->instance= name=obj00000000000000000006 instance= ns= 2024-08-18T16:19:20.874 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.858+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10/obj00000000000000000006 2024-08-18T16:19:20.874 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.858+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: entry->timestamp=2024-08-18T16:19:17.356244+0000 2024-08-18T16:19:20.874 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.858+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:entry[obj00000000000000000006]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.175.6 op=0 op_state=1 2024-08-18T16:19:20.875 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.858+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.875 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.858+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:entry[obj00000000000000000006]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000006[0] 2024-08-18T16:19:20.875 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.858+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: listing bilog for incremental sync; position=00000000002.175.6 2024-08-18T16:19:20.875 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.858+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: parsed entry: id=00000000001.115.6 iter->object=obj00000000000000000039 iter->instance= name=obj00000000000000000039 instance= ns= 2024-08-18T16:19:20.875 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.858+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8/obj00000000000000000039 2024-08-18T16:19:20.875 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.858+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: entry->timestamp=2024-08-18T16:19:16.975932+0000 2024-08-18T16:19:20.875 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.858+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:entry[obj00000000000000000039]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.115.6 op=0 op_state=1 2024-08-18T16:19:20.875 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.858+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:entry[obj00000000000000000039]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000039[0] 2024-08-18T16:19:20.875 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.858+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: listing bilog for incremental sync; position=00000000001.115.6 2024-08-18T16:19:20.875 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.858+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: parsed entry: id=00000000002.61.6 iter->object=obj00000000000000000007 iter->instance= name=obj00000000000000000007 instance= ns= 2024-08-18T16:19:20.875 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.858+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9/obj00000000000000000007 2024-08-18T16:19:20.875 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.858+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: entry->timestamp=2024-08-18T16:19:17.365163+0000 2024-08-18T16:19:20.876 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.858+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:entry[obj00000000000000000007]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.61.6 op=0 op_state=1 2024-08-18T16:19:20.876 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.858+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.876 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.858+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:entry[obj00000000000000000007]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000007[0] 2024-08-18T16:19:20.876 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.858+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: listing bilog for incremental sync; position=00000000002.61.6 2024-08-18T16:19:20.876 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.858+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: sync status for source bucket shard: 0 2024-08-18T16:19:20.876 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.859+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.876 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.859+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.877 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.859+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.877 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.859+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]: parsed entry: id=00000000001.85.6 iter->object=obj00000000000000000040 iter->instance= name=obj00000000000000000040 instance= ns= 2024-08-18T16:19:20.877 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.859+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11/obj00000000000000000040 2024-08-18T16:19:20.878 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.859+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]: entry->timestamp=2024-08-18T16:19:17.233102+0000 2024-08-18T16:19:20.878 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.859+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:entry[obj00000000000000000040]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.85.6 op=0 op_state=1 2024-08-18T16:19:20.878 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.859+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: parsed entry: id=00000000002.141.6 iter->object=obj00000000000000000009 iter->instance= name=obj00000000000000000009 instance= ns= 2024-08-18T16:19:20.878 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.859+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12/obj00000000000000000009 2024-08-18T16:19:20.878 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.859+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: entry->timestamp=2024-08-18T16:19:16.982932+0000 2024-08-18T16:19:20.878 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.859+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:entry[obj00000000000000000009]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.141.6 op=0 op_state=1 2024-08-18T16:19:20.878 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.859+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:entry[obj00000000000000000040]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000040[0] 2024-08-18T16:19:20.878 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.859+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]: listing bilog for incremental sync; position=00000000001.85.6 2024-08-18T16:19:20.878 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.859+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:entry[obj00000000000000000009]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000009[0] 2024-08-18T16:19:20.878 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.859+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: listing bilog for incremental sync; position=00000000002.141.6 2024-08-18T16:19:20.879 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.859+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]: sync status for source bucket shard: 0 2024-08-18T16:19:20.879 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.859+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.879 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.859+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]: sync status for source bucket shard: 0 2024-08-18T16:19:20.879 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.859+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.879 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.859+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]: sync status for source bucket shard: 0 2024-08-18T16:19:20.879 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.859+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.879 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.860+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]: sync status for source bucket shard: 0 2024-08-18T16:19:20.879 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.860+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.879 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.861+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:entry[obj00000000000000000006]: success 2024-08-18T16:19:20.879 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.861+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10:3 marker=00000000002.175.6 timestamp=2024-08-18T16:19:17.356244+0000 2024-08-18T16:19:20.879 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.862+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]: parsed entry: id=00000000002.80.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2024-08-18T16:19:20.879 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.862+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19/obj00000000000000000021 2024-08-18T16:19:20.880 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.862+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]: entry->timestamp=2024-08-18T16:19:17.248676+0000 2024-08-18T16:19:20.880 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.862+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.80.6 op=0 op_state=1 2024-08-18T16:19:20.880 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.862+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:entry[obj00000000000000000021]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000021[0] 2024-08-18T16:19:20.880 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.862+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]: listing bilog for incremental sync; position=00000000002.80.6 2024-08-18T16:19:20.880 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.862+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: parsed entry: id=00000000002.101.6 iter->object=obj00000000000000000000 iter->instance= name=obj00000000000000000000 instance= ns= 2024-08-18T16:19:20.880 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.862+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18/obj00000000000000000000 2024-08-18T16:19:20.880 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.862+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: entry->timestamp=2024-08-18T16:19:17.048772+0000 2024-08-18T16:19:20.880 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.862+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:entry[obj00000000000000000000]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.101.6 op=0 op_state=1 2024-08-18T16:19:20.880 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.862+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:entry[obj00000000000000000000]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000000[0] 2024-08-18T16:19:20.880 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.862+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: listing bilog for incremental sync; position=00000000002.101.6 2024-08-18T16:19:20.880 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.862+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]: parsed entry: id=00000000001.82.6 iter->object=obj00000000000000000055 iter->instance= name=obj00000000000000000055 instance= ns= 2024-08-18T16:19:20.880 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.862+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21/obj00000000000000000055 2024-08-18T16:19:20.881 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.862+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]: entry->timestamp=2024-08-18T16:19:17.295995+0000 2024-08-18T16:19:20.881 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.862+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:entry[obj00000000000000000055]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.82.6 op=0 op_state=1 2024-08-18T16:19:20.881 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.862+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]: parsed entry: id=00000000002.80.6 iter->object=obj00000000000000000008 iter->instance= name=obj00000000000000000008 instance= ns= 2024-08-18T16:19:20.881 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.862+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22/obj00000000000000000008 2024-08-18T16:19:20.881 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.862+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]: entry->timestamp=2024-08-18T16:19:17.240648+0000 2024-08-18T16:19:20.881 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.862+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:entry[obj00000000000000000008]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.80.6 op=0 op_state=1 2024-08-18T16:19:20.881 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.862+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:entry[obj00000000000000000055]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000055[0] 2024-08-18T16:19:20.881 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.862+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]: listing bilog for incremental sync; position=00000000001.82.6 2024-08-18T16:19:20.881 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.862+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:entry[obj00000000000000000008]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000008[0] 2024-08-18T16:19:20.881 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.862+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]: listing bilog for incremental sync; position=00000000002.80.6 2024-08-18T16:19:20.881 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.862+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:entry[obj00000000000000000050]: success 2024-08-18T16:19:20.881 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.862+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5:3 marker=00000000001.71.6 timestamp=2024-08-18T16:19:17.125461+0000 2024-08-18T16:19:20.882 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.863+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:entry[obj00000000000000000007]: success 2024-08-18T16:19:20.882 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.863+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9:3 marker=00000000002.61.6 timestamp=2024-08-18T16:19:17.365163+0000 2024-08-18T16:19:20.882 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.863+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:entry[obj00000000000000000009]: success 2024-08-18T16:19:20.882 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.863+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12:3 marker=00000000002.141.6 timestamp=2024-08-18T16:19:16.982932+0000 2024-08-18T16:19:20.882 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.864+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:entry[obj00000000000000000021]: success 2024-08-18T16:19:20.882 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.864+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19:3 marker=00000000002.80.6 timestamp=2024-08-18T16:19:17.248676+0000 2024-08-18T16:19:20.882 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.865+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:entry[obj00000000000000000000]: success 2024-08-18T16:19:20.882 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.865+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18:3 marker=00000000002.101.6 timestamp=2024-08-18T16:19:17.048772+0000 2024-08-18T16:19:20.882 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.868+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:entry[obj00000000000000000039]: success 2024-08-18T16:19:20.882 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.868+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8:3 marker=00000000001.115.6 timestamp=2024-08-18T16:19:16.975932+0000 2024-08-18T16:19:20.882 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.869+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:entry[obj00000000000000000040]: success 2024-08-18T16:19:20.883 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.869+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11:3 marker=00000000001.85.6 timestamp=2024-08-18T16:19:17.233102+0000 2024-08-18T16:19:20.883 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.870+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:entry[obj00000000000000000008]: success 2024-08-18T16:19:20.883 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.870+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22:3 marker=00000000002.80.6 timestamp=2024-08-18T16:19:17.240648+0000 2024-08-18T16:19:20.883 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.871+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:entry[obj00000000000000000055]: success 2024-08-18T16:19:20.883 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.871+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21:3 marker=00000000001.82.6 timestamp=2024-08-18T16:19:17.295995+0000 2024-08-18T16:19:20.896 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.896+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: parsed entry: id=00000000001.71.6 iter->object=obj00000000000000000058 iter->instance= name=obj00000000000000000058 instance= ns= 2024-08-18T16:19:20.896 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.896+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0/obj00000000000000000058 2024-08-18T16:19:20.897 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.896+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: entry->timestamp=2024-08-18T16:19:17.303274+0000 2024-08-18T16:19:20.897 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.896+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:entry[obj00000000000000000058]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.71.6 op=0 op_state=1 2024-08-18T16:19:20.897 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.896+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:entry[obj00000000000000000058]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000058[0] 2024-08-18T16:19:20.897 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.896+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: listing bilog for incremental sync; position=00000000001.71.6 2024-08-18T16:19:20.898 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.897+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: parsed entry: id=00000000002.68.6 iter->object=obj00000000000000000014 iter->instance= name=obj00000000000000000014 instance= ns= 2024-08-18T16:19:20.898 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.897+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4/obj00000000000000000014 2024-08-18T16:19:20.898 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.897+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: entry->timestamp=2024-08-18T16:19:17.108165+0000 2024-08-18T16:19:20.899 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.897+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:entry[obj00000000000000000014]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.68.6 op=0 op_state=1 2024-08-18T16:19:20.899 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.897+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:entry[obj00000000000000000014]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000014[0] 2024-08-18T16:19:20.899 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.897+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: listing bilog for incremental sync; position=00000000002.68.6 2024-08-18T16:19:20.899 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.897+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: parsed entry: id=00000000002.90.6 iter->object=obj00000000000000000026 iter->instance= name=obj00000000000000000026 instance= ns= 2024-08-18T16:19:20.899 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.897+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1/obj00000000000000000026 2024-08-18T16:19:20.899 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.897+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: entry->timestamp=2024-08-18T16:19:16.991700+0000 2024-08-18T16:19:20.899 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.897+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:entry[obj00000000000000000026]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.90.6 op=0 op_state=1 2024-08-18T16:19:20.899 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.897+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: parsed entry: id=00000000001.92.6 iter->object=obj00000000000000000046 iter->instance= name=obj00000000000000000046 instance= ns= 2024-08-18T16:19:20.899 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.897+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3/obj00000000000000000046 2024-08-18T16:19:20.899 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.897+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: entry->timestamp=2024-08-18T16:19:17.181145+0000 2024-08-18T16:19:20.899 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.897+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:entry[obj00000000000000000046]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.92.6 op=0 op_state=1 2024-08-18T16:19:20.900 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.897+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: parsed entry: id=00000000002.70.6 iter->object=obj00000000000000000011 iter->instance= name=obj00000000000000000011 instance= ns= 2024-08-18T16:19:20.900 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.897+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2/obj00000000000000000011 2024-08-18T16:19:20.900 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.897+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: entry->timestamp=2024-08-18T16:19:17.309985+0000 2024-08-18T16:19:20.900 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.897+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:entry[obj00000000000000000011]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.70.6 op=0 op_state=1 2024-08-18T16:19:20.900 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.897+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:entry[obj00000000000000000026]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000026[0] 2024-08-18T16:19:20.900 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.897+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: listing bilog for incremental sync; position=00000000002.90.6 2024-08-18T16:19:20.900 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.897+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:entry[obj00000000000000000046]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000046[0] 2024-08-18T16:19:20.900 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.897+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: listing bilog for incremental sync; position=00000000001.92.6 2024-08-18T16:19:20.900 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.897+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:entry[obj00000000000000000011]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000011[0] 2024-08-18T16:19:20.900 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.897+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: listing bilog for incremental sync; position=00000000002.70.6 2024-08-18T16:19:20.900 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.898+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: parsed entry: id=00000000002.173.6 iter->object=obj00000000000000000032 iter->instance= name=obj00000000000000000032 instance= ns= 2024-08-18T16:19:20.901 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.898+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6/obj00000000000000000032 2024-08-18T16:19:20.901 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.898+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: entry->timestamp=2024-08-18T16:19:17.116603+0000 2024-08-18T16:19:20.901 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.898+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:entry[obj00000000000000000032]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.173.6 op=0 op_state=1 2024-08-18T16:19:20.901 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.898+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:entry[obj00000000000000000032]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000032[0] 2024-08-18T16:19:20.901 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.898+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: listing bilog for incremental sync; position=00000000002.173.6 2024-08-18T16:19:20.903 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.899+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]: parsed entry: id=00000000001.94.6 iter->object=obj00000000000000000045 iter->instance= name=obj00000000000000000045 instance= ns= 2024-08-18T16:19:20.903 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.899+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13/obj00000000000000000045 2024-08-18T16:19:20.903 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.899+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]: entry->timestamp=2024-08-18T16:19:17.173028+0000 2024-08-18T16:19:20.903 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.899+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:entry[obj00000000000000000045]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.94.6 op=0 op_state=1 2024-08-18T16:19:20.903 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.899+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:entry[obj00000000000000000045]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000045[0] 2024-08-18T16:19:20.903 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.899+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]: listing bilog for incremental sync; position=00000000001.94.6 2024-08-18T16:19:20.903 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.899+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:entry[obj00000000000000000026]: success 2024-08-18T16:19:20.903 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.899+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1:3 marker=00000000002.90.6 timestamp=2024-08-18T16:19:16.991700+0000 2024-08-18T16:19:20.903 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.899+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:entry[obj00000000000000000011]: success 2024-08-18T16:19:20.903 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.899+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2:3 marker=00000000002.70.6 timestamp=2024-08-18T16:19:17.309985+0000 2024-08-18T16:19:20.903 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.900+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:entry[obj00000000000000000014]: success 2024-08-18T16:19:20.904 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.900+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4:3 marker=00000000002.68.6 timestamp=2024-08-18T16:19:17.108165+0000 2024-08-18T16:19:20.904 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.900+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:entry[obj00000000000000000032]: success 2024-08-18T16:19:20.904 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.900+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6:3 marker=00000000002.173.6 timestamp=2024-08-18T16:19:17.116603+0000 2024-08-18T16:19:20.904 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.900+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: parsed entry: id=00000000002.92.6 iter->object=obj00000000000000000013 iter->instance= name=obj00000000000000000013 instance= ns= 2024-08-18T16:19:20.904 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.900+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14/obj00000000000000000013 2024-08-18T16:19:20.904 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.900+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: entry->timestamp=2024-08-18T16:19:17.040343+0000 2024-08-18T16:19:20.904 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.900+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:entry[obj00000000000000000013]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.92.6 op=0 op_state=1 2024-08-18T16:19:20.904 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.900+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:entry[obj00000000000000000013]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000013[0] 2024-08-18T16:19:20.904 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.900+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: listing bilog for incremental sync; position=00000000002.92.6 2024-08-18T16:19:20.904 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.900+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: parsed entry: id=00000000002.98.6 iter->object=obj00000000000000000012 iter->instance= name=obj00000000000000000012 instance= ns= 2024-08-18T16:19:20.904 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.900+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15/obj00000000000000000012 2024-08-18T16:19:20.904 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.900+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: entry->timestamp=2024-08-18T16:19:16.929534+0000 2024-08-18T16:19:20.905 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.900+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:entry[obj00000000000000000012]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.98.6 op=0 op_state=1 2024-08-18T16:19:20.905 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.900+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:entry[obj00000000000000000012]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000012[0] 2024-08-18T16:19:20.905 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.900+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: listing bilog for incremental sync; position=00000000002.98.6 2024-08-18T16:19:20.905 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.901+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: parsed entry: id=00000000002.69.6 iter->object=obj00000000000000000033 iter->instance= name=obj00000000000000000033 instance= ns= 2024-08-18T16:19:20.905 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.901+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16/obj00000000000000000033 2024-08-18T16:19:20.905 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.901+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: entry->timestamp=2024-08-18T16:19:16.920927+0000 2024-08-18T16:19:20.905 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.901+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:entry[obj00000000000000000033]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.69.6 op=0 op_state=1 2024-08-18T16:19:20.905 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.901+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:entry[obj00000000000000000033]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000033[0] 2024-08-18T16:19:20.905 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.901+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: listing bilog for incremental sync; position=00000000002.69.6 2024-08-18T16:19:20.905 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.901+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: parsed entry: id=00000000002.76.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2024-08-18T16:19:20.905 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.901+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17/obj00000000000000000001 2024-08-18T16:19:20.905 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.901+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: entry->timestamp=2024-08-18T16:19:17.060727+0000 2024-08-18T16:19:20.905 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.901+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.76.6 op=0 op_state=1 2024-08-18T16:19:20.906 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.901+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:entry[obj00000000000000000001]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000001[0] 2024-08-18T16:19:20.906 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.901+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: listing bilog for incremental sync; position=00000000002.76.6 2024-08-18T16:19:20.906 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.902+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:entry[obj00000000000000000012]: success 2024-08-18T16:19:20.906 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.902+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15:3 marker=00000000002.98.6 timestamp=2024-08-18T16:19:16.929534+0000 2024-08-18T16:19:20.906 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.902+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:entry[obj00000000000000000013]: success 2024-08-18T16:19:20.906 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.902+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14:3 marker=00000000002.92.6 timestamp=2024-08-18T16:19:17.040343+0000 2024-08-18T16:19:20.907 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.903+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:entry[obj00000000000000000058]: success 2024-08-18T16:19:20.907 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.903+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0:3 marker=00000000001.71.6 timestamp=2024-08-18T16:19:17.303274+0000 2024-08-18T16:19:20.907 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.903+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:entry[obj00000000000000000001]: success 2024-08-18T16:19:20.907 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.903+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17:3 marker=00000000002.76.6 timestamp=2024-08-18T16:19:17.060727+0000 2024-08-18T16:19:20.907 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.903+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:entry[obj00000000000000000033]: success 2024-08-18T16:19:20.907 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.903+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16:3 marker=00000000002.69.6 timestamp=2024-08-18T16:19:16.920927+0000 2024-08-18T16:19:20.945 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.945+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:entry[obj00000000000000000046]: success 2024-08-18T16:19:20.945 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.945+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3:3 marker=00000000001.92.6 timestamp=2024-08-18T16:19:17.181145+0000 2024-08-18T16:19:20.947 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.947+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:entry[obj00000000000000000045]: success 2024-08-18T16:19:20.947 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.947+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13:3 marker=00000000001.94.6 timestamp=2024-08-18T16:19:17.173028+0000 2024-08-18T16:19:20.978 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.970+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.978 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.970+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.978 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.970+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.978 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.970+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.979 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.970+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: sync status for source bucket shard: 0 2024-08-18T16:19:20.979 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.971+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.979 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.971+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: sync status for source bucket shard: 0 2024-08-18T16:19:20.979 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.971+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.979 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.971+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: sync status for source bucket shard: 0 2024-08-18T16:19:20.979 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.971+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.979 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.971+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: sync status for source bucket shard: 0 2024-08-18T16:19:20.979 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.971+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.979 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.971+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.979 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.971+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.979 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.971+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.979 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.971+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: sync status for source bucket shard: 0 2024-08-18T16:19:20.980 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.971+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.980 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.971+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.980 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.971+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: sync status for source bucket shard: 0 2024-08-18T16:19:20.980 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.971+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.980 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.971+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.980 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.972+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: sync status for source bucket shard: 0 2024-08-18T16:19:20.980 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.972+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.980 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.972+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.980 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.972+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: sync status for source bucket shard: 0 2024-08-18T16:19:20.980 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.972+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.980 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.972+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.980 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.972+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: sync status for source bucket shard: 0 2024-08-18T16:19:20.981 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.972+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.981 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.972+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.981 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.972+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]: sync status for source bucket shard: 0 2024-08-18T16:19:20.981 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.972+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.981 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.972+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.981 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.972+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: sync status for source bucket shard: 0 2024-08-18T16:19:20.981 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.972+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.981 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.973+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.981 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.973+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]: sync status for source bucket shard: 0 2024-08-18T16:19:20.981 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.973+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.981 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.973+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.981 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.973+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: sync status for source bucket shard: 0 2024-08-18T16:19:20.982 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.973+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.982 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.973+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.982 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.973+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: sync status for source bucket shard: 0 2024-08-18T16:19:20.982 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.973+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.982 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.973+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: sync status for source bucket shard: 0 2024-08-18T16:19:20.982 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.973+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.982 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.973+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.982 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.973+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.982 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.974+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.982 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.974+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: sync status for source bucket shard: 0 2024-08-18T16:19:20.982 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.974+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.983 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.974+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: sync status for source bucket shard: 0 2024-08-18T16:19:20.983 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.974+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.983 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.974+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.983 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.974+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: sync status for source bucket shard: 0 2024-08-18T16:19:20.983 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.974+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.983 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.974+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.983 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.975+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: parsed entry: id=00000000002.141.6 iter->object=obj00000000000000000008 iter->instance= name=obj00000000000000000008 instance= ns= 2024-08-18T16:19:20.983 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.975+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9/obj00000000000000000008 2024-08-18T16:19:20.983 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.975+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: entry->timestamp=2024-08-18T16:19:18.113243+0000 2024-08-18T16:19:20.983 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.975+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:entry[obj00000000000000000008]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.141.6 op=0 op_state=1 2024-08-18T16:19:20.983 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.975+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.984 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.975+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:entry[obj00000000000000000008]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000008[0] 2024-08-18T16:19:20.984 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.975+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: listing bilog for incremental sync; position=00000000002.141.6 2024-08-18T16:19:20.984 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.975+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]: sync status for source bucket shard: 0 2024-08-18T16:19:20.984 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.975+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.984 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.975+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: parsed entry: id=00000000002.105.6 iter->object=obj00000000000000000020 iter->instance= name=obj00000000000000000020 instance= ns= 2024-08-18T16:19:20.984 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.975+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14/obj00000000000000000020 2024-08-18T16:19:20.984 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.975+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: entry->timestamp=2024-08-18T16:19:17.856631+0000 2024-08-18T16:19:20.984 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.975+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:entry[obj00000000000000000020]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.105.6 op=0 op_state=1 2024-08-18T16:19:20.984 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.975+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.984 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.975+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: parsed entry: id=00000000001.78.6 iter->object=obj00000000000000000037 iter->instance= name=obj00000000000000000037 instance= ns= 2024-08-18T16:19:20.984 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.975+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15/obj00000000000000000037 2024-08-18T16:19:20.984 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.975+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: entry->timestamp=2024-08-18T16:19:17.746196+0000 2024-08-18T16:19:20.985 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.975+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:entry[obj00000000000000000037]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.78.6 op=0 op_state=1 2024-08-18T16:19:20.985 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.975+0000 7f63ec126880 0 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:entry[obj00000000000000000020]: : injecting data sync error on key=obj00000000000000000020 2024-08-18T16:19:20.985 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.975+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: listing bilog for incremental sync; position=00000000002.105.6 2024-08-18T16:19:20.985 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.975+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:entry[obj00000000000000000037]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000037[0] 2024-08-18T16:19:20.985 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.975+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: listing bilog for incremental sync; position=00000000001.78.6 2024-08-18T16:19:20.985 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.975+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:entry[obj00000000000000000020]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:19:20.985 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.975+0000 7f63ec126880 0 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:entry[obj00000000000000000020]: ERROR: failed to sync object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14/obj00000000000000000020 2024-08-18T16:19:20.985 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.975+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.985 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.975+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]: sync status for source bucket shard: 0 2024-08-18T16:19:20.985 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.975+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.985 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.975+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.985 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.976+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]: sync status for source bucket shard: 0 2024-08-18T16:19:20.986 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.976+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.986 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.976+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.986 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.976+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:28]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.986 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.976+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]: sync status for source bucket shard: 0 2024-08-18T16:19:20.986 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.976+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.986 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.976+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.986 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.977+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]: sync status for source bucket shard: 0 2024-08-18T16:19:20.986 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.977+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25]: sync status for source bucket shard: 0 2024-08-18T16:19:20.986 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.977+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: sync status for source bucket shard: 0 2024-08-18T16:19:20.986 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.977+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]: sync status for source bucket shard: 0 2024-08-18T16:19:20.986 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.977+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:28]: sync status for source bucket shard: 0 2024-08-18T16:19:20.987 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.977+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.987 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.977+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.987 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.977+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.987 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.977+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.987 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.977+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:28]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:28]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.987 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.977+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.987 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.977+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:entry[obj00000000000000000008]: success 2024-08-18T16:19:20.987 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.977+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9:4 marker=00000000002.141.6 timestamp=2024-08-18T16:19:18.113243+0000 2024-08-18T16:19:20.987 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.977+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]: parsed entry: id=00000000001.145.6 iter->object=obj00000000000000000042 iter->instance= name=obj00000000000000000042 instance= ns= 2024-08-18T16:19:20.987 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.977+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19/obj00000000000000000042 2024-08-18T16:19:20.987 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.977+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]: entry->timestamp=2024-08-18T16:19:18.280618+0000 2024-08-18T16:19:20.987 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.977+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:entry[obj00000000000000000042]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.145.6 op=0 op_state=1 2024-08-18T16:19:20.988 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.977+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:entry[obj00000000000000000042]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000042[0] 2024-08-18T16:19:20.988 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.977+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]: listing bilog for incremental sync; position=00000000001.145.6 2024-08-18T16:19:20.988 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.977+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]: sync status for source bucket shard: 0 2024-08-18T16:19:20.988 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.977+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.988 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.978+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: a sync operation returned error 2024-08-18T16:19:20.988 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.978+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: backing out with sync_status=-5 2024-08-18T16:19:20.988 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.978+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: incremental sync on bucket failed, retcode=-5 2024-08-18T16:19:20.988 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.978+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: ERROR: incremental sync failed. error: -5 2024-08-18T16:19:20.988 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.978+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]: sync status for source bucket shard: 0 2024-08-18T16:19:20.988 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.978+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.992 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.978+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]: parsed entry: id=00000000002.80.6 iter->object=obj00000000000000000009 iter->instance= name=obj00000000000000000009 instance= ns= 2024-08-18T16:19:20.992 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.978+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22/obj00000000000000000009 2024-08-18T16:19:20.992 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.978+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]: entry->timestamp=2024-08-18T16:19:17.919915+0000 2024-08-18T16:19:20.992 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.978+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:entry[obj00000000000000000009]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.80.6 op=0 op_state=1 2024-08-18T16:19:20.992 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.978+0000 7f63ec126880 0 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:entry[obj00000000000000000009]: : injecting data sync error on key=obj00000000000000000009 2024-08-18T16:19:20.993 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.978+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]: listing bilog for incremental sync; position=00000000002.80.6 2024-08-18T16:19:20.993 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.978+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:entry[obj00000000000000000009]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:19:20.993 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.978+0000 7f63ec126880 0 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:entry[obj00000000000000000009]: ERROR: failed to sync object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22/obj00000000000000000009 2024-08-18T16:19:20.993 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.978+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]: parsed entry: id=00000000002.82.6 iter->object=obj00000000000000000010 iter->instance= name=obj00000000000000000010 instance= ns= 2024-08-18T16:19:20.993 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.978+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20/obj00000000000000000010 2024-08-18T16:19:20.994 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.978+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]: entry->timestamp=2024-08-18T16:19:18.176758+0000 2024-08-18T16:19:20.994 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.978+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:entry[obj00000000000000000010]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.82.6 op=0 op_state=1 2024-08-18T16:19:20.994 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.978+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:entry[obj00000000000000000010]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000010[0] 2024-08-18T16:19:20.994 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.978+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]: listing bilog for incremental sync; position=00000000002.82.6 2024-08-18T16:19:20.994 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.979+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]: parsed entry: id=00000000002.52.6 iter->object=obj00000000000000000025 iter->instance= name=obj00000000000000000025 instance= ns= 2024-08-18T16:19:20.994 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.979+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23/obj00000000000000000025 2024-08-18T16:19:20.995 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.979+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]: entry->timestamp=2024-08-18T16:19:17.792369+0000 2024-08-18T16:19:20.995 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.979+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:entry[obj00000000000000000025]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.52.6 op=0 op_state=1 2024-08-18T16:19:20.995 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.979+0000 7f63ec126880 0 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:entry[obj00000000000000000025]: : injecting data sync error on key=obj00000000000000000025 2024-08-18T16:19:20.995 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.979+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]: listing bilog for incremental sync; position=00000000002.52.6 2024-08-18T16:19:20.995 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.979+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:entry[obj00000000000000000025]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:19:20.995 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.979+0000 7f63ec126880 0 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:entry[obj00000000000000000025]: ERROR: failed to sync object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23/obj00000000000000000025 2024-08-18T16:19:20.995 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.979+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]: parsed entry: id=00000000002.99.6 iter->object=obj00000000000000000030 iter->instance= name=obj00000000000000000030 instance= ns= 2024-08-18T16:19:20.995 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.979+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27/obj00000000000000000030 2024-08-18T16:19:20.995 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.979+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]: entry->timestamp=2024-08-18T16:19:17.681589+0000 2024-08-18T16:19:20.995 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.980+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]:entry[obj00000000000000000030]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.99.6 op=0 op_state=1 2024-08-18T16:19:20.996 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.980+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:20.996 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.980+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]:entry[obj00000000000000000030]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000030[0] 2024-08-18T16:19:20.996 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.980+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]: listing bilog for incremental sync; position=00000000002.99.6 2024-08-18T16:19:20.996 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.980+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: parsed entry: id=00000000001.180.6 iter->object=obj00000000000000000027 iter->instance= name=obj00000000000000000027 instance= ns= 2024-08-18T16:19:20.996 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.980+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2/obj00000000000000000027 2024-08-18T16:19:20.996 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.980+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: entry->timestamp=2024-08-18T16:19:18.160384+0000 2024-08-18T16:19:20.996 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.980+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:entry[obj00000000000000000027]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.180.6 op=0 op_state=1 2024-08-18T16:19:20.996 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.980+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:entry[obj00000000000000000027]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000027[0] 2024-08-18T16:19:20.996 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.980+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: listing bilog for incremental sync; position=00000000001.180.6 2024-08-18T16:19:20.996 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.980+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]: parsed entry: id=00000000002.101.6 iter->object=obj00000000000000000016 iter->instance= name=obj00000000000000000016 instance= ns= 2024-08-18T16:19:20.996 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.980+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24/obj00000000000000000016 2024-08-18T16:19:20.996 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.980+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]: entry->timestamp=2024-08-18T16:19:17.809180+0000 2024-08-18T16:19:20.997 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.980+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]:entry[obj00000000000000000016]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.101.6 op=0 op_state=1 2024-08-18T16:19:20.997 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.980+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:28]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:28]: parsed entry: id=00000000002.76.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2024-08-18T16:19:20.997 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.980+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:28]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:28]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:28/obj00000000000000000021 2024-08-18T16:19:20.997 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.980+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:28]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:28]: entry->timestamp=2024-08-18T16:19:17.992108+0000 2024-08-18T16:19:20.997 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.980+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:28]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:28]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.76.6 op=0 op_state=1 2024-08-18T16:19:20.997 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.980+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]:entry[obj00000000000000000016]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000016[0] 2024-08-18T16:19:20.997 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.980+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]: listing bilog for incremental sync; position=00000000002.101.6 2024-08-18T16:19:20.997 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.980+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:28]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:28]:entry[obj00000000000000000021]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000021[0] 2024-08-18T16:19:20.997 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.980+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:28]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:28]: listing bilog for incremental sync; position=00000000002.76.6 2024-08-18T16:19:20.997 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.981+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:entry[obj00000000000000000010]: success 2024-08-18T16:19:20.997 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.981+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20:4 marker=00000000002.82.6 timestamp=2024-08-18T16:19:18.176758+0000 2024-08-18T16:19:20.997 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.982+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: sync status for source bucket shard: 0 2024-08-18T16:19:20.998 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.982+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: listing bilog for incremental sync; position= 2024-08-18T16:19:20.998 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.982+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]:entry[obj00000000000000000016]: success 2024-08-18T16:19:20.998 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.982+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24:4 marker=00000000002.101.6 timestamp=2024-08-18T16:19:17.809180+0000 2024-08-18T16:19:20.998 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.982+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]:entry[obj00000000000000000030]: success 2024-08-18T16:19:20.998 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.982+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27:4 marker=00000000002.99.6 timestamp=2024-08-18T16:19:17.681589+0000 2024-08-18T16:19:20.998 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.983+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:28]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:28]:entry[obj00000000000000000021]: success 2024-08-18T16:19:20.998 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.983+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:28]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:28]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:28:4 marker=00000000002.76.6 timestamp=2024-08-18T16:19:17.992108+0000 2024-08-18T16:19:20.998 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.984+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:entry[obj00000000000000000037]: success 2024-08-18T16:19:20.998 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.984+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15:4 marker=00000000001.78.6 timestamp=2024-08-18T16:19:17.746196+0000 2024-08-18T16:19:20.998 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.984+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:entry[obj00000000000000000042]: success 2024-08-18T16:19:20.998 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.984+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19:4 marker=00000000001.145.6 timestamp=2024-08-18T16:19:18.280618+0000 2024-08-18T16:19:20.998 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.987+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:entry[obj00000000000000000027]: success 2024-08-18T16:19:20.999 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:20.987+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2:4 marker=00000000001.180.6 timestamp=2024-08-18T16:19:18.160384+0000 2024-08-18T16:19:21.015 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.013+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: parsed entry: id=00000000002.122.6 iter->object=obj00000000000000000006 iter->instance= name=obj00000000000000000006 instance= ns= 2024-08-18T16:19:21.015 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.013+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0/obj00000000000000000006 2024-08-18T16:19:21.015 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.013+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: entry->timestamp=2024-08-18T16:19:17.976117+0000 2024-08-18T16:19:21.015 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.013+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:entry[obj00000000000000000006]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.122.6 op=0 op_state=1 2024-08-18T16:19:21.015 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.013+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:entry[obj00000000000000000006]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000006[0] 2024-08-18T16:19:21.015 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.013+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: listing bilog for incremental sync; position=00000000002.122.6 2024-08-18T16:19:21.015 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.013+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: parsed entry: id=00000000002.103.6 iter->object=obj00000000000000000002 iter->instance= name=obj00000000000000000002 instance= ns= 2024-08-18T16:19:21.015 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.013+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5/obj00000000000000000002 2024-08-18T16:19:21.016 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.013+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: entry->timestamp=2024-08-18T16:19:18.224388+0000 2024-08-18T16:19:21.016 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.013+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:entry[obj00000000000000000002]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.103.6 op=0 op_state=1 2024-08-18T16:19:21.016 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.013+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: parsed entry: id=00000000001.54.6 iter->object=obj00000000000000000059 iter->instance= name=obj00000000000000000059 instance= ns= 2024-08-18T16:19:21.016 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.013+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1/obj00000000000000000059 2024-08-18T16:19:21.016 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.013+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: entry->timestamp=2024-08-18T16:19:18.040381+0000 2024-08-18T16:19:21.016 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.013+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:entry[obj00000000000000000059]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.54.6 op=0 op_state=1 2024-08-18T16:19:21.016 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.013+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: parsed entry: id=00000000002.70.6 iter->object=obj00000000000000000004 iter->instance= name=obj00000000000000000004 instance= ns= 2024-08-18T16:19:21.016 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.013+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3/obj00000000000000000004 2024-08-18T16:19:21.016 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.013+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: entry->timestamp=2024-08-18T16:19:17.671427+0000 2024-08-18T16:19:21.016 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.013+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:entry[obj00000000000000000004]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.70.6 op=0 op_state=1 2024-08-18T16:19:21.016 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.013+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:entry[obj00000000000000000002]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000002[0] 2024-08-18T16:19:21.016 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.013+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: listing bilog for incremental sync; position=00000000002.103.6 2024-08-18T16:19:21.017 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.013+0000 7f63ec126880 0 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:entry[obj00000000000000000059]: : injecting data sync error on key=obj00000000000000000059 2024-08-18T16:19:21.017 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.013+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: listing bilog for incremental sync; position=00000000001.54.6 2024-08-18T16:19:21.017 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.013+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:entry[obj00000000000000000004]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000004[0] 2024-08-18T16:19:21.017 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.013+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: listing bilog for incremental sync; position=00000000002.70.6 2024-08-18T16:19:21.017 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.013+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:entry[obj00000000000000000059]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:19:21.017 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.013+0000 7f63ec126880 0 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:entry[obj00000000000000000059]: ERROR: failed to sync object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1/obj00000000000000000059 2024-08-18T16:19:21.017 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.014+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: parsed entry: id=00000000002.88.6 iter->object=obj00000000000000000011 iter->instance= name=obj00000000000000000011 instance= ns= 2024-08-18T16:19:21.017 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.014+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4/obj00000000000000000011 2024-08-18T16:19:21.017 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.014+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: entry->timestamp=2024-08-18T16:19:18.048298+0000 2024-08-18T16:19:21.017 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.014+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:entry[obj00000000000000000011]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.88.6 op=0 op_state=1 2024-08-18T16:19:21.017 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.014+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:entry[obj00000000000000000011]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000011[0] 2024-08-18T16:19:21.017 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.014+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: listing bilog for incremental sync; position=00000000002.88.6 2024-08-18T16:19:21.017 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.014+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: parsed entry: id=00000000002.75.6 iter->object=obj00000000000000000019 iter->instance= name=obj00000000000000000019 instance= ns= 2024-08-18T16:19:21.018 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.014+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6/obj00000000000000000019 2024-08-18T16:19:21.018 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.014+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: entry->timestamp=2024-08-18T16:19:18.233851+0000 2024-08-18T16:19:21.018 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.014+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:entry[obj00000000000000000019]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.75.6 op=0 op_state=1 2024-08-18T16:19:21.018 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.014+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:entry[obj00000000000000000019]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000019[0] 2024-08-18T16:19:21.018 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.014+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: listing bilog for incremental sync; position=00000000002.75.6 2024-08-18T16:19:21.021 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.015+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: parsed entry: id=00000000001.155.6 iter->object=obj00000000000000000049 iter->instance= name=obj00000000000000000049 instance= ns= 2024-08-18T16:19:21.021 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.015+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7/obj00000000000000000049 2024-08-18T16:19:21.021 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.015+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: entry->timestamp=2024-08-18T16:19:17.984613+0000 2024-08-18T16:19:21.021 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.015+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:entry[obj00000000000000000049]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.155.6 op=0 op_state=1 2024-08-18T16:19:21.021 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.015+0000 7f63ec126880 0 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:entry[obj00000000000000000049]: : injecting data sync error on key=obj00000000000000000049 2024-08-18T16:19:21.022 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.015+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: listing bilog for incremental sync; position=00000000001.155.6 2024-08-18T16:19:21.022 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.015+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: parsed entry: id=00000000001.80.6 iter->object=obj00000000000000000017 iter->instance= name=obj00000000000000000017 instance= ns= 2024-08-18T16:19:21.022 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.015+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8/obj00000000000000000017 2024-08-18T16:19:21.022 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.015+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: entry->timestamp=2024-08-18T16:19:17.865804+0000 2024-08-18T16:19:21.022 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.015+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:entry[obj00000000000000000017]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.80.6 op=0 op_state=1 2024-08-18T16:19:21.022 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.015+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: parsed entry: id=00000000002.139.6 iter->object=obj00000000000000000015 iter->instance= name=obj00000000000000000015 instance= ns= 2024-08-18T16:19:21.022 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.015+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10/obj00000000000000000015 2024-08-18T16:19:21.022 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.015+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: entry->timestamp=2024-08-18T16:19:18.057904+0000 2024-08-18T16:19:21.022 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.015+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:entry[obj00000000000000000015]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.139.6 op=0 op_state=1 2024-08-18T16:19:21.022 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.015+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]: parsed entry: id=00000000001.130.6 iter->object=obj00000000000000000031 iter->instance= name=obj00000000000000000031 instance= ns= 2024-08-18T16:19:21.022 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.015+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11/obj00000000000000000031 2024-08-18T16:19:21.023 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.015+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]: entry->timestamp=2024-08-18T16:19:17.912320+0000 2024-08-18T16:19:21.023 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.015+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:entry[obj00000000000000000031]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.130.6 op=0 op_state=1 2024-08-18T16:19:21.023 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.015+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: parsed entry: id=00000000002.66.6 iter->object=obj00000000000000000022 iter->instance= name=obj00000000000000000022 instance= ns= 2024-08-18T16:19:21.023 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.015+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12/obj00000000000000000022 2024-08-18T16:19:21.023 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.015+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: entry->timestamp=2024-08-18T16:19:17.801074+0000 2024-08-18T16:19:21.023 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.015+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:entry[obj00000000000000000022]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.66.6 op=0 op_state=1 2024-08-18T16:19:21.023 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.015+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:entry[obj00000000000000000049]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:19:21.023 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.015+0000 7f63ec126880 0 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:entry[obj00000000000000000049]: ERROR: failed to sync object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7/obj00000000000000000049 2024-08-18T16:19:21.023 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.015+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:entry[obj00000000000000000017]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000017[0] 2024-08-18T16:19:21.023 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.015+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: listing bilog for incremental sync; position=00000000001.80.6 2024-08-18T16:19:21.023 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.015+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:entry[obj00000000000000000015]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000015[0] 2024-08-18T16:19:21.023 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.015+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: listing bilog for incremental sync; position=00000000002.139.6 2024-08-18T16:19:21.024 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.015+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:entry[obj00000000000000000031]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000031[0] 2024-08-18T16:19:21.024 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.015+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]: listing bilog for incremental sync; position=00000000001.130.6 2024-08-18T16:19:21.024 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.015+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:entry[obj00000000000000000022]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000022[0] 2024-08-18T16:19:21.024 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.015+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: listing bilog for incremental sync; position=00000000002.66.6 2024-08-18T16:19:21.024 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.016+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:entry[obj00000000000000000002]: success 2024-08-18T16:19:21.024 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.016+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5:4 marker=00000000002.103.6 timestamp=2024-08-18T16:19:18.224388+0000 2024-08-18T16:19:21.024 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.016+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: parsed entry: id=00000000002.48.6 iter->object=obj00000000000000000003 iter->instance= name=obj00000000000000000003 instance= ns= 2024-08-18T16:19:21.024 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.016+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18/obj00000000000000000003 2024-08-18T16:19:21.024 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.016+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: entry->timestamp=2024-08-18T16:19:18.167730+0000 2024-08-18T16:19:21.024 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.016+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:entry[obj00000000000000000003]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.48.6 op=0 op_state=1 2024-08-18T16:19:21.024 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.016+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:entry[obj00000000000000000003]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000003[0] 2024-08-18T16:19:21.024 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.016+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: listing bilog for incremental sync; position=00000000002.48.6 2024-08-18T16:19:21.025 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.016+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: parsed entry: id=00000000002.153.6 iter->object=obj00000000000000000005 iter->instance= name=obj00000000000000000005 instance= ns= 2024-08-18T16:19:21.025 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.016+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16/obj00000000000000000005 2024-08-18T16:19:21.025 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.016+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: entry->timestamp=2024-08-18T16:19:17.690778+0000 2024-08-18T16:19:21.025 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.016+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:entry[obj00000000000000000005]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.153.6 op=0 op_state=1 2024-08-18T16:19:21.025 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.016+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: parsed entry: id=00000000002.98.6 iter->object=obj00000000000000000012 iter->instance= name=obj00000000000000000012 instance= ns= 2024-08-18T16:19:21.025 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.016+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17/obj00000000000000000012 2024-08-18T16:19:21.025 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.016+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: entry->timestamp=2024-08-18T16:19:18.104377+0000 2024-08-18T16:19:21.025 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.016+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:entry[obj00000000000000000012]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.98.6 op=0 op_state=1 2024-08-18T16:19:21.025 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.016+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:entry[obj00000000000000000006]: success 2024-08-18T16:19:21.025 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.016+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0:4 marker=00000000002.122.6 timestamp=2024-08-18T16:19:17.976117+0000 2024-08-18T16:19:21.025 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.016+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:entry[obj00000000000000000005]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000005[0] 2024-08-18T16:19:21.025 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.016+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: listing bilog for incremental sync; position=00000000002.153.6 2024-08-18T16:19:21.025 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.016+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:entry[obj00000000000000000012]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000012[0] 2024-08-18T16:19:21.026 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.016+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: listing bilog for incremental sync; position=00000000002.98.6 2024-08-18T16:19:21.026 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.017+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:entry[obj00000000000000000004]: success 2024-08-18T16:19:21.026 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.017+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3:4 marker=00000000002.70.6 timestamp=2024-08-18T16:19:17.671427+0000 2024-08-18T16:19:21.026 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.017+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:entry[obj00000000000000000019]: success 2024-08-18T16:19:21.026 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.017+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6:4 marker=00000000002.75.6 timestamp=2024-08-18T16:19:18.233851+0000 2024-08-18T16:19:21.026 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.017+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:entry[obj00000000000000000011]: success 2024-08-18T16:19:21.026 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.017+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4:4 marker=00000000002.88.6 timestamp=2024-08-18T16:19:18.048298+0000 2024-08-18T16:19:21.026 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.018+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:entry[obj00000000000000000015]: success 2024-08-18T16:19:21.026 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.018+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10:4 marker=00000000002.139.6 timestamp=2024-08-18T16:19:18.057904+0000 2024-08-18T16:19:21.026 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.019+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:entry[obj00000000000000000003]: success 2024-08-18T16:19:21.026 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.019+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18:4 marker=00000000002.48.6 timestamp=2024-08-18T16:19:18.167730+0000 2024-08-18T16:19:21.026 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.020+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]: parsed entry: id=00000000002.82.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2024-08-18T16:19:21.027 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.020+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21/obj00000000000000000001 2024-08-18T16:19:21.027 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.020+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]: entry->timestamp=2024-08-18T16:19:17.928464+0000 2024-08-18T16:19:21.027 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.020+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.82.6 op=0 op_state=1 2024-08-18T16:19:21.027 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.020+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]: parsed entry: id=00000000002.100.6 iter->object=obj00000000000000000014 iter->instance= name=obj00000000000000000014 instance= ns= 2024-08-18T16:19:21.027 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.020+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26/obj00000000000000000014 2024-08-18T16:19:21.027 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.020+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]: entry->timestamp=2024-08-18T16:19:17.736743+0000 2024-08-18T16:19:21.027 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.020+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]:entry[obj00000000000000000014]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.100.6 op=0 op_state=1 2024-08-18T16:19:21.027 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.020+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:entry[obj00000000000000000001]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000001[0] 2024-08-18T16:19:21.027 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.020+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]: listing bilog for incremental sync; position=00000000002.82.6 2024-08-18T16:19:21.027 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.020+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]:entry[obj00000000000000000014]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000014[0] 2024-08-18T16:19:21.027 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.020+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]: listing bilog for incremental sync; position=00000000002.100.6 2024-08-18T16:19:21.027 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.020+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:entry[obj00000000000000000012]: success 2024-08-18T16:19:21.027 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.020+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17:4 marker=00000000002.98.6 timestamp=2024-08-18T16:19:18.104377+0000 2024-08-18T16:19:21.027 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.021+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]: a sync operation returned error 2024-08-18T16:19:21.028 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.021+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]: a sync operation returned error 2024-08-18T16:19:21.028 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.021+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]: backing out with sync_status=-5 2024-08-18T16:19:21.028 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.021+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]: incremental sync on bucket failed, retcode=-5 2024-08-18T16:19:21.028 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.021+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]: backing out with sync_status=-5 2024-08-18T16:19:21.028 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.021+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]: ERROR: incremental sync failed. error: -5 2024-08-18T16:19:21.028 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.021+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]: incremental sync on bucket failed, retcode=-5 2024-08-18T16:19:21.028 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.021+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]: ERROR: incremental sync failed. error: -5 2024-08-18T16:19:21.029 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.022+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:entry[obj00000000000000000001]: success 2024-08-18T16:19:21.029 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.022+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21:4 marker=00000000002.82.6 timestamp=2024-08-18T16:19:17.928464+0000 2024-08-18T16:19:21.029 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.022+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:21.029 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.022+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:21.029 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.023+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]:entry[obj00000000000000000014]: success 2024-08-18T16:19:21.030 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.023+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26:4 marker=00000000002.100.6 timestamp=2024-08-18T16:19:17.736743+0000 2024-08-18T16:19:21.030 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.023+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]: sync status for source bucket shard: 0 2024-08-18T16:19:21.030 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.023+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]: listing bilog for incremental sync; position= 2024-08-18T16:19:21.030 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.024+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]: sync status for source bucket shard: 0 2024-08-18T16:19:21.030 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.024+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]: listing bilog for incremental sync; position= 2024-08-18T16:19:21.030 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.024+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: parsed entry: id=00000000002.105.6 iter->object=obj00000000000000000020 iter->instance= name=obj00000000000000000020 instance= ns= 2024-08-18T16:19:21.030 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.024+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14/obj00000000000000000020 2024-08-18T16:19:21.030 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.024+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: entry->timestamp=2024-08-18T16:19:17.856631+0000 2024-08-18T16:19:21.030 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.024+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:entry[obj00000000000000000020]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.105.6 op=0 op_state=1 2024-08-18T16:19:21.030 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.024+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:entry[obj00000000000000000020]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000020[0] 2024-08-18T16:19:21.030 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.024+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: listing bilog for incremental sync; position=00000000002.105.6 2024-08-18T16:19:21.030 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.025+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:entry[obj00000000000000000022]: success 2024-08-18T16:19:21.030 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.025+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12:4 marker=00000000002.66.6 timestamp=2024-08-18T16:19:17.801074+0000 2024-08-18T16:19:21.031 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.025+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:entry[obj00000000000000000005]: success 2024-08-18T16:19:21.031 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.025+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16:4 marker=00000000002.153.6 timestamp=2024-08-18T16:19:17.690778+0000 2024-08-18T16:19:21.031 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.025+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:entry[obj00000000000000000031]: success 2024-08-18T16:19:21.031 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.025+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11:4 marker=00000000001.130.6 timestamp=2024-08-18T16:19:17.912320+0000 2024-08-18T16:19:21.031 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.026+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:entry[obj00000000000000000020]: success 2024-08-18T16:19:21.031 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.026+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14:4 marker=00000000002.105.6 timestamp=2024-08-18T16:19:17.856631+0000 2024-08-18T16:19:21.056 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.056+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: a sync operation returned error 2024-08-18T16:19:21.056 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.056+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: backing out with sync_status=-5 2024-08-18T16:19:21.056 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.056+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: incremental sync on bucket failed, retcode=-5 2024-08-18T16:19:21.056 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.056+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: ERROR: incremental sync failed. error: -5 2024-08-18T16:19:21.057 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.057+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:21.057 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.057+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: sync status for source bucket shard: 0 2024-08-18T16:19:21.058 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.057+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: listing bilog for incremental sync; position= 2024-08-18T16:19:21.058 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.058+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: a sync operation returned error 2024-08-18T16:19:21.058 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.058+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: backing out with sync_status=-5 2024-08-18T16:19:21.058 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.058+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: incremental sync on bucket failed, retcode=-5 2024-08-18T16:19:21.058 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.058+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: ERROR: incremental sync failed. error: -5 2024-08-18T16:19:21.059 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.059+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:21.060 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.059+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: sync status for source bucket shard: 0 2024-08-18T16:19:21.060 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.059+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: listing bilog for incremental sync; position= 2024-08-18T16:19:21.066 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.066+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]: parsed entry: id=00000000002.52.6 iter->object=obj00000000000000000025 iter->instance= name=obj00000000000000000025 instance= ns= 2024-08-18T16:19:21.066 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.066+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23/obj00000000000000000025 2024-08-18T16:19:21.066 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.066+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]: entry->timestamp=2024-08-18T16:19:17.792369+0000 2024-08-18T16:19:21.066 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.066+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:entry[obj00000000000000000025]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.52.6 op=0 op_state=1 2024-08-18T16:19:21.067 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.066+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:entry[obj00000000000000000025]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000025[0] 2024-08-18T16:19:21.067 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.066+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]: listing bilog for incremental sync; position=00000000002.52.6 2024-08-18T16:19:21.067 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.067+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]: parsed entry: id=00000000002.80.6 iter->object=obj00000000000000000009 iter->instance= name=obj00000000000000000009 instance= ns= 2024-08-18T16:19:21.067 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.067+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22/obj00000000000000000009 2024-08-18T16:19:21.067 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.067+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]: entry->timestamp=2024-08-18T16:19:17.919915+0000 2024-08-18T16:19:21.067 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.067+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:entry[obj00000000000000000009]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.80.6 op=0 op_state=1 2024-08-18T16:19:21.067 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.067+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:entry[obj00000000000000000009]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000009[0] 2024-08-18T16:19:21.068 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.067+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]: listing bilog for incremental sync; position=00000000002.80.6 2024-08-18T16:19:21.068 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.067+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:entry[obj00000000000000000025]: success 2024-08-18T16:19:21.068 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.067+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23:4 marker=00000000002.52.6 timestamp=2024-08-18T16:19:17.792369+0000 2024-08-18T16:19:21.069 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.068+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:entry[obj00000000000000000009]: success 2024-08-18T16:19:21.069 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.068+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22:4 marker=00000000002.80.6 timestamp=2024-08-18T16:19:17.919915+0000 2024-08-18T16:19:21.069 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.069+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:entry[obj00000000000000000017]: success 2024-08-18T16:19:21.069 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.069+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8:4 marker=00000000001.80.6 timestamp=2024-08-18T16:19:17.865804+0000 2024-08-18T16:19:21.100 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.100+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: parsed entry: id=00000000001.54.6 iter->object=obj00000000000000000059 iter->instance= name=obj00000000000000000059 instance= ns= 2024-08-18T16:19:21.100 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.100+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1/obj00000000000000000059 2024-08-18T16:19:21.101 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.100+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: entry->timestamp=2024-08-18T16:19:18.040381+0000 2024-08-18T16:19:21.101 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.100+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:entry[obj00000000000000000059]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.54.6 op=0 op_state=1 2024-08-18T16:19:21.101 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.100+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:entry[obj00000000000000000059]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000059[0] 2024-08-18T16:19:21.101 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.100+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: listing bilog for incremental sync; position=00000000001.54.6 2024-08-18T16:19:21.102 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.102+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: parsed entry: id=00000000001.155.6 iter->object=obj00000000000000000049 iter->instance= name=obj00000000000000000049 instance= ns= 2024-08-18T16:19:21.102 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.102+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7/obj00000000000000000049 2024-08-18T16:19:21.102 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.102+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: entry->timestamp=2024-08-18T16:19:17.984613+0000 2024-08-18T16:19:21.102 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.102+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:entry[obj00000000000000000049]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.155.6 op=0 op_state=1 2024-08-18T16:19:21.102 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.102+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:entry[obj00000000000000000049]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000049[0] 2024-08-18T16:19:21.102 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.102+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: listing bilog for incremental sync; position=00000000001.155.6 2024-08-18T16:19:21.106 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.106+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:entry[obj00000000000000000049]: success 2024-08-18T16:19:21.107 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.106+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7:4 marker=00000000001.155.6 timestamp=2024-08-18T16:19:17.984613+0000 2024-08-18T16:19:21.145 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.144+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:entry[obj00000000000000000059]: success 2024-08-18T16:19:21.145 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.144+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1:4 marker=00000000001.54.6 timestamp=2024-08-18T16:19:18.040381+0000 2024-08-18T16:19:21.159 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.150+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:21.159 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.150+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:21.159 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.150+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:21.159 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.150+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:21.159 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.150+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:21.159 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.150+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: sync status for source bucket shard: 0 2024-08-18T16:19:21.159 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.150+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: listing bilog for incremental sync; position= 2024-08-18T16:19:21.160 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.150+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: sync status for source bucket shard: 0 2024-08-18T16:19:21.160 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.151+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: listing bilog for incremental sync; position= 2024-08-18T16:19:21.160 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.151+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: sync status for source bucket shard: 0 2024-08-18T16:19:21.160 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.151+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: sync status for source bucket shard: 0 2024-08-18T16:19:21.160 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.151+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: listing bilog for incremental sync; position= 2024-08-18T16:19:21.160 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.151+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: listing bilog for incremental sync; position= 2024-08-18T16:19:21.160 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.151+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:21.160 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.151+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: sync status for source bucket shard: 0 2024-08-18T16:19:21.160 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.151+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: listing bilog for incremental sync; position= 2024-08-18T16:19:21.160 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.151+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:21.161 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.151+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:21.161 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.151+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: sync status for source bucket shard: 0 2024-08-18T16:19:21.161 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.151+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: listing bilog for incremental sync; position= 2024-08-18T16:19:21.161 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.151+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:21.161 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.151+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: sync status for source bucket shard: 0 2024-08-18T16:19:21.161 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.151+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: sync status for source bucket shard: 0 2024-08-18T16:19:21.161 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.151+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: listing bilog for incremental sync; position= 2024-08-18T16:19:21.161 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.151+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: listing bilog for incremental sync; position= 2024-08-18T16:19:21.161 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.151+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:21.161 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.152+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:21.161 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.152+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: sync status for source bucket shard: 0 2024-08-18T16:19:21.162 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.152+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: listing bilog for incremental sync; position= 2024-08-18T16:19:21.162 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.152+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:21.162 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.152+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: sync status for source bucket shard: 0 2024-08-18T16:19:21.162 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.152+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: listing bilog for incremental sync; position= 2024-08-18T16:19:21.162 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.152+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: sync status for source bucket shard: 0 2024-08-18T16:19:21.162 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.152+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: listing bilog for incremental sync; position= 2024-08-18T16:19:21.162 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.152+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:21.162 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.152+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]: sync status for source bucket shard: 0 2024-08-18T16:19:21.162 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.152+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]: listing bilog for incremental sync; position= 2024-08-18T16:19:21.162 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.152+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:21.163 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.152+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]: sync status for source bucket shard: 0 2024-08-18T16:19:21.163 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.152+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]: listing bilog for incremental sync; position= 2024-08-18T16:19:21.163 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.153+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:21.163 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.153+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: parsed entry: id=00000000002.75.6 iter->object=obj00000000000000000005 iter->instance= name=obj00000000000000000005 instance= ns= 2024-08-18T16:19:21.163 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.153+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3/obj00000000000000000005 2024-08-18T16:19:21.163 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.153+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: entry->timestamp=2024-08-18T16:19:18.560050+0000 2024-08-18T16:19:21.163 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.153+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:entry[obj00000000000000000005]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.75.6 op=0 op_state=1 2024-08-18T16:19:21.163 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.153+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:entry[obj00000000000000000005]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000005[0] 2024-08-18T16:19:21.163 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.153+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: listing bilog for incremental sync; position=00000000002.75.6 2024-08-18T16:19:21.163 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.153+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: sync status for source bucket shard: 0 2024-08-18T16:19:21.164 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.153+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: listing bilog for incremental sync; position= 2024-08-18T16:19:21.164 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.153+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: sync status for source bucket shard: 0 2024-08-18T16:19:21.164 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.153+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: parsed entry: id=00000000002.73.6 iter->object=obj00000000000000000006 iter->instance= name=obj00000000000000000006 instance= ns= 2024-08-18T16:19:21.164 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.153+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1/obj00000000000000000006 2024-08-18T16:19:21.164 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.153+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: entry->timestamp=2024-08-18T16:19:18.673659+0000 2024-08-18T16:19:21.164 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.153+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:entry[obj00000000000000000006]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.73.6 op=0 op_state=1 2024-08-18T16:19:21.164 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.153+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: listing bilog for incremental sync; position= 2024-08-18T16:19:21.164 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.153+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:entry[obj00000000000000000006]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000006[0] 2024-08-18T16:19:21.164 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.153+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: listing bilog for incremental sync; position=00000000002.73.6 2024-08-18T16:19:21.165 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.153+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: parsed entry: id=00000000001.90.6 iter->object=obj00000000000000000056 iter->instance= name=obj00000000000000000056 instance= ns= 2024-08-18T16:19:21.165 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.153+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4/obj00000000000000000056 2024-08-18T16:19:21.165 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.153+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: entry->timestamp=2024-08-18T16:19:19.144248+0000 2024-08-18T16:19:21.165 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.153+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:entry[obj00000000000000000056]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.90.6 op=0 op_state=1 2024-08-18T16:19:21.165 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.153+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:21.165 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.153+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: parsed entry: id=00000000002.75.6 iter->object=obj00000000000000000004 iter->instance= name=obj00000000000000000004 instance= ns= 2024-08-18T16:19:21.165 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.153+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5/obj00000000000000000004 2024-08-18T16:19:21.165 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.153+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: entry->timestamp=2024-08-18T16:19:19.204480+0000 2024-08-18T16:19:21.165 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.153+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:entry[obj00000000000000000004]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.75.6 op=0 op_state=1 2024-08-18T16:19:21.165 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.153+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:entry[obj00000000000000000056]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000056[0] 2024-08-18T16:19:21.166 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.153+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: listing bilog for incremental sync; position=00000000001.90.6 2024-08-18T16:19:21.166 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.153+0000 7f63ec126880 0 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:entry[obj00000000000000000004]: : injecting data sync error on key=obj00000000000000000004 2024-08-18T16:19:21.166 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.153+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: listing bilog for incremental sync; position=00000000002.75.6 2024-08-18T16:19:21.166 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.154+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:entry[obj00000000000000000004]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:19:21.166 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.154+0000 7f63ec126880 0 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:entry[obj00000000000000000004]: ERROR: failed to sync object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5/obj00000000000000000004 2024-08-18T16:19:21.166 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.154+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:21.166 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.154+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: parsed entry: id=00000000002.126.6 iter->object=obj00000000000000000023 iter->instance= name=obj00000000000000000023 instance= ns= 2024-08-18T16:19:21.166 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.154+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6/obj00000000000000000023 2024-08-18T16:19:21.166 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.154+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: entry->timestamp=2024-08-18T16:19:18.904686+0000 2024-08-18T16:19:21.166 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.154+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:entry[obj00000000000000000023]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.126.6 op=0 op_state=1 2024-08-18T16:19:21.167 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.154+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:21.167 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.154+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:21.167 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.154+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:entry[obj00000000000000000023]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000023[0] 2024-08-18T16:19:21.167 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.154+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: listing bilog for incremental sync; position=00000000002.126.6 2024-08-18T16:19:21.167 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.154+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: parsed entry: id=00000000002.83.6 iter->object=obj00000000000000000000 iter->instance= name=obj00000000000000000000 instance= ns= 2024-08-18T16:19:21.167 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.154+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9/obj00000000000000000000 2024-08-18T16:19:21.167 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.154+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: entry->timestamp=2024-08-18T16:19:18.665114+0000 2024-08-18T16:19:21.167 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.154+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:entry[obj00000000000000000000]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.83.6 op=0 op_state=1 2024-08-18T16:19:21.167 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.154+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: parsed entry: id=00000000001.88.6 iter->object=obj00000000000000000317 iter->instance= name=obj00000000000000000317 instance= ns= 2024-08-18T16:19:21.167 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.154+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7/obj00000000000000000317 2024-08-18T16:19:21.167 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.154+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: entry->timestamp=2024-08-18T16:19:18.784378+0000 2024-08-18T16:19:21.168 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.154+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:entry[obj00000000000000000317]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.88.6 op=0 op_state=1 2024-08-18T16:19:21.168 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.154+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:21.168 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.154+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: parsed entry: id=00000000002.122.6 iter->object=obj00000000000000000022 iter->instance= name=obj00000000000000000022 instance= ns= 2024-08-18T16:19:21.168 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.154+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8/obj00000000000000000022 2024-08-18T16:19:21.168 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.154+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: entry->timestamp=2024-08-18T16:19:18.551431+0000 2024-08-18T16:19:21.168 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.154+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:entry[obj00000000000000000022]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.122.6 op=0 op_state=1 2024-08-18T16:19:21.168 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.154+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: parsed entry: id=00000000002.85.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2024-08-18T16:19:21.168 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.154+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10/obj00000000000000000021 2024-08-18T16:19:21.168 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.154+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: entry->timestamp=2024-08-18T16:19:18.968460+0000 2024-08-18T16:19:21.168 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.154+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.85.6 op=0 op_state=1 2024-08-18T16:19:21.169 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.154+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:21.169 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.154+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:entry[obj00000000000000000000]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000000[0] 2024-08-18T16:19:21.169 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.154+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: listing bilog for incremental sync; position=00000000002.83.6 2024-08-18T16:19:21.169 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.154+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:entry[obj00000000000000000317]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000317[0] 2024-08-18T16:19:21.169 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.154+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: listing bilog for incremental sync; position=00000000001.88.6 2024-08-18T16:19:21.169 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.154+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:21.169 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.154+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:entry[obj00000000000000000022]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000022[0] 2024-08-18T16:19:21.169 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.155+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: listing bilog for incremental sync; position=00000000002.122.6 2024-08-18T16:19:21.169 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.155+0000 7f63ec126880 0 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:entry[obj00000000000000000021]: : injecting data sync error on key=obj00000000000000000021 2024-08-18T16:19:21.169 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.155+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: listing bilog for incremental sync; position=00000000002.85.6 2024-08-18T16:19:21.170 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.155+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:entry[obj00000000000000000021]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:19:21.170 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.155+0000 7f63ec126880 0 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:entry[obj00000000000000000021]: ERROR: failed to sync object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10/obj00000000000000000021 2024-08-18T16:19:21.170 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.155+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:21.170 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.155+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:21.170 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.155+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: sync status for source bucket shard: 0 2024-08-18T16:19:21.170 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.155+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: listing bilog for incremental sync; position= 2024-08-18T16:19:21.170 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.155+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:21.170 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.155+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: sync status for source bucket shard: 0 2024-08-18T16:19:21.170 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.155+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: sync status for source bucket shard: 0 2024-08-18T16:19:21.170 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.155+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:21.171 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.156+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: listing bilog for incremental sync; position= 2024-08-18T16:19:21.171 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.156+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: listing bilog for incremental sync; position= 2024-08-18T16:19:21.171 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.156+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:21.171 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.156+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:21.171 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.156+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:entry[obj00000000000000000005]: success 2024-08-18T16:19:21.171 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.156+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3:5 marker=00000000002.75.6 timestamp=2024-08-18T16:19:18.560050+0000 2024-08-18T16:19:21.171 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.156+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:28]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:21.171 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.156+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: sync status for source bucket shard: 0 2024-08-18T16:19:21.171 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.156+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: listing bilog for incremental sync; position= 2024-08-18T16:19:21.171 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.156+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:21.171 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.156+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]: sync status for source bucket shard: 0 2024-08-18T16:19:21.171 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.156+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]: listing bilog for incremental sync; position= 2024-08-18T16:19:21.172 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.156+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]: sync status for source bucket shard: 0 2024-08-18T16:19:21.172 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.156+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]: listing bilog for incremental sync; position= 2024-08-18T16:19:21.172 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.156+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]: sync status for source bucket shard: 0 2024-08-18T16:19:21.172 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.156+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]: sync status for source bucket shard: 0 2024-08-18T16:19:21.172 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.156+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]: sync status for source bucket shard: 0 2024-08-18T16:19:21.172 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.156+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]: sync status for source bucket shard: 0 2024-08-18T16:19:21.172 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.156+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]: listing bilog for incremental sync; position= 2024-08-18T16:19:21.172 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.156+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]: listing bilog for incremental sync; position= 2024-08-18T16:19:21.172 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.156+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]: listing bilog for incremental sync; position= 2024-08-18T16:19:21.172 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.156+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]: listing bilog for incremental sync; position= 2024-08-18T16:19:21.172 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.156+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:30]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:21.173 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.157+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:28]: sync status for source bucket shard: 0 2024-08-18T16:19:21.173 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.157+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:28]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:28]: listing bilog for incremental sync; position= 2024-08-18T16:19:21.173 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.157+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]: sync status for source bucket shard: 0 2024-08-18T16:19:21.173 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.157+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]: listing bilog for incremental sync; position= 2024-08-18T16:19:21.173 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.157+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:entry[obj00000000000000000006]: success 2024-08-18T16:19:21.173 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.157+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1:5 marker=00000000002.73.6 timestamp=2024-08-18T16:19:18.673659+0000 2024-08-18T16:19:21.173 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.157+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25]: sync status for source bucket shard: 0 2024-08-18T16:19:21.173 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.157+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]: sync status for source bucket shard: 0 2024-08-18T16:19:21.173 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.157+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25]: listing bilog for incremental sync; position= 2024-08-18T16:19:21.173 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.158+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]: listing bilog for incremental sync; position= 2024-08-18T16:19:21.174 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.158+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]: sync status for source bucket shard: 0 2024-08-18T16:19:21.174 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.158+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]: listing bilog for incremental sync; position= 2024-08-18T16:19:21.174 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.158+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: parsed entry: id=00000000002.53.6 iter->object=obj00000000000000000007 iter->instance= name=obj00000000000000000007 instance= ns= 2024-08-18T16:19:21.174 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.158+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15/obj00000000000000000007 2024-08-18T16:19:21.174 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.158+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: entry->timestamp=2024-08-18T16:19:18.912352+0000 2024-08-18T16:19:21.174 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.158+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:entry[obj00000000000000000007]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.53.6 op=0 op_state=1 2024-08-18T16:19:21.174 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.158+0000 7f63ec126880 0 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:entry[obj00000000000000000007]: : injecting data sync error on key=obj00000000000000000007 2024-08-18T16:19:21.174 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.158+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: listing bilog for incremental sync; position=00000000002.53.6 2024-08-18T16:19:21.174 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.158+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:entry[obj00000000000000000007]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:19:21.175 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.158+0000 7f63ec126880 0 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:entry[obj00000000000000000007]: ERROR: failed to sync object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15/obj00000000000000000007 2024-08-18T16:19:21.175 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.158+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:30]: sync status for source bucket shard: 0 2024-08-18T16:19:21.175 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.158+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:30]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:30]: listing bilog for incremental sync; position= 2024-08-18T16:19:21.175 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.158+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: a sync operation returned error 2024-08-18T16:19:21.175 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.158+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: backing out with sync_status=-5 2024-08-18T16:19:21.175 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.158+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: incremental sync on bucket failed, retcode=-5 2024-08-18T16:19:21.175 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.158+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: ERROR: incremental sync failed. error: -5 2024-08-18T16:19:21.178 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.159+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: parsed entry: id=00000000002.106.6 iter->object=obj00000000000000000017 iter->instance= name=obj00000000000000000017 instance= ns= 2024-08-18T16:19:21.179 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.159+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16/obj00000000000000000017 2024-08-18T16:19:21.179 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.159+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: entry->timestamp=2024-08-18T16:19:18.792258+0000 2024-08-18T16:19:21.179 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.159+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:entry[obj00000000000000000017]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.106.6 op=0 op_state=1 2024-08-18T16:19:21.179 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.159+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:entry[obj00000000000000000017]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000017[0] 2024-08-18T16:19:21.179 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.159+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: listing bilog for incremental sync; position=00000000002.106.6 2024-08-18T16:19:21.179 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.159+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: parsed entry: id=00000000002.110.6 iter->object=obj00000000000000000016 iter->instance= name=obj00000000000000000016 instance= ns= 2024-08-18T16:19:21.179 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.159+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18/obj00000000000000000016 2024-08-18T16:19:21.179 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.159+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: entry->timestamp=2024-08-18T16:19:18.848534+0000 2024-08-18T16:19:21.179 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.159+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:entry[obj00000000000000000016]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.110.6 op=0 op_state=1 2024-08-18T16:19:21.179 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.159+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: parsed entry: id=00000000001.82.6 iter->object=obj00000000000000000162 iter->instance= name=obj00000000000000000162 instance= ns= 2024-08-18T16:19:21.179 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.159+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17/obj00000000000000000162 2024-08-18T16:19:21.180 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.159+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: entry->timestamp=2024-08-18T16:19:18.617486+0000 2024-08-18T16:19:21.180 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.159+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:entry[obj00000000000000000162]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.82.6 op=0 op_state=1 2024-08-18T16:19:21.180 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.159+0000 7f63ec126880 0 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:entry[obj00000000000000000016]: : injecting data sync error on key=obj00000000000000000016 2024-08-18T16:19:21.180 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.159+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: listing bilog for incremental sync; position=00000000002.110.6 2024-08-18T16:19:21.180 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.159+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:entry[obj00000000000000000162]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000162[0] 2024-08-18T16:19:21.180 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.159+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: listing bilog for incremental sync; position=00000000001.82.6 2024-08-18T16:19:21.180 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.159+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:entry[obj00000000000000000016]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:19:21.180 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.159+0000 7f63ec126880 0 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:entry[obj00000000000000000016]: ERROR: failed to sync object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18/obj00000000000000000016 2024-08-18T16:19:21.180 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.160+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]: parsed entry: id=00000000001.88.6 iter->object=obj00000000000000000029 iter->instance= name=obj00000000000000000029 instance= ns= 2024-08-18T16:19:21.181 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.160+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29/obj00000000000000000029 2024-08-18T16:19:21.181 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.160+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]: entry->timestamp=2024-08-18T16:19:18.977845+0000 2024-08-18T16:19:21.181 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.160+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]:entry[obj00000000000000000029]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.88.6 op=0 op_state=1 2024-08-18T16:19:21.181 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.160+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]:entry[obj00000000000000000029]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000029[0] 2024-08-18T16:19:21.181 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.160+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]: listing bilog for incremental sync; position=00000000001.88.6 2024-08-18T16:19:21.181 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.160+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]: parsed entry: id=00000000002.185.6 iter->object=obj00000000000000000012 iter->instance= name=obj00000000000000000012 instance= ns= 2024-08-18T16:19:21.181 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.160+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22/obj00000000000000000012 2024-08-18T16:19:21.181 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.160+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]: entry->timestamp=2024-08-18T16:19:19.024938+0000 2024-08-18T16:19:21.181 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.160+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:entry[obj00000000000000000012]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.185.6 op=0 op_state=1 2024-08-18T16:19:21.182 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.160+0000 7f63ec126880 0 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:entry[obj00000000000000000012]: : injecting data sync error on key=obj00000000000000000012 2024-08-18T16:19:21.182 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.160+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]: listing bilog for incremental sync; position=00000000002.185.6 2024-08-18T16:19:21.182 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.160+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:entry[obj00000000000000000012]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:19:21.182 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.160+0000 7f63ec126880 0 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:entry[obj00000000000000000012]: ERROR: failed to sync object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22/obj00000000000000000012 2024-08-18T16:19:21.182 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.160+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:entry[obj00000000000000000022]: success 2024-08-18T16:19:21.182 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.160+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8:5 marker=00000000002.122.6 timestamp=2024-08-18T16:19:18.551431+0000 2024-08-18T16:19:21.182 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.161+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]: parsed entry: id=00000000002.82.6 iter->object=obj00000000000000000003 iter->instance= name=obj00000000000000000003 instance= ns= 2024-08-18T16:19:21.182 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.161+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19/obj00000000000000000003 2024-08-18T16:19:21.182 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.161+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]: entry->timestamp=2024-08-18T16:19:19.080473+0000 2024-08-18T16:19:21.182 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.161+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:entry[obj00000000000000000003]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.82.6 op=0 op_state=1 2024-08-18T16:19:21.182 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.161+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:entry[obj00000000000000000003]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000003[0] 2024-08-18T16:19:21.183 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.161+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]: listing bilog for incremental sync; position=00000000002.82.6 2024-08-18T16:19:21.183 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.161+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]: parsed entry: id=00000000002.86.6 iter->object=obj00000000000000000010 iter->instance= name=obj00000000000000000010 instance= ns= 2024-08-18T16:19:21.183 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.161+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26/obj00000000000000000010 2024-08-18T16:19:21.183 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.161+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]: entry->timestamp=2024-08-18T16:19:18.608847+0000 2024-08-18T16:19:21.183 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.161+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]:entry[obj00000000000000000010]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.86.6 op=0 op_state=1 2024-08-18T16:19:21.183 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.161+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]:entry[obj00000000000000000010]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000010[0] 2024-08-18T16:19:21.183 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.161+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]: listing bilog for incremental sync; position=00000000002.86.6 2024-08-18T16:19:21.183 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.161+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:entry[obj00000000000000000000]: success 2024-08-18T16:19:21.183 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.161+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9:5 marker=00000000002.83.6 timestamp=2024-08-18T16:19:18.665114+0000 2024-08-18T16:19:21.183 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.161+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: a sync operation returned error 2024-08-18T16:19:21.183 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.161+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]: parsed entry: id=00000000002.103.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2024-08-18T16:19:21.184 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.161+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23/obj00000000000000000001 2024-08-18T16:19:21.184 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.161+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]: entry->timestamp=2024-08-18T16:19:19.089586+0000 2024-08-18T16:19:21.184 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.161+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.103.6 op=0 op_state=1 2024-08-18T16:19:21.184 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.161+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: backing out with sync_status=-5 2024-08-18T16:19:21.184 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.161+0000 7f63ec126880 0 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:entry[obj00000000000000000001]: : injecting data sync error on key=obj00000000000000000001 2024-08-18T16:19:21.184 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.162+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]: listing bilog for incremental sync; position=00000000002.103.6 2024-08-18T16:19:21.184 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.162+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: incremental sync on bucket failed, retcode=-5 2024-08-18T16:19:21.184 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.162+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:entry[obj00000000000000000001]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:19:21.184 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.162+0000 7f63ec126880 0 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:entry[obj00000000000000000001]: ERROR: failed to sync object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23/obj00000000000000000001 2024-08-18T16:19:21.184 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.162+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: ERROR: incremental sync failed. error: -5 2024-08-18T16:19:21.185 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.162+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]: parsed entry: id=00000000002.112.6 iter->object=obj00000000000000000011 iter->instance= name=obj00000000000000000011 instance= ns= 2024-08-18T16:19:21.185 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.162+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24/obj00000000000000000011 2024-08-18T16:19:21.185 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.162+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]: entry->timestamp=2024-08-18T16:19:19.033586+0000 2024-08-18T16:19:21.185 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.162+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]:entry[obj00000000000000000011]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.112.6 op=0 op_state=1 2024-08-18T16:19:21.185 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.162+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]:entry[obj00000000000000000011]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000011[0] 2024-08-18T16:19:21.185 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.162+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]: listing bilog for incremental sync; position=00000000002.112.6 2024-08-18T16:19:21.185 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.162+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]: parsed entry: id=00000000001.124.6 iter->object=obj00000000000000000423 iter->instance= name=obj00000000000000000423 instance= ns= 2024-08-18T16:19:21.185 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.162+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20/obj00000000000000000423 2024-08-18T16:19:21.185 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.162+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]: entry->timestamp=2024-08-18T16:19:18.720502+0000 2024-08-18T16:19:21.186 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.162+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:entry[obj00000000000000000423]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.124.6 op=0 op_state=1 2024-08-18T16:19:21.186 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.162+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:30]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:30]: parsed entry: id=00000000001.114.6 iter->object=obj00000000000000000133 iter->instance= name=obj00000000000000000133 instance= ns= 2024-08-18T16:19:21.186 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.162+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:30]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:30]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:30/obj00000000000000000133 2024-08-18T16:19:21.186 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.162+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:30]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:30]: entry->timestamp=2024-08-18T16:19:19.158889+0000 2024-08-18T16:19:21.186 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.162+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:30]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:30]:entry[obj00000000000000000133]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.114.6 op=0 op_state=1 2024-08-18T16:19:21.186 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.162+0000 7f63ec126880 0 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:entry[obj00000000000000000423]: : injecting data sync error on key=obj00000000000000000423 2024-08-18T16:19:21.186 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.162+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]: listing bilog for incremental sync; position=00000000001.124.6 2024-08-18T16:19:21.186 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.162+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:30]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:30]:entry[obj00000000000000000133]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000133[0] 2024-08-18T16:19:21.186 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.162+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:30]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:30]: listing bilog for incremental sync; position=00000000001.114.6 2024-08-18T16:19:21.186 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.162+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]: parsed entry: id=00000000002.150.6 iter->object=obj00000000000000000002 iter->instance= name=obj00000000000000000002 instance= ns= 2024-08-18T16:19:21.186 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.162+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21/obj00000000000000000002 2024-08-18T16:19:21.187 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.162+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]: entry->timestamp=2024-08-18T16:19:18.921737+0000 2024-08-18T16:19:21.187 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.162+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:entry[obj00000000000000000002]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.150.6 op=0 op_state=1 2024-08-18T16:19:21.187 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.162+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:21.187 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.162+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:entry[obj00000000000000000056]: success 2024-08-18T16:19:21.187 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.162+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4:5 marker=00000000001.90.6 timestamp=2024-08-18T16:19:19.144248+0000 2024-08-18T16:19:21.187 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.162+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:entry[obj00000000000000000423]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:19:21.187 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.162+0000 7f63ec126880 0 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:entry[obj00000000000000000423]: ERROR: failed to sync object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20/obj00000000000000000423 2024-08-18T16:19:21.187 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.162+0000 7f63ec126880 0 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:entry[obj00000000000000000002]: : injecting data sync error on key=obj00000000000000000002 2024-08-18T16:19:21.187 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.162+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]: listing bilog for incremental sync; position=00000000002.150.6 2024-08-18T16:19:21.187 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.163+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:entry[obj00000000000000000017]: success 2024-08-18T16:19:21.187 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.163+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16:5 marker=00000000002.106.6 timestamp=2024-08-18T16:19:18.792258+0000 2024-08-18T16:19:21.188 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.163+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:entry[obj00000000000000000002]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:19:21.188 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.163+0000 7f63ec126880 0 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:entry[obj00000000000000000002]: ERROR: failed to sync object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21/obj00000000000000000002 2024-08-18T16:19:21.188 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.164+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]:entry[obj00000000000000000010]: success 2024-08-18T16:19:21.188 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.164+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26:5 marker=00000000002.86.6 timestamp=2024-08-18T16:19:18.608847+0000 2024-08-18T16:19:21.188 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.164+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: sync status for source bucket shard: 0 2024-08-18T16:19:21.188 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.164+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: listing bilog for incremental sync; position= 2024-08-18T16:19:21.188 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.164+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:entry[obj00000000000000000003]: success 2024-08-18T16:19:21.188 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.164+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19:5 marker=00000000002.82.6 timestamp=2024-08-18T16:19:19.080473+0000 2024-08-18T16:19:21.188 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.165+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:21.188 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.166+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: sync status for source bucket shard: 0 2024-08-18T16:19:21.189 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.166+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: listing bilog for incremental sync; position= 2024-08-18T16:19:21.189 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.167+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]:entry[obj00000000000000000011]: success 2024-08-18T16:19:21.189 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.167+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24:5 marker=00000000002.112.6 timestamp=2024-08-18T16:19:19.033586+0000 2024-08-18T16:19:21.189 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.167+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: parsed entry: id=00000000002.53.6 iter->object=obj00000000000000000007 iter->instance= name=obj00000000000000000007 instance= ns= 2024-08-18T16:19:21.189 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.167+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15/obj00000000000000000007 2024-08-18T16:19:21.189 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.167+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: entry->timestamp=2024-08-18T16:19:18.912352+0000 2024-08-18T16:19:21.189 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.167+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:entry[obj00000000000000000007]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.53.6 op=0 op_state=1 2024-08-18T16:19:21.189 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.167+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:entry[obj00000000000000000007]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000007[0] 2024-08-18T16:19:21.189 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.167+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: listing bilog for incremental sync; position=00000000002.53.6 2024-08-18T16:19:21.189 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.168+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:entry[obj00000000000000000023]: success 2024-08-18T16:19:21.190 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.168+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6:5 marker=00000000002.126.6 timestamp=2024-08-18T16:19:18.904686+0000 2024-08-18T16:19:21.190 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.169+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:entry[obj00000000000000000162]: success 2024-08-18T16:19:21.190 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.169+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17:5 marker=00000000001.82.6 timestamp=2024-08-18T16:19:18.617486+0000 2024-08-18T16:19:21.190 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.169+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]:entry[obj00000000000000000029]: success 2024-08-18T16:19:21.190 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.169+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29:5 marker=00000000001.88.6 timestamp=2024-08-18T16:19:18.977845+0000 2024-08-18T16:19:21.190 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.169+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:entry[obj00000000000000000007]: success 2024-08-18T16:19:21.190 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.169+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15:5 marker=00000000002.53.6 timestamp=2024-08-18T16:19:18.912352+0000 2024-08-18T16:19:21.190 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.171+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:30]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:30]:entry[obj00000000000000000133]: success 2024-08-18T16:19:21.190 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.171+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:30]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:30]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:30:5 marker=00000000001.114.6 timestamp=2024-08-18T16:19:19.158889+0000 2024-08-18T16:19:21.194 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.193+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: parsed entry: id=00000000001.82.6 iter->object=obj00000000000000000467 iter->instance= name=obj00000000000000000467 instance= ns= 2024-08-18T16:19:21.194 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.193+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2/obj00000000000000000467 2024-08-18T16:19:21.194 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.193+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: entry->timestamp=2024-08-18T16:19:19.151651+0000 2024-08-18T16:19:21.194 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.193+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:entry[obj00000000000000000467]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.82.6 op=0 op_state=1 2024-08-18T16:19:21.194 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.193+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:entry[obj00000000000000000467]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000467[0] 2024-08-18T16:19:21.194 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.193+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: listing bilog for incremental sync; position=00000000001.82.6 2024-08-18T16:19:21.194 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.193+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: parsed entry: id=00000000002.163.6 iter->object=obj00000000000000000019 iter->instance= name=obj00000000000000000019 instance= ns= 2024-08-18T16:19:21.194 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.193+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0/obj00000000000000000019 2024-08-18T16:19:21.195 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.193+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: entry->timestamp=2024-08-18T16:19:18.736467+0000 2024-08-18T16:19:21.195 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.193+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:entry[obj00000000000000000019]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.163.6 op=0 op_state=1 2024-08-18T16:19:21.195 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.193+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:entry[obj00000000000000000019]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000019[0] 2024-08-18T16:19:21.195 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.193+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: listing bilog for incremental sync; position=00000000002.163.6 2024-08-18T16:19:21.195 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.195+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:entry[obj00000000000000000019]: success 2024-08-18T16:19:21.195 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.195+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0:5 marker=00000000002.163.6 timestamp=2024-08-18T16:19:18.736467+0000 2024-08-18T16:19:21.196 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.196+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: parsed entry: id=00000000002.165.6 iter->object=obj00000000000000000018 iter->instance= name=obj00000000000000000018 instance= ns= 2024-08-18T16:19:21.196 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.196+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14/obj00000000000000000018 2024-08-18T16:19:21.197 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.196+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: entry->timestamp=2024-08-18T16:19:19.136367+0000 2024-08-18T16:19:21.197 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.196+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:entry[obj00000000000000000018]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.165.6 op=0 op_state=1 2024-08-18T16:19:21.197 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.196+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:entry[obj00000000000000000018]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000018[0] 2024-08-18T16:19:21.197 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.196+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: listing bilog for incremental sync; position=00000000002.165.6 2024-08-18T16:19:21.198 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.197+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: parsed entry: id=00000000001.88.6 iter->object=obj00000000000000000271 iter->instance= name=obj00000000000000000271 instance= ns= 2024-08-18T16:19:21.198 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.197+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12/obj00000000000000000271 2024-08-18T16:19:21.198 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.197+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: entry->timestamp=2024-08-18T16:19:18.858093+0000 2024-08-18T16:19:21.199 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.197+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:entry[obj00000000000000000271]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.88.6 op=0 op_state=1 2024-08-18T16:19:21.199 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.197+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:entry[obj00000000000000000271]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000271[0] 2024-08-18T16:19:21.199 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.197+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: listing bilog for incremental sync; position=00000000001.88.6 2024-08-18T16:19:21.199 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.197+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: a sync operation returned error 2024-08-18T16:19:21.199 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.197+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: backing out with sync_status=-5 2024-08-18T16:19:21.199 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.197+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: incremental sync on bucket failed, retcode=-5 2024-08-18T16:19:21.199 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.197+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: ERROR: incremental sync failed. error: -5 2024-08-18T16:19:21.199 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.198+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:entry[obj00000000000000000467]: success 2024-08-18T16:19:21.199 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.198+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2:5 marker=00000000001.82.6 timestamp=2024-08-18T16:19:19.151651+0000 2024-08-18T16:19:21.200 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.199+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:21.200 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.199+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: sync status for source bucket shard: 0 2024-08-18T16:19:21.200 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.199+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: listing bilog for incremental sync; position= 2024-08-18T16:19:21.201 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.201+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]: parsed entry: id=00000000001.110.6 iter->object=obj00000000000000000061 iter->instance= name=obj00000000000000000061 instance= ns= 2024-08-18T16:19:21.201 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.201+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27/obj00000000000000000061 2024-08-18T16:19:21.201 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.201+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]: entry->timestamp=2024-08-18T16:19:18.800570+0000 2024-08-18T16:19:21.202 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.201+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]:entry[obj00000000000000000061]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.110.6 op=0 op_state=1 2024-08-18T16:19:21.202 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.201+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]:entry[obj00000000000000000061]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000061[0] 2024-08-18T16:19:21.202 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.201+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]: listing bilog for incremental sync; position=00000000001.110.6 2024-08-18T16:19:21.202 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.201+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25]: parsed entry: id=00000000001.104.6 iter->object=obj00000000000000000283 iter->instance= name=obj00000000000000000283 instance= ns= 2024-08-18T16:19:21.202 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.201+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25/obj00000000000000000283 2024-08-18T16:19:21.202 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.201+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25]: entry->timestamp=2024-08-18T16:19:18.728571+0000 2024-08-18T16:19:21.202 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.201+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25]:entry[obj00000000000000000283]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.104.6 op=0 op_state=1 2024-08-18T16:19:21.202 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.201+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25]:entry[obj00000000000000000283]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000283[0] 2024-08-18T16:19:21.202 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.201+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25]: listing bilog for incremental sync; position=00000000001.104.6 2024-08-18T16:19:21.203 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.202+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: a sync operation returned error 2024-08-18T16:19:21.203 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.202+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: backing out with sync_status=-5 2024-08-18T16:19:21.203 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.202+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: incremental sync on bucket failed, retcode=-5 2024-08-18T16:19:21.203 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.202+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: ERROR: incremental sync failed. error: -5 2024-08-18T16:19:21.203 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.202+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:entry[obj00000000000000000018]: success 2024-08-18T16:19:21.203 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.202+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14:5 marker=00000000002.165.6 timestamp=2024-08-18T16:19:19.136367+0000 2024-08-18T16:19:21.204 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.203+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]: a sync operation returned error 2024-08-18T16:19:21.204 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.203+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]: backing out with sync_status=-5 2024-08-18T16:19:21.204 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.203+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]: incremental sync on bucket failed, retcode=-5 2024-08-18T16:19:21.204 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.203+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]: ERROR: incremental sync failed. error: -5 2024-08-18T16:19:21.204 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.204+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]: a sync operation returned error 2024-08-18T16:19:21.205 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.204+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]: backing out with sync_status=-5 2024-08-18T16:19:21.205 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.204+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]: incremental sync on bucket failed, retcode=-5 2024-08-18T16:19:21.205 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.204+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]: ERROR: incremental sync failed. error: -5 2024-08-18T16:19:21.205 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.204+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:entry[obj00000000000000000317]: success 2024-08-18T16:19:21.205 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.204+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7:5 marker=00000000001.88.6 timestamp=2024-08-18T16:19:18.784378+0000 2024-08-18T16:19:21.205 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.205+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]: a sync operation returned error 2024-08-18T16:19:21.205 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.205+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]: a sync operation returned error 2024-08-18T16:19:21.205 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.205+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]: backing out with sync_status=-5 2024-08-18T16:19:21.205 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.205+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]: incremental sync on bucket failed, retcode=-5 2024-08-18T16:19:21.206 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.205+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]: backing out with sync_status=-5 2024-08-18T16:19:21.206 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.205+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]: ERROR: incremental sync failed. error: -5 2024-08-18T16:19:21.206 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.205+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]: incremental sync on bucket failed, retcode=-5 2024-08-18T16:19:21.206 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.205+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]: ERROR: incremental sync failed. error: -5 2024-08-18T16:19:21.206 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.206+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: parsed entry: id=00000000002.85.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2024-08-18T16:19:21.206 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.206+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10/obj00000000000000000021 2024-08-18T16:19:21.207 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.206+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: entry->timestamp=2024-08-18T16:19:18.968460+0000 2024-08-18T16:19:21.207 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.206+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.85.6 op=0 op_state=1 2024-08-18T16:19:21.207 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.206+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:entry[obj00000000000000000021]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000021[0] 2024-08-18T16:19:21.207 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.206+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: listing bilog for incremental sync; position=00000000002.85.6 2024-08-18T16:19:21.208 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.207+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:21.208 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.208+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:entry[obj00000000000000000021]: success 2024-08-18T16:19:21.208 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.208+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10:5 marker=00000000002.85.6 timestamp=2024-08-18T16:19:18.968460+0000 2024-08-18T16:19:21.208 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.208+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: sync status for source bucket shard: 0 2024-08-18T16:19:21.209 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.208+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: listing bilog for incremental sync; position= 2024-08-18T16:19:21.209 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.208+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:entry[obj00000000000000000271]: success 2024-08-18T16:19:21.209 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.208+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12:5 marker=00000000001.88.6 timestamp=2024-08-18T16:19:18.858093+0000 2024-08-18T16:19:21.209 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.209+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25]:entry[obj00000000000000000283]: success 2024-08-18T16:19:21.210 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.209+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25:5 marker=00000000001.104.6 timestamp=2024-08-18T16:19:18.728571+0000 2024-08-18T16:19:21.210 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.210+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:21.210 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.210+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]: sync status for source bucket shard: 0 2024-08-18T16:19:21.210 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.210+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]: listing bilog for incremental sync; position= 2024-08-18T16:19:21.211 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.211+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:21.211 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.211+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]: sync status for source bucket shard: 0 2024-08-18T16:19:21.211 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.211+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]: listing bilog for incremental sync; position= 2024-08-18T16:19:21.212 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.211+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:21.212 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.212+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:21.212 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.212+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]: sync status for source bucket shard: 0 2024-08-18T16:19:21.212 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.212+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]: listing bilog for incremental sync; position= 2024-08-18T16:19:21.212 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.212+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]: sync status for source bucket shard: 0 2024-08-18T16:19:21.212 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.212+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]: listing bilog for incremental sync; position= 2024-08-18T16:19:21.242 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.242+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: parsed entry: id=00000000002.75.6 iter->object=obj00000000000000000004 iter->instance= name=obj00000000000000000004 instance= ns= 2024-08-18T16:19:21.243 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.242+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5/obj00000000000000000004 2024-08-18T16:19:21.243 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.242+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: entry->timestamp=2024-08-18T16:19:19.204480+0000 2024-08-18T16:19:21.243 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.242+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:entry[obj00000000000000000004]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.75.6 op=0 op_state=1 2024-08-18T16:19:21.243 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.242+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:entry[obj00000000000000000004]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000004[0] 2024-08-18T16:19:21.243 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.242+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: listing bilog for incremental sync; position=00000000002.75.6 2024-08-18T16:19:21.244 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.243+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:entry[obj00000000000000000004]: success 2024-08-18T16:19:21.244 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.243+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5:5 marker=00000000002.75.6 timestamp=2024-08-18T16:19:19.204480+0000 2024-08-18T16:19:21.249 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.248+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]:entry[obj00000000000000000061]: success 2024-08-18T16:19:21.249 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.248+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27:5 marker=00000000001.110.6 timestamp=2024-08-18T16:19:18.800570+0000 2024-08-18T16:19:21.251 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.251+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: parsed entry: id=00000000002.110.6 iter->object=obj00000000000000000016 iter->instance= name=obj00000000000000000016 instance= ns= 2024-08-18T16:19:21.251 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.251+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18/obj00000000000000000016 2024-08-18T16:19:21.251 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.251+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: entry->timestamp=2024-08-18T16:19:18.848534+0000 2024-08-18T16:19:21.252 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.251+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:entry[obj00000000000000000016]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.110.6 op=0 op_state=1 2024-08-18T16:19:21.252 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.251+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:entry[obj00000000000000000016]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000016[0] 2024-08-18T16:19:21.252 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.251+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: listing bilog for incremental sync; position=00000000002.110.6 2024-08-18T16:19:21.252 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.252+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]: parsed entry: id=00000000002.185.6 iter->object=obj00000000000000000012 iter->instance= name=obj00000000000000000012 instance= ns= 2024-08-18T16:19:21.252 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.252+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22/obj00000000000000000012 2024-08-18T16:19:21.252 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.252+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]: entry->timestamp=2024-08-18T16:19:19.024938+0000 2024-08-18T16:19:21.252 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.252+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:entry[obj00000000000000000012]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.185.6 op=0 op_state=1 2024-08-18T16:19:21.253 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.252+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:entry[obj00000000000000000012]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000012[0] 2024-08-18T16:19:21.253 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.252+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]: listing bilog for incremental sync; position=00000000002.185.6 2024-08-18T16:19:21.254 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.252+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:entry[obj00000000000000000016]: success 2024-08-18T16:19:21.254 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.252+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18:5 marker=00000000002.110.6 timestamp=2024-08-18T16:19:18.848534+0000 2024-08-18T16:19:21.254 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.253+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]: parsed entry: id=00000000002.103.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2024-08-18T16:19:21.254 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.253+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23/obj00000000000000000001 2024-08-18T16:19:21.254 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.253+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]: entry->timestamp=2024-08-18T16:19:19.089586+0000 2024-08-18T16:19:21.254 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.253+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.103.6 op=0 op_state=1 2024-08-18T16:19:21.254 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.253+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:entry[obj00000000000000000001]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000001[0] 2024-08-18T16:19:21.255 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.253+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]: listing bilog for incremental sync; position=00000000002.103.6 2024-08-18T16:19:21.255 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.253+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]: parsed entry: id=00000000002.150.6 iter->object=obj00000000000000000002 iter->instance= name=obj00000000000000000002 instance= ns= 2024-08-18T16:19:21.255 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.253+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21/obj00000000000000000002 2024-08-18T16:19:21.255 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.253+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]: entry->timestamp=2024-08-18T16:19:18.921737+0000 2024-08-18T16:19:21.255 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.253+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:entry[obj00000000000000000002]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.150.6 op=0 op_state=1 2024-08-18T16:19:21.255 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.253+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:entry[obj00000000000000000002]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000002[0] 2024-08-18T16:19:21.255 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.253+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]: listing bilog for incremental sync; position=00000000002.150.6 2024-08-18T16:19:21.256 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.253+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:entry[obj00000000000000000012]: success 2024-08-18T16:19:21.256 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.253+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22:5 marker=00000000002.185.6 timestamp=2024-08-18T16:19:19.024938+0000 2024-08-18T16:19:21.257 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.255+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]: parsed entry: id=00000000001.124.6 iter->object=obj00000000000000000423 iter->instance= name=obj00000000000000000423 instance= ns= 2024-08-18T16:19:21.257 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.255+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20/obj00000000000000000423 2024-08-18T16:19:21.257 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.255+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]: entry->timestamp=2024-08-18T16:19:18.720502+0000 2024-08-18T16:19:21.257 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.255+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:entry[obj00000000000000000423]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.124.6 op=0 op_state=1 2024-08-18T16:19:21.257 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.255+0000 7f63ec126880 0 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:entry[obj00000000000000000423]: : injecting data sync error on key=obj00000000000000000423 2024-08-18T16:19:21.258 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.255+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]: listing bilog for incremental sync; position=00000000001.124.6 2024-08-18T16:19:21.258 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.255+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:entry[obj00000000000000000423]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:19:21.258 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.255+0000 7f63ec126880 0 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:entry[obj00000000000000000423]: ERROR: failed to sync object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20/obj00000000000000000423 2024-08-18T16:19:21.258 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.255+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:entry[obj00000000000000000001]: success 2024-08-18T16:19:21.258 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.255+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23:5 marker=00000000002.103.6 timestamp=2024-08-18T16:19:19.089586+0000 2024-08-18T16:19:21.259 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.255+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:entry[obj00000000000000000002]: success 2024-08-18T16:19:21.259 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.255+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21:5 marker=00000000002.150.6 timestamp=2024-08-18T16:19:18.921737+0000 2024-08-18T16:19:21.297 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.297+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]: a sync operation returned error 2024-08-18T16:19:21.297 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.297+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]: backing out with sync_status=-5 2024-08-18T16:19:21.298 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.297+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]: incremental sync on bucket failed, retcode=-5 2024-08-18T16:19:21.298 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.297+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]: ERROR: incremental sync failed. error: -5 2024-08-18T16:19:21.298 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.298+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:21.298 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.298+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]: sync status for source bucket shard: 0 2024-08-18T16:19:21.299 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.298+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]: listing bilog for incremental sync; position= 2024-08-18T16:19:21.340 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.340+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]: parsed entry: id=00000000001.124.6 iter->object=obj00000000000000000423 iter->instance= name=obj00000000000000000423 instance= ns= 2024-08-18T16:19:21.341 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.340+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20/obj00000000000000000423 2024-08-18T16:19:21.341 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.340+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]: entry->timestamp=2024-08-18T16:19:18.720502+0000 2024-08-18T16:19:21.341 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.340+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:entry[obj00000000000000000423]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.124.6 op=0 op_state=1 2024-08-18T16:19:21.341 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.340+0000 7f63ec126880 0 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:entry[obj00000000000000000423]: : injecting data sync error on key=obj00000000000000000423 2024-08-18T16:19:21.341 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.340+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]: listing bilog for incremental sync; position=00000000001.124.6 2024-08-18T16:19:21.341 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.340+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:entry[obj00000000000000000423]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:19:21.341 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.340+0000 7f63ec126880 0 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:entry[obj00000000000000000423]: ERROR: failed to sync object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20/obj00000000000000000423 2024-08-18T16:19:21.382 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.382+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]: a sync operation returned error 2024-08-18T16:19:21.383 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.382+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]: backing out with sync_status=-5 2024-08-18T16:19:21.383 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.382+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]: incremental sync on bucket failed, retcode=-5 2024-08-18T16:19:21.383 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.382+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]: ERROR: incremental sync failed. error: -5 2024-08-18T16:19:21.383 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.382+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:21.384 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.383+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]: sync status for source bucket shard: 0 2024-08-18T16:19:21.384 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.383+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]: listing bilog for incremental sync; position= 2024-08-18T16:19:21.426 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.425+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]: parsed entry: id=00000000001.124.6 iter->object=obj00000000000000000423 iter->instance= name=obj00000000000000000423 instance= ns= 2024-08-18T16:19:21.426 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.425+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20/obj00000000000000000423 2024-08-18T16:19:21.427 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.425+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]: entry->timestamp=2024-08-18T16:19:18.720502+0000 2024-08-18T16:19:21.427 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.425+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:entry[obj00000000000000000423]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.124.6 op=0 op_state=1 2024-08-18T16:19:21.427 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.425+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:entry[obj00000000000000000423]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000423[0] 2024-08-18T16:19:21.427 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.425+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]: listing bilog for incremental sync; position=00000000001.124.6 2024-08-18T16:19:21.430 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.429+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:entry[obj00000000000000000423]: success 2024-08-18T16:19:21.430 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.429+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20:5 marker=00000000001.124.6 timestamp=2024-08-18T16:19:18.720502+0000 2024-08-18T16:19:21.475 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.472+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:21.476 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.472+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:21.476 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.472+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:21.476 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.472+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:21.476 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.472+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:21.476 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.472+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:21.476 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.472+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:21.476 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.473+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: sync status for source bucket shard: 0 2024-08-18T16:19:21.476 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.473+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: listing bilog for incremental sync; position= 2024-08-18T16:19:21.476 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.473+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: sync status for source bucket shard: 0 2024-08-18T16:19:21.476 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.473+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: listing bilog for incremental sync; position= 2024-08-18T16:19:21.476 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.473+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: sync status for source bucket shard: 0 2024-08-18T16:19:21.477 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.473+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: sync status for source bucket shard: 0 2024-08-18T16:19:21.477 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.473+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: sync status for source bucket shard: 0 2024-08-18T16:19:21.477 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.473+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: sync status for source bucket shard: 0 2024-08-18T16:19:21.477 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.473+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: sync status for source bucket shard: 0 2024-08-18T16:19:21.477 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.473+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: listing bilog for incremental sync; position= 2024-08-18T16:19:21.477 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.473+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: listing bilog for incremental sync; position= 2024-08-18T16:19:21.477 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.473+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: listing bilog for incremental sync; position= 2024-08-18T16:19:21.477 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.473+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: listing bilog for incremental sync; position= 2024-08-18T16:19:21.477 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.473+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: listing bilog for incremental sync; position= 2024-08-18T16:19:21.477 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.473+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:21.477 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.473+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:21.478 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.473+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:21.478 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.473+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:21.478 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.473+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:21.478 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.474+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:21.478 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.474+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: sync status for source bucket shard: 0 2024-08-18T16:19:21.478 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.474+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: listing bilog for incremental sync; position= 2024-08-18T16:19:21.478 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.474+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: sync status for source bucket shard: 0 2024-08-18T16:19:21.479 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.474+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: listing bilog for incremental sync; position= 2024-08-18T16:19:21.479 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.474+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: sync status for source bucket shard: 0 2024-08-18T16:19:21.479 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.474+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: listing bilog for incremental sync; position= 2024-08-18T16:19:21.479 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.474+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: sync status for source bucket shard: 0 2024-08-18T16:19:21.479 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.474+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: sync status for source bucket shard: 0 2024-08-18T16:19:21.479 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.474+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: listing bilog for incremental sync; position= 2024-08-18T16:19:21.479 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.474+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: listing bilog for incremental sync; position= 2024-08-18T16:19:21.479 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.474+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]: sync status for source bucket shard: 0 2024-08-18T16:19:21.479 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.474+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:21.479 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.474+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]: listing bilog for incremental sync; position= 2024-08-18T16:19:21.479 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.474+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:21.479 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.474+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:21.480 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.474+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:21.480 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.474+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:21.480 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.475+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: parsed entry: id=00000000002.128.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2024-08-18T16:19:21.480 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.475+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3/obj00000000000000000001 2024-08-18T16:19:21.480 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.475+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: entry->timestamp=2024-08-18T16:19:20.116900+0000 2024-08-18T16:19:21.480 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.475+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.128.6 op=0 op_state=1 2024-08-18T16:19:21.480 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.475+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:entry[obj00000000000000000001]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000001[0] 2024-08-18T16:19:21.480 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.475+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: listing bilog for incremental sync; position=00000000002.128.6 2024-08-18T16:19:21.480 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.475+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]: sync status for source bucket shard: 0 2024-08-18T16:19:21.480 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.475+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]: listing bilog for incremental sync; position= 2024-08-18T16:19:21.489 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.476+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: sync status for source bucket shard: 0 2024-08-18T16:19:21.489 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.476+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: sync status for source bucket shard: 0 2024-08-18T16:19:21.489 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.476+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: sync status for source bucket shard: 0 2024-08-18T16:19:21.489 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.476+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: sync status for source bucket shard: 0 2024-08-18T16:19:21.489 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.476+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: listing bilog for incremental sync; position= 2024-08-18T16:19:21.489 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.476+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: listing bilog for incremental sync; position= 2024-08-18T16:19:21.489 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.476+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: listing bilog for incremental sync; position= 2024-08-18T16:19:21.489 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.476+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: listing bilog for incremental sync; position= 2024-08-18T16:19:21.489 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.476+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: parsed entry: id=00000000002.81.6 iter->object=obj00000000000000000016 iter->instance= name=obj00000000000000000016 instance= ns= 2024-08-18T16:19:21.490 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.476+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7/obj00000000000000000016 2024-08-18T16:19:21.490 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.476+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: entry->timestamp=2024-08-18T16:19:20.228897+0000 2024-08-18T16:19:21.490 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.476+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:entry[obj00000000000000000016]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.81.6 op=0 op_state=1 2024-08-18T16:19:21.490 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.476+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: parsed entry: id=00000000002.108.6 iter->object=obj00000000000000000011 iter->instance= name=obj00000000000000000011 instance= ns= 2024-08-18T16:19:21.490 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.476+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2/obj00000000000000000011 2024-08-18T16:19:21.490 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.476+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: entry->timestamp=2024-08-18T16:19:19.775284+0000 2024-08-18T16:19:21.490 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.476+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:entry[obj00000000000000000011]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.108.6 op=0 op_state=1 2024-08-18T16:19:21.490 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.476+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:21.490 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.476+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:21.490 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.476+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: parsed entry: id=00000000001.81.6 iter->object=obj00000000000000000065 iter->instance= name=obj00000000000000000065 instance= ns= 2024-08-18T16:19:21.490 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.476+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5/obj00000000000000000065 2024-08-18T16:19:21.491 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.476+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: entry->timestamp=2024-08-18T16:19:19.716103+0000 2024-08-18T16:19:21.491 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.476+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:entry[obj00000000000000000065]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.81.6 op=0 op_state=1 2024-08-18T16:19:21.491 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.476+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: parsed entry: id=00000000002.112.6 iter->object=obj00000000000000000033 iter->instance= name=obj00000000000000000033 instance= ns= 2024-08-18T16:19:21.491 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.476+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4/obj00000000000000000033 2024-08-18T16:19:21.491 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.476+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: entry->timestamp=2024-08-18T16:19:19.593080+0000 2024-08-18T16:19:21.491 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.476+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:entry[obj00000000000000000033]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.112.6 op=0 op_state=1 2024-08-18T16:19:21.491 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.476+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:21.491 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.476+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:entry[obj00000000000000000016]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000016[0] 2024-08-18T16:19:21.491 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.476+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: listing bilog for incremental sync; position=00000000002.81.6 2024-08-18T16:19:21.492 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.476+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: parsed entry: id=00000000001.140.6 iter->object=obj00000000000000000060 iter->instance= name=obj00000000000000000060 instance= ns= 2024-08-18T16:19:21.492 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.476+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0/obj00000000000000000060 2024-08-18T16:19:21.492 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.476+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: entry->timestamp=2024-08-18T16:19:19.996680+0000 2024-08-18T16:19:21.492 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.476+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:entry[obj00000000000000000060]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.140.6 op=0 op_state=1 2024-08-18T16:19:21.492 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.477+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:entry[obj00000000000000000011]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000011[0] 2024-08-18T16:19:21.492 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.477+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: listing bilog for incremental sync; position=00000000002.108.6 2024-08-18T16:19:21.492 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.477+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:entry[obj00000000000000000065]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000065[0] 2024-08-18T16:19:21.492 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.477+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: listing bilog for incremental sync; position=00000000001.81.6 2024-08-18T16:19:21.492 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.477+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:entry[obj00000000000000000033]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000033[0] 2024-08-18T16:19:21.492 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.477+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: listing bilog for incremental sync; position=00000000002.112.6 2024-08-18T16:19:21.493 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.477+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:entry[obj00000000000000000060]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000060[0] 2024-08-18T16:19:21.493 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.477+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: listing bilog for incremental sync; position=00000000001.140.6 2024-08-18T16:19:21.493 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.477+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: parsed entry: id=00000000002.83.6 iter->object=obj00000000000000000015 iter->instance= name=obj00000000000000000015 instance= ns= 2024-08-18T16:19:21.493 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.477+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9/obj00000000000000000015 2024-08-18T16:19:21.493 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.477+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: entry->timestamp=2024-08-18T16:19:20.126126+0000 2024-08-18T16:19:21.493 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.477+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:entry[obj00000000000000000015]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.83.6 op=0 op_state=1 2024-08-18T16:19:21.493 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.477+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:21.493 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.477+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:21.494 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.477+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:entry[obj00000000000000000015]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000015[0] 2024-08-18T16:19:21.494 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.477+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: listing bilog for incremental sync; position=00000000002.83.6 2024-08-18T16:19:21.494 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.477+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: parsed entry: id=00000000001.112.6 iter->object=obj00000000000000000062 iter->instance= name=obj00000000000000000062 instance= ns= 2024-08-18T16:19:21.494 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.477+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12/obj00000000000000000062 2024-08-18T16:19:21.494 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.477+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: entry->timestamp=2024-08-18T16:19:20.061021+0000 2024-08-18T16:19:21.494 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.477+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:entry[obj00000000000000000062]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.112.6 op=0 op_state=1 2024-08-18T16:19:21.494 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.477+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:21.494 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.477+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: parsed entry: id=00000000001.170.6 iter->object=obj00000000000000000048 iter->instance= name=obj00000000000000000048 instance= ns= 2024-08-18T16:19:21.494 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.477+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8/obj00000000000000000048 2024-08-18T16:19:21.494 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.477+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: entry->timestamp=2024-08-18T16:19:19.833816+0000 2024-08-18T16:19:21.494 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.477+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:entry[obj00000000000000000048]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.170.6 op=0 op_state=1 2024-08-18T16:19:21.494 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.477+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: parsed entry: id=00000000002.167.6 iter->object=obj00000000000000000005 iter->instance= name=obj00000000000000000005 instance= ns= 2024-08-18T16:19:21.495 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.477+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6/obj00000000000000000005 2024-08-18T16:19:21.495 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.477+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: entry->timestamp=2024-08-18T16:19:20.349953+0000 2024-08-18T16:19:21.495 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.477+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:entry[obj00000000000000000005]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.167.6 op=0 op_state=1 2024-08-18T16:19:21.495 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.477+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:21.495 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.477+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:entry[obj00000000000000000062]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000062[0] 2024-08-18T16:19:21.495 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.477+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: listing bilog for incremental sync; position=00000000001.112.6 2024-08-18T16:19:21.495 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.477+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: parsed entry: id=00000000002.163.6 iter->object=obj00000000000000000026 iter->instance= name=obj00000000000000000026 instance= ns= 2024-08-18T16:19:21.495 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.477+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10/obj00000000000000000026 2024-08-18T16:19:21.495 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.477+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: entry->timestamp=2024-08-18T16:19:20.238134+0000 2024-08-18T16:19:21.495 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.477+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:entry[obj00000000000000000026]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.163.6 op=0 op_state=1 2024-08-18T16:19:21.495 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.477+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:21.495 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.478+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:entry[obj00000000000000000048]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000048[0] 2024-08-18T16:19:21.495 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.478+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: listing bilog for incremental sync; position=00000000001.170.6 2024-08-18T16:19:21.496 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.478+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:entry[obj00000000000000000005]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000005[0] 2024-08-18T16:19:21.496 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.478+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: listing bilog for incremental sync; position=00000000002.167.6 2024-08-18T16:19:21.496 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.478+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:entry[obj00000000000000000026]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000026[0] 2024-08-18T16:19:21.496 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.478+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: listing bilog for incremental sync; position=00000000002.163.6 2024-08-18T16:19:21.496 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.478+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]: parsed entry: id=00000000002.138.6 iter->object=obj00000000000000000030 iter->instance= name=obj00000000000000000030 instance= ns= 2024-08-18T16:19:21.496 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.478+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11/obj00000000000000000030 2024-08-18T16:19:21.496 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.478+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]: entry->timestamp=2024-08-18T16:19:19.545863+0000 2024-08-18T16:19:21.496 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.478+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:entry[obj00000000000000000030]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.138.6 op=0 op_state=1 2024-08-18T16:19:21.496 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.478+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:21.496 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.478+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:21.497 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.478+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:entry[obj00000000000000000030]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000030[0] 2024-08-18T16:19:21.497 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.478+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]: listing bilog for incremental sync; position=00000000002.138.6 2024-08-18T16:19:21.497 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.478+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:21.497 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.478+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:28]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:21.497 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.478+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:30]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:21.497 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.478+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:entry[obj00000000000000000001]: success 2024-08-18T16:19:21.497 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.478+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3:6 marker=00000000002.128.6 timestamp=2024-08-18T16:19:20.116900+0000 2024-08-18T16:19:21.497 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.479+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:31]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:21.497 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.479+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: sync status for source bucket shard: 0 2024-08-18T16:19:21.497 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.479+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]: sync status for source bucket shard: 0 2024-08-18T16:19:21.497 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.479+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: listing bilog for incremental sync; position= 2024-08-18T16:19:21.497 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.479+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:32]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:21.497 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.479+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]: sync status for source bucket shard: 0 2024-08-18T16:19:21.498 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.479+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:34]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:21.498 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.479+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]: listing bilog for incremental sync; position= 2024-08-18T16:19:21.498 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.479+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:33]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:21.498 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.479+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]: listing bilog for incremental sync; position= 2024-08-18T16:19:21.498 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.479+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]: sync status for source bucket shard: 0 2024-08-18T16:19:21.498 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.479+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]: sync status for source bucket shard: 0 2024-08-18T16:19:21.498 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.479+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:35]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:21.498 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.479+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:36]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:21.498 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.479+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]: sync status for source bucket shard: 0 2024-08-18T16:19:21.498 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.479+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]: listing bilog for incremental sync; position= 2024-08-18T16:19:21.498 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.479+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]: listing bilog for incremental sync; position= 2024-08-18T16:19:21.498 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.479+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]: listing bilog for incremental sync; position= 2024-08-18T16:19:21.499 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.480+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25]: sync status for source bucket shard: 0 2024-08-18T16:19:21.499 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.480+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]: sync status for source bucket shard: 0 2024-08-18T16:19:21.499 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.480+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:28]: sync status for source bucket shard: 0 2024-08-18T16:19:21.499 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.480+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]: sync status for source bucket shard: 0 2024-08-18T16:19:21.499 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.480+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:30]: sync status for source bucket shard: 0 2024-08-18T16:19:21.499 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.480+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:34]: sync status for source bucket shard: 0 2024-08-18T16:19:21.499 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.480+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:33]: sync status for source bucket shard: 0 2024-08-18T16:19:21.499 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.480+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25]: listing bilog for incremental sync; position= 2024-08-18T16:19:21.499 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.480+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]: listing bilog for incremental sync; position= 2024-08-18T16:19:21.499 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.480+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:28]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:28]: listing bilog for incremental sync; position= 2024-08-18T16:19:21.499 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.480+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]: listing bilog for incremental sync; position= 2024-08-18T16:19:21.499 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.480+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:30]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:30]: listing bilog for incremental sync; position= 2024-08-18T16:19:21.499 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.480+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:34]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:34]: listing bilog for incremental sync; position= 2024-08-18T16:19:21.500 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.480+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:33]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:33]: listing bilog for incremental sync; position= 2024-08-18T16:19:21.500 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.480+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:entry[obj00000000000000000016]: success 2024-08-18T16:19:21.500 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.480+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7:6 marker=00000000002.81.6 timestamp=2024-08-18T16:19:20.228897+0000 2024-08-18T16:19:21.500 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.481+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:36]: sync status for source bucket shard: 0 2024-08-18T16:19:21.500 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.481+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:36]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:36]: listing bilog for incremental sync; position= 2024-08-18T16:19:21.500 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.482+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]: sync status for source bucket shard: 0 2024-08-18T16:19:21.500 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.482+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]: sync status for source bucket shard: 0 2024-08-18T16:19:21.500 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.482+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:31]: sync status for source bucket shard: 0 2024-08-18T16:19:21.500 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.482+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:32]: sync status for source bucket shard: 0 2024-08-18T16:19:21.500 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.482+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:35]: sync status for source bucket shard: 0 2024-08-18T16:19:21.500 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.482+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]: listing bilog for incremental sync; position= 2024-08-18T16:19:21.500 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.482+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]: listing bilog for incremental sync; position= 2024-08-18T16:19:21.501 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.482+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:31]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:31]: listing bilog for incremental sync; position= 2024-08-18T16:19:21.501 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.482+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:32]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:32]: listing bilog for incremental sync; position= 2024-08-18T16:19:21.501 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.482+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:35]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:35]: listing bilog for incremental sync; position= 2024-08-18T16:19:21.501 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.482+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:entry[obj00000000000000000033]: success 2024-08-18T16:19:21.501 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.482+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4:6 marker=00000000002.112.6 timestamp=2024-08-18T16:19:19.593080+0000 2024-08-18T16:19:21.501 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.482+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:entry[obj00000000000000000015]: success 2024-08-18T16:19:21.501 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.482+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9:6 marker=00000000002.83.6 timestamp=2024-08-18T16:19:20.126126+0000 2024-08-18T16:19:21.501 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.482+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:entry[obj00000000000000000011]: success 2024-08-18T16:19:21.501 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.482+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2:6 marker=00000000002.108.6 timestamp=2024-08-18T16:19:19.775284+0000 2024-08-18T16:19:21.501 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.483+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:entry[obj00000000000000000005]: success 2024-08-18T16:19:21.501 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.483+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6:6 marker=00000000002.167.6 timestamp=2024-08-18T16:19:20.349953+0000 2024-08-18T16:19:21.501 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.484+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]: parsed entry: id=00000000002.98.6 iter->object=obj00000000000000000039 iter->instance= name=obj00000000000000000039 instance= ns= 2024-08-18T16:19:21.501 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.484+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23/obj00000000000000000039 2024-08-18T16:19:21.502 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.484+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]: entry->timestamp=2024-08-18T16:19:19.824786+0000 2024-08-18T16:19:21.502 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.484+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:entry[obj00000000000000000039]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.98.6 op=0 op_state=1 2024-08-18T16:19:21.502 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.484+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:entry[obj00000000000000000039]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000039[0] 2024-08-18T16:19:21.502 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.484+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]: listing bilog for incremental sync; position=00000000002.98.6 2024-08-18T16:19:21.502 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.484+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:entry[obj00000000000000000030]: success 2024-08-18T16:19:21.502 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.484+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11:6 marker=00000000002.138.6 timestamp=2024-08-18T16:19:19.545863+0000 2024-08-18T16:19:21.502 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.485+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:entry[obj00000000000000000026]: success 2024-08-18T16:19:21.502 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.485+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10:6 marker=00000000002.163.6 timestamp=2024-08-18T16:19:20.238134+0000 2024-08-18T16:19:21.502 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.485+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:entry[obj00000000000000000065]: success 2024-08-18T16:19:21.502 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.485+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5:6 marker=00000000001.81.6 timestamp=2024-08-18T16:19:19.716103+0000 2024-08-18T16:19:21.502 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.486+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]: parsed entry: id=00000000002.106.6 iter->object=obj00000000000000000045 iter->instance= name=obj00000000000000000045 instance= ns= 2024-08-18T16:19:21.502 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.486+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19/obj00000000000000000045 2024-08-18T16:19:21.503 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.486+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]: entry->timestamp=2024-08-18T16:19:20.340703+0000 2024-08-18T16:19:21.503 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.486+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:entry[obj00000000000000000045]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.106.6 op=0 op_state=1 2024-08-18T16:19:21.503 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.486+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:entry[obj00000000000000000045]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000045[0] 2024-08-18T16:19:21.503 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.486+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]: listing bilog for incremental sync; position=00000000002.106.6 2024-08-18T16:19:21.503 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.486+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:35]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:35]: parsed entry: id=00000000002.93.6 iter->object=obj00000000000000000018 iter->instance= name=obj00000000000000000018 instance= ns= 2024-08-18T16:19:21.503 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.486+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:35]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:35]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:35/obj00000000000000000018 2024-08-18T16:19:21.503 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.486+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:35]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:35]: entry->timestamp=2024-08-18T16:19:20.284525+0000 2024-08-18T16:19:21.503 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.486+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:35]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:35]:entry[obj00000000000000000018]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.93.6 op=0 op_state=1 2024-08-18T16:19:21.503 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.486+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:35]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:35]:entry[obj00000000000000000018]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000018[0] 2024-08-18T16:19:21.503 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.486+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:35]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:35]: listing bilog for incremental sync; position=00000000002.93.6 2024-08-18T16:19:21.503 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.486+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: parsed entry: id=00000000002.100.6 iter->object=obj00000000000000000013 iter->instance= name=obj00000000000000000013 instance= ns= 2024-08-18T16:19:21.503 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.486+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18/obj00000000000000000013 2024-08-18T16:19:21.504 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.486+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: entry->timestamp=2024-08-18T16:19:20.004845+0000 2024-08-18T16:19:21.504 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.486+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:entry[obj00000000000000000013]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.100.6 op=0 op_state=1 2024-08-18T16:19:21.504 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.486+0000 7f63ec126880 0 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:entry[obj00000000000000000013]: : injecting data sync error on key=obj00000000000000000013 2024-08-18T16:19:21.504 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.486+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: listing bilog for incremental sync; position=00000000002.100.6 2024-08-18T16:19:21.504 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.486+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:entry[obj00000000000000000013]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:19:21.504 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.486+0000 7f63ec126880 0 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:entry[obj00000000000000000013]: ERROR: failed to sync object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18/obj00000000000000000013 2024-08-18T16:19:21.504 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.487+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:30]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:30]: parsed entry: id=00000000001.86.6 iter->object=obj00000000000000000036 iter->instance= name=obj00000000000000000036 instance= ns= 2024-08-18T16:19:21.504 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.487+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:30]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:30]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:30/obj00000000000000000036 2024-08-18T16:19:21.504 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.487+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:30]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:30]: entry->timestamp=2024-08-18T16:19:20.013736+0000 2024-08-18T16:19:21.504 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.487+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:30]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:30]:entry[obj00000000000000000036]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.86.6 op=0 op_state=1 2024-08-18T16:19:21.504 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.487+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]: parsed entry: id=00000000002.97.6 iter->object=obj00000000000000000004 iter->instance= name=obj00000000000000000004 instance= ns= 2024-08-18T16:19:21.504 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.487+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29/obj00000000000000000004 2024-08-18T16:19:21.504 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.487+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]: entry->timestamp=2024-08-18T16:19:19.764610+0000 2024-08-18T16:19:21.505 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.487+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]:entry[obj00000000000000000004]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.97.6 op=0 op_state=1 2024-08-18T16:19:21.505 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.487+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:30]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:30]:entry[obj00000000000000000036]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000036[0] 2024-08-18T16:19:21.505 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.487+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:30]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:30]: listing bilog for incremental sync; position=00000000001.86.6 2024-08-18T16:19:21.505 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.487+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]:entry[obj00000000000000000004]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000004[0] 2024-08-18T16:19:21.505 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.487+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]: listing bilog for incremental sync; position=00000000002.97.6 2024-08-18T16:19:21.505 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.487+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]: parsed entry: id=00000000002.115.6 iter->object=obj00000000000000000007 iter->instance= name=obj00000000000000000007 instance= ns= 2024-08-18T16:19:21.505 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.487+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22/obj00000000000000000007 2024-08-18T16:19:21.505 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.487+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]: entry->timestamp=2024-08-18T16:19:19.491062+0000 2024-08-18T16:19:21.505 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.487+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:entry[obj00000000000000000007]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.115.6 op=0 op_state=1 2024-08-18T16:19:21.505 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.487+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:34]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:34]: parsed entry: id=00000000001.94.6 iter->object=obj00000000000000000057 iter->instance= name=obj00000000000000000057 instance= ns= 2024-08-18T16:19:21.505 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.487+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:34]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:34]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:34/obj00000000000000000057 2024-08-18T16:19:21.505 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.487+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:34]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:34]: entry->timestamp=2024-08-18T16:19:19.657944+0000 2024-08-18T16:19:21.506 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.487+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:34]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:34]:entry[obj00000000000000000057]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.94.6 op=0 op_state=1 2024-08-18T16:19:21.506 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.487+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:33]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:33]: parsed entry: id=00000000002.114.6 iter->object=obj00000000000000000019 iter->instance= name=obj00000000000000000019 instance= ns= 2024-08-18T16:19:21.506 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.487+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:33]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:33]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:33/obj00000000000000000019 2024-08-18T16:19:21.506 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.487+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:33]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:33]: entry->timestamp=2024-08-18T16:19:19.884831+0000 2024-08-18T16:19:21.506 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.487+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:33]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:33]:entry[obj00000000000000000019]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.114.6 op=0 op_state=1 2024-08-18T16:19:21.506 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.487+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:entry[obj00000000000000000007]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000007[0] 2024-08-18T16:19:21.506 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.487+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]: listing bilog for incremental sync; position=00000000002.115.6 2024-08-18T16:19:21.506 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.487+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:34]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:34]:entry[obj00000000000000000057]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000057[0] 2024-08-18T16:19:21.506 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.487+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:34]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:34]: listing bilog for incremental sync; position=00000000001.94.6 2024-08-18T16:19:21.506 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.487+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:33]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:33]:entry[obj00000000000000000019]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000019[0] 2024-08-18T16:19:21.506 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.487+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:33]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:33]: listing bilog for incremental sync; position=00000000002.114.6 2024-08-18T16:19:21.506 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.488+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:36]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:36]: parsed entry: id=00000000002.161.6 iter->object=obj00000000000000000008 iter->instance= name=obj00000000000000000008 instance= ns= 2024-08-18T16:19:21.506 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.488+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:36]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:36]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:36/obj00000000000000000008 2024-08-18T16:19:21.507 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.488+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:36]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:36]: entry->timestamp=2024-08-18T16:19:19.536516+0000 2024-08-18T16:19:21.507 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.488+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:36]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:36]:entry[obj00000000000000000008]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.161.6 op=0 op_state=1 2024-08-18T16:19:21.507 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.488+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:36]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:36]:entry[obj00000000000000000008]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000008[0] 2024-08-18T16:19:21.507 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.488+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:36]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:36]: listing bilog for incremental sync; position=00000000002.161.6 2024-08-18T16:19:21.507 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.488+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]: parsed entry: id=00000000002.94.6 iter->object=obj00000000000000000006 iter->instance= name=obj00000000000000000006 instance= ns= 2024-08-18T16:19:21.507 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.488+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20/obj00000000000000000006 2024-08-18T16:19:21.507 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.488+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]: entry->timestamp=2024-08-18T16:19:19.949473+0000 2024-08-18T16:19:21.507 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.488+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:entry[obj00000000000000000006]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.94.6 op=0 op_state=1 2024-08-18T16:19:21.507 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.488+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]: parsed entry: id=00000000002.100.6 iter->object=obj00000000000000000053 iter->instance= name=obj00000000000000000053 instance= ns= 2024-08-18T16:19:21.507 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.488+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27/obj00000000000000000053 2024-08-18T16:19:21.507 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.488+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]: entry->timestamp=2024-08-18T16:19:19.482927+0000 2024-08-18T16:19:21.507 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.488+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]:entry[obj00000000000000000053]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.100.6 op=0 op_state=1 2024-08-18T16:19:21.507 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.488+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:entry[obj00000000000000000039]: success 2024-08-18T16:19:21.508 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.488+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23:6 marker=00000000002.98.6 timestamp=2024-08-18T16:19:19.824786+0000 2024-08-18T16:19:21.508 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.488+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:entry[obj00000000000000000006]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000006[0] 2024-08-18T16:19:21.508 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.488+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]: listing bilog for incremental sync; position=00000000002.94.6 2024-08-18T16:19:21.508 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.488+0000 7f63ec126880 0 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]:entry[obj00000000000000000053]: : injecting data sync error on key=obj00000000000000000053 2024-08-18T16:19:21.508 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.488+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]: listing bilog for incremental sync; position=00000000002.100.6 2024-08-18T16:19:21.508 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.488+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]:entry[obj00000000000000000053]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:19:21.508 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.488+0000 7f63ec126880 0 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]:entry[obj00000000000000000053]: ERROR: failed to sync object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27/obj00000000000000000053 2024-08-18T16:19:21.510 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.489+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:entry[obj00000000000000000060]: success 2024-08-18T16:19:21.510 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.489+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0:6 marker=00000000001.140.6 timestamp=2024-08-18T16:19:19.996680+0000 2024-08-18T16:19:21.510 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.489+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:entry[obj00000000000000000062]: success 2024-08-18T16:19:21.510 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.489+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12:6 marker=00000000001.112.6 timestamp=2024-08-18T16:19:20.061021+0000 2024-08-18T16:19:21.510 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.490+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:35]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:35]:entry[obj00000000000000000018]: success 2024-08-18T16:19:21.510 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.490+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:35]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:35]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:35:6 marker=00000000002.93.6 timestamp=2024-08-18T16:19:20.284525+0000 2024-08-18T16:19:21.510 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.490+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]:entry[obj00000000000000000004]: success 2024-08-18T16:19:21.510 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.490+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29:6 marker=00000000002.97.6 timestamp=2024-08-18T16:19:19.764610+0000 2024-08-18T16:19:21.510 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.491+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:33]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:33]:entry[obj00000000000000000019]: success 2024-08-18T16:19:21.510 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.491+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:33]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:33]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:33:6 marker=00000000002.114.6 timestamp=2024-08-18T16:19:19.884831+0000 2024-08-18T16:19:21.510 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.491+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:entry[obj00000000000000000007]: success 2024-08-18T16:19:21.511 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.491+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22:6 marker=00000000002.115.6 timestamp=2024-08-18T16:19:19.491062+0000 2024-08-18T16:19:21.511 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.492+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:36]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:36]:entry[obj00000000000000000008]: success 2024-08-18T16:19:21.511 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.492+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:36]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:36]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:36:6 marker=00000000002.161.6 timestamp=2024-08-18T16:19:19.536516+0000 2024-08-18T16:19:21.511 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.492+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:entry[obj00000000000000000045]: success 2024-08-18T16:19:21.511 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.492+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19:6 marker=00000000002.106.6 timestamp=2024-08-18T16:19:20.340703+0000 2024-08-18T16:19:21.511 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.492+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:entry[obj00000000000000000048]: success 2024-08-18T16:19:21.511 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.492+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8:6 marker=00000000001.170.6 timestamp=2024-08-18T16:19:19.833816+0000 2024-08-18T16:19:21.511 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.493+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:entry[obj00000000000000000006]: success 2024-08-18T16:19:21.511 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.493+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20:6 marker=00000000002.94.6 timestamp=2024-08-18T16:19:19.949473+0000 2024-08-18T16:19:21.511 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.496+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:30]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:30]:entry[obj00000000000000000036]: success 2024-08-18T16:19:21.511 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.496+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:30]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:30]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:30:6 marker=00000000001.86.6 timestamp=2024-08-18T16:19:20.013736+0000 2024-08-18T16:19:21.515 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.515+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: parsed entry: id=00000000002.100.6 iter->object=obj00000000000000000000 iter->instance= name=obj00000000000000000000 instance= ns= 2024-08-18T16:19:21.515 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.515+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1/obj00000000000000000000 2024-08-18T16:19:21.516 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.515+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: entry->timestamp=2024-08-18T16:19:19.893792+0000 2024-08-18T16:19:21.516 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.515+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:entry[obj00000000000000000000]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.100.6 op=0 op_state=1 2024-08-18T16:19:21.516 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.515+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:entry[obj00000000000000000000]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000000[0] 2024-08-18T16:19:21.516 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.515+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: listing bilog for incremental sync; position=00000000002.100.6 2024-08-18T16:19:21.517 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.517+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:entry[obj00000000000000000000]: success 2024-08-18T16:19:21.517 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.517+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1:6 marker=00000000002.100.6 timestamp=2024-08-18T16:19:19.893792+0000 2024-08-18T16:19:21.518 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.518+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]: parsed entry: id=00000000002.90.6 iter->object=obj00000000000000000009 iter->instance= name=obj00000000000000000009 instance= ns= 2024-08-18T16:19:21.518 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.518+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13/obj00000000000000000009 2024-08-18T16:19:21.518 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.518+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]: entry->timestamp=2024-08-18T16:19:20.069388+0000 2024-08-18T16:19:21.518 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.518+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:entry[obj00000000000000000009]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.90.6 op=0 op_state=1 2024-08-18T16:19:21.518 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.518+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:entry[obj00000000000000000009]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000009[0] 2024-08-18T16:19:21.519 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.518+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]: listing bilog for incremental sync; position=00000000002.90.6 2024-08-18T16:19:21.520 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.519+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: parsed entry: id=00000000002.91.6 iter->object=obj00000000000000000012 iter->instance= name=obj00000000000000000012 instance= ns= 2024-08-18T16:19:21.520 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.519+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16/obj00000000000000000012 2024-08-18T16:19:21.520 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.519+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: entry->timestamp=2024-08-18T16:19:19.602880+0000 2024-08-18T16:19:21.520 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.519+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:entry[obj00000000000000000012]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.91.6 op=0 op_state=1 2024-08-18T16:19:21.520 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.519+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:entry[obj00000000000000000012]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000012[0] 2024-08-18T16:19:21.520 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.519+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: listing bilog for incremental sync; position=00000000002.91.6 2024-08-18T16:19:21.520 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.519+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: parsed entry: id=00000000002.95.6 iter->object=obj00000000000000000024 iter->instance= name=obj00000000000000000024 instance= ns= 2024-08-18T16:19:21.520 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.519+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15/obj00000000000000000024 2024-08-18T16:19:21.520 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.519+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: entry->timestamp=2024-08-18T16:19:19.704871+0000 2024-08-18T16:19:21.520 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.519+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:entry[obj00000000000000000024]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.95.6 op=0 op_state=1 2024-08-18T16:19:21.521 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.519+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:entry[obj00000000000000000024]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000024[0] 2024-08-18T16:19:21.521 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.519+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: listing bilog for incremental sync; position=00000000002.95.6 2024-08-18T16:19:21.521 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.519+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:entry[obj00000000000000000009]: success 2024-08-18T16:19:21.521 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.519+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13:6 marker=00000000002.90.6 timestamp=2024-08-18T16:19:20.069388+0000 2024-08-18T16:19:21.522 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.520+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: parsed entry: id=00000000002.165.6 iter->object=obj00000000000000000040 iter->instance= name=obj00000000000000000040 instance= ns= 2024-08-18T16:19:21.522 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.520+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14/obj00000000000000000040 2024-08-18T16:19:21.522 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.520+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: entry->timestamp=2024-08-18T16:19:20.293593+0000 2024-08-18T16:19:21.522 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.520+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:entry[obj00000000000000000040]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.165.6 op=0 op_state=1 2024-08-18T16:19:21.523 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.520+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:entry[obj00000000000000000040]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000040[0] 2024-08-18T16:19:21.523 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.520+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: listing bilog for incremental sync; position=00000000002.165.6 2024-08-18T16:19:21.523 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.520+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: parsed entry: id=00000000002.116.6 iter->object=obj00000000000000000002 iter->instance= name=obj00000000000000000002 instance= ns= 2024-08-18T16:19:21.523 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.520+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17/obj00000000000000000002 2024-08-18T16:19:21.523 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.520+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: entry->timestamp=2024-08-18T16:19:19.940549+0000 2024-08-18T16:19:21.523 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.520+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:entry[obj00000000000000000002]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.116.6 op=0 op_state=1 2024-08-18T16:19:21.523 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.520+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:entry[obj00000000000000000002]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000002[0] 2024-08-18T16:19:21.523 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.520+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: listing bilog for incremental sync; position=00000000002.116.6 2024-08-18T16:19:21.523 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.521+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:entry[obj00000000000000000024]: success 2024-08-18T16:19:21.523 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.521+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15:6 marker=00000000002.95.6 timestamp=2024-08-18T16:19:19.704871+0000 2024-08-18T16:19:21.523 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.521+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:entry[obj00000000000000000012]: success 2024-08-18T16:19:21.523 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.521+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16:6 marker=00000000002.91.6 timestamp=2024-08-18T16:19:19.602880+0000 2024-08-18T16:19:21.524 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.522+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:entry[obj00000000000000000040]: success 2024-08-18T16:19:21.524 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.522+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14:6 marker=00000000002.165.6 timestamp=2024-08-18T16:19:20.293593+0000 2024-08-18T16:19:21.524 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.522+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:entry[obj00000000000000000002]: success 2024-08-18T16:19:21.524 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.522+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17:6 marker=00000000002.116.6 timestamp=2024-08-18T16:19:19.940549+0000 2024-08-18T16:19:21.525 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.523+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25]: parsed entry: id=00000000002.114.6 iter->object=obj00000000000000000010 iter->instance= name=obj00000000000000000010 instance= ns= 2024-08-18T16:19:21.525 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.523+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25/obj00000000000000000010 2024-08-18T16:19:21.525 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.523+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25]: entry->timestamp=2024-08-18T16:19:20.182223+0000 2024-08-18T16:19:21.525 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.523+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25]:entry[obj00000000000000000010]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.114.6 op=0 op_state=1 2024-08-18T16:19:21.525 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.523+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25]:entry[obj00000000000000000010]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000010[0] 2024-08-18T16:19:21.526 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.523+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25]: listing bilog for incremental sync; position=00000000002.114.6 2024-08-18T16:19:21.526 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.524+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]: parsed entry: id=00000000002.110.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2024-08-18T16:19:21.526 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.524+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26/obj00000000000000000021 2024-08-18T16:19:21.526 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.524+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]: entry->timestamp=2024-08-18T16:19:19.648761+0000 2024-08-18T16:19:21.526 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.524+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.110.6 op=0 op_state=1 2024-08-18T16:19:21.526 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.524+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]:entry[obj00000000000000000021]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000021[0] 2024-08-18T16:19:21.526 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.524+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]: listing bilog for incremental sync; position=00000000002.110.6 2024-08-18T16:19:21.526 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.524+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:32]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:32]: parsed entry: id=00000000002.87.6 iter->object=obj00000000000000000014 iter->instance= name=obj00000000000000000014 instance= ns= 2024-08-18T16:19:21.526 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.524+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:32]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:32]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:32/obj00000000000000000014 2024-08-18T16:19:21.526 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.524+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:32]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:32]: entry->timestamp=2024-08-18T16:19:20.396690+0000 2024-08-18T16:19:21.526 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.524+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:32]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:32]:entry[obj00000000000000000014]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.87.6 op=0 op_state=1 2024-08-18T16:19:21.527 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.524+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:32]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:32]:entry[obj00000000000000000014]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000014[0] 2024-08-18T16:19:21.527 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.524+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:32]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:32]: listing bilog for incremental sync; position=00000000002.87.6 2024-08-18T16:19:21.527 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.524+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25]:entry[obj00000000000000000010]: success 2024-08-18T16:19:21.527 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.524+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25:6 marker=00000000002.114.6 timestamp=2024-08-18T16:19:20.182223+0000 2024-08-18T16:19:21.527 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.525+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:31]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:31]: parsed entry: id=00000000002.142.6 iter->object=obj00000000000000000003 iter->instance= name=obj00000000000000000003 instance= ns= 2024-08-18T16:19:21.527 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.525+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:31]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:31]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:31/obj00000000000000000003 2024-08-18T16:19:21.527 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.525+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:31]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:31]: entry->timestamp=2024-08-18T16:19:20.172927+0000 2024-08-18T16:19:21.527 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.525+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:31]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:31]:entry[obj00000000000000000003]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.142.6 op=0 op_state=1 2024-08-18T16:19:21.527 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.525+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:31]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:31]:entry[obj00000000000000000003]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000003[0] 2024-08-18T16:19:21.527 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.525+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:31]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:31]: listing bilog for incremental sync; position=00000000002.142.6 2024-08-18T16:19:21.528 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.525+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]:entry[obj00000000000000000021]: success 2024-08-18T16:19:21.528 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.525+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26:6 marker=00000000002.110.6 timestamp=2024-08-18T16:19:19.648761+0000 2024-08-18T16:19:21.528 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.526+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:32]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:32]:entry[obj00000000000000000014]: success 2024-08-18T16:19:21.529 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.526+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:32]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:32]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:32:6 marker=00000000002.87.6 timestamp=2024-08-18T16:19:20.396690+0000 2024-08-18T16:19:21.529 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.526+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:31]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:31]:entry[obj00000000000000000003]: success 2024-08-18T16:19:21.529 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.526+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:31]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:31]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:31:6 marker=00000000002.142.6 timestamp=2024-08-18T16:19:20.172927+0000 2024-08-18T16:19:21.530 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.530+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: a sync operation returned error 2024-08-18T16:19:21.530 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.530+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: backing out with sync_status=-5 2024-08-18T16:19:21.530 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.530+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: incremental sync on bucket failed, retcode=-5 2024-08-18T16:19:21.531 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.530+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: ERROR: incremental sync failed. error: -5 2024-08-18T16:19:21.531 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.530+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:21.531 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.530+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: sync status for source bucket shard: 0 2024-08-18T16:19:21.531 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.530+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: listing bilog for incremental sync; position= 2024-08-18T16:19:21.532 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.532+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]: a sync operation returned error 2024-08-18T16:19:21.532 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.532+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]: backing out with sync_status=-5 2024-08-18T16:19:21.532 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.532+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]: incremental sync on bucket failed, retcode=-5 2024-08-18T16:19:21.532 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.532+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]: ERROR: incremental sync failed. error: -5 2024-08-18T16:19:21.533 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.532+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:21.533 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.533+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]: sync status for source bucket shard: 0 2024-08-18T16:19:21.533 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.533+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]: listing bilog for incremental sync; position= 2024-08-18T16:19:21.534 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.534+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:34]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:34]:entry[obj00000000000000000057]: success 2024-08-18T16:19:21.534 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.534+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:34]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:34]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:34:6 marker=00000000001.94.6 timestamp=2024-08-18T16:19:19.657944+0000 2024-08-18T16:19:21.573 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.572+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: parsed entry: id=00000000002.100.6 iter->object=obj00000000000000000013 iter->instance= name=obj00000000000000000013 instance= ns= 2024-08-18T16:19:21.573 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.572+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18/obj00000000000000000013 2024-08-18T16:19:21.573 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.572+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: entry->timestamp=2024-08-18T16:19:20.004845+0000 2024-08-18T16:19:21.573 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.572+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:entry[obj00000000000000000013]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.100.6 op=0 op_state=1 2024-08-18T16:19:21.573 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.572+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:entry[obj00000000000000000013]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000013[0] 2024-08-18T16:19:21.573 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.572+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: listing bilog for incremental sync; position=00000000002.100.6 2024-08-18T16:19:21.574 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.574+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:entry[obj00000000000000000013]: success 2024-08-18T16:19:21.574 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.574+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18:6 marker=00000000002.100.6 timestamp=2024-08-18T16:19:20.004845+0000 2024-08-18T16:19:21.575 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.575+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]: parsed entry: id=00000000002.100.6 iter->object=obj00000000000000000053 iter->instance= name=obj00000000000000000053 instance= ns= 2024-08-18T16:19:21.575 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.575+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27/obj00000000000000000053 2024-08-18T16:19:21.575 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.575+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]: entry->timestamp=2024-08-18T16:19:19.482927+0000 2024-08-18T16:19:21.575 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.575+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]:entry[obj00000000000000000053]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.100.6 op=0 op_state=1 2024-08-18T16:19:21.575 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.575+0000 7f63ec126880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]:entry[obj00000000000000000053]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000053[0] 2024-08-18T16:19:21.575 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.575+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]: listing bilog for incremental sync; position=00000000002.100.6 2024-08-18T16:19:21.579 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.579+0000 7f63ec126880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]:entry[obj00000000000000000053]: success 2024-08-18T16:19:21.579 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:21.579+0000 7f63ec126880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27:6 marker=00000000002.100.6 timestamp=2024-08-18T16:19:19.482927+0000 2024-08-18T16:19:21.639 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'list', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-77', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:19:21.640 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'list', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-77', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:19:21.640 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket list --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-77 --max-entries 10000 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:19:21.689 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:19:21.689 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:19:21.724 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T16:19:21.724 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:21.725 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000000", 2024-08-18T16:19:21.725 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:21.725 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:21.725 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:21.725 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 27 2024-08-18T16:19:21.725 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.726 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:21.726 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:21.726 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:21.727 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:21.727 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 5, 2024-08-18T16:19:21.727 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:19.893792Z", 2024-08-18T16:19:21.727 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:21.727 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:21.727 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:21.727 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:21.727 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:21.727 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:21.727 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:21.727 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:21.727 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.727 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.12694390134036672627", 2024-08-18T16:19:21.728 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:21.728 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:21.728 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:21.728 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.728 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:21.728 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000001", 2024-08-18T16:19:21.728 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:21.728 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:21.728 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:21.728 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 29 2024-08-18T16:19:21.728 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.728 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:21.728 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:21.728 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:21.729 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:21.729 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:19:21.729 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:20.116900Z", 2024-08-18T16:19:21.729 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:21.729 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:21.729 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:21.729 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:21.729 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:21.729 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:21.729 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:21.729 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:21.729 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.729 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.1775643464440672235", 2024-08-18T16:19:21.729 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:21.730 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:21.730 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:21.730 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.730 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:21.730 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000002", 2024-08-18T16:19:21.730 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:21.730 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:21.731 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:21.731 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 27 2024-08-18T16:19:21.731 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.731 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:21.731 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:21.731 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:21.731 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:21.731 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:19:21.731 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:19.940549Z", 2024-08-18T16:19:21.731 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:21.731 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:21.732 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:21.732 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:21.732 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:21.732 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:21.733 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:21.733 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:21.733 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.733 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.793237549756425130", 2024-08-18T16:19:21.733 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:21.733 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:21.733 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:21.733 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.733 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:21.733 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000003", 2024-08-18T16:19:21.733 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:21.733 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:21.733 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:21.733 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 11 2024-08-18T16:19:21.734 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.734 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:21.734 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:21.734 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:21.734 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:21.734 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 6, 2024-08-18T16:19:21.734 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:20.172927Z", 2024-08-18T16:19:21.734 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:21.734 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:21.734 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:21.734 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:21.734 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:21.735 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:21.735 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:21.735 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:21.735 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.735 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.7219353317483614029", 2024-08-18T16:19:21.735 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:21.735 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:21.735 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:21.735 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.735 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:21.735 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000004", 2024-08-18T16:19:21.736 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:21.736 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:21.736 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:21.736 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 20 2024-08-18T16:19:21.736 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.736 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:21.736 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:21.736 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:21.736 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:21.736 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-08-18T16:19:21.736 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:19.764610Z", 2024-08-18T16:19:21.736 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:21.736 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:21.737 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:21.737 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:21.737 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:21.737 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:21.737 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:21.737 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:21.737 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.737 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.9740686867748003437", 2024-08-18T16:19:21.737 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:21.737 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:21.737 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:21.737 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.737 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:21.737 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000005", 2024-08-18T16:19:21.738 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:21.738 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:21.738 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:21.738 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 17 2024-08-18T16:19:21.738 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.738 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:21.738 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:21.738 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:21.738 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:21.738 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-08-18T16:19:21.738 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:20.349953Z", 2024-08-18T16:19:21.738 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:21.738 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:21.739 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:21.739 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:21.739 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:21.739 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:21.739 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:21.739 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:21.739 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.739 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.14675195546044874488", 2024-08-18T16:19:21.739 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:21.739 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:21.739 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:21.739 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.739 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:21.739 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000006", 2024-08-18T16:19:21.740 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:21.740 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:21.740 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:21.740 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 15 2024-08-18T16:19:21.740 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.740 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:21.740 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:21.740 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:21.740 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:21.740 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 6, 2024-08-18T16:19:21.740 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:19.949473Z", 2024-08-18T16:19:21.740 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:21.740 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:21.740 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:21.741 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:21.741 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:21.741 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:21.741 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:21.741 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:21.741 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.741 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.5927078113473162591", 2024-08-18T16:19:21.742 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:21.742 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:21.742 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:21.742 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.742 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:21.743 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000007", 2024-08-18T16:19:21.743 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:21.743 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:21.743 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:21.743 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 26 2024-08-18T16:19:21.743 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.743 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:21.743 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:21.743 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:21.743 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:21.743 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:19:21.743 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:19.491062Z", 2024-08-18T16:19:21.743 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:21.744 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:21.744 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:21.744 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:21.744 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:21.744 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:21.744 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:21.744 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:21.744 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.744 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.6096458032678042161", 2024-08-18T16:19:21.744 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:21.745 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:21.745 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:21.745 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.745 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:21.745 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000008", 2024-08-18T16:19:21.745 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:21.745 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:21.745 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:21.745 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 12 2024-08-18T16:19:21.745 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.745 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:21.745 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:21.745 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:21.746 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:21.746 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 5, 2024-08-18T16:19:21.746 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:19.536516Z", 2024-08-18T16:19:21.746 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:21.746 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:21.746 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:21.746 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:21.746 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:21.746 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:21.746 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:21.746 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:21.747 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.747 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.8062311990328899558", 2024-08-18T16:19:21.747 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:21.747 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:21.747 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:21.747 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.747 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:21.747 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000009", 2024-08-18T16:19:21.747 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:21.747 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:21.748 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:21.748 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 15 2024-08-18T16:19:21.748 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.748 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:21.748 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:21.748 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:21.748 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:21.748 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 5, 2024-08-18T16:19:21.748 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:20.069388Z", 2024-08-18T16:19:21.748 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:21.748 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:21.748 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:21.749 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:21.749 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:21.749 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:21.749 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:21.749 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:21.749 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.749 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.14028552126077933249", 2024-08-18T16:19:21.749 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:21.749 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:21.749 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:21.749 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.749 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:21.750 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000010", 2024-08-18T16:19:21.750 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:21.750 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:21.750 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:21.750 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 26 2024-08-18T16:19:21.750 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.750 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:21.750 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:21.751 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:21.751 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:21.751 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 6, 2024-08-18T16:19:21.751 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:20.182223Z", 2024-08-18T16:19:21.751 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:21.751 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:21.751 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:21.751 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:21.751 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:21.751 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:21.751 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:21.751 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:21.751 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.751 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.3142064944146292633", 2024-08-18T16:19:21.752 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:21.752 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:21.752 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:21.752 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.752 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:21.752 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000011", 2024-08-18T16:19:21.752 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:21.752 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:21.752 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:21.752 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 21 2024-08-18T16:19:21.752 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.752 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:21.752 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:21.752 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:21.752 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:21.753 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:19:21.753 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:19.775284Z", 2024-08-18T16:19:21.753 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:21.753 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:21.753 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:21.753 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:21.753 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:21.753 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:21.753 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:21.753 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:21.753 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.753 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.12809818972740143366", 2024-08-18T16:19:21.753 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:21.753 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:21.754 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:21.754 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.754 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:21.754 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000012", 2024-08-18T16:19:21.754 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:21.754 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:21.754 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:21.754 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 14 2024-08-18T16:19:21.754 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.754 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:21.754 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:21.754 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:21.755 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:21.755 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-08-18T16:19:21.755 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:19.602880Z", 2024-08-18T16:19:21.755 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:21.755 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:21.755 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:21.755 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:21.755 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:21.756 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:21.756 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:21.758 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:21.758 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.758 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.17262977424700118057", 2024-08-18T16:19:21.758 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:21.758 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:21.758 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:21.758 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.758 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:21.759 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000013", 2024-08-18T16:19:21.759 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:21.759 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:21.759 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:21.759 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 22 2024-08-18T16:19:21.759 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.759 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:21.759 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:21.759 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:21.759 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:21.759 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:19:21.759 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:20.004845Z", 2024-08-18T16:19:21.759 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:21.760 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:21.760 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:21.760 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:21.760 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:21.760 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:21.760 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:21.760 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:21.760 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.760 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.10984888302709676165", 2024-08-18T16:19:21.760 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:21.760 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:21.760 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:21.761 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.761 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:21.761 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000014", 2024-08-18T16:19:21.761 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:21.761 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:21.761 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:21.761 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 11 2024-08-18T16:19:21.761 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.761 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:21.761 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:21.761 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:21.761 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:21.761 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 5, 2024-08-18T16:19:21.761 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:20.396690Z", 2024-08-18T16:19:21.761 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:21.762 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:21.762 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:21.762 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:21.762 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:21.762 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:21.762 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:21.762 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:21.762 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.762 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.6617372424373138853", 2024-08-18T16:19:21.762 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:21.762 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:21.763 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:21.763 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.763 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:21.763 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000015", 2024-08-18T16:19:21.763 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:21.763 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:21.763 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:21.763 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 19 2024-08-18T16:19:21.763 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.763 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:21.763 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:21.763 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:21.763 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:21.763 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 5, 2024-08-18T16:19:21.764 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:20.126126Z", 2024-08-18T16:19:21.764 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:21.764 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:21.764 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:21.764 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:21.764 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:21.764 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:21.764 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:21.765 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:21.765 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.765 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.11736278763128699882", 2024-08-18T16:19:21.765 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:21.765 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:21.765 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:21.765 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.765 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:21.765 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000016", 2024-08-18T16:19:21.765 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:21.765 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:21.765 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:21.765 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 15 2024-08-18T16:19:21.765 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.766 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:21.766 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:21.766 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:21.766 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:21.766 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:19:21.766 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:20.228897Z", 2024-08-18T16:19:21.766 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:21.766 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:21.766 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:21.766 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:21.767 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:21.767 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:21.767 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:21.767 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:21.767 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.767 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.5916908414622314875", 2024-08-18T16:19:21.767 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:21.767 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:21.767 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:21.767 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.767 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:21.767 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000017", 2024-08-18T16:19:21.768 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:21.768 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:21.768 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:21.768 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 5 2024-08-18T16:19:21.768 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.768 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:21.768 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:21.768 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:21.768 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:21.769 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-08-18T16:19:21.769 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:18.792258Z", 2024-08-18T16:19:21.769 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:21.769 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:21.769 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:21.769 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:21.770 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:21.770 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:21.770 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:21.770 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:21.770 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.771 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.9410209081733305536", 2024-08-18T16:19:21.771 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:21.771 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:21.771 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:21.771 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.771 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:21.772 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000018", 2024-08-18T16:19:21.772 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:21.772 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:21.772 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:21.772 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 14 2024-08-18T16:19:21.772 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.773 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:21.773 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:21.773 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:21.773 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:21.773 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 6, 2024-08-18T16:19:21.773 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:20.284525Z", 2024-08-18T16:19:21.774 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:21.774 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:21.774 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:21.774 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:21.774 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:21.774 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:21.775 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:21.775 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:21.775 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.775 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.5742324590714405983", 2024-08-18T16:19:21.775 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:21.775 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:21.776 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:21.776 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.776 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:21.776 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000019", 2024-08-18T16:19:21.776 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:21.776 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:21.776 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:21.777 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 16 2024-08-18T16:19:21.777 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.777 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:21.777 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:21.777 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:21.777 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:21.777 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:19:21.777 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:19.884831Z", 2024-08-18T16:19:21.778 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:21.778 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:21.778 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:21.778 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:21.778 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:21.778 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:21.779 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:21.779 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:21.779 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.779 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.4545522631134190464", 2024-08-18T16:19:21.779 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:21.780 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:21.780 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:21.780 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.780 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:21.780 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000020", 2024-08-18T16:19:21.780 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:21.780 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:21.780 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:21.781 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 7 2024-08-18T16:19:21.781 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.781 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:21.781 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:21.781 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:21.781 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:21.781 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 6, 2024-08-18T16:19:21.781 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:17.856631Z", 2024-08-18T16:19:21.782 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:21.782 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:21.782 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:21.782 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:21.782 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:21.783 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:21.783 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:21.783 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:21.783 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.783 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.15037230926380855934", 2024-08-18T16:19:21.783 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:21.784 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:21.784 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:21.784 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.784 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:21.784 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000021", 2024-08-18T16:19:21.784 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:21.784 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:21.784 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:21.785 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 24 2024-08-18T16:19:21.785 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.785 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:21.785 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:21.785 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:21.785 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:21.785 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 5, 2024-08-18T16:19:21.785 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:19.648761Z", 2024-08-18T16:19:21.786 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:21.786 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:21.786 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:21.786 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:21.786 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:21.786 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:21.786 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:21.786 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:21.787 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.787 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.17107778765257565441", 2024-08-18T16:19:21.787 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:21.787 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:21.787 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:21.787 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.787 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:21.788 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000022", 2024-08-18T16:19:21.788 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:21.788 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:21.788 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:21.788 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 8 2024-08-18T16:19:21.788 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.789 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:21.789 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:21.789 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:21.789 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:21.789 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 5, 2024-08-18T16:19:21.789 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:18.551431Z", 2024-08-18T16:19:21.790 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:21.790 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:21.790 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:21.790 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:21.790 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:21.791 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:21.791 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:21.791 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:21.791 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.791 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.16936946981126341225", 2024-08-18T16:19:21.791 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:21.792 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:21.792 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:21.792 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.792 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:21.792 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000023", 2024-08-18T16:19:21.793 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:21.793 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:21.793 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:21.793 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 6 2024-08-18T16:19:21.793 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.793 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:21.794 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:21.794 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:21.794 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:21.794 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:19:21.795 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:18.904686Z", 2024-08-18T16:19:21.795 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:21.795 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:21.795 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:21.795 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:21.796 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:21.796 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:21.796 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:21.796 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:21.796 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.796 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.13289786810085350536", 2024-08-18T16:19:21.797 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:21.797 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:21.797 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:21.797 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.797 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:21.798 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000024", 2024-08-18T16:19:21.798 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:21.798 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:21.798 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:21.798 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 7 2024-08-18T16:19:21.799 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.799 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:21.799 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:21.799 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:21.799 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:21.799 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-08-18T16:19:21.800 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:19.704871Z", 2024-08-18T16:19:21.800 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:21.800 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:21.800 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:21.800 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:21.800 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:21.801 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:21.801 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:21.801 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:21.801 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.801 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.9646989395737438153", 2024-08-18T16:19:21.801 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:21.802 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:21.802 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:21.802 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.802 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:21.802 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000025", 2024-08-18T16:19:21.803 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:21.803 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:21.803 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:21.803 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 12 2024-08-18T16:19:21.803 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.803 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:21.804 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:21.804 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:21.804 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:21.804 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:19:21.804 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:17.792369Z", 2024-08-18T16:19:21.804 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:21.805 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:21.805 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:21.805 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:21.805 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:21.805 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:21.805 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:21.805 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:21.806 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.806 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.5464753145558169266", 2024-08-18T16:19:21.806 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:21.806 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:21.806 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:21.806 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.807 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:21.807 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000026", 2024-08-18T16:19:21.807 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:21.807 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:21.807 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:21.807 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 28 2024-08-18T16:19:21.807 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.807 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:21.808 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:21.808 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:21.808 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:21.808 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 6, 2024-08-18T16:19:21.808 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:20.238134Z", 2024-08-18T16:19:21.808 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:21.808 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:21.809 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:21.809 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:21.809 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:21.809 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:21.809 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:21.809 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:21.809 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.810 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.14265561417317866991", 2024-08-18T16:19:21.810 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:21.810 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:21.810 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:21.810 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.811 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:21.811 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000027", 2024-08-18T16:19:21.811 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:21.811 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:21.811 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:21.811 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 15 2024-08-18T16:19:21.812 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.812 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:21.812 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:21.812 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:21.812 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:21.813 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 5, 2024-08-18T16:19:21.813 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:18.160384Z", 2024-08-18T16:19:21.813 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:21.813 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:21.813 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:21.813 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:21.813 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:21.814 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:21.814 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:21.814 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:21.814 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.814 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.6696940155157261459", 2024-08-18T16:19:21.814 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:21.814 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:21.815 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:21.815 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.815 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:21.815 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000028", 2024-08-18T16:19:21.815 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:21.815 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:21.815 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:21.815 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 19 2024-08-18T16:19:21.816 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.816 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:21.816 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:21.816 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:21.816 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:21.816 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:19:21.816 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:15.858769Z", 2024-08-18T16:19:21.817 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:21.817 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:21.817 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:21.817 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:21.817 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:21.817 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:21.818 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:21.818 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:21.818 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.818 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.8082080429937621815", 2024-08-18T16:19:21.818 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:21.818 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:21.819 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:21.819 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.819 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:21.819 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000029", 2024-08-18T16:19:21.819 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:21.819 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:21.820 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:21.820 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 13 2024-08-18T16:19:21.820 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.820 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:21.820 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:21.820 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:21.821 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:21.821 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 5, 2024-08-18T16:19:21.821 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:18.977845Z", 2024-08-18T16:19:21.821 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:21.821 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:21.822 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:21.822 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:21.822 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:21.822 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:21.822 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:21.822 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:21.823 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.823 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.2095082254158583061", 2024-08-18T16:19:21.823 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:21.823 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:21.823 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:21.823 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.823 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:21.823 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000030", 2024-08-18T16:19:21.824 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:21.824 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:21.824 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:21.824 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 9 2024-08-18T16:19:21.824 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.824 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:21.824 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:21.824 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:21.824 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:21.824 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:19:21.824 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:19.545863Z", 2024-08-18T16:19:21.824 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:21.825 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:21.825 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:21.825 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:21.825 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:21.825 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:21.825 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:21.825 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:21.825 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.825 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.6389725217193624987", 2024-08-18T16:19:21.825 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:21.825 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:21.825 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:21.825 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.826 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:21.826 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000031", 2024-08-18T16:19:21.826 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:21.826 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:21.826 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:21.826 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 14 2024-08-18T16:19:21.826 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.826 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:21.826 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:21.826 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:21.826 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:21.826 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:19:21.827 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:17.912320Z", 2024-08-18T16:19:21.827 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:21.827 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:21.827 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:21.827 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:21.827 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:21.827 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:21.827 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:21.827 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:21.827 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.827 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.1323283957345927255", 2024-08-18T16:19:21.827 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:21.827 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:21.827 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:21.828 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.828 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:21.828 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000032", 2024-08-18T16:19:21.828 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:21.828 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:21.828 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:21.828 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 17 2024-08-18T16:19:21.828 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.828 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:21.828 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:21.828 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:21.828 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:21.828 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-08-18T16:19:21.829 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:17.116603Z", 2024-08-18T16:19:21.829 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:21.829 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:21.829 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:21.829 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:21.829 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:21.829 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:21.829 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:21.829 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:21.829 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.829 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.2466765093447699000", 2024-08-18T16:19:21.829 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:21.829 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:21.830 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:21.830 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.830 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:21.830 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000033", 2024-08-18T16:19:21.830 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:21.830 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:21.830 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:21.830 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 25 2024-08-18T16:19:21.830 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.830 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:21.830 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:21.830 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:21.830 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:21.831 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 6, 2024-08-18T16:19:21.831 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:19.593080Z", 2024-08-18T16:19:21.831 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:21.831 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:21.831 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:21.831 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:21.831 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:21.831 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:21.831 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:21.831 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:21.831 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.831 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.15803267565377777934", 2024-08-18T16:19:21.831 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:21.832 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:21.832 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:21.832 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.832 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:21.832 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000036", 2024-08-18T16:19:21.832 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:21.832 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:21.832 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:21.832 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 10 2024-08-18T16:19:21.832 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.832 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:21.832 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:21.832 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:21.833 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:21.833 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:19:21.833 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:20.013736Z", 2024-08-18T16:19:21.833 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:21.833 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:21.833 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:21.833 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:21.833 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:21.833 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:21.833 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:21.833 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:21.834 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.834 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.7641068590117292671", 2024-08-18T16:19:21.834 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:21.834 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:21.834 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:21.834 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.835 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:21.835 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000037", 2024-08-18T16:19:21.835 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:21.835 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:21.835 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:21.835 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 14 2024-08-18T16:19:21.835 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.835 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:21.836 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:21.836 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:21.836 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:21.836 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:19:21.836 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:17.746196Z", 2024-08-18T16:19:21.836 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:21.836 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:21.836 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:21.836 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:21.836 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:21.836 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:21.837 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:21.837 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:21.837 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.837 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.3958915118237578826", 2024-08-18T16:19:21.837 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:21.837 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:21.837 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:21.837 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.837 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:21.837 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000039", 2024-08-18T16:19:21.837 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:21.837 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:21.838 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:21.838 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 16 2024-08-18T16:19:21.838 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.838 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:21.838 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:21.838 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:21.838 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:21.838 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 6, 2024-08-18T16:19:21.838 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:19.824786Z", 2024-08-18T16:19:21.838 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:21.838 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:21.838 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:21.839 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:21.839 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:21.839 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:21.839 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:21.839 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:21.839 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.839 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.17774398092704509620", 2024-08-18T16:19:21.839 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:21.839 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:21.839 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:21.840 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.840 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:21.840 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000040", 2024-08-18T16:19:21.840 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:21.840 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:21.840 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:21.840 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 9 2024-08-18T16:19:21.840 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.840 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:21.840 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:21.840 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:21.840 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:21.840 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 5, 2024-08-18T16:19:21.840 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:20.293593Z", 2024-08-18T16:19:21.841 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:21.841 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:21.841 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:21.841 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:21.841 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:21.841 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:21.841 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:21.841 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:21.841 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.841 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.14997593642927215052", 2024-08-18T16:19:21.841 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:21.841 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:21.841 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:21.842 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.842 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:21.842 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000042", 2024-08-18T16:19:21.842 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:21.842 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:21.842 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:21.842 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 7 2024-08-18T16:19:21.842 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.842 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:21.842 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:21.842 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:21.842 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:21.842 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:19:21.843 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:18.280618Z", 2024-08-18T16:19:21.843 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:21.843 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:21.843 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:21.843 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:21.843 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:21.843 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:21.843 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:21.843 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:21.843 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.843 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.7641885035429566441", 2024-08-18T16:19:21.843 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:21.843 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:21.844 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:21.844 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.844 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:21.844 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000045", 2024-08-18T16:19:21.844 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:21.844 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:21.844 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:21.844 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 12 2024-08-18T16:19:21.844 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.844 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:21.844 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:21.844 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:21.844 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:21.845 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-08-18T16:19:21.845 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:20.340703Z", 2024-08-18T16:19:21.845 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:21.845 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:21.845 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:21.845 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:21.845 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:21.845 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:21.845 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:21.845 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:21.845 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.846 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.10308047693451003210", 2024-08-18T16:19:21.846 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:21.846 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:21.846 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:21.846 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.846 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:21.846 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000046", 2024-08-18T16:19:21.846 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:21.846 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:21.846 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:21.847 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 5 2024-08-18T16:19:21.847 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.847 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:21.847 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:21.847 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:21.847 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:21.847 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-08-18T16:19:21.847 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:17.181145Z", 2024-08-18T16:19:21.847 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:21.847 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:21.847 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:21.847 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:21.847 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:21.848 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:21.848 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:21.848 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:21.848 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.848 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.7654529219595040550", 2024-08-18T16:19:21.848 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:21.848 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:21.848 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:21.848 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.848 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:21.848 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000048", 2024-08-18T16:19:21.848 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:21.848 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:21.849 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:21.849 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 8 2024-08-18T16:19:21.849 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.849 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:21.849 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:21.849 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:21.849 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:21.849 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:19:21.849 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:19.833816Z", 2024-08-18T16:19:21.849 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:21.849 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:21.849 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:21.849 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:21.849 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:21.850 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:21.850 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:21.850 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:21.850 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.850 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.15237095668940641587", 2024-08-18T16:19:21.850 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:21.850 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:21.850 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:21.850 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.850 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:21.851 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000049", 2024-08-18T16:19:21.851 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:21.851 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:21.851 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:21.851 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 10 2024-08-18T16:19:21.851 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.851 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:21.851 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:21.851 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:21.851 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:21.851 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:19:21.852 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:17.984613Z", 2024-08-18T16:19:21.852 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:21.852 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:21.852 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:21.852 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:21.852 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:21.852 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:21.852 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:21.852 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:21.852 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.852 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.2272497271796380264", 2024-08-18T16:19:21.852 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:21.853 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:21.853 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:21.853 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.853 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:21.853 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000050", 2024-08-18T16:19:21.853 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:21.853 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:21.853 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:21.853 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 13 2024-08-18T16:19:21.853 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.853 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:21.853 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:21.854 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:21.854 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:21.854 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:19:21.854 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:17.125461Z", 2024-08-18T16:19:21.854 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:21.854 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:21.854 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:21.854 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:21.854 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:21.855 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:21.855 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:21.855 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:21.855 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.855 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.4477381748113686257", 2024-08-18T16:19:21.855 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:21.855 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:21.855 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:21.855 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.855 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:21.856 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000053", 2024-08-18T16:19:21.856 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:21.856 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:21.856 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:21.856 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 23 2024-08-18T16:19:21.856 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.856 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:21.856 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:21.857 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:21.857 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:21.857 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:19:21.857 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:19.482927Z", 2024-08-18T16:19:21.857 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:21.857 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:21.857 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:21.857 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:21.857 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:21.857 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:21.857 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:21.858 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:21.858 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.858 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.11468292518739981541", 2024-08-18T16:19:21.858 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:21.858 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:21.858 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:21.858 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.858 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:21.858 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000055", 2024-08-18T16:19:21.858 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:21.858 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:21.859 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:21.859 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 49 2024-08-18T16:19:21.859 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.859 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:21.859 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:21.859 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:21.859 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:21.859 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 5, 2024-08-18T16:19:21.860 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:17.295995Z", 2024-08-18T16:19:21.860 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:21.860 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:21.860 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:21.860 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:21.860 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:21.860 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:21.860 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:21.860 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:21.860 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.860 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.7944227825297655296", 2024-08-18T16:19:21.860 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:21.861 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:21.861 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:21.861 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.861 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:21.861 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000056", 2024-08-18T16:19:21.861 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:21.861 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:21.861 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:21.861 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 10 2024-08-18T16:19:21.861 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.861 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:21.861 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:21.861 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:21.862 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:21.862 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 6, 2024-08-18T16:19:21.862 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:19.144248Z", 2024-08-18T16:19:21.862 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:21.862 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:21.862 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:21.862 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:21.862 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:21.862 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:21.862 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:21.862 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:21.862 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.863 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.5024964257853397233", 2024-08-18T16:19:21.863 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:21.863 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:21.863 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:21.863 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.863 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:21.863 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000057", 2024-08-18T16:19:21.863 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:21.863 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:21.863 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:21.863 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 11 2024-08-18T16:19:21.863 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.863 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:21.863 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:21.864 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:21.864 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:21.864 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-08-18T16:19:21.864 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:19.657944Z", 2024-08-18T16:19:21.864 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:21.864 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:21.864 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:21.864 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:21.864 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:21.864 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:21.864 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:21.864 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:21.864 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.865 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.10146843407256219658", 2024-08-18T16:19:21.865 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:21.865 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:21.865 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:21.865 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.865 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:21.865 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000058", 2024-08-18T16:19:21.865 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:21.865 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:21.865 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:21.866 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 11 2024-08-18T16:19:21.866 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.866 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:21.866 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:21.866 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:21.866 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:21.866 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:19:21.866 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:17.303274Z", 2024-08-18T16:19:21.866 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:21.866 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:21.866 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:21.866 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:21.866 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:21.867 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:21.867 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:21.867 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:21.867 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.867 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.1248125027812078921", 2024-08-18T16:19:21.867 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:21.867 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:21.867 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:21.867 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.867 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:21.868 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000059", 2024-08-18T16:19:21.868 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:21.868 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:21.868 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:21.868 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 9 2024-08-18T16:19:21.868 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.868 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:21.868 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:21.868 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:21.868 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:21.868 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-08-18T16:19:21.868 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:18.040381Z", 2024-08-18T16:19:21.868 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:21.868 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:21.869 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:21.869 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:21.869 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:21.869 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:21.869 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:21.869 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:21.869 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.869 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.1559192484951112292", 2024-08-18T16:19:21.869 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:21.869 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:21.869 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:21.869 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.869 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:21.870 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000060", 2024-08-18T16:19:21.870 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:21.870 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:21.870 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:21.870 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 22 2024-08-18T16:19:21.870 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.870 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:21.870 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:21.870 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:21.870 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:21.870 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:19:21.870 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:19.996680Z", 2024-08-18T16:19:21.871 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:21.871 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:21.871 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:21.871 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:21.871 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:21.871 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:21.871 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:21.871 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:21.871 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.871 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.11751167038022683473", 2024-08-18T16:19:21.872 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:21.872 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:21.872 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:21.872 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.872 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:21.872 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000061", 2024-08-18T16:19:21.872 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:21.872 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:21.872 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:21.872 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 9 2024-08-18T16:19:21.872 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.872 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:21.872 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:21.873 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:21.873 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:21.873 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-08-18T16:19:21.873 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:18.800570Z", 2024-08-18T16:19:21.873 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:21.873 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:21.873 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:21.873 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:21.873 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:21.873 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:21.873 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:21.874 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:21.874 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.874 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.18125434188322097673", 2024-08-18T16:19:21.874 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:21.874 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:21.874 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:21.874 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.874 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:21.874 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000062", 2024-08-18T16:19:21.875 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:21.875 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:21.875 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:21.875 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 7 2024-08-18T16:19:21.875 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.875 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:21.875 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:21.875 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:21.875 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:21.875 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 5, 2024-08-18T16:19:21.875 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:20.061021Z", 2024-08-18T16:19:21.876 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:21.876 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:21.876 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:21.876 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:21.876 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:21.876 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:21.876 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:21.876 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:21.876 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.876 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.8171122739445927861", 2024-08-18T16:19:21.877 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:21.877 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:21.877 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:21.877 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.877 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:21.877 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000065", 2024-08-18T16:19:21.877 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:21.877 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:21.877 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:21.877 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 11 2024-08-18T16:19:21.877 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.877 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:21.877 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:21.878 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:21.878 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:21.878 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 5, 2024-08-18T16:19:21.878 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:19.716103Z", 2024-08-18T16:19:21.878 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:21.878 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:21.878 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:21.878 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:21.878 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:21.878 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:21.878 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:21.878 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:21.878 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.879 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.14973101582796792823", 2024-08-18T16:19:21.879 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:21.879 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:21.879 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:21.879 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.879 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:21.879 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000076", 2024-08-18T16:19:21.879 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:21.879 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:21.879 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:21.879 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 10 2024-08-18T16:19:21.880 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.880 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:21.880 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:21.880 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:21.880 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:21.880 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-08-18T16:19:21.880 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:15.743831Z", 2024-08-18T16:19:21.880 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:21.880 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:21.880 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:21.880 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:21.880 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:21.881 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:21.881 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:21.881 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:21.881 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.881 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.14675623907960268010", 2024-08-18T16:19:21.881 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:21.881 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:21.881 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:21.881 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.881 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:21.881 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000133", 2024-08-18T16:19:21.881 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:21.881 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:21.881 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:21.882 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 22 2024-08-18T16:19:21.882 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.882 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:21.882 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:21.882 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:21.882 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:21.882 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 5, 2024-08-18T16:19:21.882 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:19.158889Z", 2024-08-18T16:19:21.882 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:21.882 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:21.882 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:21.882 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:21.883 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:21.883 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:21.883 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:21.883 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:21.883 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.883 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.7134523735991721996", 2024-08-18T16:19:21.883 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:21.883 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:21.883 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:21.883 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.883 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:21.883 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000162", 2024-08-18T16:19:21.883 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:21.883 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:21.884 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:21.884 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 11 2024-08-18T16:19:21.884 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.884 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:21.884 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:21.884 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:21.884 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:21.884 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:19:21.884 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:18.617486Z", 2024-08-18T16:19:21.884 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:21.884 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:21.884 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:21.884 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:21.885 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:21.885 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:21.885 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:21.885 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:21.885 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.885 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.15849648118366767075", 2024-08-18T16:19:21.885 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:21.885 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:21.885 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:21.885 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.885 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:21.885 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000271", 2024-08-18T16:19:21.886 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:21.886 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:21.886 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:21.886 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 10 2024-08-18T16:19:21.886 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.886 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:21.886 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:21.886 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:21.886 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:21.886 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 6, 2024-08-18T16:19:21.886 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:18.858093Z", 2024-08-18T16:19:21.886 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:21.887 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:21.887 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:21.887 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:21.887 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:21.887 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:21.887 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:21.887 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:21.887 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.887 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.11174342658862842768", 2024-08-18T16:19:21.887 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:21.887 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:21.887 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:21.887 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.887 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:21.888 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000283", 2024-08-18T16:19:21.888 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:21.888 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:21.888 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:21.888 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 9 2024-08-18T16:19:21.888 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.888 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:21.888 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:21.888 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:21.888 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:21.888 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 5, 2024-08-18T16:19:21.888 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:18.728571Z", 2024-08-18T16:19:21.888 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:21.889 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:21.889 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:21.889 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:21.889 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:21.889 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:21.889 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:21.889 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:21.889 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.889 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.12018170805249087996", 2024-08-18T16:19:21.889 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:21.889 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:21.891 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:21.891 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.891 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:21.891 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000317", 2024-08-18T16:19:21.891 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:21.891 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:21.891 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:21.891 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 9 2024-08-18T16:19:21.891 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.891 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:21.892 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:21.892 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:21.892 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:21.892 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-08-18T16:19:21.892 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:18.784378Z", 2024-08-18T16:19:21.892 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:21.892 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:21.892 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:21.892 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:21.892 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:21.892 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:21.892 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:21.893 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:21.893 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.893 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.5571667273815001199", 2024-08-18T16:19:21.893 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:21.893 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:21.893 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:21.893 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.893 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:21.893 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000423", 2024-08-18T16:19:21.894 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:21.894 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:21.894 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:21.894 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 12 2024-08-18T16:19:21.894 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.894 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:21.894 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:21.894 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:21.895 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:21.895 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-08-18T16:19:21.895 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:18.720502Z", 2024-08-18T16:19:21.895 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:21.895 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:21.895 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:21.895 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:21.895 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:21.895 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:21.895 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:21.895 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:21.895 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.895 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.14231938979835901774", 2024-08-18T16:19:21.896 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:21.896 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:21.896 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:21.896 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.896 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:21.896 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000467", 2024-08-18T16:19:21.896 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:21.896 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:21.896 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:21.896 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 5 2024-08-18T16:19:21.896 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.896 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:21.896 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:21.896 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:21.897 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:21.897 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 5, 2024-08-18T16:19:21.897 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:19.151651Z", 2024-08-18T16:19:21.897 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:21.897 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:21.897 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:21.897 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:21.897 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:21.897 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:21.897 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:21.897 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:21.897 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:21.897 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.13528487640187220054", 2024-08-18T16:19:21.898 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:21.898 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:21.898 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:21.898 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:19:21.898 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T16:19:21.899 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'list', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-77', '--max-entries', '10000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:19:21.899 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'list', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-77', '--max-entries', '10000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:19:21.899 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket list --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-77 --max-entries 10000 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:19:21.952 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:19:21.953 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:19:21.987 INFO:teuthology.orchestra.run.smithi193.stdout:[ 2024-08-18T16:19:21.988 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:19:21.988 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj00000000000000000000", 2024-08-18T16:19:21.988 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:19:21.988 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:19:21.988 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:19:21.988 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 10 2024-08-18T16:19:21.988 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:21.988 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:19:21.988 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:19:21.988 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:19:21.989 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:19:21.989 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 5, 2024-08-18T16:19:21.989 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:19:19.893792Z", 2024-08-18T16:19:21.989 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:21.989 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:19:21.989 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:21.989 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:21.989 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:21.989 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:19:21.989 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:19:21.989 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:19:21.990 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:21.990 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_VBP895OC47VueQqXjXrD4b_-LtP8seg", 2024-08-18T16:19:21.990 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:19:21.990 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:19:21.990 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:19:21.990 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:21.990 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:19:21.990 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj00000000000000000001", 2024-08-18T16:19:21.990 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:19:21.990 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:19:21.990 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:19:21.990 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 11 2024-08-18T16:19:21.991 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:21.991 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:19:21.991 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:19:21.991 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:19:21.991 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:19:21.991 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 16, 2024-08-18T16:19:21.991 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:19:20.116900Z", 2024-08-18T16:19:21.991 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:21.991 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:19:21.991 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:21.991 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:21.991 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:21.991 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:19:21.992 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:19:21.992 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:19:21.992 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:21.992 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_2RNtVSbSJyAlo-gfqcKs_34o8jqxg6C", 2024-08-18T16:19:21.992 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:19:21.992 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:19:21.992 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:19:21.992 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:21.992 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:19:21.992 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj00000000000000000002", 2024-08-18T16:19:21.992 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:19:21.992 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:19:21.992 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:19:21.993 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 10 2024-08-18T16:19:21.993 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:21.993 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:19:21.993 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:19:21.993 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:19:21.993 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:19:21.993 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 16, 2024-08-18T16:19:21.993 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:19:19.940549Z", 2024-08-18T16:19:21.993 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:21.993 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:19:21.994 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:21.994 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:21.994 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:21.994 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:19:21.994 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:19:21.994 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:19:21.994 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:21.994 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_U_z7tmNzcbmpo8aCiCNzXFrcnoM13dK", 2024-08-18T16:19:21.994 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:19:21.994 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:19:21.994 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:19:21.995 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:21.995 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:19:21.995 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj00000000000000000003", 2024-08-18T16:19:21.995 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:19:21.995 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:19:21.995 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:19:21.995 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 6 2024-08-18T16:19:21.995 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:21.995 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:19:21.996 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:19:21.996 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:19:21.996 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:19:21.996 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 6, 2024-08-18T16:19:21.996 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:19:20.172927Z", 2024-08-18T16:19:21.996 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:21.997 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:19:21.997 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:21.997 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:21.997 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:21.997 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:19:21.997 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:19:21.997 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:19:21.997 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:21.997 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_WViuJJp46Cps1LhpONYLFpjXnTkQwjN", 2024-08-18T16:19:21.997 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:19:21.997 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:19:21.998 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:19:21.998 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:21.998 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:19:21.998 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj00000000000000000004", 2024-08-18T16:19:21.998 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:19:21.998 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:19:21.998 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:19:21.998 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 9 2024-08-18T16:19:21.998 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:21.998 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:19:21.998 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:19:21.998 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:19:21.999 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:19:21.999 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 3, 2024-08-18T16:19:21.999 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:19:19.764610Z", 2024-08-18T16:19:21.999 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:21.999 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:19:21.999 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:21.999 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:21.999 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:21.999 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:19:21.999 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:19:21.999 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:19:21.999 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:21.999 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_8jHt-hKetWH_ZIfoUm4o6eHfirrlbHW", 2024-08-18T16:19:22.000 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:19:22.000 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:19:22.000 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:19:22.000 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.000 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:19:22.000 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj00000000000000000005", 2024-08-18T16:19:22.000 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:19:22.000 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:19:22.000 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:19:22.000 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 5 2024-08-18T16:19:22.000 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.000 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:19:22.000 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:19:22.001 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:19:22.001 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:19:22.001 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 6, 2024-08-18T16:19:22.001 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:19:20.349953Z", 2024-08-18T16:19:22.001 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:22.001 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:19:22.001 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:22.001 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:22.001 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:22.001 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:19:22.001 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:19:22.001 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:19:22.001 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.001 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_g-G5S9Wi4FmGfIN08Q36sqAlDYuWVa8", 2024-08-18T16:19:22.002 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:19:22.002 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:19:22.002 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:19:22.002 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.002 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:19:22.002 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj00000000000000000006", 2024-08-18T16:19:22.002 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:19:22.002 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:19:22.002 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:19:22.002 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 6 2024-08-18T16:19:22.002 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.002 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:19:22.003 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:19:22.003 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:19:22.003 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:19:22.003 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 6, 2024-08-18T16:19:22.003 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:19:19.949473Z", 2024-08-18T16:19:22.003 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:22.003 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:19:22.003 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:22.003 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:22.003 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:22.003 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:19:22.003 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:19:22.003 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:19:22.003 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.004 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_IdUaNZFmDihLvaki0PwIj2KatYQMH-W", 2024-08-18T16:19:22.004 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:19:22.004 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:19:22.004 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:19:22.004 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.004 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:19:22.004 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj00000000000000000007", 2024-08-18T16:19:22.004 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:19:22.004 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:19:22.004 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:19:22.004 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 10 2024-08-18T16:19:22.004 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.004 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:19:22.005 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:19:22.005 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:19:22.005 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:19:22.005 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 16, 2024-08-18T16:19:22.005 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:19:19.491062Z", 2024-08-18T16:19:22.005 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:22.005 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:19:22.005 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:22.005 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:22.005 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:22.005 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:19:22.005 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:19:22.005 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:19:22.006 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.006 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_pW52w5A9VKYzXcCCn6bK2gPb98lu_PK", 2024-08-18T16:19:22.006 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:19:22.006 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:19:22.006 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:19:22.006 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.006 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:19:22.006 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj00000000000000000008", 2024-08-18T16:19:22.007 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:19:22.007 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:19:22.007 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:19:22.007 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 6 2024-08-18T16:19:22.007 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.007 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:19:22.007 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:19:22.007 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:19:22.007 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:19:22.007 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 5, 2024-08-18T16:19:22.007 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:19:19.536516Z", 2024-08-18T16:19:22.007 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:22.007 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:19:22.008 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:22.008 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:22.008 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:22.008 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:19:22.008 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:19:22.008 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:19:22.008 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.008 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_19tq-JuObu4vSjdnt2HhkRgDQMvCsp8", 2024-08-18T16:19:22.008 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:19:22.008 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:19:22.008 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:19:22.009 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.009 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:19:22.009 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj00000000000000000009", 2024-08-18T16:19:22.009 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:19:22.009 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:19:22.009 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:19:22.009 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 4 2024-08-18T16:19:22.009 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.009 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:19:22.009 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:19:22.009 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:19:22.009 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:19:22.010 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 6, 2024-08-18T16:19:22.010 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:19:20.069388Z", 2024-08-18T16:19:22.010 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:22.010 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:19:22.010 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:22.010 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:22.010 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:22.010 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:19:22.010 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:19:22.010 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:19:22.010 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.010 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_UhxbckjEIPNC48LV375sl1NfVg5oAX4", 2024-08-18T16:19:22.011 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:19:22.011 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:19:22.011 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:19:22.011 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.011 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:19:22.011 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj00000000000000000010", 2024-08-18T16:19:22.011 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:19:22.012 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:19:22.012 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:19:22.012 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 7 2024-08-18T16:19:22.012 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.012 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:19:22.012 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:19:22.012 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:19:22.012 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:19:22.012 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 16, 2024-08-18T16:19:22.012 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:19:20.182223Z", 2024-08-18T16:19:22.012 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:22.012 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:19:22.013 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:22.013 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:22.013 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:22.013 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:19:22.013 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:19:22.013 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:19:22.013 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.013 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_2FfIXsceCqEdDYM9OOlAOEnw4S--_Vc", 2024-08-18T16:19:22.013 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:19:22.013 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:19:22.014 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:19:22.014 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.014 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:19:22.014 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj00000000000000000011", 2024-08-18T16:19:22.014 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:19:22.014 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:19:22.014 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:19:22.014 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 7 2024-08-18T16:19:22.014 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.014 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:19:22.014 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:19:22.015 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:19:22.015 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:19:22.015 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 6, 2024-08-18T16:19:22.015 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:19:19.775284Z", 2024-08-18T16:19:22.015 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:22.015 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:19:22.015 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:22.015 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:22.015 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:22.015 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:19:22.015 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:19:22.015 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:19:22.016 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.016 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_jWTBJAJhSwoT4CfkJxe6y7y0dp61Jiy", 2024-08-18T16:19:22.016 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:19:22.016 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:19:22.016 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:19:22.016 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.016 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:19:22.016 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj00000000000000000012", 2024-08-18T16:19:22.016 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:19:22.016 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:19:22.016 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:19:22.016 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 4 2024-08-18T16:19:22.016 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.017 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:19:22.017 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:19:22.017 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:19:22.017 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:19:22.017 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 16, 2024-08-18T16:19:22.017 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:19:19.602880Z", 2024-08-18T16:19:22.017 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:22.017 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:19:22.017 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:22.017 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:22.017 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:22.017 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:19:22.017 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:19:22.018 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:19:22.018 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.018 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_PYZk2dJShgC3pttGHraOmpYsOVl6kKt", 2024-08-18T16:19:22.018 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:19:22.018 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:19:22.018 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:19:22.018 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.018 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:19:22.018 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj00000000000000000013", 2024-08-18T16:19:22.018 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:19:22.018 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:19:22.018 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:19:22.019 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 8 2024-08-18T16:19:22.019 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.019 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:19:22.019 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:19:22.019 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:19:22.019 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:19:22.019 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 5, 2024-08-18T16:19:22.019 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:19:20.004845Z", 2024-08-18T16:19:22.019 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:22.019 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:19:22.019 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:22.019 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:22.019 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:22.019 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:19:22.020 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:19:22.020 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:19:22.020 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.020 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_kav58jyXmxwMZP7bgE0nqweGni7X2iK", 2024-08-18T16:19:22.020 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:19:22.020 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:19:22.020 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:19:22.020 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.020 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:19:22.020 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj00000000000000000014", 2024-08-18T16:19:22.020 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:19:22.020 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:19:22.020 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:19:22.020 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 4 2024-08-18T16:19:22.021 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.021 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:19:22.021 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:19:22.021 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:19:22.021 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:19:22.021 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 3, 2024-08-18T16:19:22.021 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:19:20.396690Z", 2024-08-18T16:19:22.021 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:22.021 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:19:22.021 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:22.021 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:22.021 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:22.021 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:19:22.022 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:19:22.022 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:19:22.022 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.022 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_1TwyCx49Wu6itBXT-_1zyJWQFN_WSY1", 2024-08-18T16:19:22.022 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:19:22.022 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:19:22.022 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:19:22.022 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.022 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:19:22.022 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj00000000000000000015", 2024-08-18T16:19:22.022 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:19:22.022 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:19:22.022 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:19:22.023 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 11 2024-08-18T16:19:22.023 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.023 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:19:22.023 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:19:22.023 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:19:22.023 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:19:22.023 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 3, 2024-08-18T16:19:22.023 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:19:20.126126Z", 2024-08-18T16:19:22.023 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:22.023 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:19:22.023 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:22.024 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:22.024 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:22.024 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:19:22.024 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:19:22.024 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:19:22.024 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.024 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_b3kVDyGHnGXekzlaUUv3gArwjU4HPXw", 2024-08-18T16:19:22.024 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:19:22.024 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:19:22.024 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:19:22.024 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.024 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:19:22.025 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj00000000000000000016", 2024-08-18T16:19:22.025 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:19:22.025 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:19:22.025 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:19:22.025 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 5 2024-08-18T16:19:22.025 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.025 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:19:22.025 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:19:22.025 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:19:22.025 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:19:22.025 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 5, 2024-08-18T16:19:22.025 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:19:20.228897Z", 2024-08-18T16:19:22.025 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:22.025 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:19:22.026 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:22.026 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:22.026 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:22.026 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:19:22.026 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:19:22.026 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:19:22.026 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.026 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_9RxjZpvh2-4VveLErDMSj4hTXqYviRY", 2024-08-18T16:19:22.027 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:19:22.027 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:19:22.027 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:19:22.027 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.027 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:19:22.027 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj00000000000000000017", 2024-08-18T16:19:22.027 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:19:22.027 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:19:22.028 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:19:22.028 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 4 2024-08-18T16:19:22.028 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.028 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:19:22.028 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:19:22.028 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:19:22.028 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:19:22.028 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 5, 2024-08-18T16:19:22.028 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:19:18.792258Z", 2024-08-18T16:19:22.028 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:22.028 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:19:22.029 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:22.029 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:22.029 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:22.029 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:19:22.029 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:19:22.029 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:19:22.029 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.029 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_L_FCXq0I2eUNyR7xziPTZEX52NcEIKs", 2024-08-18T16:19:22.029 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:19:22.029 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:19:22.029 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:19:22.029 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.029 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:19:22.029 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj00000000000000000018", 2024-08-18T16:19:22.030 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:19:22.030 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:19:22.030 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:19:22.030 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 7 2024-08-18T16:19:22.030 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.030 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:19:22.030 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:19:22.030 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:19:22.030 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:19:22.030 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 5, 2024-08-18T16:19:22.030 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:19:20.284525Z", 2024-08-18T16:19:22.030 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:22.031 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:19:22.031 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:22.031 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:22.031 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:22.031 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:19:22.031 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:19:22.031 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:19:22.031 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.031 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_1u6-RA6unJLf-CywtDpvdJIsVCi0Toj", 2024-08-18T16:19:22.031 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:19:22.031 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:19:22.031 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:19:22.031 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.032 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:19:22.032 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj00000000000000000019", 2024-08-18T16:19:22.032 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:19:22.032 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:19:22.032 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:19:22.032 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 7 2024-08-18T16:19:22.032 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.032 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:19:22.032 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:19:22.032 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:19:22.032 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:19:22.032 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 16, 2024-08-18T16:19:22.032 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:19:19.884831Z", 2024-08-18T16:19:22.033 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:22.033 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:19:22.033 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:22.033 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:22.033 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:22.033 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:19:22.033 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:19:22.033 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:19:22.033 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.033 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_Ij2L8zIaX_EgrML3h4QNm0Vg6p6GQYM", 2024-08-18T16:19:22.033 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:19:22.033 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:19:22.033 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:19:22.033 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.034 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:19:22.034 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj00000000000000000020", 2024-08-18T16:19:22.034 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:19:22.034 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:19:22.034 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:19:22.034 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 3 2024-08-18T16:19:22.034 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.034 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:19:22.034 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:19:22.034 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:19:22.034 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:19:22.034 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 16, 2024-08-18T16:19:22.035 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:19:17.856631Z", 2024-08-18T16:19:22.035 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:22.035 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:19:22.035 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:22.035 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:22.035 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:22.035 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:19:22.035 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:19:22.035 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:19:22.035 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.035 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "__4GMENhTNlm5-h2j5AIOB4HSHqYYyL1", 2024-08-18T16:19:22.036 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:19:22.036 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:19:22.036 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:19:22.036 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.036 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:19:22.036 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj00000000000000000021", 2024-08-18T16:19:22.036 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:19:22.036 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:19:22.036 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:19:22.036 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 6 2024-08-18T16:19:22.037 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.037 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:19:22.037 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:19:22.037 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:19:22.037 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:19:22.037 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 16, 2024-08-18T16:19:22.037 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:19:19.648761Z", 2024-08-18T16:19:22.037 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:22.037 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:19:22.037 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:22.037 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:22.037 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:22.038 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:19:22.038 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:19:22.038 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:19:22.038 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.038 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_wcPuMH_DLZiyiQr61ozhBUvB8qVE6BZ", 2024-08-18T16:19:22.038 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:19:22.038 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:19:22.038 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:19:22.038 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.038 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:19:22.038 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj00000000000000000022", 2024-08-18T16:19:22.038 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:19:22.038 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:19:22.039 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:19:22.039 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 7 2024-08-18T16:19:22.039 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.039 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:19:22.039 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:19:22.039 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:19:22.039 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:19:22.039 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 6, 2024-08-18T16:19:22.039 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:19:18.551431Z", 2024-08-18T16:19:22.039 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:22.039 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:19:22.039 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:22.040 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:22.040 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:22.040 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:19:22.040 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:19:22.040 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:19:22.040 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.040 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_kMxgL-IpTOnZL5OJihRb9ATNVrKEPLx", 2024-08-18T16:19:22.040 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:19:22.040 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:19:22.040 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:19:22.040 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.040 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:19:22.041 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj00000000000000000023", 2024-08-18T16:19:22.041 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:19:22.041 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:19:22.041 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:19:22.041 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 5 2024-08-18T16:19:22.041 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.041 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:19:22.041 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:19:22.041 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:19:22.041 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:19:22.042 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 16, 2024-08-18T16:19:22.042 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:19:18.904686Z", 2024-08-18T16:19:22.042 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:22.042 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:19:22.042 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:22.042 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:22.042 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:22.042 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:19:22.042 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:19:22.042 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:19:22.042 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.043 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_Qp6OPCP-6I-Ca7iF97d3vP3TT26k5-E", 2024-08-18T16:19:22.043 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:19:22.043 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:19:22.043 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:19:22.043 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.043 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:19:22.043 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj00000000000000000024", 2024-08-18T16:19:22.043 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:19:22.043 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:19:22.043 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:19:22.043 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 5 2024-08-18T16:19:22.043 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.043 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:19:22.044 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:19:22.044 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:19:22.044 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:19:22.044 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 16, 2024-08-18T16:19:22.044 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:19:19.704871Z", 2024-08-18T16:19:22.044 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:22.044 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:19:22.044 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:22.044 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:22.044 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:22.044 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:19:22.044 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:19:22.044 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:19:22.045 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.045 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_FUSrZ_O2YHuTDYPhm42fqmi2zxOsOBA", 2024-08-18T16:19:22.045 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:19:22.045 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:19:22.045 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:19:22.045 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.045 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:19:22.045 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj00000000000000000025", 2024-08-18T16:19:22.045 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:19:22.045 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:19:22.045 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:19:22.045 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 8 2024-08-18T16:19:22.045 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.045 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:19:22.046 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:19:22.046 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:19:22.046 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:19:22.046 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 16, 2024-08-18T16:19:22.046 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:19:17.792369Z", 2024-08-18T16:19:22.046 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:22.046 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:19:22.046 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:22.046 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:22.046 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:22.046 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:19:22.046 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:19:22.047 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:19:22.047 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.047 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_DxzvNmy7jqB2XEiMKUKhv9DTgV2yISu", 2024-08-18T16:19:22.047 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:19:22.047 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:19:22.047 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:19:22.047 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.047 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:19:22.047 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj00000000000000000026", 2024-08-18T16:19:22.047 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:19:22.048 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:19:22.048 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:19:22.048 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 9 2024-08-18T16:19:22.048 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.048 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:19:22.048 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:19:22.048 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:19:22.048 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:19:22.048 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 3, 2024-08-18T16:19:22.048 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:19:20.238134Z", 2024-08-18T16:19:22.048 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:22.048 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:19:22.049 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:22.049 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:22.049 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:22.049 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:19:22.049 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:19:22.049 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:19:22.049 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.049 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_qKf3KclmcdbaZTWt8Tyu_cZeYkZm_Di", 2024-08-18T16:19:22.049 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:19:22.049 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:19:22.049 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:19:22.049 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.050 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:19:22.050 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj00000000000000000027", 2024-08-18T16:19:22.050 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:19:22.050 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:19:22.050 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:19:22.050 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 9 2024-08-18T16:19:22.050 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.050 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:19:22.050 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:19:22.050 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:19:22.050 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:19:22.051 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 16, 2024-08-18T16:19:22.051 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:19:18.160384Z", 2024-08-18T16:19:22.051 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:22.051 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:19:22.051 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:22.051 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:22.051 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:22.051 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:19:22.051 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:19:22.051 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:19:22.051 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.051 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_DHFU-4lBeBNw3Uvqv03wbKpDvTm4MBT", 2024-08-18T16:19:22.051 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:19:22.052 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:19:22.052 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:19:22.052 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.052 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:19:22.052 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj00000000000000000028", 2024-08-18T16:19:22.052 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:19:22.052 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:19:22.052 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:19:22.052 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 11 2024-08-18T16:19:22.052 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.052 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:19:22.052 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:19:22.053 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:19:22.053 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:19:22.053 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 16, 2024-08-18T16:19:22.053 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:19:15.858769Z", 2024-08-18T16:19:22.053 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:22.053 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:19:22.053 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:22.053 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:22.053 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:22.053 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:19:22.053 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:19:22.054 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:19:22.054 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.054 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_eQZoQbJm0cH0-w8m4YTl35DE10LY7CC", 2024-08-18T16:19:22.054 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:19:22.054 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:19:22.054 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:19:22.054 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.054 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:19:22.054 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj00000000000000000029", 2024-08-18T16:19:22.054 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:19:22.054 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:19:22.054 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:19:22.054 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 6 2024-08-18T16:19:22.055 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.055 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:19:22.055 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:19:22.055 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:19:22.055 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:19:22.055 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 5, 2024-08-18T16:19:22.055 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:19:18.977845Z", 2024-08-18T16:19:22.055 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:22.055 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:19:22.055 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:22.055 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:22.055 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:22.055 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:19:22.056 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:19:22.056 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:19:22.056 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.056 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_n-RBpVVPFuCS06tM_GuvWwopjvEVGLc", 2024-08-18T16:19:22.056 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:19:22.056 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:19:22.056 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:19:22.056 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.056 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:19:22.056 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj00000000000000000030", 2024-08-18T16:19:22.056 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:19:22.056 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:19:22.056 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:19:22.056 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 3 2024-08-18T16:19:22.057 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.057 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:19:22.057 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:19:22.057 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:19:22.057 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:19:22.057 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 5, 2024-08-18T16:19:22.057 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:19:19.545863Z", 2024-08-18T16:19:22.057 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:22.057 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:19:22.058 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:22.058 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:22.058 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:22.058 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:19:22.058 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:19:22.058 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:19:22.058 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.058 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_Y9u0PUzE9gxqB2FDsxOCEgVJPrzAwhA", 2024-08-18T16:19:22.058 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:19:22.058 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:19:22.058 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:19:22.059 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.059 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:19:22.059 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj00000000000000000031", 2024-08-18T16:19:22.059 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:19:22.059 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:19:22.059 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:19:22.059 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 13 2024-08-18T16:19:22.059 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.059 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:19:22.059 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:19:22.059 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:19:22.059 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:19:22.059 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 3, 2024-08-18T16:19:22.060 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:19:17.912320Z", 2024-08-18T16:19:22.060 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:22.060 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:19:22.060 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:22.060 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:22.060 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:22.060 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:19:22.060 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:19:22.060 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:19:22.060 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.060 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_AxdVafkD31yg-ryPR08Jhku_Vu44sY-", 2024-08-18T16:19:22.061 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:19:22.061 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:19:22.061 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:19:22.061 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.061 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:19:22.061 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj00000000000000000032", 2024-08-18T16:19:22.061 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:19:22.061 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:19:22.061 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:19:22.061 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 9 2024-08-18T16:19:22.061 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.061 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:19:22.061 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:19:22.062 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:19:22.062 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:19:22.062 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 16, 2024-08-18T16:19:22.062 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:19:17.116603Z", 2024-08-18T16:19:22.062 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:22.062 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:19:22.062 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:22.062 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:22.062 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:22.062 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:19:22.062 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:19:22.062 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:19:22.062 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.063 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_xBqxzkjFpFcESYppQJS_VLM-rz2Tm-Z", 2024-08-18T16:19:22.063 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:19:22.063 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:19:22.063 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:19:22.063 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.063 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:19:22.063 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj00000000000000000033", 2024-08-18T16:19:22.063 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:19:22.063 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:19:22.063 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:19:22.063 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 8 2024-08-18T16:19:22.063 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.063 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:19:22.064 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:19:22.064 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:19:22.064 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:19:22.064 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 6, 2024-08-18T16:19:22.064 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:19:19.593080Z", 2024-08-18T16:19:22.064 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:22.064 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:19:22.064 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:22.064 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:22.064 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:22.064 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:19:22.064 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:19:22.064 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:19:22.065 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.065 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_MCNybtIWdjcevmJLdM6lMP-fREwBTdA", 2024-08-18T16:19:22.065 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:19:22.065 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:19:22.065 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:19:22.065 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.065 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:19:22.065 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj00000000000000000036", 2024-08-18T16:19:22.065 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:19:22.065 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:19:22.065 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:19:22.065 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 4 2024-08-18T16:19:22.065 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.066 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:19:22.066 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:19:22.066 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:19:22.066 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:19:22.066 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 16, 2024-08-18T16:19:22.066 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:19:20.013736Z", 2024-08-18T16:19:22.066 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:22.066 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:19:22.066 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:22.067 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:22.067 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:22.067 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:19:22.067 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:19:22.067 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:19:22.067 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.067 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_q7TjkmMH6oFfwqI3JvY-e9rzkEAXRHu", 2024-08-18T16:19:22.067 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:19:22.067 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:19:22.067 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:19:22.067 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.067 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:19:22.068 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj00000000000000000037", 2024-08-18T16:19:22.068 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:19:22.068 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:19:22.068 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:19:22.068 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 10 2024-08-18T16:19:22.068 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.068 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:19:22.068 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:19:22.068 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:19:22.068 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:19:22.068 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 5, 2024-08-18T16:19:22.069 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:19:17.746196Z", 2024-08-18T16:19:22.069 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:22.069 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:19:22.069 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:22.069 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:22.069 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:22.069 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:19:22.069 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:19:22.069 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:19:22.069 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.069 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_ehrHsbROUdHfjXJsNSArC_6eJdKdq9u", 2024-08-18T16:19:22.069 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:19:22.069 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:19:22.069 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:19:22.070 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.070 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:19:22.070 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj00000000000000000039", 2024-08-18T16:19:22.070 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:19:22.070 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:19:22.070 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:19:22.070 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 12 2024-08-18T16:19:22.070 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.070 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:19:22.070 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:19:22.070 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:19:22.070 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:19:22.071 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 16, 2024-08-18T16:19:22.071 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:19:19.824786Z", 2024-08-18T16:19:22.071 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:22.071 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:19:22.071 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:22.071 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:22.071 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:22.072 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:19:22.072 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:19:22.072 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:19:22.072 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.072 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_njv_YkMmC8wmQquz7hP1HX96mFvG60b", 2024-08-18T16:19:22.072 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:19:22.072 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:19:22.072 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:19:22.072 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.072 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:19:22.072 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj00000000000000000040", 2024-08-18T16:19:22.073 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:19:22.073 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:19:22.073 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:19:22.073 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 5 2024-08-18T16:19:22.073 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.073 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:19:22.073 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:19:22.073 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:19:22.073 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:19:22.073 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 5, 2024-08-18T16:19:22.073 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:19:20.293593Z", 2024-08-18T16:19:22.073 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:22.073 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:19:22.074 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:22.074 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:22.074 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:22.074 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:19:22.074 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:19:22.074 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:19:22.074 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.074 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_EuVTQIeRYkP1osSBQj8Pl-adSnjelRA", 2024-08-18T16:19:22.074 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:19:22.074 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:19:22.074 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:19:22.074 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.075 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:19:22.075 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj00000000000000000042", 2024-08-18T16:19:22.075 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:19:22.075 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:19:22.075 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:19:22.075 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 7 2024-08-18T16:19:22.075 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.075 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:19:22.075 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:19:22.075 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:19:22.075 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:19:22.075 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 5, 2024-08-18T16:19:22.075 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:19:18.280618Z", 2024-08-18T16:19:22.076 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:22.076 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:19:22.076 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:22.076 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:22.076 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:22.076 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:19:22.076 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:19:22.076 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:19:22.076 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.076 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_mCd172tzIkVe7PgWj0thVhFURYKe5BG", 2024-08-18T16:19:22.076 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:19:22.077 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:19:22.077 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:19:22.077 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.077 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:19:22.077 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj00000000000000000045", 2024-08-18T16:19:22.077 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:19:22.077 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:19:22.077 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:19:22.077 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 14 2024-08-18T16:19:22.077 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.078 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:19:22.078 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:19:22.078 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:19:22.078 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:19:22.078 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 16, 2024-08-18T16:19:22.078 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:19:20.340703Z", 2024-08-18T16:19:22.078 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:22.078 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:19:22.078 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:22.078 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:22.078 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:22.078 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:19:22.078 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:19:22.079 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:19:22.079 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.079 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_25WRF3OeXgedKGZwmdFxL-Fs43VCK4B", 2024-08-18T16:19:22.079 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:19:22.079 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:19:22.079 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:19:22.079 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.079 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:19:22.079 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj00000000000000000046", 2024-08-18T16:19:22.079 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:19:22.079 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:19:22.079 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:19:22.079 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 6 2024-08-18T16:19:22.080 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.080 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:19:22.080 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:19:22.080 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:19:22.080 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:19:22.080 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 16, 2024-08-18T16:19:22.080 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:19:17.181145Z", 2024-08-18T16:19:22.080 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:22.080 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:19:22.080 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:22.081 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:22.081 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:22.081 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:19:22.081 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:19:22.081 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:19:22.081 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.081 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_v_wCE-YO0EbWFa1nGxH10MtzBkAE5uC", 2024-08-18T16:19:22.081 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:19:22.081 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:19:22.081 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:19:22.082 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.082 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:19:22.082 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj00000000000000000048", 2024-08-18T16:19:22.082 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:19:22.082 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:19:22.082 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:19:22.082 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 9 2024-08-18T16:19:22.083 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.083 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:19:22.083 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:19:22.083 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:19:22.083 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:19:22.083 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 16, 2024-08-18T16:19:22.083 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:19:19.833816Z", 2024-08-18T16:19:22.083 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:22.083 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:19:22.083 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:22.084 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:22.084 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:22.084 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:19:22.084 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:19:22.084 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:19:22.085 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.085 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_4YaADzspPGm9U6_Y23VU_pziuX0m57H", 2024-08-18T16:19:22.085 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:19:22.085 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:19:22.085 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:19:22.085 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.086 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:19:22.086 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj00000000000000000049", 2024-08-18T16:19:22.086 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:19:22.086 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:19:22.086 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:19:22.087 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 6 2024-08-18T16:19:22.087 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.087 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:19:22.087 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:19:22.087 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:19:22.088 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:19:22.088 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 5, 2024-08-18T16:19:22.088 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:19:17.984613Z", 2024-08-18T16:19:22.088 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:22.088 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:19:22.088 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:22.088 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:22.088 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:22.088 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:19:22.088 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:19:22.088 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:19:22.088 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.089 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_laz18roO-D89r7GKSZ-gPHjXeGVPrPq", 2024-08-18T16:19:22.089 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:19:22.089 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:19:22.089 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:19:22.089 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.089 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:19:22.089 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj00000000000000000050", 2024-08-18T16:19:22.089 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:19:22.089 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:19:22.089 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:19:22.089 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 8 2024-08-18T16:19:22.089 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.089 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:19:22.090 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:19:22.090 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:19:22.090 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:19:22.090 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 16, 2024-08-18T16:19:22.090 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:19:17.125461Z", 2024-08-18T16:19:22.090 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:22.090 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:19:22.090 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:22.090 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:22.090 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:22.090 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:19:22.091 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:19:22.091 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:19:22.091 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.091 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_ro8K2lRAOLSOztbEPWmnoEG5mDBXYLC", 2024-08-18T16:19:22.091 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:19:22.091 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:19:22.091 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:19:22.091 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.091 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:19:22.091 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj00000000000000000053", 2024-08-18T16:19:22.091 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:19:22.091 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:19:22.091 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:19:22.092 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 12 2024-08-18T16:19:22.092 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.092 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:19:22.092 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:19:22.092 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:19:22.092 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:19:22.092 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 16, 2024-08-18T16:19:22.092 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:19:19.482927Z", 2024-08-18T16:19:22.092 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:22.092 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:19:22.092 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:22.092 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:22.092 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:22.093 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:19:22.093 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:19:22.093 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:19:22.093 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.093 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_MQAn90nhjQ2MMg-szlj6gud4_olurNX", 2024-08-18T16:19:22.093 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:19:22.093 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:19:22.093 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:19:22.093 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.093 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:19:22.093 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj00000000000000000055", 2024-08-18T16:19:22.093 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:19:22.094 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:19:22.094 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:19:22.094 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 47 2024-08-18T16:19:22.094 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.094 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:19:22.094 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:19:22.094 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:19:22.094 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:19:22.095 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 3, 2024-08-18T16:19:22.095 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:19:17.295995Z", 2024-08-18T16:19:22.095 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:22.095 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:19:22.095 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:22.095 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:22.095 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:22.096 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:19:22.096 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:19:22.096 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:19:22.096 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.096 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_JNwEepeoH2h9nZqB9-6GBSmCCmKNQ0b", 2024-08-18T16:19:22.097 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:19:22.097 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:19:22.097 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:19:22.097 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.097 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:19:22.097 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj00000000000000000056", 2024-08-18T16:19:22.098 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:19:22.098 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:19:22.098 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:19:22.098 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 10 2024-08-18T16:19:22.098 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.098 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:19:22.099 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:19:22.099 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:19:22.099 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:19:22.099 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 16, 2024-08-18T16:19:22.099 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:19:19.144248Z", 2024-08-18T16:19:22.099 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:22.100 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:19:22.100 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:22.100 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:22.100 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:22.100 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:19:22.101 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:19:22.101 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:19:22.101 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.101 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_wMYtcXtfx_xWz3E5TbVaIjKC72uEas2", 2024-08-18T16:19:22.101 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:19:22.101 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:19:22.102 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:19:22.102 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.102 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:19:22.102 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj00000000000000000057", 2024-08-18T16:19:22.102 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:19:22.103 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:19:22.103 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:19:22.103 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 10 2024-08-18T16:19:22.103 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.103 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:19:22.104 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:19:22.104 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:19:22.104 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:19:22.104 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 6, 2024-08-18T16:19:22.104 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:19:19.657944Z", 2024-08-18T16:19:22.105 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:22.105 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:19:22.105 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:22.105 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:22.105 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:22.105 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:19:22.105 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:19:22.105 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:19:22.105 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.105 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "__lrCHshVctnv6vHrD7ZE7aXz6M3cxT-", 2024-08-18T16:19:22.106 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:19:22.106 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:19:22.106 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:19:22.106 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.106 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:19:22.106 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj00000000000000000058", 2024-08-18T16:19:22.106 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:19:22.106 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:19:22.106 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:19:22.106 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 11 2024-08-18T16:19:22.106 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.106 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:19:22.106 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:19:22.107 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:19:22.107 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:19:22.107 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 3, 2024-08-18T16:19:22.107 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:19:17.303274Z", 2024-08-18T16:19:22.107 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:22.107 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:19:22.107 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:22.107 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:22.107 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:22.107 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:19:22.107 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:19:22.107 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:19:22.107 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.108 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_RaqD7tf77g4_VPN7MpwfLfREkzlHuyN", 2024-08-18T16:19:22.108 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:19:22.108 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:19:22.108 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:19:22.108 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.108 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:19:22.108 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj00000000000000000059", 2024-08-18T16:19:22.108 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:19:22.108 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:19:22.108 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:19:22.108 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 7 2024-08-18T16:19:22.108 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.109 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:19:22.109 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:19:22.109 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:19:22.109 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:19:22.109 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 16, 2024-08-18T16:19:22.109 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:19:18.040381Z", 2024-08-18T16:19:22.109 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:22.109 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:19:22.109 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:22.109 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:22.109 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:22.109 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:19:22.109 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:19:22.110 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:19:22.110 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.110 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_Tz049J1eh9aeEpbw4nhMeGFwpfKL35Z", 2024-08-18T16:19:22.110 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:19:22.110 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:19:22.110 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:19:22.110 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.110 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:19:22.110 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj00000000000000000060", 2024-08-18T16:19:22.110 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:19:22.110 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:19:22.110 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:19:22.110 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 9 2024-08-18T16:19:22.111 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.111 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:19:22.111 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:19:22.111 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:19:22.111 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:19:22.111 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 3, 2024-08-18T16:19:22.111 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:19:19.996680Z", 2024-08-18T16:19:22.111 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:22.111 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:19:22.111 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:22.111 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:22.111 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:22.111 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:19:22.111 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:19:22.112 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:19:22.112 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.112 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_De8dE4jBsdLFMGvtKQT6aHSlZhXIug-", 2024-08-18T16:19:22.112 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:19:22.112 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:19:22.112 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:19:22.112 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.112 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:19:22.112 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj00000000000000000061", 2024-08-18T16:19:22.112 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:19:22.112 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:19:22.112 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:19:22.112 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 17 2024-08-18T16:19:22.113 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.113 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:19:22.113 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:19:22.113 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:19:22.113 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:19:22.113 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 6, 2024-08-18T16:19:22.113 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:19:18.800570Z", 2024-08-18T16:19:22.113 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:22.113 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:19:22.113 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:22.113 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:22.113 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:22.114 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:19:22.114 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:19:22.114 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:19:22.114 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.114 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_W4Kg4ONo6xeRFCiR5I2qcpR-f3tqmsC", 2024-08-18T16:19:22.114 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:19:22.114 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:19:22.114 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:19:22.114 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.114 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:19:22.114 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj00000000000000000062", 2024-08-18T16:19:22.114 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:19:22.114 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:19:22.115 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:19:22.115 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 5 2024-08-18T16:19:22.115 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.115 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:19:22.115 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:19:22.115 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:19:22.115 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:19:22.115 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 16, 2024-08-18T16:19:22.115 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:19:20.061021Z", 2024-08-18T16:19:22.115 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:22.115 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:19:22.115 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:22.115 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:22.115 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:22.116 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:19:22.116 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:19:22.116 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:19:22.116 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.116 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_gOQbsXvzTO5XEZlJ3vtFPpAf8VkkcqV", 2024-08-18T16:19:22.116 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:19:22.116 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:19:22.116 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:19:22.116 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.116 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:19:22.116 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj00000000000000000065", 2024-08-18T16:19:22.116 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:19:22.116 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:19:22.117 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:19:22.117 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 11 2024-08-18T16:19:22.117 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.117 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:19:22.117 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:19:22.117 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:19:22.117 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:19:22.117 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 3, 2024-08-18T16:19:22.117 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:19:19.716103Z", 2024-08-18T16:19:22.117 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:22.117 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:19:22.117 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:22.117 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:22.117 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:22.118 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:19:22.118 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:19:22.118 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:19:22.118 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.118 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_aiGS-XsjZRYubl5jXBl35g2USrD2wBs", 2024-08-18T16:19:22.118 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:19:22.118 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:19:22.118 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:19:22.118 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.118 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:19:22.118 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj00000000000000000076", 2024-08-18T16:19:22.118 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:19:22.118 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:19:22.119 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:19:22.119 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 10 2024-08-18T16:19:22.119 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.119 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:19:22.119 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:19:22.119 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:19:22.119 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:19:22.119 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 6, 2024-08-18T16:19:22.119 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:19:15.743831Z", 2024-08-18T16:19:22.119 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:22.119 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:19:22.119 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:22.119 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:22.120 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:22.120 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:19:22.120 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:19:22.120 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:19:22.120 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.120 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_GLHdFSiaIQcwtm0HoltYAKsvn0JGc6W", 2024-08-18T16:19:22.120 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:19:22.120 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:19:22.120 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:19:22.120 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.120 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:19:22.120 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj00000000000000000133", 2024-08-18T16:19:22.120 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:19:22.120 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:19:22.121 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:19:22.121 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 11 2024-08-18T16:19:22.121 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.121 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:19:22.121 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:19:22.121 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:19:22.121 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:19:22.121 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 3, 2024-08-18T16:19:22.121 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:19:19.158889Z", 2024-08-18T16:19:22.121 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:22.121 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:19:22.121 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:22.121 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:22.121 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:22.122 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:19:22.122 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:19:22.122 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:19:22.122 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.122 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_7YWAe_AUpsmbQ53rDuu1DiHITaSupkp", 2024-08-18T16:19:22.122 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:19:22.122 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:19:22.122 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:19:22.122 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.122 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:19:22.122 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj00000000000000000162", 2024-08-18T16:19:22.122 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:19:22.122 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:19:22.123 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:19:22.123 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 7 2024-08-18T16:19:22.123 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.123 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:19:22.123 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:19:22.123 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:19:22.123 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:19:22.123 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 3, 2024-08-18T16:19:22.123 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:19:18.617486Z", 2024-08-18T16:19:22.123 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:22.123 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:19:22.123 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:22.123 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:22.124 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:22.124 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:19:22.124 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:19:22.124 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:19:22.124 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.124 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_BEr72hcPz0edFMV5fNA7eRLbUnBk_Xn", 2024-08-18T16:19:22.124 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:19:22.124 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:19:22.124 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:19:22.124 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.124 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:19:22.124 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj00000000000000000271", 2024-08-18T16:19:22.125 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:19:22.125 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:19:22.125 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:19:22.125 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 5 2024-08-18T16:19:22.125 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.125 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:19:22.125 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:19:22.125 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:19:22.125 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:19:22.125 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 5, 2024-08-18T16:19:22.125 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:19:18.858093Z", 2024-08-18T16:19:22.125 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:22.125 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:19:22.125 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:22.126 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:22.126 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:22.126 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:19:22.126 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:19:22.126 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:19:22.126 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.126 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_KGsmRF4krISthsS7ZctFK_G_UyzYwCG", 2024-08-18T16:19:22.126 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:19:22.126 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:19:22.126 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:19:22.126 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.126 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:19:22.126 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj00000000000000000283", 2024-08-18T16:19:22.127 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:19:22.127 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:19:22.127 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:19:22.127 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 5 2024-08-18T16:19:22.127 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.127 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:19:22.127 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:19:22.127 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:19:22.127 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:19:22.127 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 16, 2024-08-18T16:19:22.127 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:19:18.728571Z", 2024-08-18T16:19:22.127 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:22.127 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:19:22.127 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:22.128 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:22.128 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:22.128 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:19:22.128 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:19:22.128 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:19:22.128 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.128 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_efR2glNP7KwkXOe-SnS2isnZmSHVs15", 2024-08-18T16:19:22.128 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:19:22.128 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:19:22.128 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:19:22.128 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.129 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:19:22.129 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj00000000000000000317", 2024-08-18T16:19:22.129 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:19:22.129 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:19:22.129 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:19:22.129 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 8 2024-08-18T16:19:22.129 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.129 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:19:22.129 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:19:22.129 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:19:22.129 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:19:22.129 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 3, 2024-08-18T16:19:22.129 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:19:18.784378Z", 2024-08-18T16:19:22.130 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:22.130 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:19:22.130 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:22.130 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:22.130 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:22.130 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:19:22.130 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:19:22.130 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:19:22.130 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.130 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_MRF9WlbXLpG5Vl4AK3csFTmc7-aYbr-", 2024-08-18T16:19:22.130 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:19:22.130 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:19:22.130 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:19:22.131 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.131 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:19:22.131 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj00000000000000000423", 2024-08-18T16:19:22.131 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:19:22.131 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:19:22.131 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:19:22.131 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 12 2024-08-18T16:19:22.131 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.131 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:19:22.131 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:19:22.131 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:19:22.131 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:19:22.131 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 6, 2024-08-18T16:19:22.132 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:19:18.720502Z", 2024-08-18T16:19:22.132 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:22.132 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:19:22.132 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:22.132 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:22.132 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:22.132 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:19:22.132 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:19:22.132 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:19:22.132 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.132 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_oOLXIAgrRHhTMm3jsmVz0BP9UmO1XZV", 2024-08-18T16:19:22.132 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:19:22.132 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:19:22.133 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:19:22.133 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.133 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:19:22.133 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj00000000000000000467", 2024-08-18T16:19:22.133 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:19:22.133 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:19:22.133 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:19:22.134 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 5 2024-08-18T16:19:22.134 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.134 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:19:22.134 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:19:22.135 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:19:22.135 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:19:22.135 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 6, 2024-08-18T16:19:22.135 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:19:19.151651Z", 2024-08-18T16:19:22.135 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:22.135 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:19:22.135 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:22.135 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:22.135 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:22.135 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:19:22.135 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:19:22.135 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:19:22.135 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:19:22.136 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_mmxH7sHSTaOJ7J2bbQjZI_iwo4fWyvF", 2024-08-18T16:19:22.136 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:19:22.136 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:19:22.136 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:19:22.136 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:19:22.136 INFO:teuthology.orchestra.run.smithi193.stdout:] 2024-08-18T16:19:22.137 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'run', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-77', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:19:22.137 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'run', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-77', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:19:22.137 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync run --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-77 --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:19:22.188 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:19:22.188 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:19:22.261 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.259+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.261 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.259+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.261 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.259+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.261 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.259+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: sync status for source bucket shard: 2 2024-08-18T16:19:22.261 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.259+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: listing bilog for incremental sync; position=00000000001.83.6 2024-08-18T16:19:22.261 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.259+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.261 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.259+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: sync status for source bucket shard: 2 2024-08-18T16:19:22.261 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.259+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.261 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.260+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: sync status for source bucket shard: 2 2024-08-18T16:19:22.261 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.260+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: listing bilog for incremental sync; position=00000000001.85.6 2024-08-18T16:19:22.261 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.260+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.262 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.260+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: sync status for source bucket shard: 2 2024-08-18T16:19:22.262 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.260+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: listing bilog for incremental sync; position=00000000001.95.6 2024-08-18T16:19:22.262 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.260+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.262 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.260+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.262 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.260+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: sync status for source bucket shard: 2 2024-08-18T16:19:22.262 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.260+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: listing bilog for incremental sync; position=00000000001.60.6 2024-08-18T16:19:22.264 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.261+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.264 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.261+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: sync status for source bucket shard: 2 2024-08-18T16:19:22.264 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.261+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: sync status for source bucket shard: 2 2024-08-18T16:19:22.264 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.261+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: listing bilog for incremental sync; position=00000000001.74.6 2024-08-18T16:19:22.264 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.261+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: listing bilog for incremental sync; position=00000000001.76.6 2024-08-18T16:19:22.264 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.261+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.264 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.261+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.264 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.261+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.264 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.261+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: sync status for source bucket shard: 2 2024-08-18T16:19:22.264 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.262+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: listing bilog for incremental sync; position=00000000001.43.6 2024-08-18T16:19:22.264 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.262+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: sync status for source bucket shard: 2 2024-08-18T16:19:22.264 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.262+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: listing bilog for incremental sync; position=00000000001.84.6 2024-08-18T16:19:22.264 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.262+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: sync status for source bucket shard: 2 2024-08-18T16:19:22.265 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.262+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: listing bilog for incremental sync; position=00000000001.93.6 2024-08-18T16:19:22.265 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.262+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: sync status for source bucket shard: 2 2024-08-18T16:19:22.265 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.262+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: listing bilog for incremental sync; position=00000000001.61.6 2024-08-18T16:19:22.314 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.312+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.314 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.312+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.314 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.312+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.314 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.312+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: sync status for source bucket shard: 0 2024-08-18T16:19:22.314 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.312+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.314 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.312+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: sync status for source bucket shard: 0 2024-08-18T16:19:22.314 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.312+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.314 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.312+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.314 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.313+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: sync status for source bucket shard: 0 2024-08-18T16:19:22.315 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.313+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.315 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.313+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.315 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.313+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: sync status for source bucket shard: 0 2024-08-18T16:19:22.315 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.313+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.315 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.313+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.315 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.313+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.319 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.314+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.319 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.314+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.319 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.314+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: sync status for source bucket shard: 0 2024-08-18T16:19:22.319 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.314+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.319 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.314+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: sync status for source bucket shard: 0 2024-08-18T16:19:22.319 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.314+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.319 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.314+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.319 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.315+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: sync status for source bucket shard: 0 2024-08-18T16:19:22.319 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.315+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.319 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.315+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: sync status for source bucket shard: 0 2024-08-18T16:19:22.319 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.315+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.319 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.315+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.320 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.315+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.320 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.315+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.320 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.315+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.320 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.315+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.320 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.316+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: sync status for source bucket shard: 0 2024-08-18T16:19:22.320 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.316+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: sync status for source bucket shard: 0 2024-08-18T16:19:22.320 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.316+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]: sync status for source bucket shard: 0 2024-08-18T16:19:22.320 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.316+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: sync status for source bucket shard: 0 2024-08-18T16:19:22.320 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.316+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: sync status for source bucket shard: 0 2024-08-18T16:19:22.320 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.316+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.320 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.316+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.320 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.316+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.320 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.316+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.321 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.316+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.321 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.316+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: parsed entry: id=00000000001.161.6 iter->object=obj00000000000000000032 iter->instance= name=obj00000000000000000032 instance= ns= 2024-08-18T16:19:22.321 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.316+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6/obj00000000000000000032 2024-08-18T16:19:22.321 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.316+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: entry->timestamp=2024-08-18T16:19:15.971787+0000 2024-08-18T16:19:22.321 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.316+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:entry[obj00000000000000000032]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.161.6 op=0 op_state=1 2024-08-18T16:19:22.321 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.316+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.321 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.316+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.321 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.316+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:entry[obj00000000000000000032]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000032[0] 2024-08-18T16:19:22.321 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.316+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: listing bilog for incremental sync; position=00000000001.161.6 2024-08-18T16:19:22.321 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.317+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: sync status for source bucket shard: 0 2024-08-18T16:19:22.321 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.317+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]: sync status for source bucket shard: 0 2024-08-18T16:19:22.321 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.317+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.322 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.317+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.322 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.317+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: parsed entry: id=00000000002.138.6 iter->object=obj00000000000000000010 iter->instance= name=obj00000000000000000010 instance= ns= 2024-08-18T16:19:22.322 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.317+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8/obj00000000000000000010 2024-08-18T16:19:22.322 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.317+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: entry->timestamp=2024-08-18T16:19:15.866337+0000 2024-08-18T16:19:22.322 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.317+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:entry[obj00000000000000000010]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.138.6 op=0 op_state=1 2024-08-18T16:19:22.322 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.318+0000 7fbf28131880 0 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:entry[obj00000000000000000010]: : injecting data sync error on key=obj00000000000000000010 2024-08-18T16:19:22.322 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.318+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: listing bilog for incremental sync; position=00000000002.138.6 2024-08-18T16:19:22.322 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.318+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:entry[obj00000000000000000010]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:19:22.322 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.318+0000 7fbf28131880 0 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:entry[obj00000000000000000010]: ERROR: failed to sync object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8/obj00000000000000000010 2024-08-18T16:19:22.322 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.318+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: sync status for source bucket shard: 0 2024-08-18T16:19:22.322 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.318+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: sync status for source bucket shard: 0 2024-08-18T16:19:22.322 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.318+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.323 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.318+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.325 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.319+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]: parsed entry: id=00000000001.87.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2024-08-18T16:19:22.325 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.319+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11/obj00000000000000000021 2024-08-18T16:19:22.325 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.319+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]: entry->timestamp=2024-08-18T16:19:15.911763+0000 2024-08-18T16:19:22.325 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.319+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.87.6 op=0 op_state=1 2024-08-18T16:19:22.325 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.319+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:entry[obj00000000000000000021]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000021[0] 2024-08-18T16:19:22.325 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.319+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]: listing bilog for incremental sync; position=00000000001.87.6 2024-08-18T16:19:22.325 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.319+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: parsed entry: id=00000000002.70.6 iter->object=obj00000000000000000000 iter->instance= name=obj00000000000000000000 instance= ns= 2024-08-18T16:19:22.325 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.319+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5/obj00000000000000000000 2024-08-18T16:19:22.326 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.319+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: entry->timestamp=2024-08-18T16:19:15.979352+0000 2024-08-18T16:19:22.326 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.319+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:entry[obj00000000000000000000]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.70.6 op=0 op_state=1 2024-08-18T16:19:22.326 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.319+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: parsed entry: id=00000000001.85.6 iter->object=obj00000000000000000030 iter->instance= name=obj00000000000000000030 instance= ns= 2024-08-18T16:19:22.326 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.319+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12/obj00000000000000000030 2024-08-18T16:19:22.326 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.319+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: entry->timestamp=2024-08-18T16:19:15.806045+0000 2024-08-18T16:19:22.326 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.319+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:entry[obj00000000000000000030]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.85.6 op=0 op_state=1 2024-08-18T16:19:22.326 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.319+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:entry[obj00000000000000000000]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000000[0] 2024-08-18T16:19:22.326 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.319+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: listing bilog for incremental sync; position=00000000002.70.6 2024-08-18T16:19:22.326 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.319+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:entry[obj00000000000000000030]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000030[0] 2024-08-18T16:19:22.326 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.319+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: listing bilog for incremental sync; position=00000000001.85.6 2024-08-18T16:19:22.326 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.321+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: parsed entry: id=00000000001.109.6 iter->object=obj00000000000000000033 iter->instance= name=obj00000000000000000033 instance= ns= 2024-08-18T16:19:22.326 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.321+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14/obj00000000000000000033 2024-08-18T16:19:22.327 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.321+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: entry->timestamp=2024-08-18T16:19:15.918835+0000 2024-08-18T16:19:22.327 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.321+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:entry[obj00000000000000000033]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.109.6 op=0 op_state=1 2024-08-18T16:19:22.327 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.321+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:entry[obj00000000000000000033]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000033[0] 2024-08-18T16:19:22.327 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.321+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: listing bilog for incremental sync; position=00000000001.109.6 2024-08-18T16:19:22.327 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.322+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]: parsed entry: id=00000000001.90.6 iter->object=obj00000000000000000026 iter->instance= name=obj00000000000000000026 instance= ns= 2024-08-18T16:19:22.327 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.322+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13/obj00000000000000000026 2024-08-18T16:19:22.327 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.322+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]: entry->timestamp=2024-08-18T16:19:15.798609+0000 2024-08-18T16:19:22.327 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.322+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:entry[obj00000000000000000026]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.90.6 op=0 op_state=1 2024-08-18T16:19:22.327 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.322+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:entry[obj00000000000000000026]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000026[0] 2024-08-18T16:19:22.327 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.322+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]: listing bilog for incremental sync; position=00000000001.90.6 2024-08-18T16:19:22.330 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.330+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:entry[obj00000000000000000032]: success 2024-08-18T16:19:22.330 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.330+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6:1 marker=00000000001.161.6 timestamp=2024-08-18T16:19:15.971787+0000 2024-08-18T16:19:22.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.330+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:entry[obj00000000000000000033]: success 2024-08-18T16:19:22.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.330+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14:1 marker=00000000001.109.6 timestamp=2024-08-18T16:19:15.918835+0000 2024-08-18T16:19:22.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.330+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:entry[obj00000000000000000000]: success 2024-08-18T16:19:22.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.330+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5:1 marker=00000000002.70.6 timestamp=2024-08-18T16:19:15.979352+0000 2024-08-18T16:19:22.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.332+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:entry[obj00000000000000000030]: success 2024-08-18T16:19:22.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.332+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12:1 marker=00000000001.85.6 timestamp=2024-08-18T16:19:15.806045+0000 2024-08-18T16:19:22.356 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.355+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: parsed entry: id=00000000001.103.6 iter->object=obj00000000000000000007 iter->instance= name=obj00000000000000000007 instance= ns= 2024-08-18T16:19:22.356 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.355+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0/obj00000000000000000007 2024-08-18T16:19:22.356 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.355+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: entry->timestamp=2024-08-18T16:19:15.668069+0000 2024-08-18T16:19:22.356 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.355+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:entry[obj00000000000000000007]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.103.6 op=0 op_state=1 2024-08-18T16:19:22.356 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.355+0000 7fbf28131880 0 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:entry[obj00000000000000000007]: : injecting data sync error on key=obj00000000000000000007 2024-08-18T16:19:22.356 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.355+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: listing bilog for incremental sync; position=00000000001.103.6 2024-08-18T16:19:22.356 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.355+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: parsed entry: id=00000000001.105.6 iter->object=obj00000000000000000016 iter->instance= name=obj00000000000000000016 instance= ns= 2024-08-18T16:19:22.356 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.355+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1/obj00000000000000000016 2024-08-18T16:19:22.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.355+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: entry->timestamp=2024-08-18T16:19:15.677788+0000 2024-08-18T16:19:22.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.355+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:entry[obj00000000000000000016]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.105.6 op=0 op_state=1 2024-08-18T16:19:22.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.355+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: parsed entry: id=00000000001.87.6 iter->object=obj00000000000000000024 iter->instance= name=obj00000000000000000024 instance= ns= 2024-08-18T16:19:22.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.355+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2/obj00000000000000000024 2024-08-18T16:19:22.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.355+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: entry->timestamp=2024-08-18T16:19:15.851851+0000 2024-08-18T16:19:22.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.355+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:entry[obj00000000000000000024]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.87.6 op=0 op_state=1 2024-08-18T16:19:22.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.355+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:entry[obj00000000000000000007]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:19:22.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.355+0000 7fbf28131880 0 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:entry[obj00000000000000000007]: ERROR: failed to sync object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0/obj00000000000000000007 2024-08-18T16:19:22.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.355+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:entry[obj00000000000000000016]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000016[0] 2024-08-18T16:19:22.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.355+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: listing bilog for incremental sync; position=00000000001.105.6 2024-08-18T16:19:22.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.355+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:entry[obj00000000000000000024]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000024[0] 2024-08-18T16:19:22.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.355+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: listing bilog for incremental sync; position=00000000001.87.6 2024-08-18T16:19:22.359 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.356+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: parsed entry: id=00000000001.86.6 iter->object=obj00000000000000000028 iter->instance= name=obj00000000000000000028 instance= ns= 2024-08-18T16:19:22.360 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.356+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4/obj00000000000000000028 2024-08-18T16:19:22.360 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.356+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: entry->timestamp=2024-08-18T16:19:15.858769+0000 2024-08-18T16:19:22.360 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.356+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:entry[obj00000000000000000028]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.86.6 op=0 op_state=1 2024-08-18T16:19:22.360 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.356+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:entry[obj00000000000000000028]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000028[0] 2024-08-18T16:19:22.360 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.356+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: listing bilog for incremental sync; position=00000000001.86.6 2024-08-18T16:19:22.360 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.357+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: parsed entry: id=00000000001.56.6 iter->object=obj00000000000000000019 iter->instance= name=obj00000000000000000019 instance= ns= 2024-08-18T16:19:22.360 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.357+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9/obj00000000000000000019 2024-08-18T16:19:22.360 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.357+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: entry->timestamp=2024-08-18T16:19:15.728112+0000 2024-08-18T16:19:22.360 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.357+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:entry[obj00000000000000000019]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.56.6 op=0 op_state=1 2024-08-18T16:19:22.360 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.357+0000 7fbf28131880 0 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:entry[obj00000000000000000019]: : injecting data sync error on key=obj00000000000000000019 2024-08-18T16:19:22.360 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.357+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: listing bilog for incremental sync; position=00000000001.56.6 2024-08-18T16:19:22.360 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.357+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:entry[obj00000000000000000019]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:19:22.361 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.357+0000 7fbf28131880 0 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:entry[obj00000000000000000019]: ERROR: failed to sync object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9/obj00000000000000000019 2024-08-18T16:19:22.361 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.358+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: parsed entry: id=00000000002.69.6 iter->object=obj00000000000000000013 iter->instance= name=obj00000000000000000013 instance= ns= 2024-08-18T16:19:22.361 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.358+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10/obj00000000000000000013 2024-08-18T16:19:22.361 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.358+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: entry->timestamp=2024-08-18T16:19:15.734909+0000 2024-08-18T16:19:22.361 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.358+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:entry[obj00000000000000000013]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.69.6 op=0 op_state=1 2024-08-18T16:19:22.361 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.358+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:entry[obj00000000000000000013]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000013[0] 2024-08-18T16:19:22.361 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.358+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: listing bilog for incremental sync; position=00000000002.69.6 2024-08-18T16:19:22.361 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.359+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: parsed entry: id=00000000001.117.6 iter->object=obj00000000000000000020 iter->instance= name=obj00000000000000000020 instance= ns= 2024-08-18T16:19:22.361 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.359+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3/obj00000000000000000020 2024-08-18T16:19:22.361 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.359+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: entry->timestamp=2024-08-18T16:19:15.791409+0000 2024-08-18T16:19:22.361 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.359+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:entry[obj00000000000000000020]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.117.6 op=0 op_state=1 2024-08-18T16:19:22.361 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.359+0000 7fbf28131880 0 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:entry[obj00000000000000000020]: : injecting data sync error on key=obj00000000000000000020 2024-08-18T16:19:22.361 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.359+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: listing bilog for incremental sync; position=00000000001.117.6 2024-08-18T16:19:22.362 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.359+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:entry[obj00000000000000000020]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:19:22.362 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.359+0000 7fbf28131880 0 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:entry[obj00000000000000000020]: ERROR: failed to sync object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3/obj00000000000000000020 2024-08-18T16:19:22.363 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.361+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: a sync operation returned error 2024-08-18T16:19:22.363 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.361+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: parsed entry: id=00000000002.131.6 iter->object=obj00000000000000000004 iter->instance= name=obj00000000000000000004 instance= ns= 2024-08-18T16:19:22.363 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.361+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15/obj00000000000000000004 2024-08-18T16:19:22.363 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.361+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: entry->timestamp=2024-08-18T16:19:15.925675+0000 2024-08-18T16:19:22.363 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.361+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:entry[obj00000000000000000004]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.131.6 op=0 op_state=1 2024-08-18T16:19:22.363 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.361+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: parsed entry: id=00000000001.88.6 iter->object=obj00000000000000000076 iter->instance= name=obj00000000000000000076 instance= ns= 2024-08-18T16:19:22.363 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.361+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16/obj00000000000000000076 2024-08-18T16:19:22.363 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.361+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: entry->timestamp=2024-08-18T16:19:15.743831+0000 2024-08-18T16:19:22.363 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.361+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:entry[obj00000000000000000076]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.88.6 op=0 op_state=1 2024-08-18T16:19:22.363 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.361+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:entry[obj00000000000000000004]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000004[0] 2024-08-18T16:19:22.364 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.361+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: listing bilog for incremental sync; position=00000000002.131.6 2024-08-18T16:19:22.364 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.361+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:entry[obj00000000000000000076]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000076[0] 2024-08-18T16:19:22.364 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.361+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: listing bilog for incremental sync; position=00000000001.88.6 2024-08-18T16:19:22.364 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.361+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: backing out with sync_status=-5 2024-08-18T16:19:22.364 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.361+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: incremental sync on bucket failed, retcode=-5 2024-08-18T16:19:22.364 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.361+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: ERROR: incremental sync failed. error: -5 2024-08-18T16:19:22.364 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.362+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.364 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.363+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: sync status for source bucket shard: 0 2024-08-18T16:19:22.364 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.363+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.366 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.365+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:entry[obj00000000000000000028]: success 2024-08-18T16:19:22.366 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.365+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4:1 marker=00000000001.86.6 timestamp=2024-08-18T16:19:15.858769+0000 2024-08-18T16:19:22.366 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.365+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:entry[obj00000000000000000013]: success 2024-08-18T16:19:22.366 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.365+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10:1 marker=00000000002.69.6 timestamp=2024-08-18T16:19:15.734909+0000 2024-08-18T16:19:22.368 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.367+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:entry[obj00000000000000000021]: success 2024-08-18T16:19:22.368 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.367+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11:1 marker=00000000001.87.6 timestamp=2024-08-18T16:19:15.911763+0000 2024-08-18T16:19:22.372 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.372+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:entry[obj00000000000000000026]: success 2024-08-18T16:19:22.372 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.372+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13:1 marker=00000000001.90.6 timestamp=2024-08-18T16:19:15.798609+0000 2024-08-18T16:19:22.399 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.398+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: a sync operation returned error 2024-08-18T16:19:22.399 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.398+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: backing out with sync_status=-5 2024-08-18T16:19:22.399 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.398+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: incremental sync on bucket failed, retcode=-5 2024-08-18T16:19:22.399 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.398+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: ERROR: incremental sync failed. error: -5 2024-08-18T16:19:22.399 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.398+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.400 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.399+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: a sync operation returned error 2024-08-18T16:19:22.400 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.399+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: backing out with sync_status=-5 2024-08-18T16:19:22.400 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.399+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: incremental sync on bucket failed, retcode=-5 2024-08-18T16:19:22.400 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.399+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: ERROR: incremental sync failed. error: -5 2024-08-18T16:19:22.400 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.399+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: sync status for source bucket shard: 0 2024-08-18T16:19:22.400 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.399+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.400 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.399+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.401 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.400+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: sync status for source bucket shard: 0 2024-08-18T16:19:22.401 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.400+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.401 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.401+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: a sync operation returned error 2024-08-18T16:19:22.402 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.401+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: backing out with sync_status=-5 2024-08-18T16:19:22.402 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.401+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: incremental sync on bucket failed, retcode=-5 2024-08-18T16:19:22.402 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.401+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: ERROR: incremental sync failed. error: -5 2024-08-18T16:19:22.402 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.401+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:entry[obj00000000000000000016]: success 2024-08-18T16:19:22.402 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.401+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1:1 marker=00000000001.105.6 timestamp=2024-08-18T16:19:15.677788+0000 2024-08-18T16:19:22.403 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.402+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.403 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.402+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: sync status for source bucket shard: 0 2024-08-18T16:19:22.403 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.402+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.403 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.402+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:entry[obj00000000000000000024]: success 2024-08-18T16:19:22.403 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.402+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2:1 marker=00000000001.87.6 timestamp=2024-08-18T16:19:15.851851+0000 2024-08-18T16:19:22.406 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.406+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: parsed entry: id=00000000002.138.6 iter->object=obj00000000000000000010 iter->instance= name=obj00000000000000000010 instance= ns= 2024-08-18T16:19:22.406 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.406+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8/obj00000000000000000010 2024-08-18T16:19:22.406 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.406+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: entry->timestamp=2024-08-18T16:19:15.866337+0000 2024-08-18T16:19:22.406 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.406+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:entry[obj00000000000000000010]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.138.6 op=0 op_state=1 2024-08-18T16:19:22.406 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.406+0000 7fbf28131880 0 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:entry[obj00000000000000000010]: : injecting data sync error on key=obj00000000000000000010 2024-08-18T16:19:22.407 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.406+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: listing bilog for incremental sync; position=00000000002.138.6 2024-08-18T16:19:22.407 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.406+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:entry[obj00000000000000000010]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:19:22.407 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.406+0000 7fbf28131880 0 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:entry[obj00000000000000000010]: ERROR: failed to sync object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8/obj00000000000000000010 2024-08-18T16:19:22.409 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.409+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:entry[obj00000000000000000004]: success 2024-08-18T16:19:22.409 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.409+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15:1 marker=00000000002.131.6 timestamp=2024-08-18T16:19:15.925675+0000 2024-08-18T16:19:22.409 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.409+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:entry[obj00000000000000000076]: success 2024-08-18T16:19:22.409 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.409+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16:1 marker=00000000001.88.6 timestamp=2024-08-18T16:19:15.743831+0000 2024-08-18T16:19:22.441 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.441+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: parsed entry: id=00000000001.103.6 iter->object=obj00000000000000000007 iter->instance= name=obj00000000000000000007 instance= ns= 2024-08-18T16:19:22.442 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.441+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0/obj00000000000000000007 2024-08-18T16:19:22.442 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.441+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: entry->timestamp=2024-08-18T16:19:15.668069+0000 2024-08-18T16:19:22.442 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.441+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:entry[obj00000000000000000007]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.103.6 op=0 op_state=1 2024-08-18T16:19:22.442 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.441+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:entry[obj00000000000000000007]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000007[0] 2024-08-18T16:19:22.442 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.441+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: listing bilog for incremental sync; position=00000000001.103.6 2024-08-18T16:19:22.443 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.443+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: parsed entry: id=00000000001.56.6 iter->object=obj00000000000000000019 iter->instance= name=obj00000000000000000019 instance= ns= 2024-08-18T16:19:22.443 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.443+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9/obj00000000000000000019 2024-08-18T16:19:22.443 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.443+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: entry->timestamp=2024-08-18T16:19:15.728112+0000 2024-08-18T16:19:22.443 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.443+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:entry[obj00000000000000000019]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.56.6 op=0 op_state=1 2024-08-18T16:19:22.444 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.443+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:entry[obj00000000000000000019]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000019[0] 2024-08-18T16:19:22.444 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.443+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: listing bilog for incremental sync; position=00000000001.56.6 2024-08-18T16:19:22.445 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.445+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:entry[obj00000000000000000007]: success 2024-08-18T16:19:22.445 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.445+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0:1 marker=00000000001.103.6 timestamp=2024-08-18T16:19:15.668069+0000 2024-08-18T16:19:22.446 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.446+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: parsed entry: id=00000000001.117.6 iter->object=obj00000000000000000020 iter->instance= name=obj00000000000000000020 instance= ns= 2024-08-18T16:19:22.446 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.446+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3/obj00000000000000000020 2024-08-18T16:19:22.446 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.446+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: entry->timestamp=2024-08-18T16:19:15.791409+0000 2024-08-18T16:19:22.446 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.446+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:entry[obj00000000000000000020]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.117.6 op=0 op_state=1 2024-08-18T16:19:22.446 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.446+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:entry[obj00000000000000000020]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000020[0] 2024-08-18T16:19:22.446 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.446+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: listing bilog for incremental sync; position=00000000001.117.6 2024-08-18T16:19:22.448 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.448+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: a sync operation returned error 2024-08-18T16:19:22.448 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.448+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: backing out with sync_status=-5 2024-08-18T16:19:22.448 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.448+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: incremental sync on bucket failed, retcode=-5 2024-08-18T16:19:22.448 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.448+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: ERROR: incremental sync failed. error: -5 2024-08-18T16:19:22.449 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.448+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.449 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.448+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: sync status for source bucket shard: 0 2024-08-18T16:19:22.449 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.448+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.451 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.451+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:entry[obj00000000000000000020]: success 2024-08-18T16:19:22.451 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.451+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3:1 marker=00000000001.117.6 timestamp=2024-08-18T16:19:15.791409+0000 2024-08-18T16:19:22.489 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.488+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:entry[obj00000000000000000019]: success 2024-08-18T16:19:22.489 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.488+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9:1 marker=00000000001.56.6 timestamp=2024-08-18T16:19:15.728112+0000 2024-08-18T16:19:22.490 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.490+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: parsed entry: id=00000000002.138.6 iter->object=obj00000000000000000010 iter->instance= name=obj00000000000000000010 instance= ns= 2024-08-18T16:19:22.490 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.490+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8/obj00000000000000000010 2024-08-18T16:19:22.490 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.490+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: entry->timestamp=2024-08-18T16:19:15.866337+0000 2024-08-18T16:19:22.490 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.490+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:entry[obj00000000000000000010]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.138.6 op=0 op_state=1 2024-08-18T16:19:22.491 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.490+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:entry[obj00000000000000000010]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000010[0] 2024-08-18T16:19:22.491 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.490+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: listing bilog for incremental sync; position=00000000002.138.6 2024-08-18T16:19:22.536 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.535+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:entry[obj00000000000000000010]: success 2024-08-18T16:19:22.536 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.535+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8:1 marker=00000000002.138.6 timestamp=2024-08-18T16:19:15.866337+0000 2024-08-18T16:19:22.541 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.540+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.541 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.540+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.541 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.540+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.541 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.540+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: sync status for source bucket shard: 0 2024-08-18T16:19:22.541 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.540+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.542 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.540+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: sync status for source bucket shard: 0 2024-08-18T16:19:22.542 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.540+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.542 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.540+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.542 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.541+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: sync status for source bucket shard: 0 2024-08-18T16:19:22.542 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.541+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.542 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.541+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.546 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.541+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: sync status for source bucket shard: 0 2024-08-18T16:19:22.546 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.541+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.546 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.541+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.546 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.541+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: sync status for source bucket shard: 0 2024-08-18T16:19:22.547 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.541+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.547 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.541+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.547 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.541+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.547 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.542+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: sync status for source bucket shard: 0 2024-08-18T16:19:22.547 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.542+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.547 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.542+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.547 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.542+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: sync status for source bucket shard: 0 2024-08-18T16:19:22.547 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.542+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.547 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.542+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: sync status for source bucket shard: 0 2024-08-18T16:19:22.547 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.542+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.547 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.542+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.547 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.542+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: parsed entry: id=00000000002.126.6 iter->object=obj00000000000000000010 iter->instance= name=obj00000000000000000010 instance= ns= 2024-08-18T16:19:22.548 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.542+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1/obj00000000000000000010 2024-08-18T16:19:22.548 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.542+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: entry->timestamp=2024-08-18T16:19:16.412010+0000 2024-08-18T16:19:22.548 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.542+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:entry[obj00000000000000000010]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.126.6 op=0 op_state=1 2024-08-18T16:19:22.548 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.542+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.548 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.542+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:entry[obj00000000000000000010]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000010[0] 2024-08-18T16:19:22.548 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.542+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: listing bilog for incremental sync; position=00000000002.126.6 2024-08-18T16:19:22.548 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.543+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.548 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.544+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: sync status for source bucket shard: 0 2024-08-18T16:19:22.548 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.544+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.548 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.544+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: sync status for source bucket shard: 0 2024-08-18T16:19:22.548 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.544+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.548 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.544+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: sync status for source bucket shard: 0 2024-08-18T16:19:22.548 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.544+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: parsed entry: id=00000000002.166.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2024-08-18T16:19:22.549 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.544+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3/obj00000000000000000021 2024-08-18T16:19:22.549 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.544+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: entry->timestamp=2024-08-18T16:19:16.547723+0000 2024-08-18T16:19:22.549 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.544+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.166.6 op=0 op_state=1 2024-08-18T16:19:22.549 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.544+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.549 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.544+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.549 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.544+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.549 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.544+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.549 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.544+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.549 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.544+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:entry[obj00000000000000000021]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000021[0] 2024-08-18T16:19:22.549 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.544+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: listing bilog for incremental sync; position=00000000002.166.6 2024-08-18T16:19:22.549 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.544+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: parsed entry: id=00000000001.144.6 iter->object=obj00000000000000000025 iter->instance= name=obj00000000000000000025 instance= ns= 2024-08-18T16:19:22.549 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.544+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5/obj00000000000000000025 2024-08-18T16:19:22.550 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.544+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: entry->timestamp=2024-08-18T16:19:16.429591+0000 2024-08-18T16:19:22.550 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.544+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:entry[obj00000000000000000025]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.144.6 op=0 op_state=1 2024-08-18T16:19:22.550 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.544+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.550 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.544+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: parsed entry: id=00000000002.67.6 iter->object=obj00000000000000000004 iter->instance= name=obj00000000000000000004 instance= ns= 2024-08-18T16:19:22.550 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.544+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4/obj00000000000000000004 2024-08-18T16:19:22.550 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.544+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: entry->timestamp=2024-08-18T16:19:16.233425+0000 2024-08-18T16:19:22.550 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.544+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:entry[obj00000000000000000004]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.67.6 op=0 op_state=1 2024-08-18T16:19:22.550 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.544+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: parsed entry: id=00000000002.65.6 iter->object=obj00000000000000000016 iter->instance= name=obj00000000000000000016 instance= ns= 2024-08-18T16:19:22.550 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.544+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2/obj00000000000000000016 2024-08-18T16:19:22.550 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.544+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: entry->timestamp=2024-08-18T16:19:16.360394+0000 2024-08-18T16:19:22.550 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.544+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:entry[obj00000000000000000016]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.65.6 op=0 op_state=1 2024-08-18T16:19:22.550 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.544+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.551 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.544+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:entry[obj00000000000000000025]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000025[0] 2024-08-18T16:19:22.551 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.544+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: listing bilog for incremental sync; position=00000000001.144.6 2024-08-18T16:19:22.551 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.544+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:entry[obj00000000000000000004]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000004[0] 2024-08-18T16:19:22.551 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.544+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: listing bilog for incremental sync; position=00000000002.67.6 2024-08-18T16:19:22.551 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.544+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:entry[obj00000000000000000016]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000016[0] 2024-08-18T16:19:22.551 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.544+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: listing bilog for incremental sync; position=00000000002.65.6 2024-08-18T16:19:22.551 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.544+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: parsed entry: id=00000000002.146.6 iter->object=obj00000000000000000015 iter->instance= name=obj00000000000000000015 instance= ns= 2024-08-18T16:19:22.551 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.545+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7/obj00000000000000000015 2024-08-18T16:19:22.551 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.545+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: entry->timestamp=2024-08-18T16:19:16.555569+0000 2024-08-18T16:19:22.551 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.545+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:entry[obj00000000000000000015]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.146.6 op=0 op_state=1 2024-08-18T16:19:22.551 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.545+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.551 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.545+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:entry[obj00000000000000000015]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000015[0] 2024-08-18T16:19:22.551 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.545+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: listing bilog for incremental sync; position=00000000002.146.6 2024-08-18T16:19:22.552 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.545+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: sync status for source bucket shard: 0 2024-08-18T16:19:22.552 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.545+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.552 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.546+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]: sync status for source bucket shard: 0 2024-08-18T16:19:22.552 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.546+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.552 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.546+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:entry[obj00000000000000000010]: success 2024-08-18T16:19:22.552 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.546+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1:2 marker=00000000002.126.6 timestamp=2024-08-18T16:19:16.412010+0000 2024-08-18T16:19:22.554 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.546+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: sync status for source bucket shard: 0 2024-08-18T16:19:22.554 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.546+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.554 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.546+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: sync status for source bucket shard: 0 2024-08-18T16:19:22.554 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.546+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.555 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.547+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: sync status for source bucket shard: 0 2024-08-18T16:19:22.555 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.547+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.555 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.547+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: sync status for source bucket shard: 0 2024-08-18T16:19:22.555 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.548+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.555 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.548+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]: sync status for source bucket shard: 0 2024-08-18T16:19:22.555 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.548+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: sync status for source bucket shard: 0 2024-08-18T16:19:22.555 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.548+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.555 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.548+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.555 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.548+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:entry[obj00000000000000000004]: success 2024-08-18T16:19:22.555 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.548+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4:2 marker=00000000002.67.6 timestamp=2024-08-18T16:19:16.233425+0000 2024-08-18T16:19:22.555 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.549+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:entry[obj00000000000000000021]: success 2024-08-18T16:19:22.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.549+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3:2 marker=00000000002.166.6 timestamp=2024-08-18T16:19:16.547723+0000 2024-08-18T16:19:22.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.551+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]: parsed entry: id=00000000002.136.6 iter->object=obj00000000000000000002 iter->instance= name=obj00000000000000000002 instance= ns= 2024-08-18T16:19:22.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.551+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13/obj00000000000000000002 2024-08-18T16:19:22.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.551+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]: entry->timestamp=2024-08-18T16:19:16.343786+0000 2024-08-18T16:19:22.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.551+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:entry[obj00000000000000000002]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.136.6 op=0 op_state=1 2024-08-18T16:19:22.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.551+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]: parsed entry: id=00000000001.81.6 iter->object=obj00000000000000000014 iter->instance= name=obj00000000000000000014 instance= ns= 2024-08-18T16:19:22.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.551+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11/obj00000000000000000014 2024-08-18T16:19:22.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.551+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]: entry->timestamp=2024-08-18T16:19:16.298314+0000 2024-08-18T16:19:22.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.551+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:entry[obj00000000000000000014]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.81.6 op=0 op_state=1 2024-08-18T16:19:22.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.551+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: parsed entry: id=00000000002.93.6 iter->object=obj00000000000000000007 iter->instance= name=obj00000000000000000007 instance= ns= 2024-08-18T16:19:22.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.551+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18/obj00000000000000000007 2024-08-18T16:19:22.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.551+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: entry->timestamp=2024-08-18T16:19:16.290628+0000 2024-08-18T16:19:22.557 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.551+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:entry[obj00000000000000000007]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.93.6 op=0 op_state=1 2024-08-18T16:19:22.557 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.551+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:entry[obj00000000000000000002]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000002[0] 2024-08-18T16:19:22.557 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.551+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]: listing bilog for incremental sync; position=00000000002.136.6 2024-08-18T16:19:22.557 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.551+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: parsed entry: id=00000000002.111.6 iter->object=obj00000000000000000012 iter->instance= name=obj00000000000000000012 instance= ns= 2024-08-18T16:19:22.557 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.551+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12/obj00000000000000000012 2024-08-18T16:19:22.557 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.551+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: entry->timestamp=2024-08-18T16:19:16.351846+0000 2024-08-18T16:19:22.557 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.551+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:entry[obj00000000000000000012]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.111.6 op=0 op_state=1 2024-08-18T16:19:22.557 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.551+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:entry[obj00000000000000000014]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000014[0] 2024-08-18T16:19:22.557 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.551+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]: listing bilog for incremental sync; position=00000000001.81.6 2024-08-18T16:19:22.557 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.551+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:entry[obj00000000000000000016]: success 2024-08-18T16:19:22.557 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.551+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2:2 marker=00000000002.65.6 timestamp=2024-08-18T16:19:16.360394+0000 2024-08-18T16:19:22.557 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.551+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:entry[obj00000000000000000007]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000007[0] 2024-08-18T16:19:22.558 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.551+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: listing bilog for incremental sync; position=00000000002.93.6 2024-08-18T16:19:22.558 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.551+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:entry[obj00000000000000000012]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000012[0] 2024-08-18T16:19:22.558 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.551+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: listing bilog for incremental sync; position=00000000002.111.6 2024-08-18T16:19:22.558 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.552+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: parsed entry: id=00000000002.114.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2024-08-18T16:19:22.558 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.552+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17/obj00000000000000000001 2024-08-18T16:19:22.558 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.552+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: entry->timestamp=2024-08-18T16:19:16.621827+0000 2024-08-18T16:19:22.558 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.552+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.114.6 op=0 op_state=1 2024-08-18T16:19:22.558 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.552+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:entry[obj00000000000000000001]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000001[0] 2024-08-18T16:19:22.558 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.552+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: listing bilog for incremental sync; position=00000000002.114.6 2024-08-18T16:19:22.559 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.556+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:entry[obj00000000000000000007]: success 2024-08-18T16:19:22.559 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.556+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18:2 marker=00000000002.93.6 timestamp=2024-08-18T16:19:16.290628+0000 2024-08-18T16:19:22.559 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.557+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:entry[obj00000000000000000015]: success 2024-08-18T16:19:22.559 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.557+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7:2 marker=00000000002.146.6 timestamp=2024-08-18T16:19:16.555569+0000 2024-08-18T16:19:22.560 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.560+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:entry[obj00000000000000000001]: success 2024-08-18T16:19:22.560 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.560+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17:2 marker=00000000002.114.6 timestamp=2024-08-18T16:19:16.621827+0000 2024-08-18T16:19:22.585 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.585+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: parsed entry: id=00000000002.130.6 iter->object=obj00000000000000000030 iter->instance= name=obj00000000000000000030 instance= ns= 2024-08-18T16:19:22.586 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.585+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6/obj00000000000000000030 2024-08-18T16:19:22.586 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.585+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: entry->timestamp=2024-08-18T16:19:16.612682+0000 2024-08-18T16:19:22.586 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.585+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:entry[obj00000000000000000030]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.130.6 op=0 op_state=1 2024-08-18T16:19:22.586 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.585+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:entry[obj00000000000000000030]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000030[0] 2024-08-18T16:19:22.586 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.585+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: listing bilog for incremental sync; position=00000000002.130.6 2024-08-18T16:19:22.587 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.586+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: parsed entry: id=00000000002.74.6 iter->object=obj00000000000000000019 iter->instance= name=obj00000000000000000019 instance= ns= 2024-08-18T16:19:22.587 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.586+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8/obj00000000000000000019 2024-08-18T16:19:22.587 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.586+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: entry->timestamp=2024-08-18T16:19:16.421104+0000 2024-08-18T16:19:22.587 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.586+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:entry[obj00000000000000000019]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.74.6 op=0 op_state=1 2024-08-18T16:19:22.587 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.586+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:entry[obj00000000000000000019]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000019[0] 2024-08-18T16:19:22.587 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.586+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: listing bilog for incremental sync; position=00000000002.74.6 2024-08-18T16:19:22.588 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.587+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:entry[obj00000000000000000030]: success 2024-08-18T16:19:22.588 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.587+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6:2 marker=00000000002.130.6 timestamp=2024-08-18T16:19:16.612682+0000 2024-08-18T16:19:22.589 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.588+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: parsed entry: id=00000000002.61.6 iter->object=obj00000000000000000003 iter->instance= name=obj00000000000000000003 instance= ns= 2024-08-18T16:19:22.589 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.588+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9/obj00000000000000000003 2024-08-18T16:19:22.589 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.588+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: entry->timestamp=2024-08-18T16:19:16.604379+0000 2024-08-18T16:19:22.589 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.588+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:entry[obj00000000000000000003]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.61.6 op=0 op_state=1 2024-08-18T16:19:22.589 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.588+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:entry[obj00000000000000000003]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000003[0] 2024-08-18T16:19:22.590 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.588+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: listing bilog for incremental sync; position=00000000002.61.6 2024-08-18T16:19:22.590 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.588+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:entry[obj00000000000000000019]: success 2024-08-18T16:19:22.590 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.588+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8:2 marker=00000000002.74.6 timestamp=2024-08-18T16:19:16.421104+0000 2024-08-18T16:19:22.590 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.590+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: parsed entry: id=00000000001.95.6 iter->object=obj00000000000000000009 iter->instance= name=obj00000000000000000009 instance= ns= 2024-08-18T16:19:22.590 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.590+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10/obj00000000000000000009 2024-08-18T16:19:22.590 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.590+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: entry->timestamp=2024-08-18T16:19:16.283741+0000 2024-08-18T16:19:22.590 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.590+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:entry[obj00000000000000000009]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.95.6 op=0 op_state=1 2024-08-18T16:19:22.590 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.590+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:entry[obj00000000000000000009]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000009[0] 2024-08-18T16:19:22.591 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.590+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: listing bilog for incremental sync; position=00000000001.95.6 2024-08-18T16:19:22.592 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.592+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: parsed entry: id=00000000002.95.6 iter->object=obj00000000000000000011 iter->instance= name=obj00000000000000000011 instance= ns= 2024-08-18T16:19:22.592 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.592+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16/obj00000000000000000011 2024-08-18T16:19:22.592 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.592+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: entry->timestamp=2024-08-18T16:19:16.480839+0000 2024-08-18T16:19:22.593 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.592+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:entry[obj00000000000000000011]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.95.6 op=0 op_state=1 2024-08-18T16:19:22.593 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.592+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:entry[obj00000000000000000011]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000011[0] 2024-08-18T16:19:22.593 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.592+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: listing bilog for incremental sync; position=00000000002.95.6 2024-08-18T16:19:22.593 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.592+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: parsed entry: id=00000000002.128.6 iter->object=obj00000000000000000006 iter->instance= name=obj00000000000000000006 instance= ns= 2024-08-18T16:19:22.593 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.592+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15/obj00000000000000000006 2024-08-18T16:19:22.593 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.592+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: entry->timestamp=2024-08-18T16:19:16.489682+0000 2024-08-18T16:19:22.593 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.592+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:entry[obj00000000000000000006]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.128.6 op=0 op_state=1 2024-08-18T16:19:22.593 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.592+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:entry[obj00000000000000000006]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000006[0] 2024-08-18T16:19:22.593 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.592+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: listing bilog for incremental sync; position=00000000002.128.6 2024-08-18T16:19:22.594 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.593+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: parsed entry: id=00000000002.93.6 iter->object=obj00000000000000000000 iter->instance= name=obj00000000000000000000 instance= ns= 2024-08-18T16:19:22.595 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.593+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14/obj00000000000000000000 2024-08-18T16:19:22.595 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.593+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: entry->timestamp=2024-08-18T16:19:16.539475+0000 2024-08-18T16:19:22.595 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.593+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:entry[obj00000000000000000000]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.93.6 op=0 op_state=1 2024-08-18T16:19:22.595 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.593+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:entry[obj00000000000000000000]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000000[0] 2024-08-18T16:19:22.595 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.593+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: listing bilog for incremental sync; position=00000000002.93.6 2024-08-18T16:19:22.595 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.593+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:entry[obj00000000000000000025]: success 2024-08-18T16:19:22.595 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.593+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5:2 marker=00000000001.144.6 timestamp=2024-08-18T16:19:16.429591+0000 2024-08-18T16:19:22.596 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.596+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:entry[obj00000000000000000000]: success 2024-08-18T16:19:22.596 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.596+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14:2 marker=00000000002.93.6 timestamp=2024-08-18T16:19:16.539475+0000 2024-08-18T16:19:22.599 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.599+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:entry[obj00000000000000000014]: success 2024-08-18T16:19:22.600 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.599+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11:2 marker=00000000001.81.6 timestamp=2024-08-18T16:19:16.298314+0000 2024-08-18T16:19:22.600 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.600+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:entry[obj00000000000000000012]: success 2024-08-18T16:19:22.601 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.600+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12:2 marker=00000000002.111.6 timestamp=2024-08-18T16:19:16.351846+0000 2024-08-18T16:19:22.603 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.602+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:entry[obj00000000000000000002]: success 2024-08-18T16:19:22.603 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.602+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13:2 marker=00000000002.136.6 timestamp=2024-08-18T16:19:16.343786+0000 2024-08-18T16:19:22.603 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.603+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:entry[obj00000000000000000006]: success 2024-08-18T16:19:22.603 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.603+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15:2 marker=00000000002.128.6 timestamp=2024-08-18T16:19:16.489682+0000 2024-08-18T16:19:22.637 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.636+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:entry[obj00000000000000000003]: success 2024-08-18T16:19:22.637 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.636+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9:2 marker=00000000002.61.6 timestamp=2024-08-18T16:19:16.604379+0000 2024-08-18T16:19:22.637 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.637+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:entry[obj00000000000000000009]: success 2024-08-18T16:19:22.637 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.637+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10:2 marker=00000000001.95.6 timestamp=2024-08-18T16:19:16.283741+0000 2024-08-18T16:19:22.639 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.639+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:entry[obj00000000000000000011]: success 2024-08-18T16:19:22.640 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.639+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16:2 marker=00000000002.95.6 timestamp=2024-08-18T16:19:16.480839+0000 2024-08-18T16:19:22.647 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.646+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.647 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.646+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.647 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.646+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.647 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.646+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.647 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.646+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: sync status for source bucket shard: 0 2024-08-18T16:19:22.647 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.646+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.647 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.646+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: sync status for source bucket shard: 0 2024-08-18T16:19:22.647 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.646+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: sync status for source bucket shard: 0 2024-08-18T16:19:22.648 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.646+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.648 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.646+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.648 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.647+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.648 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.647+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: sync status for source bucket shard: 0 2024-08-18T16:19:22.648 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.647+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.650 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.647+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.650 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.647+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.650 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.647+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.650 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.647+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: sync status for source bucket shard: 0 2024-08-18T16:19:22.650 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.647+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.650 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.647+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.650 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.648+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.650 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.648+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: sync status for source bucket shard: 0 2024-08-18T16:19:22.651 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.648+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: sync status for source bucket shard: 0 2024-08-18T16:19:22.651 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.648+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.651 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.648+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.651 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.648+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.651 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.648+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.651 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.648+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: sync status for source bucket shard: 0 2024-08-18T16:19:22.651 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.649+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: sync status for source bucket shard: 0 2024-08-18T16:19:22.651 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.649+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: sync status for source bucket shard: 0 2024-08-18T16:19:22.651 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.649+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.651 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.649+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.651 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.649+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.651 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.649+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.651 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.649+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: sync status for source bucket shard: 0 2024-08-18T16:19:22.652 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.649+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.652 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.649+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.652 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.649+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.653 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.650+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]: sync status for source bucket shard: 0 2024-08-18T16:19:22.653 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.650+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.653 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.650+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.654 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.650+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.654 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.650+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.654 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.650+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]: sync status for source bucket shard: 0 2024-08-18T16:19:22.654 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.650+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.654 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.650+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: parsed entry: id=00000000002.173.6 iter->object=obj00000000000000000032 iter->instance= name=obj00000000000000000032 instance= ns= 2024-08-18T16:19:22.654 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.651+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6/obj00000000000000000032 2024-08-18T16:19:22.654 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.651+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: entry->timestamp=2024-08-18T16:19:17.116603+0000 2024-08-18T16:19:22.654 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.651+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:entry[obj00000000000000000032]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.173.6 op=0 op_state=1 2024-08-18T16:19:22.654 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.651+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:entry[obj00000000000000000032]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000032[0] 2024-08-18T16:19:22.654 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.651+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: listing bilog for incremental sync; position=00000000002.173.6 2024-08-18T16:19:22.654 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.651+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: sync status for source bucket shard: 0 2024-08-18T16:19:22.654 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.651+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.655 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.651+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: sync status for source bucket shard: 0 2024-08-18T16:19:22.655 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.651+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: sync status for source bucket shard: 0 2024-08-18T16:19:22.655 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.651+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.655 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.651+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.655 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.651+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.655 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.651+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.655 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.651+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.655 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.651+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.655 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.651+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: sync status for source bucket shard: 0 2024-08-18T16:19:22.655 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.651+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: sync status for source bucket shard: 0 2024-08-18T16:19:22.655 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.652+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.655 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.652+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.656 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.652+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: parsed entry: id=00000000001.115.6 iter->object=obj00000000000000000039 iter->instance= name=obj00000000000000000039 instance= ns= 2024-08-18T16:19:22.656 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.652+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8/obj00000000000000000039 2024-08-18T16:19:22.656 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.652+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: entry->timestamp=2024-08-18T16:19:16.975932+0000 2024-08-18T16:19:22.656 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.652+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:entry[obj00000000000000000039]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.115.6 op=0 op_state=1 2024-08-18T16:19:22.656 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.652+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.656 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.652+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: parsed entry: id=00000000002.61.6 iter->object=obj00000000000000000007 iter->instance= name=obj00000000000000000007 instance= ns= 2024-08-18T16:19:22.656 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.652+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9/obj00000000000000000007 2024-08-18T16:19:22.656 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.652+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: entry->timestamp=2024-08-18T16:19:17.365163+0000 2024-08-18T16:19:22.656 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.652+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:entry[obj00000000000000000007]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.61.6 op=0 op_state=1 2024-08-18T16:19:22.656 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.652+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:entry[obj00000000000000000039]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000039[0] 2024-08-18T16:19:22.656 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.652+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: listing bilog for incremental sync; position=00000000001.115.6 2024-08-18T16:19:22.656 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.652+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:entry[obj00000000000000000007]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000007[0] 2024-08-18T16:19:22.657 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.652+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: listing bilog for incremental sync; position=00000000002.61.6 2024-08-18T16:19:22.658 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.653+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]: parsed entry: id=00000000001.85.6 iter->object=obj00000000000000000040 iter->instance= name=obj00000000000000000040 instance= ns= 2024-08-18T16:19:22.658 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.653+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11/obj00000000000000000040 2024-08-18T16:19:22.659 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.653+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]: entry->timestamp=2024-08-18T16:19:17.233102+0000 2024-08-18T16:19:22.659 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.653+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:entry[obj00000000000000000040]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.85.6 op=0 op_state=1 2024-08-18T16:19:22.659 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.653+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:entry[obj00000000000000000040]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000040[0] 2024-08-18T16:19:22.659 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.653+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]: listing bilog for incremental sync; position=00000000001.85.6 2024-08-18T16:19:22.659 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.653+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]: sync status for source bucket shard: 0 2024-08-18T16:19:22.659 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.653+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: sync status for source bucket shard: 0 2024-08-18T16:19:22.659 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.653+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]: sync status for source bucket shard: 0 2024-08-18T16:19:22.659 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.653+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.659 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.653+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]: parsed entry: id=00000000001.94.6 iter->object=obj00000000000000000045 iter->instance= name=obj00000000000000000045 instance= ns= 2024-08-18T16:19:22.659 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.653+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13/obj00000000000000000045 2024-08-18T16:19:22.659 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.653+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]: entry->timestamp=2024-08-18T16:19:17.173028+0000 2024-08-18T16:19:22.659 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.653+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:entry[obj00000000000000000045]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.94.6 op=0 op_state=1 2024-08-18T16:19:22.660 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.653+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.660 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.653+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.660 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.653+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:entry[obj00000000000000000045]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000045[0] 2024-08-18T16:19:22.660 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.654+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]: listing bilog for incremental sync; position=00000000001.94.6 2024-08-18T16:19:22.660 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.654+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]: sync status for source bucket shard: 0 2024-08-18T16:19:22.660 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.654+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]: sync status for source bucket shard: 0 2024-08-18T16:19:22.660 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.654+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.660 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.654+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.660 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.654+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:entry[obj00000000000000000032]: success 2024-08-18T16:19:22.660 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.654+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6:3 marker=00000000002.173.6 timestamp=2024-08-18T16:19:17.116603+0000 2024-08-18T16:19:22.660 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.655+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:entry[obj00000000000000000007]: success 2024-08-18T16:19:22.660 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.655+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9:3 marker=00000000002.61.6 timestamp=2024-08-18T16:19:17.365163+0000 2024-08-18T16:19:22.661 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.658+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]: parsed entry: id=00000000002.80.6 iter->object=obj00000000000000000008 iter->instance= name=obj00000000000000000008 instance= ns= 2024-08-18T16:19:22.661 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.658+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22/obj00000000000000000008 2024-08-18T16:19:22.661 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.658+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]: entry->timestamp=2024-08-18T16:19:17.240648+0000 2024-08-18T16:19:22.661 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.658+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:entry[obj00000000000000000008]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.80.6 op=0 op_state=1 2024-08-18T16:19:22.661 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.658+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]: parsed entry: id=00000000001.82.6 iter->object=obj00000000000000000055 iter->instance= name=obj00000000000000000055 instance= ns= 2024-08-18T16:19:22.661 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.658+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21/obj00000000000000000055 2024-08-18T16:19:22.661 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.658+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]: entry->timestamp=2024-08-18T16:19:17.295995+0000 2024-08-18T16:19:22.661 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.658+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:entry[obj00000000000000000055]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.82.6 op=0 op_state=1 2024-08-18T16:19:22.661 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.658+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: parsed entry: id=00000000002.101.6 iter->object=obj00000000000000000000 iter->instance= name=obj00000000000000000000 instance= ns= 2024-08-18T16:19:22.661 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.658+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18/obj00000000000000000000 2024-08-18T16:19:22.661 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.658+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: entry->timestamp=2024-08-18T16:19:17.048772+0000 2024-08-18T16:19:22.661 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.658+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:entry[obj00000000000000000000]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.101.6 op=0 op_state=1 2024-08-18T16:19:22.662 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.658+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:entry[obj00000000000000000008]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000008[0] 2024-08-18T16:19:22.662 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.658+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]: listing bilog for incremental sync; position=00000000002.80.6 2024-08-18T16:19:22.662 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.658+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:entry[obj00000000000000000055]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000055[0] 2024-08-18T16:19:22.662 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.658+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]: listing bilog for incremental sync; position=00000000001.82.6 2024-08-18T16:19:22.662 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.658+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]: parsed entry: id=00000000002.80.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2024-08-18T16:19:22.662 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.658+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19/obj00000000000000000021 2024-08-18T16:19:22.662 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.658+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]: entry->timestamp=2024-08-18T16:19:17.248676+0000 2024-08-18T16:19:22.662 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.658+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.80.6 op=0 op_state=1 2024-08-18T16:19:22.662 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.658+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:entry[obj00000000000000000000]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000000[0] 2024-08-18T16:19:22.662 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.658+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: listing bilog for incremental sync; position=00000000002.101.6 2024-08-18T16:19:22.662 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.658+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:entry[obj00000000000000000021]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000021[0] 2024-08-18T16:19:22.662 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.658+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]: listing bilog for incremental sync; position=00000000002.80.6 2024-08-18T16:19:22.663 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.662+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:entry[obj00000000000000000021]: success 2024-08-18T16:19:22.663 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.662+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19:3 marker=00000000002.80.6 timestamp=2024-08-18T16:19:17.248676+0000 2024-08-18T16:19:22.663 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.663+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:entry[obj00000000000000000000]: success 2024-08-18T16:19:22.664 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.663+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18:3 marker=00000000002.101.6 timestamp=2024-08-18T16:19:17.048772+0000 2024-08-18T16:19:22.664 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.663+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:entry[obj00000000000000000039]: success 2024-08-18T16:19:22.664 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.663+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8:3 marker=00000000001.115.6 timestamp=2024-08-18T16:19:16.975932+0000 2024-08-18T16:19:22.664 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.663+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:entry[obj00000000000000000040]: success 2024-08-18T16:19:22.664 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.663+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11:3 marker=00000000001.85.6 timestamp=2024-08-18T16:19:17.233102+0000 2024-08-18T16:19:22.690 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.689+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: parsed entry: id=00000000001.71.6 iter->object=obj00000000000000000058 iter->instance= name=obj00000000000000000058 instance= ns= 2024-08-18T16:19:22.690 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.689+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0/obj00000000000000000058 2024-08-18T16:19:22.690 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.689+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: entry->timestamp=2024-08-18T16:19:17.303274+0000 2024-08-18T16:19:22.690 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.689+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:entry[obj00000000000000000058]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.71.6 op=0 op_state=1 2024-08-18T16:19:22.690 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.689+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:entry[obj00000000000000000058]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000058[0] 2024-08-18T16:19:22.690 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.689+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: listing bilog for incremental sync; position=00000000001.71.6 2024-08-18T16:19:22.690 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.689+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: parsed entry: id=00000000002.90.6 iter->object=obj00000000000000000026 iter->instance= name=obj00000000000000000026 instance= ns= 2024-08-18T16:19:22.690 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.689+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1/obj00000000000000000026 2024-08-18T16:19:22.690 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.689+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: entry->timestamp=2024-08-18T16:19:16.991700+0000 2024-08-18T16:19:22.690 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.689+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:entry[obj00000000000000000026]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.90.6 op=0 op_state=1 2024-08-18T16:19:22.690 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.689+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: parsed entry: id=00000000001.92.6 iter->object=obj00000000000000000046 iter->instance= name=obj00000000000000000046 instance= ns= 2024-08-18T16:19:22.691 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.689+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3/obj00000000000000000046 2024-08-18T16:19:22.691 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.689+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: entry->timestamp=2024-08-18T16:19:17.181145+0000 2024-08-18T16:19:22.691 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.689+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:entry[obj00000000000000000046]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.92.6 op=0 op_state=1 2024-08-18T16:19:22.691 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.689+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:entry[obj00000000000000000026]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000026[0] 2024-08-18T16:19:22.691 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.689+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: listing bilog for incremental sync; position=00000000002.90.6 2024-08-18T16:19:22.691 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.689+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:entry[obj00000000000000000046]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000046[0] 2024-08-18T16:19:22.691 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.689+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: listing bilog for incremental sync; position=00000000001.92.6 2024-08-18T16:19:22.692 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.690+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: parsed entry: id=00000000002.70.6 iter->object=obj00000000000000000011 iter->instance= name=obj00000000000000000011 instance= ns= 2024-08-18T16:19:22.692 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.690+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2/obj00000000000000000011 2024-08-18T16:19:22.692 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.690+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: entry->timestamp=2024-08-18T16:19:17.309985+0000 2024-08-18T16:19:22.692 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.690+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:entry[obj00000000000000000011]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.70.6 op=0 op_state=1 2024-08-18T16:19:22.692 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.690+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:entry[obj00000000000000000011]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000011[0] 2024-08-18T16:19:22.692 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.690+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: listing bilog for incremental sync; position=00000000002.70.6 2024-08-18T16:19:22.692 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.690+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: parsed entry: id=00000000002.68.6 iter->object=obj00000000000000000014 iter->instance= name=obj00000000000000000014 instance= ns= 2024-08-18T16:19:22.692 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.690+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4/obj00000000000000000014 2024-08-18T16:19:22.692 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.690+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: entry->timestamp=2024-08-18T16:19:17.108165+0000 2024-08-18T16:19:22.693 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.690+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:entry[obj00000000000000000014]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.68.6 op=0 op_state=1 2024-08-18T16:19:22.693 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.690+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:entry[obj00000000000000000014]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000014[0] 2024-08-18T16:19:22.693 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.690+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: listing bilog for incremental sync; position=00000000002.68.6 2024-08-18T16:19:22.693 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.691+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: parsed entry: id=00000000001.71.6 iter->object=obj00000000000000000050 iter->instance= name=obj00000000000000000050 instance= ns= 2024-08-18T16:19:22.693 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.691+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5/obj00000000000000000050 2024-08-18T16:19:22.693 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.691+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: entry->timestamp=2024-08-18T16:19:17.125461+0000 2024-08-18T16:19:22.693 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.691+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:entry[obj00000000000000000050]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.71.6 op=0 op_state=1 2024-08-18T16:19:22.693 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.691+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:entry[obj00000000000000000050]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000050[0] 2024-08-18T16:19:22.693 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.691+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: listing bilog for incremental sync; position=00000000001.71.6 2024-08-18T16:19:22.693 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.691+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: parsed entry: id=00000000002.175.6 iter->object=obj00000000000000000006 iter->instance= name=obj00000000000000000006 instance= ns= 2024-08-18T16:19:22.693 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.691+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10/obj00000000000000000006 2024-08-18T16:19:22.693 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.691+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: entry->timestamp=2024-08-18T16:19:17.356244+0000 2024-08-18T16:19:22.693 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.691+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:entry[obj00000000000000000006]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.175.6 op=0 op_state=1 2024-08-18T16:19:22.694 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.691+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:entry[obj00000000000000000006]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000006[0] 2024-08-18T16:19:22.694 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.691+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: listing bilog for incremental sync; position=00000000002.175.6 2024-08-18T16:19:22.696 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.692+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:entry[obj00000000000000000026]: success 2024-08-18T16:19:22.696 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.692+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1:3 marker=00000000002.90.6 timestamp=2024-08-18T16:19:16.991700+0000 2024-08-18T16:19:22.696 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.693+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:entry[obj00000000000000000011]: success 2024-08-18T16:19:22.696 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.693+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2:3 marker=00000000002.70.6 timestamp=2024-08-18T16:19:17.309985+0000 2024-08-18T16:19:22.696 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.693+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: parsed entry: id=00000000002.76.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2024-08-18T16:19:22.696 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.693+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17/obj00000000000000000001 2024-08-18T16:19:22.696 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.693+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: entry->timestamp=2024-08-18T16:19:17.060727+0000 2024-08-18T16:19:22.696 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.693+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.76.6 op=0 op_state=1 2024-08-18T16:19:22.696 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.693+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:entry[obj00000000000000000001]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000001[0] 2024-08-18T16:19:22.697 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.693+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: listing bilog for incremental sync; position=00000000002.76.6 2024-08-18T16:19:22.697 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.694+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: parsed entry: id=00000000002.141.6 iter->object=obj00000000000000000009 iter->instance= name=obj00000000000000000009 instance= ns= 2024-08-18T16:19:22.697 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.694+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12/obj00000000000000000009 2024-08-18T16:19:22.697 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.694+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: entry->timestamp=2024-08-18T16:19:16.982932+0000 2024-08-18T16:19:22.697 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.694+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:entry[obj00000000000000000009]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.141.6 op=0 op_state=1 2024-08-18T16:19:22.697 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.694+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:entry[obj00000000000000000009]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000009[0] 2024-08-18T16:19:22.697 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.694+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: listing bilog for incremental sync; position=00000000002.141.6 2024-08-18T16:19:22.697 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.694+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: parsed entry: id=00000000002.98.6 iter->object=obj00000000000000000012 iter->instance= name=obj00000000000000000012 instance= ns= 2024-08-18T16:19:22.697 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.694+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15/obj00000000000000000012 2024-08-18T16:19:22.697 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.694+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: entry->timestamp=2024-08-18T16:19:16.929534+0000 2024-08-18T16:19:22.697 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.694+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:entry[obj00000000000000000012]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.98.6 op=0 op_state=1 2024-08-18T16:19:22.697 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.694+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:entry[obj00000000000000000014]: success 2024-08-18T16:19:22.698 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.694+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4:3 marker=00000000002.68.6 timestamp=2024-08-18T16:19:17.108165+0000 2024-08-18T16:19:22.698 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.694+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:entry[obj00000000000000000012]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000012[0] 2024-08-18T16:19:22.698 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.694+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: listing bilog for incremental sync; position=00000000002.98.6 2024-08-18T16:19:22.698 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.694+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:entry[obj00000000000000000006]: success 2024-08-18T16:19:22.698 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.694+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10:3 marker=00000000002.175.6 timestamp=2024-08-18T16:19:17.356244+0000 2024-08-18T16:19:22.699 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.697+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: parsed entry: id=00000000002.69.6 iter->object=obj00000000000000000033 iter->instance= name=obj00000000000000000033 instance= ns= 2024-08-18T16:19:22.699 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.697+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16/obj00000000000000000033 2024-08-18T16:19:22.699 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.697+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: entry->timestamp=2024-08-18T16:19:16.920927+0000 2024-08-18T16:19:22.699 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.697+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:entry[obj00000000000000000033]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.69.6 op=0 op_state=1 2024-08-18T16:19:22.699 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.697+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:entry[obj00000000000000000033]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000033[0] 2024-08-18T16:19:22.699 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.697+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: listing bilog for incremental sync; position=00000000002.69.6 2024-08-18T16:19:22.699 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.697+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:entry[obj00000000000000000001]: success 2024-08-18T16:19:22.699 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.697+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17:3 marker=00000000002.76.6 timestamp=2024-08-18T16:19:17.060727+0000 2024-08-18T16:19:22.699 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.697+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:entry[obj00000000000000000012]: success 2024-08-18T16:19:22.699 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.697+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15:3 marker=00000000002.98.6 timestamp=2024-08-18T16:19:16.929534+0000 2024-08-18T16:19:22.700 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.697+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: parsed entry: id=00000000002.92.6 iter->object=obj00000000000000000013 iter->instance= name=obj00000000000000000013 instance= ns= 2024-08-18T16:19:22.700 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.697+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14/obj00000000000000000013 2024-08-18T16:19:22.700 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.697+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: entry->timestamp=2024-08-18T16:19:17.040343+0000 2024-08-18T16:19:22.700 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.697+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:entry[obj00000000000000000013]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.92.6 op=0 op_state=1 2024-08-18T16:19:22.700 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.697+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:entry[obj00000000000000000013]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000013[0] 2024-08-18T16:19:22.700 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.697+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: listing bilog for incremental sync; position=00000000002.92.6 2024-08-18T16:19:22.700 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.698+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:entry[obj00000000000000000009]: success 2024-08-18T16:19:22.700 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.698+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12:3 marker=00000000002.141.6 timestamp=2024-08-18T16:19:16.982932+0000 2024-08-18T16:19:22.701 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.700+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:entry[obj00000000000000000050]: success 2024-08-18T16:19:22.701 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.700+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5:3 marker=00000000001.71.6 timestamp=2024-08-18T16:19:17.125461+0000 2024-08-18T16:19:22.701 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.700+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:entry[obj00000000000000000013]: success 2024-08-18T16:19:22.701 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.700+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14:3 marker=00000000002.92.6 timestamp=2024-08-18T16:19:17.040343+0000 2024-08-18T16:19:22.701 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.701+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:entry[obj00000000000000000033]: success 2024-08-18T16:19:22.701 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.701+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16:3 marker=00000000002.69.6 timestamp=2024-08-18T16:19:16.920927+0000 2024-08-18T16:19:22.704 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.704+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:entry[obj00000000000000000045]: success 2024-08-18T16:19:22.704 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.704+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13:3 marker=00000000001.94.6 timestamp=2024-08-18T16:19:17.173028+0000 2024-08-18T16:19:22.705 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.705+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:entry[obj00000000000000000008]: success 2024-08-18T16:19:22.705 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.705+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22:3 marker=00000000002.80.6 timestamp=2024-08-18T16:19:17.240648+0000 2024-08-18T16:19:22.707 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.707+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:entry[obj00000000000000000055]: success 2024-08-18T16:19:22.707 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.707+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21:3 marker=00000000001.82.6 timestamp=2024-08-18T16:19:17.295995+0000 2024-08-18T16:19:22.735 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.735+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:entry[obj00000000000000000058]: success 2024-08-18T16:19:22.735 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.735+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0:3 marker=00000000001.71.6 timestamp=2024-08-18T16:19:17.303274+0000 2024-08-18T16:19:22.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.737+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:entry[obj00000000000000000046]: success 2024-08-18T16:19:22.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.737+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3:3 marker=00000000001.92.6 timestamp=2024-08-18T16:19:17.181145+0000 2024-08-18T16:19:22.769 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.768+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.769 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.768+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.769 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.768+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.769 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.768+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: sync status for source bucket shard: 0 2024-08-18T16:19:22.769 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.768+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.769 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.768+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: sync status for source bucket shard: 0 2024-08-18T16:19:22.769 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.768+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.769 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.768+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.770 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.768+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.771 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.769+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: sync status for source bucket shard: 0 2024-08-18T16:19:22.771 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.769+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.771 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.769+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: sync status for source bucket shard: 0 2024-08-18T16:19:22.771 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.769+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.771 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.769+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.771 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.769+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.771 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.769+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.771 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.770+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: sync status for source bucket shard: 0 2024-08-18T16:19:22.771 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.770+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: sync status for source bucket shard: 0 2024-08-18T16:19:22.771 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.770+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: sync status for source bucket shard: 0 2024-08-18T16:19:22.771 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.770+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.772 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.770+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.772 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.770+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.772 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.770+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.772 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.770+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.772 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.770+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.775 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.770+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.775 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.770+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.775 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.771+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: sync status for source bucket shard: 0 2024-08-18T16:19:22.775 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.771+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.775 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.771+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: sync status for source bucket shard: 0 2024-08-18T16:19:22.775 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.771+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: sync status for source bucket shard: 0 2024-08-18T16:19:22.775 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.771+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.776 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.771+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.776 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.771+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: sync status for source bucket shard: 0 2024-08-18T16:19:22.776 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.771+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.776 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.771+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.776 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.771+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]: sync status for source bucket shard: 0 2024-08-18T16:19:22.776 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.771+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.776 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.771+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.776 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.771+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.776 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.772+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: sync status for source bucket shard: 0 2024-08-18T16:19:22.776 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.772+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.776 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.772+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.777 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.772+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.777 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.772+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.777 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.773+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]: sync status for source bucket shard: 0 2024-08-18T16:19:22.777 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.773+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.777 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.773+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: sync status for source bucket shard: 0 2024-08-18T16:19:22.777 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.773+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.777 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.773+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.777 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.773+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.777 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.773+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.777 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.773+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.777 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.773+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.777 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.773+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.777 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.773+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: sync status for source bucket shard: 0 2024-08-18T16:19:22.778 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.773+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.778 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.773+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: sync status for source bucket shard: 0 2024-08-18T16:19:22.778 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.773+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: sync status for source bucket shard: 0 2024-08-18T16:19:22.778 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.774+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.778 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.774+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.778 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.774+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.778 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.774+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.778 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.774+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:28]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.778 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.774+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]: sync status for source bucket shard: 0 2024-08-18T16:19:22.778 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.774+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]: sync status for source bucket shard: 0 2024-08-18T16:19:22.778 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.774+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]: sync status for source bucket shard: 0 2024-08-18T16:19:22.778 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.774+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: sync status for source bucket shard: 0 2024-08-18T16:19:22.779 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.774+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]: sync status for source bucket shard: 0 2024-08-18T16:19:22.779 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.774+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.779 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.774+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.779 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.774+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.779 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.774+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.779 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.774+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.779 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.774+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.779 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.774+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]: sync status for source bucket shard: 0 2024-08-18T16:19:22.779 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.774+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.781 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.775+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25]: sync status for source bucket shard: 0 2024-08-18T16:19:22.781 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.775+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.781 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.775+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:28]: sync status for source bucket shard: 0 2024-08-18T16:19:22.781 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.775+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: parsed entry: id=00000000002.105.6 iter->object=obj00000000000000000020 iter->instance= name=obj00000000000000000020 instance= ns= 2024-08-18T16:19:22.781 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.775+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14/obj00000000000000000020 2024-08-18T16:19:22.781 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.775+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: entry->timestamp=2024-08-18T16:19:17.856631+0000 2024-08-18T16:19:22.781 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.775+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:entry[obj00000000000000000020]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.105.6 op=0 op_state=1 2024-08-18T16:19:22.781 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.775+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:28]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:28]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.781 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.775+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:entry[obj00000000000000000020]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000020[0] 2024-08-18T16:19:22.781 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.775+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: listing bilog for incremental sync; position=00000000002.105.6 2024-08-18T16:19:22.781 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.776+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]: sync status for source bucket shard: 0 2024-08-18T16:19:22.782 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.776+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]: sync status for source bucket shard: 0 2024-08-18T16:19:22.782 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.776+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.782 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.776+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]: sync status for source bucket shard: 0 2024-08-18T16:19:22.782 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.776+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.782 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.776+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.782 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.776+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: parsed entry: id=00000000001.78.6 iter->object=obj00000000000000000037 iter->instance= name=obj00000000000000000037 instance= ns= 2024-08-18T16:19:22.782 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.776+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15/obj00000000000000000037 2024-08-18T16:19:22.782 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.776+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: entry->timestamp=2024-08-18T16:19:17.746196+0000 2024-08-18T16:19:22.782 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.776+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:entry[obj00000000000000000037]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.78.6 op=0 op_state=1 2024-08-18T16:19:22.782 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.776+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: parsed entry: id=00000000002.48.6 iter->object=obj00000000000000000003 iter->instance= name=obj00000000000000000003 instance= ns= 2024-08-18T16:19:22.782 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.776+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18/obj00000000000000000003 2024-08-18T16:19:22.782 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.776+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: entry->timestamp=2024-08-18T16:19:18.167730+0000 2024-08-18T16:19:22.783 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.776+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:entry[obj00000000000000000003]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.48.6 op=0 op_state=1 2024-08-18T16:19:22.783 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.776+0000 7fbf28131880 0 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:entry[obj00000000000000000037]: : injecting data sync error on key=obj00000000000000000037 2024-08-18T16:19:22.783 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.776+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: listing bilog for incremental sync; position=00000000001.78.6 2024-08-18T16:19:22.783 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.776+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:entry[obj00000000000000000003]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000003[0] 2024-08-18T16:19:22.783 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.776+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: listing bilog for incremental sync; position=00000000002.48.6 2024-08-18T16:19:22.783 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.776+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: parsed entry: id=00000000002.153.6 iter->object=obj00000000000000000005 iter->instance= name=obj00000000000000000005 instance= ns= 2024-08-18T16:19:22.783 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.776+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16/obj00000000000000000005 2024-08-18T16:19:22.783 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.776+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: entry->timestamp=2024-08-18T16:19:17.690778+0000 2024-08-18T16:19:22.783 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.776+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:entry[obj00000000000000000005]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.153.6 op=0 op_state=1 2024-08-18T16:19:22.783 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.776+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: parsed entry: id=00000000002.98.6 iter->object=obj00000000000000000012 iter->instance= name=obj00000000000000000012 instance= ns= 2024-08-18T16:19:22.783 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.776+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17/obj00000000000000000012 2024-08-18T16:19:22.783 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.776+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: entry->timestamp=2024-08-18T16:19:18.104377+0000 2024-08-18T16:19:22.784 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.776+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:entry[obj00000000000000000012]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.98.6 op=0 op_state=1 2024-08-18T16:19:22.784 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.776+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:entry[obj00000000000000000037]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:19:22.784 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.776+0000 7fbf28131880 0 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:entry[obj00000000000000000037]: ERROR: failed to sync object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15/obj00000000000000000037 2024-08-18T16:19:22.784 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.777+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:entry[obj00000000000000000005]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000005[0] 2024-08-18T16:19:22.784 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.777+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: listing bilog for incremental sync; position=00000000002.153.6 2024-08-18T16:19:22.784 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.777+0000 7fbf28131880 0 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:entry[obj00000000000000000012]: : injecting data sync error on key=obj00000000000000000012 2024-08-18T16:19:22.784 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.777+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: listing bilog for incremental sync; position=00000000002.98.6 2024-08-18T16:19:22.784 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.777+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:entry[obj00000000000000000012]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:19:22.784 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.777+0000 7fbf28131880 0 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:entry[obj00000000000000000012]: ERROR: failed to sync object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17/obj00000000000000000012 2024-08-18T16:19:22.784 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.779+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]: parsed entry: id=00000000002.82.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2024-08-18T16:19:22.784 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.779+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21/obj00000000000000000001 2024-08-18T16:19:22.785 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.779+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]: entry->timestamp=2024-08-18T16:19:17.928464+0000 2024-08-18T16:19:22.785 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.779+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.82.6 op=0 op_state=1 2024-08-18T16:19:22.785 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.779+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:entry[obj00000000000000000001]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000001[0] 2024-08-18T16:19:22.785 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.779+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]: listing bilog for incremental sync; position=00000000002.82.6 2024-08-18T16:19:22.785 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.779+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: parsed entry: id=00000000001.180.6 iter->object=obj00000000000000000027 iter->instance= name=obj00000000000000000027 instance= ns= 2024-08-18T16:19:22.785 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.779+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2/obj00000000000000000027 2024-08-18T16:19:22.785 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.779+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: entry->timestamp=2024-08-18T16:19:18.160384+0000 2024-08-18T16:19:22.785 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.779+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:entry[obj00000000000000000027]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.180.6 op=0 op_state=1 2024-08-18T16:19:22.785 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.779+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]: parsed entry: id=00000000002.52.6 iter->object=obj00000000000000000025 iter->instance= name=obj00000000000000000025 instance= ns= 2024-08-18T16:19:22.785 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.779+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23/obj00000000000000000025 2024-08-18T16:19:22.785 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.779+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]: entry->timestamp=2024-08-18T16:19:17.792369+0000 2024-08-18T16:19:22.785 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.779+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:entry[obj00000000000000000025]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.52.6 op=0 op_state=1 2024-08-18T16:19:22.785 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.779+0000 7fbf28131880 0 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:entry[obj00000000000000000027]: : injecting data sync error on key=obj00000000000000000027 2024-08-18T16:19:22.786 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.779+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: listing bilog for incremental sync; position=00000000001.180.6 2024-08-18T16:19:22.786 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.779+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:entry[obj00000000000000000025]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000025[0] 2024-08-18T16:19:22.786 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.779+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]: listing bilog for incremental sync; position=00000000002.52.6 2024-08-18T16:19:22.786 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.779+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:entry[obj00000000000000000027]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:19:22.786 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.779+0000 7fbf28131880 0 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:entry[obj00000000000000000027]: ERROR: failed to sync object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2/obj00000000000000000027 2024-08-18T16:19:22.786 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.780+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]: parsed entry: id=00000000002.99.6 iter->object=obj00000000000000000030 iter->instance= name=obj00000000000000000030 instance= ns= 2024-08-18T16:19:22.786 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.780+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27/obj00000000000000000030 2024-08-18T16:19:22.786 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.780+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]: entry->timestamp=2024-08-18T16:19:17.681589+0000 2024-08-18T16:19:22.786 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.780+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]:entry[obj00000000000000000030]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.99.6 op=0 op_state=1 2024-08-18T16:19:22.786 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.780+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]:entry[obj00000000000000000030]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000030[0] 2024-08-18T16:19:22.786 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.780+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]: listing bilog for incremental sync; position=00000000002.99.6 2024-08-18T16:19:22.787 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.780+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]: parsed entry: id=00000000002.80.6 iter->object=obj00000000000000000009 iter->instance= name=obj00000000000000000009 instance= ns= 2024-08-18T16:19:22.787 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.780+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22/obj00000000000000000009 2024-08-18T16:19:22.787 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.780+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]: entry->timestamp=2024-08-18T16:19:17.919915+0000 2024-08-18T16:19:22.787 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.780+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:entry[obj00000000000000000009]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.80.6 op=0 op_state=1 2024-08-18T16:19:22.787 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.780+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:entry[obj00000000000000000009]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000009[0] 2024-08-18T16:19:22.787 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.780+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]: listing bilog for incremental sync; position=00000000002.80.6 2024-08-18T16:19:22.788 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.781+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: a sync operation returned error 2024-08-18T16:19:22.789 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.781+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: backing out with sync_status=-5 2024-08-18T16:19:22.789 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.781+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: incremental sync on bucket failed, retcode=-5 2024-08-18T16:19:22.789 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.781+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: ERROR: incremental sync failed. error: -5 2024-08-18T16:19:22.789 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.781+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: a sync operation returned error 2024-08-18T16:19:22.789 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.781+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: backing out with sync_status=-5 2024-08-18T16:19:22.789 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.781+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: incremental sync on bucket failed, retcode=-5 2024-08-18T16:19:22.789 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.782+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: ERROR: incremental sync failed. error: -5 2024-08-18T16:19:22.789 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.783+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:entry[obj00000000000000000003]: success 2024-08-18T16:19:22.789 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.783+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18:4 marker=00000000002.48.6 timestamp=2024-08-18T16:19:18.167730+0000 2024-08-18T16:19:22.789 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.784+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:entry[obj00000000000000000025]: success 2024-08-18T16:19:22.789 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.784+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23:4 marker=00000000002.52.6 timestamp=2024-08-18T16:19:17.792369+0000 2024-08-18T16:19:22.790 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.784+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:entry[obj00000000000000000020]: success 2024-08-18T16:19:22.790 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.784+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14:4 marker=00000000002.105.6 timestamp=2024-08-18T16:19:17.856631+0000 2024-08-18T16:19:22.790 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.784+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:entry[obj00000000000000000009]: success 2024-08-18T16:19:22.790 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.784+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22:4 marker=00000000002.80.6 timestamp=2024-08-18T16:19:17.919915+0000 2024-08-18T16:19:22.790 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.785+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:entry[obj00000000000000000001]: success 2024-08-18T16:19:22.790 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.785+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21:4 marker=00000000002.82.6 timestamp=2024-08-18T16:19:17.928464+0000 2024-08-18T16:19:22.790 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.786+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]:entry[obj00000000000000000030]: success 2024-08-18T16:19:22.790 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.786+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27:4 marker=00000000002.99.6 timestamp=2024-08-18T16:19:17.681589+0000 2024-08-18T16:19:22.790 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.787+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.790 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.787+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.790 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.787+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: sync status for source bucket shard: 0 2024-08-18T16:19:22.791 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.787+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.791 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.787+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: sync status for source bucket shard: 0 2024-08-18T16:19:22.791 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.787+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.791 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.789+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:entry[obj00000000000000000005]: success 2024-08-18T16:19:22.791 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.789+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16:4 marker=00000000002.153.6 timestamp=2024-08-18T16:19:17.690778+0000 2024-08-18T16:19:22.812 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.811+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: parsed entry: id=00000000001.54.6 iter->object=obj00000000000000000059 iter->instance= name=obj00000000000000000059 instance= ns= 2024-08-18T16:19:22.812 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.811+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1/obj00000000000000000059 2024-08-18T16:19:22.812 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.811+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: entry->timestamp=2024-08-18T16:19:18.040381+0000 2024-08-18T16:19:22.812 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.811+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:entry[obj00000000000000000059]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.54.6 op=0 op_state=1 2024-08-18T16:19:22.812 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.811+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:entry[obj00000000000000000059]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000059[0] 2024-08-18T16:19:22.812 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.811+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: listing bilog for incremental sync; position=00000000001.54.6 2024-08-18T16:19:22.812 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.811+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: parsed entry: id=00000000002.70.6 iter->object=obj00000000000000000004 iter->instance= name=obj00000000000000000004 instance= ns= 2024-08-18T16:19:22.812 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.811+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3/obj00000000000000000004 2024-08-18T16:19:22.812 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.811+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: entry->timestamp=2024-08-18T16:19:17.671427+0000 2024-08-18T16:19:22.813 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.811+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:entry[obj00000000000000000004]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.70.6 op=0 op_state=1 2024-08-18T16:19:22.813 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.811+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: parsed entry: id=00000000002.122.6 iter->object=obj00000000000000000006 iter->instance= name=obj00000000000000000006 instance= ns= 2024-08-18T16:19:22.813 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.811+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0/obj00000000000000000006 2024-08-18T16:19:22.813 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.811+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: entry->timestamp=2024-08-18T16:19:17.976117+0000 2024-08-18T16:19:22.813 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.811+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:entry[obj00000000000000000006]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.122.6 op=0 op_state=1 2024-08-18T16:19:22.813 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.811+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: parsed entry: id=00000000002.88.6 iter->object=obj00000000000000000011 iter->instance= name=obj00000000000000000011 instance= ns= 2024-08-18T16:19:22.813 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.811+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4/obj00000000000000000011 2024-08-18T16:19:22.813 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.811+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: entry->timestamp=2024-08-18T16:19:18.048298+0000 2024-08-18T16:19:22.813 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.811+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:entry[obj00000000000000000011]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.88.6 op=0 op_state=1 2024-08-18T16:19:22.813 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.811+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:entry[obj00000000000000000004]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000004[0] 2024-08-18T16:19:22.813 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.811+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: listing bilog for incremental sync; position=00000000002.70.6 2024-08-18T16:19:22.813 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.811+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:entry[obj00000000000000000006]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000006[0] 2024-08-18T16:19:22.813 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.811+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: listing bilog for incremental sync; position=00000000002.122.6 2024-08-18T16:19:22.814 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.811+0000 7fbf28131880 0 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:entry[obj00000000000000000011]: : injecting data sync error on key=obj00000000000000000011 2024-08-18T16:19:22.814 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.811+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: listing bilog for incremental sync; position=00000000002.88.6 2024-08-18T16:19:22.814 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.811+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:entry[obj00000000000000000011]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:19:22.814 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.811+0000 7fbf28131880 0 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:entry[obj00000000000000000011]: ERROR: failed to sync object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4/obj00000000000000000011 2024-08-18T16:19:22.814 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.812+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: parsed entry: id=00000000001.155.6 iter->object=obj00000000000000000049 iter->instance= name=obj00000000000000000049 instance= ns= 2024-08-18T16:19:22.814 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.812+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7/obj00000000000000000049 2024-08-18T16:19:22.815 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.812+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: entry->timestamp=2024-08-18T16:19:17.984613+0000 2024-08-18T16:19:22.815 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.812+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:entry[obj00000000000000000049]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.155.6 op=0 op_state=1 2024-08-18T16:19:22.815 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.812+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:entry[obj00000000000000000049]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000049[0] 2024-08-18T16:19:22.815 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.812+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: listing bilog for incremental sync; position=00000000001.155.6 2024-08-18T16:19:22.815 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.813+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: parsed entry: id=00000000002.103.6 iter->object=obj00000000000000000002 iter->instance= name=obj00000000000000000002 instance= ns= 2024-08-18T16:19:22.815 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.813+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5/obj00000000000000000002 2024-08-18T16:19:22.815 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.813+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: entry->timestamp=2024-08-18T16:19:18.224388+0000 2024-08-18T16:19:22.815 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.813+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:entry[obj00000000000000000002]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.103.6 op=0 op_state=1 2024-08-18T16:19:22.815 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.813+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:entry[obj00000000000000000002]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000002[0] 2024-08-18T16:19:22.815 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.813+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: listing bilog for incremental sync; position=00000000002.103.6 2024-08-18T16:19:22.815 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.813+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: parsed entry: id=00000000002.75.6 iter->object=obj00000000000000000019 iter->instance= name=obj00000000000000000019 instance= ns= 2024-08-18T16:19:22.815 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.813+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6/obj00000000000000000019 2024-08-18T16:19:22.816 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.813+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: entry->timestamp=2024-08-18T16:19:18.233851+0000 2024-08-18T16:19:22.816 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.813+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:entry[obj00000000000000000019]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.75.6 op=0 op_state=1 2024-08-18T16:19:22.816 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.813+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: parsed entry: id=00000000002.66.6 iter->object=obj00000000000000000022 iter->instance= name=obj00000000000000000022 instance= ns= 2024-08-18T16:19:22.816 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.813+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12/obj00000000000000000022 2024-08-18T16:19:22.816 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.813+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: entry->timestamp=2024-08-18T16:19:17.801074+0000 2024-08-18T16:19:22.816 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.813+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:entry[obj00000000000000000022]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.66.6 op=0 op_state=1 2024-08-18T16:19:22.816 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.813+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: parsed entry: id=00000000002.139.6 iter->object=obj00000000000000000015 iter->instance= name=obj00000000000000000015 instance= ns= 2024-08-18T16:19:22.816 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.813+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10/obj00000000000000000015 2024-08-18T16:19:22.816 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.813+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: entry->timestamp=2024-08-18T16:19:18.057904+0000 2024-08-18T16:19:22.816 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.813+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:entry[obj00000000000000000015]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.139.6 op=0 op_state=1 2024-08-18T16:19:22.816 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.813+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:entry[obj00000000000000000019]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000019[0] 2024-08-18T16:19:22.816 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.813+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: listing bilog for incremental sync; position=00000000002.75.6 2024-08-18T16:19:22.817 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.813+0000 7fbf28131880 0 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:entry[obj00000000000000000022]: : injecting data sync error on key=obj00000000000000000022 2024-08-18T16:19:22.817 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.813+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: listing bilog for incremental sync; position=00000000002.66.6 2024-08-18T16:19:22.817 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.813+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:entry[obj00000000000000000015]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000015[0] 2024-08-18T16:19:22.817 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.813+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: listing bilog for incremental sync; position=00000000002.139.6 2024-08-18T16:19:22.817 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.813+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:entry[obj00000000000000000022]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:19:22.817 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.813+0000 7fbf28131880 0 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:entry[obj00000000000000000022]: ERROR: failed to sync object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12/obj00000000000000000022 2024-08-18T16:19:22.818 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.814+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: parsed entry: id=00000000002.141.6 iter->object=obj00000000000000000008 iter->instance= name=obj00000000000000000008 instance= ns= 2024-08-18T16:19:22.818 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.814+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9/obj00000000000000000008 2024-08-18T16:19:22.818 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.814+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: entry->timestamp=2024-08-18T16:19:18.113243+0000 2024-08-18T16:19:22.818 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.814+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:entry[obj00000000000000000008]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.141.6 op=0 op_state=1 2024-08-18T16:19:22.818 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.814+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: parsed entry: id=00000000001.80.6 iter->object=obj00000000000000000017 iter->instance= name=obj00000000000000000017 instance= ns= 2024-08-18T16:19:22.818 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.814+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8/obj00000000000000000017 2024-08-18T16:19:22.818 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.814+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: entry->timestamp=2024-08-18T16:19:17.865804+0000 2024-08-18T16:19:22.818 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.814+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:entry[obj00000000000000000017]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.80.6 op=0 op_state=1 2024-08-18T16:19:22.818 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.814+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]: parsed entry: id=00000000001.130.6 iter->object=obj00000000000000000031 iter->instance= name=obj00000000000000000031 instance= ns= 2024-08-18T16:19:22.819 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.814+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11/obj00000000000000000031 2024-08-18T16:19:22.819 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.814+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]: entry->timestamp=2024-08-18T16:19:17.912320+0000 2024-08-18T16:19:22.819 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.814+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:entry[obj00000000000000000031]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.130.6 op=0 op_state=1 2024-08-18T16:19:22.819 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.814+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:entry[obj00000000000000000008]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000008[0] 2024-08-18T16:19:22.819 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.814+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: listing bilog for incremental sync; position=00000000002.141.6 2024-08-18T16:19:22.819 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.814+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:entry[obj00000000000000000017]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000017[0] 2024-08-18T16:19:22.819 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.814+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: listing bilog for incremental sync; position=00000000001.80.6 2024-08-18T16:19:22.819 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.814+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:entry[obj00000000000000000031]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000031[0] 2024-08-18T16:19:22.819 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.814+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]: listing bilog for incremental sync; position=00000000001.130.6 2024-08-18T16:19:22.819 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.814+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:entry[obj00000000000000000006]: success 2024-08-18T16:19:22.819 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.814+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0:4 marker=00000000002.122.6 timestamp=2024-08-18T16:19:17.976117+0000 2024-08-18T16:19:22.819 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.815+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:entry[obj00000000000000000004]: success 2024-08-18T16:19:22.819 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.815+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3:4 marker=00000000002.70.6 timestamp=2024-08-18T16:19:17.671427+0000 2024-08-18T16:19:22.820 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.817+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]: parsed entry: id=00000000001.145.6 iter->object=obj00000000000000000042 iter->instance= name=obj00000000000000000042 instance= ns= 2024-08-18T16:19:22.820 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.817+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19/obj00000000000000000042 2024-08-18T16:19:22.820 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.817+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]: entry->timestamp=2024-08-18T16:19:18.280618+0000 2024-08-18T16:19:22.820 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.817+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:entry[obj00000000000000000042]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.145.6 op=0 op_state=1 2024-08-18T16:19:22.820 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.817+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:entry[obj00000000000000000042]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000042[0] 2024-08-18T16:19:22.820 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.817+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]: listing bilog for incremental sync; position=00000000001.145.6 2024-08-18T16:19:22.820 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.817+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:entry[obj00000000000000000008]: success 2024-08-18T16:19:22.820 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.817+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9:4 marker=00000000002.141.6 timestamp=2024-08-18T16:19:18.113243+0000 2024-08-18T16:19:22.821 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.818+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:entry[obj00000000000000000015]: success 2024-08-18T16:19:22.821 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.818+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10:4 marker=00000000002.139.6 timestamp=2024-08-18T16:19:18.057904+0000 2024-08-18T16:19:22.821 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.819+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:entry[obj00000000000000000019]: success 2024-08-18T16:19:22.821 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.819+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6:4 marker=00000000002.75.6 timestamp=2024-08-18T16:19:18.233851+0000 2024-08-18T16:19:22.822 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.819+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:entry[obj00000000000000000002]: success 2024-08-18T16:19:22.822 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.819+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5:4 marker=00000000002.103.6 timestamp=2024-08-18T16:19:18.224388+0000 2024-08-18T16:19:22.822 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.819+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]: parsed entry: id=00000000002.101.6 iter->object=obj00000000000000000016 iter->instance= name=obj00000000000000000016 instance= ns= 2024-08-18T16:19:22.822 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.819+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24/obj00000000000000000016 2024-08-18T16:19:22.822 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.819+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]: entry->timestamp=2024-08-18T16:19:17.809180+0000 2024-08-18T16:19:22.822 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.819+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]:entry[obj00000000000000000016]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.101.6 op=0 op_state=1 2024-08-18T16:19:22.822 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.820+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]:entry[obj00000000000000000016]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000016[0] 2024-08-18T16:19:22.822 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.820+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]: listing bilog for incremental sync; position=00000000002.101.6 2024-08-18T16:19:22.822 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.820+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:28]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:28]: parsed entry: id=00000000002.76.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2024-08-18T16:19:22.822 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.820+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:28]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:28]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:28/obj00000000000000000021 2024-08-18T16:19:22.822 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.820+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:28]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:28]: entry->timestamp=2024-08-18T16:19:17.992108+0000 2024-08-18T16:19:22.823 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.820+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:28]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:28]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.76.6 op=0 op_state=1 2024-08-18T16:19:22.823 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.820+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]: parsed entry: id=00000000002.82.6 iter->object=obj00000000000000000010 iter->instance= name=obj00000000000000000010 instance= ns= 2024-08-18T16:19:22.823 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.820+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20/obj00000000000000000010 2024-08-18T16:19:22.823 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.820+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]: entry->timestamp=2024-08-18T16:19:18.176758+0000 2024-08-18T16:19:22.823 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.820+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:entry[obj00000000000000000010]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.82.6 op=0 op_state=1 2024-08-18T16:19:22.823 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.820+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]: parsed entry: id=00000000002.100.6 iter->object=obj00000000000000000014 iter->instance= name=obj00000000000000000014 instance= ns= 2024-08-18T16:19:22.823 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.820+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26/obj00000000000000000014 2024-08-18T16:19:22.823 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.820+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]: entry->timestamp=2024-08-18T16:19:17.736743+0000 2024-08-18T16:19:22.823 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.820+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]:entry[obj00000000000000000014]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.100.6 op=0 op_state=1 2024-08-18T16:19:22.823 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.820+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:28]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:28]:entry[obj00000000000000000021]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000021[0] 2024-08-18T16:19:22.823 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.820+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:28]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:28]: listing bilog for incremental sync; position=00000000002.76.6 2024-08-18T16:19:22.823 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.820+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:entry[obj00000000000000000010]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000010[0] 2024-08-18T16:19:22.824 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.820+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]: listing bilog for incremental sync; position=00000000002.82.6 2024-08-18T16:19:22.824 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.820+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]:entry[obj00000000000000000014]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000014[0] 2024-08-18T16:19:22.824 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.820+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]: listing bilog for incremental sync; position=00000000002.100.6 2024-08-18T16:19:22.825 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.822+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:28]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:28]:entry[obj00000000000000000021]: success 2024-08-18T16:19:22.825 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.822+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:28]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:28]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:28:4 marker=00000000002.76.6 timestamp=2024-08-18T16:19:17.992108+0000 2024-08-18T16:19:22.825 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.823+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]:entry[obj00000000000000000016]: success 2024-08-18T16:19:22.825 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.823+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24:4 marker=00000000002.101.6 timestamp=2024-08-18T16:19:17.809180+0000 2024-08-18T16:19:22.825 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.823+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: a sync operation returned error 2024-08-18T16:19:22.825 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.823+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: backing out with sync_status=-5 2024-08-18T16:19:22.825 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.823+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: incremental sync on bucket failed, retcode=-5 2024-08-18T16:19:22.825 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.823+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: ERROR: incremental sync failed. error: -5 2024-08-18T16:19:22.825 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.823+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:entry[obj00000000000000000010]: success 2024-08-18T16:19:22.826 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.823+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20:4 marker=00000000002.82.6 timestamp=2024-08-18T16:19:18.176758+0000 2024-08-18T16:19:22.826 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.824+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]:entry[obj00000000000000000014]: success 2024-08-18T16:19:22.826 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.824+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26:4 marker=00000000002.100.6 timestamp=2024-08-18T16:19:17.736743+0000 2024-08-18T16:19:22.826 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.825+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.827 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.827+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: sync status for source bucket shard: 0 2024-08-18T16:19:22.827 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.827+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.830 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.830+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: parsed entry: id=00000000001.78.6 iter->object=obj00000000000000000037 iter->instance= name=obj00000000000000000037 instance= ns= 2024-08-18T16:19:22.831 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.830+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15/obj00000000000000000037 2024-08-18T16:19:22.831 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.830+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: entry->timestamp=2024-08-18T16:19:17.746196+0000 2024-08-18T16:19:22.831 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.830+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:entry[obj00000000000000000037]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.78.6 op=0 op_state=1 2024-08-18T16:19:22.831 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.830+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: parsed entry: id=00000000002.98.6 iter->object=obj00000000000000000012 iter->instance= name=obj00000000000000000012 instance= ns= 2024-08-18T16:19:22.831 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.830+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17/obj00000000000000000012 2024-08-18T16:19:22.831 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.830+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: entry->timestamp=2024-08-18T16:19:18.104377+0000 2024-08-18T16:19:22.831 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.830+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:entry[obj00000000000000000012]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.98.6 op=0 op_state=1 2024-08-18T16:19:22.831 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.830+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:entry[obj00000000000000000037]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000037[0] 2024-08-18T16:19:22.831 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.830+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: listing bilog for incremental sync; position=00000000001.78.6 2024-08-18T16:19:22.831 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.830+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:entry[obj00000000000000000012]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000012[0] 2024-08-18T16:19:22.831 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.830+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: listing bilog for incremental sync; position=00000000002.98.6 2024-08-18T16:19:22.833 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.832+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:entry[obj00000000000000000012]: success 2024-08-18T16:19:22.833 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.832+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17:4 marker=00000000002.98.6 timestamp=2024-08-18T16:19:18.104377+0000 2024-08-18T16:19:22.856 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.856+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: a sync operation returned error 2024-08-18T16:19:22.856 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.856+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: backing out with sync_status=-5 2024-08-18T16:19:22.856 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.856+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: incremental sync on bucket failed, retcode=-5 2024-08-18T16:19:22.856 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.856+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: ERROR: incremental sync failed. error: -5 2024-08-18T16:19:22.857 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.856+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.857 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.857+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: sync status for source bucket shard: 0 2024-08-18T16:19:22.857 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.857+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.858 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.857+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:entry[obj00000000000000000059]: success 2024-08-18T16:19:22.858 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.857+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1:4 marker=00000000001.54.6 timestamp=2024-08-18T16:19:18.040381+0000 2024-08-18T16:19:22.858 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.858+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: a sync operation returned error 2024-08-18T16:19:22.858 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.858+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: backing out with sync_status=-5 2024-08-18T16:19:22.858 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.858+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: incremental sync on bucket failed, retcode=-5 2024-08-18T16:19:22.859 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.858+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: ERROR: incremental sync failed. error: -5 2024-08-18T16:19:22.860 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.860+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.860 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.860+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: sync status for source bucket shard: 0 2024-08-18T16:19:22.860 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.860+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.862 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.861+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:entry[obj00000000000000000049]: success 2024-08-18T16:19:22.862 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.861+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7:4 marker=00000000001.155.6 timestamp=2024-08-18T16:19:17.984613+0000 2024-08-18T16:19:22.865 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.865+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:entry[obj00000000000000000017]: success 2024-08-18T16:19:22.865 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.865+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8:4 marker=00000000001.80.6 timestamp=2024-08-18T16:19:17.865804+0000 2024-08-18T16:19:22.865 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.865+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:entry[obj00000000000000000031]: success 2024-08-18T16:19:22.865 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.865+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11:4 marker=00000000001.130.6 timestamp=2024-08-18T16:19:17.912320+0000 2024-08-18T16:19:22.869 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.868+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:entry[obj00000000000000000042]: success 2024-08-18T16:19:22.869 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.868+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19:4 marker=00000000001.145.6 timestamp=2024-08-18T16:19:18.280618+0000 2024-08-18T16:19:22.869 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.869+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: parsed entry: id=00000000001.180.6 iter->object=obj00000000000000000027 iter->instance= name=obj00000000000000000027 instance= ns= 2024-08-18T16:19:22.869 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.869+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2/obj00000000000000000027 2024-08-18T16:19:22.869 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.869+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: entry->timestamp=2024-08-18T16:19:18.160384+0000 2024-08-18T16:19:22.869 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.869+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:entry[obj00000000000000000027]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.180.6 op=0 op_state=1 2024-08-18T16:19:22.869 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.869+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:entry[obj00000000000000000027]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000027[0] 2024-08-18T16:19:22.870 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.869+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: listing bilog for incremental sync; position=00000000001.180.6 2024-08-18T16:19:22.876 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.876+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:entry[obj00000000000000000037]: success 2024-08-18T16:19:22.876 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.876+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15:4 marker=00000000001.78.6 timestamp=2024-08-18T16:19:17.746196+0000 2024-08-18T16:19:22.899 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.899+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: parsed entry: id=00000000002.88.6 iter->object=obj00000000000000000011 iter->instance= name=obj00000000000000000011 instance= ns= 2024-08-18T16:19:22.900 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.899+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4/obj00000000000000000011 2024-08-18T16:19:22.900 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.899+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: entry->timestamp=2024-08-18T16:19:18.048298+0000 2024-08-18T16:19:22.900 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.899+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:entry[obj00000000000000000011]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.88.6 op=0 op_state=1 2024-08-18T16:19:22.900 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.899+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:entry[obj00000000000000000011]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000011[0] 2024-08-18T16:19:22.900 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.899+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: listing bilog for incremental sync; position=00000000002.88.6 2024-08-18T16:19:22.901 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.901+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:entry[obj00000000000000000011]: success 2024-08-18T16:19:22.901 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.901+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4:4 marker=00000000002.88.6 timestamp=2024-08-18T16:19:18.048298+0000 2024-08-18T16:19:22.903 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.903+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: parsed entry: id=00000000002.66.6 iter->object=obj00000000000000000022 iter->instance= name=obj00000000000000000022 instance= ns= 2024-08-18T16:19:22.903 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.903+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12/obj00000000000000000022 2024-08-18T16:19:22.903 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.903+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: entry->timestamp=2024-08-18T16:19:17.801074+0000 2024-08-18T16:19:22.903 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.903+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:entry[obj00000000000000000022]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.66.6 op=0 op_state=1 2024-08-18T16:19:22.903 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.903+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:entry[obj00000000000000000022]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000022[0] 2024-08-18T16:19:22.903 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.903+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: listing bilog for incremental sync; position=00000000002.66.6 2024-08-18T16:19:22.914 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.914+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:entry[obj00000000000000000027]: success 2024-08-18T16:19:22.915 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.914+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2:4 marker=00000000001.180.6 timestamp=2024-08-18T16:19:18.160384+0000 2024-08-18T16:19:22.948 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.948+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:entry[obj00000000000000000022]: success 2024-08-18T16:19:22.948 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.948+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12:4 marker=00000000002.66.6 timestamp=2024-08-18T16:19:17.801074+0000 2024-08-18T16:19:22.954 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.953+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.954 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.953+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.954 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.953+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.954 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.953+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.954 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.953+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.954 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.953+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: sync status for source bucket shard: 0 2024-08-18T16:19:22.954 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.953+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.954 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.954+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: sync status for source bucket shard: 0 2024-08-18T16:19:22.955 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.954+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: sync status for source bucket shard: 0 2024-08-18T16:19:22.955 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.954+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.955 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.954+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.955 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.954+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: sync status for source bucket shard: 0 2024-08-18T16:19:22.955 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.954+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.955 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.954+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.955 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.954+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: sync status for source bucket shard: 0 2024-08-18T16:19:22.956 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.954+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.956 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.954+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.956 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.955+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: sync status for source bucket shard: 0 2024-08-18T16:19:22.956 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.955+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: sync status for source bucket shard: 0 2024-08-18T16:19:22.956 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.955+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.956 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.955+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.956 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.955+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.956 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.955+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.956 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.955+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.956 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.955+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.958 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.955+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: sync status for source bucket shard: 0 2024-08-18T16:19:22.958 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.955+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.958 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.956+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: sync status for source bucket shard: 0 2024-08-18T16:19:22.958 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.956+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.958 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.956+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: parsed entry: id=00000000001.90.6 iter->object=obj00000000000000000056 iter->instance= name=obj00000000000000000056 instance= ns= 2024-08-18T16:19:22.958 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.956+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4/obj00000000000000000056 2024-08-18T16:19:22.958 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.956+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: entry->timestamp=2024-08-18T16:19:19.144248+0000 2024-08-18T16:19:22.959 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.956+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:entry[obj00000000000000000056]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.90.6 op=0 op_state=1 2024-08-18T16:19:22.959 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.956+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:entry[obj00000000000000000056]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000056[0] 2024-08-18T16:19:22.959 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.956+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: listing bilog for incremental sync; position=00000000001.90.6 2024-08-18T16:19:22.959 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.956+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: sync status for source bucket shard: 0 2024-08-18T16:19:22.959 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.956+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.959 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.956+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.959 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.956+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.959 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.956+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.959 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.957+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: parsed entry: id=00000000001.82.6 iter->object=obj00000000000000000467 iter->instance= name=obj00000000000000000467 instance= ns= 2024-08-18T16:19:22.959 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.957+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2/obj00000000000000000467 2024-08-18T16:19:22.959 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.957+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: entry->timestamp=2024-08-18T16:19:19.151651+0000 2024-08-18T16:19:22.959 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.957+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:entry[obj00000000000000000467]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.82.6 op=0 op_state=1 2024-08-18T16:19:22.960 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.957+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.960 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.957+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:entry[obj00000000000000000467]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000467[0] 2024-08-18T16:19:22.960 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.957+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: listing bilog for incremental sync; position=00000000001.82.6 2024-08-18T16:19:22.960 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.957+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: parsed entry: id=00000000002.75.6 iter->object=obj00000000000000000005 iter->instance= name=obj00000000000000000005 instance= ns= 2024-08-18T16:19:22.960 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.957+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3/obj00000000000000000005 2024-08-18T16:19:22.960 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.957+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: entry->timestamp=2024-08-18T16:19:18.560050+0000 2024-08-18T16:19:22.960 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.957+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:entry[obj00000000000000000005]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.75.6 op=0 op_state=1 2024-08-18T16:19:22.960 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.957+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:entry[obj00000000000000000005]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000005[0] 2024-08-18T16:19:22.960 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.957+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: listing bilog for incremental sync; position=00000000002.75.6 2024-08-18T16:19:22.960 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.957+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.960 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.957+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: parsed entry: id=00000000002.75.6 iter->object=obj00000000000000000004 iter->instance= name=obj00000000000000000004 instance= ns= 2024-08-18T16:19:22.960 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.957+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5/obj00000000000000000004 2024-08-18T16:19:22.961 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.957+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: entry->timestamp=2024-08-18T16:19:19.204480+0000 2024-08-18T16:19:22.961 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.957+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:entry[obj00000000000000000004]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.75.6 op=0 op_state=1 2024-08-18T16:19:22.961 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.957+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.961 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.957+0000 7fbf28131880 0 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:entry[obj00000000000000000004]: : injecting data sync error on key=obj00000000000000000004 2024-08-18T16:19:22.961 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.957+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: listing bilog for incremental sync; position=00000000002.75.6 2024-08-18T16:19:22.961 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.957+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.961 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.957+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:entry[obj00000000000000000004]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:19:22.961 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.957+0000 7fbf28131880 0 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:entry[obj00000000000000000004]: ERROR: failed to sync object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5/obj00000000000000000004 2024-08-18T16:19:22.961 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.957+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.961 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.958+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]: sync status for source bucket shard: 0 2024-08-18T16:19:22.961 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.958+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: sync status for source bucket shard: 0 2024-08-18T16:19:22.961 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.958+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: parsed entry: id=00000000001.88.6 iter->object=obj00000000000000000317 iter->instance= name=obj00000000000000000317 instance= ns= 2024-08-18T16:19:22.961 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.958+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7/obj00000000000000000317 2024-08-18T16:19:22.962 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.958+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: entry->timestamp=2024-08-18T16:19:18.784378+0000 2024-08-18T16:19:22.962 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.958+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:entry[obj00000000000000000317]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.88.6 op=0 op_state=1 2024-08-18T16:19:22.962 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.958+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.962 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.958+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.962 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.958+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:entry[obj00000000000000000317]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000317[0] 2024-08-18T16:19:22.962 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.958+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: listing bilog for incremental sync; position=00000000001.88.6 2024-08-18T16:19:22.964 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.958+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: sync status for source bucket shard: 0 2024-08-18T16:19:22.964 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.958+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.964 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.960+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.964 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.960+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.964 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.960+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.964 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.960+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: sync status for source bucket shard: 0 2024-08-18T16:19:22.964 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.960+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]: sync status for source bucket shard: 0 2024-08-18T16:19:22.964 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.960+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: sync status for source bucket shard: 0 2024-08-18T16:19:22.965 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.960+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: sync status for source bucket shard: 0 2024-08-18T16:19:22.965 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.960+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.965 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.960+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.965 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.961+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.965 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.961+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.965 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.961+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: parsed entry: id=00000000002.165.6 iter->object=obj00000000000000000018 iter->instance= name=obj00000000000000000018 instance= ns= 2024-08-18T16:19:22.965 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.961+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14/obj00000000000000000018 2024-08-18T16:19:22.965 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.961+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: entry->timestamp=2024-08-18T16:19:19.136367+0000 2024-08-18T16:19:22.965 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.961+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:entry[obj00000000000000000018]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.165.6 op=0 op_state=1 2024-08-18T16:19:22.965 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.961+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.965 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.961+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.965 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.961+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.966 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.961+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.966 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.961+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.966 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.961+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.966 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.961+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:28]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.966 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.961+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.966 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.961+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:30]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.966 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.961+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:entry[obj00000000000000000018]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000018[0] 2024-08-18T16:19:22.966 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.961+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: listing bilog for incremental sync; position=00000000002.165.6 2024-08-18T16:19:22.966 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.962+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]: sync status for source bucket shard: 0 2024-08-18T16:19:22.966 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.962+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]: sync status for source bucket shard: 0 2024-08-18T16:19:22.966 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.962+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: sync status for source bucket shard: 0 2024-08-18T16:19:22.967 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.962+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]: sync status for source bucket shard: 0 2024-08-18T16:19:22.967 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.962+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.967 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.962+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.967 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.962+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.967 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.962+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.967 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.962+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]: sync status for source bucket shard: 0 2024-08-18T16:19:22.967 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.962+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:30]: sync status for source bucket shard: 0 2024-08-18T16:19:22.967 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.962+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:28]: sync status for source bucket shard: 0 2024-08-18T16:19:22.967 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.962+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]: sync status for source bucket shard: 0 2024-08-18T16:19:22.967 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.962+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]: sync status for source bucket shard: 0 2024-08-18T16:19:22.967 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.962+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.967 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.963+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:30]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:30]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.968 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.963+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:28]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:28]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.968 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.963+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.968 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.963+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.968 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.963+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]: sync status for source bucket shard: 0 2024-08-18T16:19:22.968 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.963+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]: sync status for source bucket shard: 0 2024-08-18T16:19:22.968 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.963+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.968 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.963+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.968 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.963+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:entry[obj00000000000000000005]: success 2024-08-18T16:19:22.968 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.963+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3:5 marker=00000000002.75.6 timestamp=2024-08-18T16:19:18.560050+0000 2024-08-18T16:19:22.968 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.963+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]: sync status for source bucket shard: 0 2024-08-18T16:19:22.968 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.963+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.970 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.965+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25]: sync status for source bucket shard: 0 2024-08-18T16:19:22.970 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.965+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.970 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.965+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: sync status for source bucket shard: 0 2024-08-18T16:19:22.970 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.965+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.970 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.965+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: parsed entry: id=00000000002.106.6 iter->object=obj00000000000000000017 iter->instance= name=obj00000000000000000017 instance= ns= 2024-08-18T16:19:22.970 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.965+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16/obj00000000000000000017 2024-08-18T16:19:22.970 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.965+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: entry->timestamp=2024-08-18T16:19:18.792258+0000 2024-08-18T16:19:22.971 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.965+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:entry[obj00000000000000000017]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.106.6 op=0 op_state=1 2024-08-18T16:19:22.971 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.965+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]: parsed entry: id=00000000002.82.6 iter->object=obj00000000000000000003 iter->instance= name=obj00000000000000000003 instance= ns= 2024-08-18T16:19:22.971 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.965+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19/obj00000000000000000003 2024-08-18T16:19:22.971 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.965+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]: entry->timestamp=2024-08-18T16:19:19.080473+0000 2024-08-18T16:19:22.971 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.965+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:entry[obj00000000000000000003]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.82.6 op=0 op_state=1 2024-08-18T16:19:22.971 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.965+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:entry[obj00000000000000000017]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000017[0] 2024-08-18T16:19:22.971 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.965+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: listing bilog for incremental sync; position=00000000002.106.6 2024-08-18T16:19:22.971 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.965+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:entry[obj00000000000000000003]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000003[0] 2024-08-18T16:19:22.971 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.965+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]: listing bilog for incremental sync; position=00000000002.82.6 2024-08-18T16:19:22.971 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.968+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]: parsed entry: id=00000000002.150.6 iter->object=obj00000000000000000002 iter->instance= name=obj00000000000000000002 instance= ns= 2024-08-18T16:19:22.971 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.968+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21/obj00000000000000000002 2024-08-18T16:19:22.971 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.968+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]: entry->timestamp=2024-08-18T16:19:18.921737+0000 2024-08-18T16:19:22.971 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.968+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:entry[obj00000000000000000002]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.150.6 op=0 op_state=1 2024-08-18T16:19:22.972 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.968+0000 7fbf28131880 0 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:entry[obj00000000000000000002]: : injecting data sync error on key=obj00000000000000000002 2024-08-18T16:19:22.972 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.968+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]: listing bilog for incremental sync; position=00000000002.150.6 2024-08-18T16:19:22.972 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.968+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]: parsed entry: id=00000000001.88.6 iter->object=obj00000000000000000029 iter->instance= name=obj00000000000000000029 instance= ns= 2024-08-18T16:19:22.972 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.968+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29/obj00000000000000000029 2024-08-18T16:19:22.972 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.968+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]: entry->timestamp=2024-08-18T16:19:18.977845+0000 2024-08-18T16:19:22.972 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.968+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]:entry[obj00000000000000000029]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.88.6 op=0 op_state=1 2024-08-18T16:19:22.972 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.968+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]: parsed entry: id=00000000001.110.6 iter->object=obj00000000000000000061 iter->instance= name=obj00000000000000000061 instance= ns= 2024-08-18T16:19:22.972 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.968+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27/obj00000000000000000061 2024-08-18T16:19:22.972 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.968+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]: entry->timestamp=2024-08-18T16:19:18.800570+0000 2024-08-18T16:19:22.972 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.968+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]:entry[obj00000000000000000061]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.110.6 op=0 op_state=1 2024-08-18T16:19:22.972 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.968+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]: parsed entry: id=00000000002.112.6 iter->object=obj00000000000000000011 iter->instance= name=obj00000000000000000011 instance= ns= 2024-08-18T16:19:22.972 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.968+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24/obj00000000000000000011 2024-08-18T16:19:22.973 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.968+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]: entry->timestamp=2024-08-18T16:19:19.033586+0000 2024-08-18T16:19:22.973 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.968+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]:entry[obj00000000000000000011]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.112.6 op=0 op_state=1 2024-08-18T16:19:22.973 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.968+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:entry[obj00000000000000000002]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:19:22.973 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.968+0000 7fbf28131880 0 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:entry[obj00000000000000000002]: ERROR: failed to sync object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21/obj00000000000000000002 2024-08-18T16:19:22.973 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.968+0000 7fbf28131880 0 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]:entry[obj00000000000000000029]: : injecting data sync error on key=obj00000000000000000029 2024-08-18T16:19:22.973 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.968+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]: listing bilog for incremental sync; position=00000000001.88.6 2024-08-18T16:19:22.973 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.968+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]:entry[obj00000000000000000061]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000061[0] 2024-08-18T16:19:22.973 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.968+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]: listing bilog for incremental sync; position=00000000001.110.6 2024-08-18T16:19:22.973 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.968+0000 7fbf28131880 0 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]:entry[obj00000000000000000011]: : injecting data sync error on key=obj00000000000000000011 2024-08-18T16:19:22.973 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.968+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]: listing bilog for incremental sync; position=00000000002.112.6 2024-08-18T16:19:22.973 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.968+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]:entry[obj00000000000000000029]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:19:22.973 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.968+0000 7fbf28131880 0 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]:entry[obj00000000000000000029]: ERROR: failed to sync object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29/obj00000000000000000029 2024-08-18T16:19:22.974 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.968+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]:entry[obj00000000000000000011]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:19:22.974 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.968+0000 7fbf28131880 0 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]:entry[obj00000000000000000011]: ERROR: failed to sync object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24/obj00000000000000000011 2024-08-18T16:19:22.974 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.969+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: a sync operation returned error 2024-08-18T16:19:22.974 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.969+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:entry[obj00000000000000000317]: success 2024-08-18T16:19:22.974 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.969+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7:5 marker=00000000001.88.6 timestamp=2024-08-18T16:19:18.784378+0000 2024-08-18T16:19:22.974 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.969+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:entry[obj00000000000000000003]: success 2024-08-18T16:19:22.974 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.969+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19:5 marker=00000000002.82.6 timestamp=2024-08-18T16:19:19.080473+0000 2024-08-18T16:19:22.974 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.969+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:30]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:30]: parsed entry: id=00000000001.114.6 iter->object=obj00000000000000000133 iter->instance= name=obj00000000000000000133 instance= ns= 2024-08-18T16:19:22.974 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.969+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:30]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:30]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:30/obj00000000000000000133 2024-08-18T16:19:22.974 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.969+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:30]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:30]: entry->timestamp=2024-08-18T16:19:19.158889+0000 2024-08-18T16:19:22.974 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.969+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:30]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:30]:entry[obj00000000000000000133]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.114.6 op=0 op_state=1 2024-08-18T16:19:22.975 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.969+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]: parsed entry: id=00000000002.185.6 iter->object=obj00000000000000000012 iter->instance= name=obj00000000000000000012 instance= ns= 2024-08-18T16:19:22.975 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.969+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22/obj00000000000000000012 2024-08-18T16:19:22.975 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.969+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]: entry->timestamp=2024-08-18T16:19:19.024938+0000 2024-08-18T16:19:22.975 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.969+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:entry[obj00000000000000000012]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.185.6 op=0 op_state=1 2024-08-18T16:19:22.975 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.969+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: backing out with sync_status=-5 2024-08-18T16:19:22.975 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.970+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:30]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:30]:entry[obj00000000000000000133]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000133[0] 2024-08-18T16:19:22.975 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.970+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:30]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:30]: listing bilog for incremental sync; position=00000000001.114.6 2024-08-18T16:19:22.975 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.970+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:entry[obj00000000000000000012]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000012[0] 2024-08-18T16:19:22.975 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.970+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]: listing bilog for incremental sync; position=00000000002.185.6 2024-08-18T16:19:22.975 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.970+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:entry[obj00000000000000000017]: success 2024-08-18T16:19:22.975 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.970+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16:5 marker=00000000002.106.6 timestamp=2024-08-18T16:19:18.792258+0000 2024-08-18T16:19:22.975 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.970+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: incremental sync on bucket failed, retcode=-5 2024-08-18T16:19:22.976 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.970+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: ERROR: incremental sync failed. error: -5 2024-08-18T16:19:22.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.971+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: parsed entry: id=00000000001.82.6 iter->object=obj00000000000000000162 iter->instance= name=obj00000000000000000162 instance= ns= 2024-08-18T16:19:22.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.971+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17/obj00000000000000000162 2024-08-18T16:19:22.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.971+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: entry->timestamp=2024-08-18T16:19:18.617486+0000 2024-08-18T16:19:22.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.971+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:entry[obj00000000000000000162]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.82.6 op=0 op_state=1 2024-08-18T16:19:22.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.971+0000 7fbf28131880 0 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:entry[obj00000000000000000162]: : injecting data sync error on key=obj00000000000000000162 2024-08-18T16:19:22.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.971+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: listing bilog for incremental sync; position=00000000001.82.6 2024-08-18T16:19:22.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.971+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:entry[obj00000000000000000162]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:19:22.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.971+0000 7fbf28131880 0 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:entry[obj00000000000000000162]: ERROR: failed to sync object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17/obj00000000000000000162 2024-08-18T16:19:22.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.973+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:entry[obj00000000000000000056]: success 2024-08-18T16:19:22.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.973+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4:5 marker=00000000001.90.6 timestamp=2024-08-18T16:19:19.144248+0000 2024-08-18T16:19:22.978 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.974+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:22.978 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.975+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: sync status for source bucket shard: 0 2024-08-18T16:19:22.978 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.975+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: listing bilog for incremental sync; position= 2024-08-18T16:19:22.978 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.976+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:entry[obj00000000000000000012]: success 2024-08-18T16:19:22.978 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.976+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22:5 marker=00000000002.185.6 timestamp=2024-08-18T16:19:19.024938+0000 2024-08-18T16:19:22.998 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.997+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: parsed entry: id=00000000002.73.6 iter->object=obj00000000000000000006 iter->instance= name=obj00000000000000000006 instance= ns= 2024-08-18T16:19:22.998 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.997+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1/obj00000000000000000006 2024-08-18T16:19:22.998 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.997+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: entry->timestamp=2024-08-18T16:19:18.673659+0000 2024-08-18T16:19:22.998 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.997+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:entry[obj00000000000000000006]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.73.6 op=0 op_state=1 2024-08-18T16:19:22.998 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.997+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:entry[obj00000000000000000006]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000006[0] 2024-08-18T16:19:22.999 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.997+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: listing bilog for incremental sync; position=00000000002.73.6 2024-08-18T16:19:22.999 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.997+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: parsed entry: id=00000000002.163.6 iter->object=obj00000000000000000019 iter->instance= name=obj00000000000000000019 instance= ns= 2024-08-18T16:19:22.999 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.997+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0/obj00000000000000000019 2024-08-18T16:19:22.999 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.997+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: entry->timestamp=2024-08-18T16:19:18.736467+0000 2024-08-18T16:19:22.999 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.997+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:entry[obj00000000000000000019]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.163.6 op=0 op_state=1 2024-08-18T16:19:22.999 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.997+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:entry[obj00000000000000000019]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000019[0] 2024-08-18T16:19:22.999 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.997+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: listing bilog for incremental sync; position=00000000002.163.6 2024-08-18T16:19:23.000 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.999+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: parsed entry: id=00000000002.83.6 iter->object=obj00000000000000000000 iter->instance= name=obj00000000000000000000 instance= ns= 2024-08-18T16:19:23.000 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.999+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9/obj00000000000000000000 2024-08-18T16:19:23.001 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.999+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: entry->timestamp=2024-08-18T16:19:18.665114+0000 2024-08-18T16:19:23.001 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.999+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:entry[obj00000000000000000000]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.83.6 op=0 op_state=1 2024-08-18T16:19:23.001 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.999+0000 7fbf28131880 0 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:entry[obj00000000000000000000]: : injecting data sync error on key=obj00000000000000000000 2024-08-18T16:19:23.001 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.999+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: listing bilog for incremental sync; position=00000000002.83.6 2024-08-18T16:19:23.001 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.999+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: parsed entry: id=00000000002.85.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2024-08-18T16:19:23.001 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.999+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10/obj00000000000000000021 2024-08-18T16:19:23.001 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.999+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: entry->timestamp=2024-08-18T16:19:18.968460+0000 2024-08-18T16:19:23.001 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.999+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.85.6 op=0 op_state=1 2024-08-18T16:19:23.001 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.999+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: parsed entry: id=00000000002.122.6 iter->object=obj00000000000000000022 iter->instance= name=obj00000000000000000022 instance= ns= 2024-08-18T16:19:23.001 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.999+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8/obj00000000000000000022 2024-08-18T16:19:23.001 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.999+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: entry->timestamp=2024-08-18T16:19:18.551431+0000 2024-08-18T16:19:23.001 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.999+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:entry[obj00000000000000000022]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.122.6 op=0 op_state=1 2024-08-18T16:19:23.002 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.999+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:entry[obj00000000000000000006]: success 2024-08-18T16:19:23.002 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.999+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1:5 marker=00000000002.73.6 timestamp=2024-08-18T16:19:18.673659+0000 2024-08-18T16:19:23.002 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.999+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:entry[obj00000000000000000000]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:19:23.002 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.999+0000 7fbf28131880 0 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:entry[obj00000000000000000000]: ERROR: failed to sync object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9/obj00000000000000000000 2024-08-18T16:19:23.002 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.999+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:entry[obj00000000000000000021]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000021[0] 2024-08-18T16:19:23.002 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.999+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: listing bilog for incremental sync; position=00000000002.85.6 2024-08-18T16:19:23.002 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.999+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:entry[obj00000000000000000019]: success 2024-08-18T16:19:23.002 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.999+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0:5 marker=00000000002.163.6 timestamp=2024-08-18T16:19:18.736467+0000 2024-08-18T16:19:23.002 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.999+0000 7fbf28131880 0 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:entry[obj00000000000000000022]: : injecting data sync error on key=obj00000000000000000022 2024-08-18T16:19:23.002 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.999+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: listing bilog for incremental sync; position=00000000002.122.6 2024-08-18T16:19:23.002 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.999+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:entry[obj00000000000000000022]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:19:23.002 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:22.999+0000 7fbf28131880 0 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:entry[obj00000000000000000022]: ERROR: failed to sync object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8/obj00000000000000000022 2024-08-18T16:19:23.003 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.002+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:entry[obj00000000000000000021]: success 2024-08-18T16:19:23.003 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.002+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10:5 marker=00000000002.85.6 timestamp=2024-08-18T16:19:18.968460+0000 2024-08-18T16:19:23.004 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.004+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:entry[obj00000000000000000467]: success 2024-08-18T16:19:23.005 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.004+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2:5 marker=00000000001.82.6 timestamp=2024-08-18T16:19:19.151651+0000 2024-08-18T16:19:23.005 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.005+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: parsed entry: id=00000000002.53.6 iter->object=obj00000000000000000007 iter->instance= name=obj00000000000000000007 instance= ns= 2024-08-18T16:19:23.005 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.005+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15/obj00000000000000000007 2024-08-18T16:19:23.005 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.005+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: entry->timestamp=2024-08-18T16:19:18.912352+0000 2024-08-18T16:19:23.005 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.005+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:entry[obj00000000000000000007]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.53.6 op=0 op_state=1 2024-08-18T16:19:23.005 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.005+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:entry[obj00000000000000000007]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000007[0] 2024-08-18T16:19:23.005 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.005+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: listing bilog for incremental sync; position=00000000002.53.6 2024-08-18T16:19:23.008 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.007+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: parsed entry: id=00000000001.88.6 iter->object=obj00000000000000000271 iter->instance= name=obj00000000000000000271 instance= ns= 2024-08-18T16:19:23.008 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.007+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12/obj00000000000000000271 2024-08-18T16:19:23.008 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.007+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: entry->timestamp=2024-08-18T16:19:18.858093+0000 2024-08-18T16:19:23.008 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.007+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:entry[obj00000000000000000271]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.88.6 op=0 op_state=1 2024-08-18T16:19:23.008 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.007+0000 7fbf28131880 0 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:entry[obj00000000000000000271]: : injecting data sync error on key=obj00000000000000000271 2024-08-18T16:19:23.008 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.007+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: listing bilog for incremental sync; position=00000000001.88.6 2024-08-18T16:19:23.008 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.007+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: parsed entry: id=00000000002.126.6 iter->object=obj00000000000000000023 iter->instance= name=obj00000000000000000023 instance= ns= 2024-08-18T16:19:23.008 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.007+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6/obj00000000000000000023 2024-08-18T16:19:23.008 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.007+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: entry->timestamp=2024-08-18T16:19:18.904686+0000 2024-08-18T16:19:23.008 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.007+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:entry[obj00000000000000000023]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.126.6 op=0 op_state=1 2024-08-18T16:19:23.009 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.007+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:entry[obj00000000000000000271]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:19:23.009 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.007+0000 7fbf28131880 0 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:entry[obj00000000000000000271]: ERROR: failed to sync object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12/obj00000000000000000271 2024-08-18T16:19:23.009 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.007+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:entry[obj00000000000000000023]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000023[0] 2024-08-18T16:19:23.009 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.007+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: listing bilog for incremental sync; position=00000000002.126.6 2024-08-18T16:19:23.009 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.007+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:entry[obj00000000000000000007]: success 2024-08-18T16:19:23.009 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.007+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15:5 marker=00000000002.53.6 timestamp=2024-08-18T16:19:18.912352+0000 2024-08-18T16:19:23.010 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.008+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]: parsed entry: id=00000000001.124.6 iter->object=obj00000000000000000423 iter->instance= name=obj00000000000000000423 instance= ns= 2024-08-18T16:19:23.010 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.008+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20/obj00000000000000000423 2024-08-18T16:19:23.010 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.008+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]: entry->timestamp=2024-08-18T16:19:18.720502+0000 2024-08-18T16:19:23.010 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.008+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:entry[obj00000000000000000423]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.124.6 op=0 op_state=1 2024-08-18T16:19:23.010 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.008+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:entry[obj00000000000000000423]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000423[0] 2024-08-18T16:19:23.010 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.008+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]: listing bilog for incremental sync; position=00000000001.124.6 2024-08-18T16:19:23.010 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.008+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: parsed entry: id=00000000002.110.6 iter->object=obj00000000000000000016 iter->instance= name=obj00000000000000000016 instance= ns= 2024-08-18T16:19:23.010 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.008+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18/obj00000000000000000016 2024-08-18T16:19:23.010 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.008+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: entry->timestamp=2024-08-18T16:19:18.848534+0000 2024-08-18T16:19:23.010 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.008+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:entry[obj00000000000000000016]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.110.6 op=0 op_state=1 2024-08-18T16:19:23.011 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.008+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:entry[obj00000000000000000018]: success 2024-08-18T16:19:23.011 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.008+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14:5 marker=00000000002.165.6 timestamp=2024-08-18T16:19:19.136367+0000 2024-08-18T16:19:23.011 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.008+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:entry[obj00000000000000000016]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000016[0] 2024-08-18T16:19:23.011 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.008+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: listing bilog for incremental sync; position=00000000002.110.6 2024-08-18T16:19:23.011 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.010+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]: parsed entry: id=00000000002.103.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2024-08-18T16:19:23.011 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.010+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23/obj00000000000000000001 2024-08-18T16:19:23.011 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.010+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]: entry->timestamp=2024-08-18T16:19:19.089586+0000 2024-08-18T16:19:23.011 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.010+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.103.6 op=0 op_state=1 2024-08-18T16:19:23.011 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.010+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:entry[obj00000000000000000001]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000001[0] 2024-08-18T16:19:23.011 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.010+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]: listing bilog for incremental sync; position=00000000002.103.6 2024-08-18T16:19:23.012 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.010+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]: parsed entry: id=00000000002.86.6 iter->object=obj00000000000000000010 iter->instance= name=obj00000000000000000010 instance= ns= 2024-08-18T16:19:23.012 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.010+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26/obj00000000000000000010 2024-08-18T16:19:23.012 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.010+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]: entry->timestamp=2024-08-18T16:19:18.608847+0000 2024-08-18T16:19:23.012 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.010+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]:entry[obj00000000000000000010]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.86.6 op=0 op_state=1 2024-08-18T16:19:23.012 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.010+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25]: parsed entry: id=00000000001.104.6 iter->object=obj00000000000000000283 iter->instance= name=obj00000000000000000283 instance= ns= 2024-08-18T16:19:23.012 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.010+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25/obj00000000000000000283 2024-08-18T16:19:23.012 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.010+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25]: entry->timestamp=2024-08-18T16:19:18.728571+0000 2024-08-18T16:19:23.012 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.010+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25]:entry[obj00000000000000000283]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.104.6 op=0 op_state=1 2024-08-18T16:19:23.012 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.010+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]:entry[obj00000000000000000010]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000010[0] 2024-08-18T16:19:23.012 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.010+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]: listing bilog for incremental sync; position=00000000002.86.6 2024-08-18T16:19:23.012 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.010+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25]:entry[obj00000000000000000283]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000283[0] 2024-08-18T16:19:23.012 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.010+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25]: listing bilog for incremental sync; position=00000000001.104.6 2024-08-18T16:19:23.013 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.013+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]: a sync operation returned error 2024-08-18T16:19:23.013 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.013+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]: backing out with sync_status=-5 2024-08-18T16:19:23.013 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.013+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]: incremental sync on bucket failed, retcode=-5 2024-08-18T16:19:23.014 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.013+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]:entry[obj00000000000000000010]: success 2024-08-18T16:19:23.014 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.013+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26:5 marker=00000000002.86.6 timestamp=2024-08-18T16:19:18.608847+0000 2024-08-18T16:19:23.014 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.013+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]: ERROR: incremental sync failed. error: -5 2024-08-18T16:19:23.014 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.013+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:entry[obj00000000000000000016]: success 2024-08-18T16:19:23.014 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.013+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18:5 marker=00000000002.110.6 timestamp=2024-08-18T16:19:18.848534+0000 2024-08-18T16:19:23.014 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.014+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:entry[obj00000000000000000001]: success 2024-08-18T16:19:23.014 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.014+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23:5 marker=00000000002.103.6 timestamp=2024-08-18T16:19:19.089586+0000 2024-08-18T16:19:23.015 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.014+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:23.015 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.015+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]: a sync operation returned error 2024-08-18T16:19:23.015 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.015+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]: a sync operation returned error 2024-08-18T16:19:23.015 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.015+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: a sync operation returned error 2024-08-18T16:19:23.016 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.015+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]: backing out with sync_status=-5 2024-08-18T16:19:23.016 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.015+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]: incremental sync on bucket failed, retcode=-5 2024-08-18T16:19:23.016 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.015+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]: backing out with sync_status=-5 2024-08-18T16:19:23.016 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.015+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: backing out with sync_status=-5 2024-08-18T16:19:23.016 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.015+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]: ERROR: incremental sync failed. error: -5 2024-08-18T16:19:23.016 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.015+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]: incremental sync on bucket failed, retcode=-5 2024-08-18T16:19:23.016 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.015+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: incremental sync on bucket failed, retcode=-5 2024-08-18T16:19:23.016 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.015+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]: ERROR: incremental sync failed. error: -5 2024-08-18T16:19:23.016 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.015+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: ERROR: incremental sync failed. error: -5 2024-08-18T16:19:23.017 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.015+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]: sync status for source bucket shard: 0 2024-08-18T16:19:23.017 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.015+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]: listing bilog for incremental sync; position= 2024-08-18T16:19:23.017 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.016+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:23.017 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.016+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:23.017 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.017+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]: sync status for source bucket shard: 0 2024-08-18T16:19:23.017 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.017+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]: listing bilog for incremental sync; position= 2024-08-18T16:19:23.017 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.017+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]: sync status for source bucket shard: 0 2024-08-18T16:19:23.017 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.017+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]: listing bilog for incremental sync; position= 2024-08-18T16:19:23.018 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.017+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:23.018 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.017+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25]:entry[obj00000000000000000283]: success 2024-08-18T16:19:23.018 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.018+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25:5 marker=00000000001.104.6 timestamp=2024-08-18T16:19:18.728571+0000 2024-08-18T16:19:23.018 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.018+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: parsed entry: id=00000000002.75.6 iter->object=obj00000000000000000004 iter->instance= name=obj00000000000000000004 instance= ns= 2024-08-18T16:19:23.018 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.018+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5/obj00000000000000000004 2024-08-18T16:19:23.019 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.018+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: entry->timestamp=2024-08-18T16:19:19.204480+0000 2024-08-18T16:19:23.019 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.018+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:entry[obj00000000000000000004]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.75.6 op=0 op_state=1 2024-08-18T16:19:23.019 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.018+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:entry[obj00000000000000000004]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000004[0] 2024-08-18T16:19:23.019 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.018+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: listing bilog for incremental sync; position=00000000002.75.6 2024-08-18T16:19:23.019 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.018+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: sync status for source bucket shard: 0 2024-08-18T16:19:23.019 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.018+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: listing bilog for incremental sync; position= 2024-08-18T16:19:23.021 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.020+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:entry[obj00000000000000000004]: success 2024-08-18T16:19:23.021 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.020+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5:5 marker=00000000002.75.6 timestamp=2024-08-18T16:19:19.204480+0000 2024-08-18T16:19:23.021 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.021+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]:entry[obj00000000000000000061]: success 2024-08-18T16:19:23.022 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.021+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27:5 marker=00000000001.110.6 timestamp=2024-08-18T16:19:18.800570+0000 2024-08-18T16:19:23.022 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.021+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:30]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:30]:entry[obj00000000000000000133]: success 2024-08-18T16:19:23.022 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.021+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:30]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:30]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:30:5 marker=00000000001.114.6 timestamp=2024-08-18T16:19:19.158889+0000 2024-08-18T16:19:23.041 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.041+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: a sync operation returned error 2024-08-18T16:19:23.041 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.041+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: backing out with sync_status=-5 2024-08-18T16:19:23.041 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.041+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: incremental sync on bucket failed, retcode=-5 2024-08-18T16:19:23.041 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.041+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: ERROR: incremental sync failed. error: -5 2024-08-18T16:19:23.042 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.041+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:23.042 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.041+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: sync status for source bucket shard: 0 2024-08-18T16:19:23.042 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.041+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: listing bilog for incremental sync; position= 2024-08-18T16:19:23.042 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.042+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: a sync operation returned error 2024-08-18T16:19:23.042 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.042+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: backing out with sync_status=-5 2024-08-18T16:19:23.042 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.042+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: incremental sync on bucket failed, retcode=-5 2024-08-18T16:19:23.043 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.042+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: ERROR: incremental sync failed. error: -5 2024-08-18T16:19:23.043 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.043+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:23.044 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.043+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: sync status for source bucket shard: 0 2024-08-18T16:19:23.044 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.044+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: listing bilog for incremental sync; position= 2024-08-18T16:19:23.050 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.050+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: a sync operation returned error 2024-08-18T16:19:23.050 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.050+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: backing out with sync_status=-5 2024-08-18T16:19:23.050 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.050+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: incremental sync on bucket failed, retcode=-5 2024-08-18T16:19:23.050 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.050+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: ERROR: incremental sync failed. error: -5 2024-08-18T16:19:23.050 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.050+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:23.051 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.050+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: sync status for source bucket shard: 0 2024-08-18T16:19:23.051 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.050+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: listing bilog for incremental sync; position= 2024-08-18T16:19:23.053 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.053+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:entry[obj00000000000000000023]: success 2024-08-18T16:19:23.053 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.053+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6:5 marker=00000000002.126.6 timestamp=2024-08-18T16:19:18.904686+0000 2024-08-18T16:19:23.057 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.056+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:entry[obj00000000000000000423]: success 2024-08-18T16:19:23.057 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.056+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20:5 marker=00000000001.124.6 timestamp=2024-08-18T16:19:18.720502+0000 2024-08-18T16:19:23.058 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.058+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]: parsed entry: id=00000000001.88.6 iter->object=obj00000000000000000029 iter->instance= name=obj00000000000000000029 instance= ns= 2024-08-18T16:19:23.058 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.058+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29/obj00000000000000000029 2024-08-18T16:19:23.058 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.058+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]: entry->timestamp=2024-08-18T16:19:18.977845+0000 2024-08-18T16:19:23.058 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.058+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]:entry[obj00000000000000000029]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.88.6 op=0 op_state=1 2024-08-18T16:19:23.059 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.058+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]:entry[obj00000000000000000029]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000029[0] 2024-08-18T16:19:23.059 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.058+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]: listing bilog for incremental sync; position=00000000001.88.6 2024-08-18T16:19:23.059 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.059+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]: parsed entry: id=00000000002.150.6 iter->object=obj00000000000000000002 iter->instance= name=obj00000000000000000002 instance= ns= 2024-08-18T16:19:23.059 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.059+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21/obj00000000000000000002 2024-08-18T16:19:23.059 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.059+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]: entry->timestamp=2024-08-18T16:19:18.921737+0000 2024-08-18T16:19:23.059 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.059+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:entry[obj00000000000000000002]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.150.6 op=0 op_state=1 2024-08-18T16:19:23.060 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.059+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:entry[obj00000000000000000002]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000002[0] 2024-08-18T16:19:23.060 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.059+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]: listing bilog for incremental sync; position=00000000002.150.6 2024-08-18T16:19:23.060 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.060+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]: parsed entry: id=00000000002.112.6 iter->object=obj00000000000000000011 iter->instance= name=obj00000000000000000011 instance= ns= 2024-08-18T16:19:23.060 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.060+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24/obj00000000000000000011 2024-08-18T16:19:23.060 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.060+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]: entry->timestamp=2024-08-18T16:19:19.033586+0000 2024-08-18T16:19:23.060 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.060+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]:entry[obj00000000000000000011]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.112.6 op=0 op_state=1 2024-08-18T16:19:23.060 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.060+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]:entry[obj00000000000000000011]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000011[0] 2024-08-18T16:19:23.060 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.060+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]: listing bilog for incremental sync; position=00000000002.112.6 2024-08-18T16:19:23.062 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.061+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:entry[obj00000000000000000002]: success 2024-08-18T16:19:23.062 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.061+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21:5 marker=00000000002.150.6 timestamp=2024-08-18T16:19:18.921737+0000 2024-08-18T16:19:23.062 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.061+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: parsed entry: id=00000000001.82.6 iter->object=obj00000000000000000162 iter->instance= name=obj00000000000000000162 instance= ns= 2024-08-18T16:19:23.062 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.061+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17/obj00000000000000000162 2024-08-18T16:19:23.062 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.061+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: entry->timestamp=2024-08-18T16:19:18.617486+0000 2024-08-18T16:19:23.062 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.061+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:entry[obj00000000000000000162]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.82.6 op=0 op_state=1 2024-08-18T16:19:23.062 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.061+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:entry[obj00000000000000000162]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000162[0] 2024-08-18T16:19:23.062 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.061+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: listing bilog for incremental sync; position=00000000001.82.6 2024-08-18T16:19:23.063 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.062+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]:entry[obj00000000000000000011]: success 2024-08-18T16:19:23.063 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.062+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24:5 marker=00000000002.112.6 timestamp=2024-08-18T16:19:19.033586+0000 2024-08-18T16:19:23.084 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.084+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: parsed entry: id=00000000002.122.6 iter->object=obj00000000000000000022 iter->instance= name=obj00000000000000000022 instance= ns= 2024-08-18T16:19:23.084 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.084+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8/obj00000000000000000022 2024-08-18T16:19:23.084 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.084+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: entry->timestamp=2024-08-18T16:19:18.551431+0000 2024-08-18T16:19:23.085 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.084+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:entry[obj00000000000000000022]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.122.6 op=0 op_state=1 2024-08-18T16:19:23.085 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.084+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:entry[obj00000000000000000022]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000022[0] 2024-08-18T16:19:23.085 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.084+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: listing bilog for incremental sync; position=00000000002.122.6 2024-08-18T16:19:23.086 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.086+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: parsed entry: id=00000000002.83.6 iter->object=obj00000000000000000000 iter->instance= name=obj00000000000000000000 instance= ns= 2024-08-18T16:19:23.086 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.086+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9/obj00000000000000000000 2024-08-18T16:19:23.086 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.086+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: entry->timestamp=2024-08-18T16:19:18.665114+0000 2024-08-18T16:19:23.086 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.086+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:entry[obj00000000000000000000]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.83.6 op=0 op_state=1 2024-08-18T16:19:23.086 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.086+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:entry[obj00000000000000000000]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000000[0] 2024-08-18T16:19:23.086 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.086+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: listing bilog for incremental sync; position=00000000002.83.6 2024-08-18T16:19:23.086 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.086+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:entry[obj00000000000000000022]: success 2024-08-18T16:19:23.087 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.086+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8:5 marker=00000000002.122.6 timestamp=2024-08-18T16:19:18.551431+0000 2024-08-18T16:19:23.088 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.087+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:entry[obj00000000000000000000]: success 2024-08-18T16:19:23.088 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.087+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9:5 marker=00000000002.83.6 timestamp=2024-08-18T16:19:18.665114+0000 2024-08-18T16:19:23.092 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.092+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: parsed entry: id=00000000001.88.6 iter->object=obj00000000000000000271 iter->instance= name=obj00000000000000000271 instance= ns= 2024-08-18T16:19:23.092 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.092+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12/obj00000000000000000271 2024-08-18T16:19:23.092 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.092+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: entry->timestamp=2024-08-18T16:19:18.858093+0000 2024-08-18T16:19:23.092 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.092+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:entry[obj00000000000000000271]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.88.6 op=0 op_state=1 2024-08-18T16:19:23.092 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.092+0000 7fbf28131880 0 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:entry[obj00000000000000000271]: : injecting data sync error on key=obj00000000000000000271 2024-08-18T16:19:23.092 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.092+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: listing bilog for incremental sync; position=00000000001.88.6 2024-08-18T16:19:23.093 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.092+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:entry[obj00000000000000000271]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:19:23.093 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.092+0000 7fbf28131880 0 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:entry[obj00000000000000000271]: ERROR: failed to sync object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12/obj00000000000000000271 2024-08-18T16:19:23.105 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.105+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]:entry[obj00000000000000000029]: success 2024-08-18T16:19:23.105 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.105+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29:5 marker=00000000001.88.6 timestamp=2024-08-18T16:19:18.977845+0000 2024-08-18T16:19:23.107 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.107+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:entry[obj00000000000000000162]: success 2024-08-18T16:19:23.107 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.107+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17:5 marker=00000000001.82.6 timestamp=2024-08-18T16:19:18.617486+0000 2024-08-18T16:19:23.134 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.134+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: a sync operation returned error 2024-08-18T16:19:23.134 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.134+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: backing out with sync_status=-5 2024-08-18T16:19:23.134 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.134+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: incremental sync on bucket failed, retcode=-5 2024-08-18T16:19:23.134 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.134+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: ERROR: incremental sync failed. error: -5 2024-08-18T16:19:23.135 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.134+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:23.135 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.134+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: sync status for source bucket shard: 0 2024-08-18T16:19:23.135 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.134+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: listing bilog for incremental sync; position= 2024-08-18T16:19:23.177 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.177+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: parsed entry: id=00000000001.88.6 iter->object=obj00000000000000000271 iter->instance= name=obj00000000000000000271 instance= ns= 2024-08-18T16:19:23.178 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.177+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12/obj00000000000000000271 2024-08-18T16:19:23.178 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.177+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: entry->timestamp=2024-08-18T16:19:18.858093+0000 2024-08-18T16:19:23.178 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.177+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:entry[obj00000000000000000271]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.88.6 op=0 op_state=1 2024-08-18T16:19:23.178 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.177+0000 7fbf28131880 0 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:entry[obj00000000000000000271]: : injecting data sync error on key=obj00000000000000000271 2024-08-18T16:19:23.178 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.177+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: listing bilog for incremental sync; position=00000000001.88.6 2024-08-18T16:19:23.178 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.177+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:entry[obj00000000000000000271]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:19:23.178 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.177+0000 7fbf28131880 0 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:entry[obj00000000000000000271]: ERROR: failed to sync object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12/obj00000000000000000271 2024-08-18T16:19:23.219 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.219+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: a sync operation returned error 2024-08-18T16:19:23.219 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.219+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: backing out with sync_status=-5 2024-08-18T16:19:23.219 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.219+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: incremental sync on bucket failed, retcode=-5 2024-08-18T16:19:23.219 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.219+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: ERROR: incremental sync failed. error: -5 2024-08-18T16:19:23.220 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.219+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:23.220 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.219+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: sync status for source bucket shard: 0 2024-08-18T16:19:23.220 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.219+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: listing bilog for incremental sync; position= 2024-08-18T16:19:23.262 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.261+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: parsed entry: id=00000000001.88.6 iter->object=obj00000000000000000271 iter->instance= name=obj00000000000000000271 instance= ns= 2024-08-18T16:19:23.262 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.261+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12/obj00000000000000000271 2024-08-18T16:19:23.262 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.261+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: entry->timestamp=2024-08-18T16:19:18.858093+0000 2024-08-18T16:19:23.262 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.261+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:entry[obj00000000000000000271]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.88.6 op=0 op_state=1 2024-08-18T16:19:23.262 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.261+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:entry[obj00000000000000000271]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000271[0] 2024-08-18T16:19:23.262 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.261+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: listing bilog for incremental sync; position=00000000001.88.6 2024-08-18T16:19:23.265 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.265+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:entry[obj00000000000000000271]: success 2024-08-18T16:19:23.265 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.265+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12:5 marker=00000000001.88.6 timestamp=2024-08-18T16:19:18.858093+0000 2024-08-18T16:19:23.308 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.307+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:23.309 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.307+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:23.309 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.307+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:23.309 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.307+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:23.309 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.307+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: sync status for source bucket shard: 0 2024-08-18T16:19:23.309 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.307+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: listing bilog for incremental sync; position= 2024-08-18T16:19:23.309 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.307+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: sync status for source bucket shard: 0 2024-08-18T16:19:23.309 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.307+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: listing bilog for incremental sync; position= 2024-08-18T16:19:23.309 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.307+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:23.309 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.308+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: sync status for source bucket shard: 0 2024-08-18T16:19:23.309 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.308+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: sync status for source bucket shard: 0 2024-08-18T16:19:23.309 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.308+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: listing bilog for incremental sync; position= 2024-08-18T16:19:23.309 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.308+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: listing bilog for incremental sync; position= 2024-08-18T16:19:23.310 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.308+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:23.310 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.308+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:23.311 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.308+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:23.311 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.309+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: sync status for source bucket shard: 0 2024-08-18T16:19:23.311 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.309+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: listing bilog for incremental sync; position= 2024-08-18T16:19:23.311 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.309+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: sync status for source bucket shard: 0 2024-08-18T16:19:23.311 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.309+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: listing bilog for incremental sync; position= 2024-08-18T16:19:23.311 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.309+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:23.311 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.309+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:23.311 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.309+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: parsed entry: id=00000000002.100.6 iter->object=obj00000000000000000000 iter->instance= name=obj00000000000000000000 instance= ns= 2024-08-18T16:19:23.311 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.309+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1/obj00000000000000000000 2024-08-18T16:19:23.311 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.309+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: entry->timestamp=2024-08-18T16:19:19.893792+0000 2024-08-18T16:19:23.311 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.309+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:entry[obj00000000000000000000]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.100.6 op=0 op_state=1 2024-08-18T16:19:23.312 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.309+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:23.312 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.309+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:23.312 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.309+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:entry[obj00000000000000000000]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000000[0] 2024-08-18T16:19:23.312 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.309+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: listing bilog for incremental sync; position=00000000002.100.6 2024-08-18T16:19:23.312 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.309+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: sync status for source bucket shard: 0 2024-08-18T16:19:23.312 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.309+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: sync status for source bucket shard: 0 2024-08-18T16:19:23.312 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.309+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: sync status for source bucket shard: 0 2024-08-18T16:19:23.312 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.310+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: listing bilog for incremental sync; position= 2024-08-18T16:19:23.312 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.310+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: listing bilog for incremental sync; position= 2024-08-18T16:19:23.312 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.310+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: listing bilog for incremental sync; position= 2024-08-18T16:19:23.312 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.310+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:23.312 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.310+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: sync status for source bucket shard: 0 2024-08-18T16:19:23.312 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.310+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: listing bilog for incremental sync; position= 2024-08-18T16:19:23.315 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.311+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: sync status for source bucket shard: 0 2024-08-18T16:19:23.315 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.311+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: listing bilog for incremental sync; position= 2024-08-18T16:19:23.315 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.311+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: parsed entry: id=00000000002.112.6 iter->object=obj00000000000000000033 iter->instance= name=obj00000000000000000033 instance= ns= 2024-08-18T16:19:23.315 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.311+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4/obj00000000000000000033 2024-08-18T16:19:23.315 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.311+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: entry->timestamp=2024-08-18T16:19:19.593080+0000 2024-08-18T16:19:23.315 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.311+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:entry[obj00000000000000000033]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.112.6 op=0 op_state=1 2024-08-18T16:19:23.315 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.311+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: parsed entry: id=00000000002.128.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2024-08-18T16:19:23.315 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.311+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3/obj00000000000000000001 2024-08-18T16:19:23.315 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.311+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: entry->timestamp=2024-08-18T16:19:20.116900+0000 2024-08-18T16:19:23.315 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.311+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.128.6 op=0 op_state=1 2024-08-18T16:19:23.315 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.311+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:entry[obj00000000000000000033]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000033[0] 2024-08-18T16:19:23.315 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.311+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: listing bilog for incremental sync; position=00000000002.112.6 2024-08-18T16:19:23.316 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.311+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:entry[obj00000000000000000001]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000001[0] 2024-08-18T16:19:23.316 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.311+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: listing bilog for incremental sync; position=00000000002.128.6 2024-08-18T16:19:23.316 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.311+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]: sync status for source bucket shard: 0 2024-08-18T16:19:23.316 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.311+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]: listing bilog for incremental sync; position= 2024-08-18T16:19:23.316 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.312+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: sync status for source bucket shard: 0 2024-08-18T16:19:23.316 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.312+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: listing bilog for incremental sync; position= 2024-08-18T16:19:23.316 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.312+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: parsed entry: id=00000000001.140.6 iter->object=obj00000000000000000060 iter->instance= name=obj00000000000000000060 instance= ns= 2024-08-18T16:19:23.316 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.312+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0/obj00000000000000000060 2024-08-18T16:19:23.316 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.312+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: entry->timestamp=2024-08-18T16:19:19.996680+0000 2024-08-18T16:19:23.316 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.312+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:entry[obj00000000000000000060]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.140.6 op=0 op_state=1 2024-08-18T16:19:23.316 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.312+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:23.316 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.312+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:23.317 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.312+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:23.317 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.312+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:23.317 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.312+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:23.317 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.312+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:23.317 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.312+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:23.317 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.312+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:23.317 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.312+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:23.317 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.312+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:23.317 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.312+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:entry[obj00000000000000000060]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000060[0] 2024-08-18T16:19:23.317 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.312+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: listing bilog for incremental sync; position=00000000001.140.6 2024-08-18T16:19:23.317 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.312+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: parsed entry: id=00000000002.167.6 iter->object=obj00000000000000000005 iter->instance= name=obj00000000000000000005 instance= ns= 2024-08-18T16:19:23.317 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.312+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6/obj00000000000000000005 2024-08-18T16:19:23.317 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.312+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: entry->timestamp=2024-08-18T16:19:20.349953+0000 2024-08-18T16:19:23.318 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.312+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:entry[obj00000000000000000005]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.167.6 op=0 op_state=1 2024-08-18T16:19:23.318 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.312+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:entry[obj00000000000000000005]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000005[0] 2024-08-18T16:19:23.318 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.312+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: listing bilog for incremental sync; position=00000000002.167.6 2024-08-18T16:19:23.318 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.313+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:23.318 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.313+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:23.318 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.313+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:23.318 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.313+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:23.318 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.313+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:23.318 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.313+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]: sync status for source bucket shard: 0 2024-08-18T16:19:23.318 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.313+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: sync status for source bucket shard: 0 2024-08-18T16:19:23.318 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.313+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]: listing bilog for incremental sync; position= 2024-08-18T16:19:23.318 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.313+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: listing bilog for incremental sync; position= 2024-08-18T16:19:23.318 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.313+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: parsed entry: id=00000000001.81.6 iter->object=obj00000000000000000065 iter->instance= name=obj00000000000000000065 instance= ns= 2024-08-18T16:19:23.319 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.313+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5/obj00000000000000000065 2024-08-18T16:19:23.319 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.313+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: entry->timestamp=2024-08-18T16:19:19.716103+0000 2024-08-18T16:19:23.319 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.313+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:entry[obj00000000000000000065]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.81.6 op=0 op_state=1 2024-08-18T16:19:23.319 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.313+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: parsed entry: id=00000000002.81.6 iter->object=obj00000000000000000016 iter->instance= name=obj00000000000000000016 instance= ns= 2024-08-18T16:19:23.319 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.313+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7/obj00000000000000000016 2024-08-18T16:19:23.319 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.313+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: entry->timestamp=2024-08-18T16:19:20.228897+0000 2024-08-18T16:19:23.319 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.313+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:entry[obj00000000000000000016]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.81.6 op=0 op_state=1 2024-08-18T16:19:23.319 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.313+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: parsed entry: id=00000000001.170.6 iter->object=obj00000000000000000048 iter->instance= name=obj00000000000000000048 instance= ns= 2024-08-18T16:19:23.319 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.313+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8/obj00000000000000000048 2024-08-18T16:19:23.319 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.313+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: entry->timestamp=2024-08-18T16:19:19.833816+0000 2024-08-18T16:19:23.319 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.313+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:entry[obj00000000000000000048]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.170.6 op=0 op_state=1 2024-08-18T16:19:23.319 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.313+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:entry[obj00000000000000000065]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000065[0] 2024-08-18T16:19:23.320 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.313+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: listing bilog for incremental sync; position=00000000001.81.6 2024-08-18T16:19:23.320 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.313+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:entry[obj00000000000000000016]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000016[0] 2024-08-18T16:19:23.320 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.313+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: listing bilog for incremental sync; position=00000000002.81.6 2024-08-18T16:19:23.320 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.313+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:entry[obj00000000000000000048]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000048[0] 2024-08-18T16:19:23.320 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.313+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: listing bilog for incremental sync; position=00000000001.170.6 2024-08-18T16:19:23.324 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.315+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]: sync status for source bucket shard: 0 2024-08-18T16:19:23.324 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.315+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]: listing bilog for incremental sync; position= 2024-08-18T16:19:23.324 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.315+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: sync status for source bucket shard: 0 2024-08-18T16:19:23.324 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.315+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]: sync status for source bucket shard: 0 2024-08-18T16:19:23.324 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.315+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]: sync status for source bucket shard: 0 2024-08-18T16:19:23.324 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.315+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]: sync status for source bucket shard: 0 2024-08-18T16:19:23.324 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.315+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: listing bilog for incremental sync; position= 2024-08-18T16:19:23.325 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.315+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:24]: listing bilog for incremental sync; position= 2024-08-18T16:19:23.325 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.315+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]: listing bilog for incremental sync; position= 2024-08-18T16:19:23.325 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.315+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]: listing bilog for incremental sync; position= 2024-08-18T16:19:23.325 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.316+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:entry[obj00000000000000000000]: success 2024-08-18T16:19:23.325 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.316+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:1:6 marker=00000000002.100.6 timestamp=2024-08-18T16:19:19.893792+0000 2024-08-18T16:19:23.325 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.316+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:entry[obj00000000000000000033]: success 2024-08-18T16:19:23.325 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.316+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:4:6 marker=00000000002.112.6 timestamp=2024-08-18T16:19:19.593080+0000 2024-08-18T16:19:23.325 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.316+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:entry[obj00000000000000000001]: success 2024-08-18T16:19:23.325 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.316+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:3:6 marker=00000000002.128.6 timestamp=2024-08-18T16:19:20.116900+0000 2024-08-18T16:19:23.325 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.316+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]: parsed entry: id=00000000002.115.6 iter->object=obj00000000000000000007 iter->instance= name=obj00000000000000000007 instance= ns= 2024-08-18T16:19:23.325 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.316+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22/obj00000000000000000007 2024-08-18T16:19:23.325 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.316+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]: entry->timestamp=2024-08-18T16:19:19.491062+0000 2024-08-18T16:19:23.326 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.316+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:entry[obj00000000000000000007]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.115.6 op=0 op_state=1 2024-08-18T16:19:23.326 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.316+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:entry[obj00000000000000000007]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000007[0] 2024-08-18T16:19:23.326 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.316+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]: listing bilog for incremental sync; position=00000000002.115.6 2024-08-18T16:19:23.326 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.318+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]: sync status for source bucket shard: 0 2024-08-18T16:19:23.326 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.318+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: sync status for source bucket shard: 0 2024-08-18T16:19:23.326 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.318+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: sync status for source bucket shard: 0 2024-08-18T16:19:23.326 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.318+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25]: sync status for source bucket shard: 0 2024-08-18T16:19:23.326 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.318+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]: sync status for source bucket shard: 0 2024-08-18T16:19:23.326 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.318+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: sync status for source bucket shard: 0 2024-08-18T16:19:23.326 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.318+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]: sync status for source bucket shard: 0 2024-08-18T16:19:23.326 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.319+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]: sync status for source bucket shard: 0 2024-08-18T16:19:23.326 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.319+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:23.327 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.319+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]: listing bilog for incremental sync; position= 2024-08-18T16:19:23.327 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.319+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: listing bilog for incremental sync; position= 2024-08-18T16:19:23.327 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.319+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: listing bilog for incremental sync; position= 2024-08-18T16:19:23.327 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.319+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25]: listing bilog for incremental sync; position= 2024-08-18T16:19:23.327 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.319+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]: listing bilog for incremental sync; position= 2024-08-18T16:19:23.327 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.319+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: listing bilog for incremental sync; position= 2024-08-18T16:19:23.327 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.319+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]: listing bilog for incremental sync; position= 2024-08-18T16:19:23.327 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.319+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:21]: listing bilog for incremental sync; position= 2024-08-18T16:19:23.327 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.319+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:28]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:23.327 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.319+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:30]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:23.327 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.319+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:31]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:23.327 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.319+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:32]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:23.327 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.319+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:33]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:23.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.319+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:34]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:23.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.319+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:35]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:23.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.319+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:36]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:23.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.319+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]: parsed entry: id=00000000002.138.6 iter->object=obj00000000000000000030 iter->instance= name=obj00000000000000000030 instance= ns= 2024-08-18T16:19:23.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.319+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11/obj00000000000000000030 2024-08-18T16:19:23.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.319+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]: entry->timestamp=2024-08-18T16:19:19.545863+0000 2024-08-18T16:19:23.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.319+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:entry[obj00000000000000000030]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.138.6 op=0 op_state=1 2024-08-18T16:19:23.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.320+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:entry[obj00000000000000000030]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000030[0] 2024-08-18T16:19:23.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.320+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]: listing bilog for incremental sync; position=00000000002.138.6 2024-08-18T16:19:23.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.320+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]: parsed entry: id=00000000002.106.6 iter->object=obj00000000000000000045 iter->instance= name=obj00000000000000000045 instance= ns= 2024-08-18T16:19:23.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.320+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19/obj00000000000000000045 2024-08-18T16:19:23.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.320+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]: entry->timestamp=2024-08-18T16:19:20.340703+0000 2024-08-18T16:19:23.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.320+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:entry[obj00000000000000000045]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.106.6 op=0 op_state=1 2024-08-18T16:19:23.329 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.320+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]: parsed entry: id=00000000002.98.6 iter->object=obj00000000000000000039 iter->instance= name=obj00000000000000000039 instance= ns= 2024-08-18T16:19:23.329 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.320+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23/obj00000000000000000039 2024-08-18T16:19:23.329 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.320+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]: entry->timestamp=2024-08-18T16:19:19.824786+0000 2024-08-18T16:19:23.329 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.320+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:entry[obj00000000000000000039]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.98.6 op=0 op_state=1 2024-08-18T16:19:23.329 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.320+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:entry[obj00000000000000000045]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000045[0] 2024-08-18T16:19:23.329 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.320+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]: listing bilog for incremental sync; position=00000000002.106.6 2024-08-18T16:19:23.329 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.321+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:entry[obj00000000000000000039]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000039[0] 2024-08-18T16:19:23.329 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.321+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]: listing bilog for incremental sync; position=00000000002.98.6 2024-08-18T16:19:23.329 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.321+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: parsed entry: id=00000000002.116.6 iter->object=obj00000000000000000002 iter->instance= name=obj00000000000000000002 instance= ns= 2024-08-18T16:19:23.329 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.321+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17/obj00000000000000000002 2024-08-18T16:19:23.329 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.321+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: entry->timestamp=2024-08-18T16:19:19.940549+0000 2024-08-18T16:19:23.329 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.321+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:entry[obj00000000000000000002]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.116.6 op=0 op_state=1 2024-08-18T16:19:23.329 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.321+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]: parsed entry: id=00000000002.110.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2024-08-18T16:19:23.330 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.321+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26/obj00000000000000000021 2024-08-18T16:19:23.330 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.321+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]: entry->timestamp=2024-08-18T16:19:19.648761+0000 2024-08-18T16:19:23.330 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.321+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.110.6 op=0 op_state=1 2024-08-18T16:19:23.330 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.321+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: parsed entry: id=00000000002.100.6 iter->object=obj00000000000000000013 iter->instance= name=obj00000000000000000013 instance= ns= 2024-08-18T16:19:23.330 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.321+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18/obj00000000000000000013 2024-08-18T16:19:23.330 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.321+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: entry->timestamp=2024-08-18T16:19:20.004845+0000 2024-08-18T16:19:23.330 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.321+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:entry[obj00000000000000000013]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.100.6 op=0 op_state=1 2024-08-18T16:19:23.330 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.321+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:entry[obj00000000000000000005]: success 2024-08-18T16:19:23.330 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.321+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:6:6 marker=00000000002.167.6 timestamp=2024-08-18T16:19:20.349953+0000 2024-08-18T16:19:23.330 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.321+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:entry[obj00000000000000000002]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000002[0] 2024-08-18T16:19:23.330 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.321+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: listing bilog for incremental sync; position=00000000002.116.6 2024-08-18T16:19:23.330 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.321+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:entry[obj00000000000000000016]: success 2024-08-18T16:19:23.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.321+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:7:6 marker=00000000002.81.6 timestamp=2024-08-18T16:19:20.228897+0000 2024-08-18T16:19:23.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.321+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]:entry[obj00000000000000000021]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000021[0] 2024-08-18T16:19:23.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.321+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]: listing bilog for incremental sync; position=00000000002.110.6 2024-08-18T16:19:23.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.321+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:entry[obj00000000000000000013]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000013[0] 2024-08-18T16:19:23.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.321+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: listing bilog for incremental sync; position=00000000002.100.6 2024-08-18T16:19:23.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.322+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:31]: sync status for source bucket shard: 0 2024-08-18T16:19:23.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.322+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:31]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:31]: listing bilog for incremental sync; position= 2024-08-18T16:19:23.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.322+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:32]: sync status for source bucket shard: 0 2024-08-18T16:19:23.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.322+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:32]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:32]: listing bilog for incremental sync; position= 2024-08-18T16:19:23.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.323+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:28]: sync status for source bucket shard: 0 2024-08-18T16:19:23.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.323+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:28]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:28]: listing bilog for incremental sync; position= 2024-08-18T16:19:23.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.323+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:34]: sync status for source bucket shard: 0 2024-08-18T16:19:23.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.323+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:35]: sync status for source bucket shard: 0 2024-08-18T16:19:23.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.323+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:34]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:34]: listing bilog for incremental sync; position= 2024-08-18T16:19:23.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.323+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:35]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:35]: listing bilog for incremental sync; position= 2024-08-18T16:19:23.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.325+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]: sync status for source bucket shard: 0 2024-08-18T16:19:23.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.325+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]: listing bilog for incremental sync; position= 2024-08-18T16:19:23.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.325+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:30]: sync status for source bucket shard: 0 2024-08-18T16:19:23.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.325+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:30]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:30]: listing bilog for incremental sync; position= 2024-08-18T16:19:23.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.325+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:33]: sync status for source bucket shard: 0 2024-08-18T16:19:23.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.325+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:33]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:33]: listing bilog for incremental sync; position= 2024-08-18T16:19:23.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.325+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:36]: sync status for source bucket shard: 0 2024-08-18T16:19:23.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.325+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:36]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:36]: listing bilog for incremental sync; position= 2024-08-18T16:19:23.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.326+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:entry[obj00000000000000000030]: success 2024-08-18T16:19:23.335 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.326+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:11:6 marker=00000000002.138.6 timestamp=2024-08-18T16:19:19.545863+0000 2024-08-18T16:19:23.335 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.326+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:entry[obj00000000000000000007]: success 2024-08-18T16:19:23.335 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.326+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:22:6 marker=00000000002.115.6 timestamp=2024-08-18T16:19:19.491062+0000 2024-08-18T16:19:23.335 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.327+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]:entry[obj00000000000000000021]: success 2024-08-18T16:19:23.335 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.327+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:26:6 marker=00000000002.110.6 timestamp=2024-08-18T16:19:19.648761+0000 2024-08-18T16:19:23.335 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.327+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:entry[obj00000000000000000039]: success 2024-08-18T16:19:23.335 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.327+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:23:6 marker=00000000002.98.6 timestamp=2024-08-18T16:19:19.824786+0000 2024-08-18T16:19:23.335 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.328+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:34]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:34]: parsed entry: id=00000000001.94.6 iter->object=obj00000000000000000057 iter->instance= name=obj00000000000000000057 instance= ns= 2024-08-18T16:19:23.335 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.328+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:34]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:34]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:34/obj00000000000000000057 2024-08-18T16:19:23.335 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.328+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:34]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:34]: entry->timestamp=2024-08-18T16:19:19.657944+0000 2024-08-18T16:19:23.335 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.328+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:34]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:34]:entry[obj00000000000000000057]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.94.6 op=0 op_state=1 2024-08-18T16:19:23.335 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.328+0000 7fbf28131880 0 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:34]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:34]:entry[obj00000000000000000057]: : injecting data sync error on key=obj00000000000000000057 2024-08-18T16:19:23.335 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.328+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:34]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:34]: listing bilog for incremental sync; position=00000000001.94.6 2024-08-18T16:19:23.336 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.328+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:34]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:34]:entry[obj00000000000000000057]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:19:23.336 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.328+0000 7fbf28131880 0 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:34]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:34]:entry[obj00000000000000000057]: ERROR: failed to sync object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:34/obj00000000000000000057 2024-08-18T16:19:23.336 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.328+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:31]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:31]: parsed entry: id=00000000002.142.6 iter->object=obj00000000000000000003 iter->instance= name=obj00000000000000000003 instance= ns= 2024-08-18T16:19:23.336 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.328+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:31]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:31]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:31/obj00000000000000000003 2024-08-18T16:19:23.336 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.328+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:31]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:31]: entry->timestamp=2024-08-18T16:19:20.172927+0000 2024-08-18T16:19:23.336 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.328+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:31]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:31]:entry[obj00000000000000000003]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.142.6 op=0 op_state=1 2024-08-18T16:19:23.336 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.328+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:31]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:31]:entry[obj00000000000000000003]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000003[0] 2024-08-18T16:19:23.336 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.328+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:31]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:31]: listing bilog for incremental sync; position=00000000002.142.6 2024-08-18T16:19:23.336 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.328+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:entry[obj00000000000000000013]: success 2024-08-18T16:19:23.336 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.328+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:18:6 marker=00000000002.100.6 timestamp=2024-08-18T16:19:20.004845+0000 2024-08-18T16:19:23.336 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.329+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:33]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:33]: parsed entry: id=00000000002.114.6 iter->object=obj00000000000000000019 iter->instance= name=obj00000000000000000019 instance= ns= 2024-08-18T16:19:23.336 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.329+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:33]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:33]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:33/obj00000000000000000019 2024-08-18T16:19:23.337 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.329+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:33]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:33]: entry->timestamp=2024-08-18T16:19:19.884831+0000 2024-08-18T16:19:23.337 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.329+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:33]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:33]:entry[obj00000000000000000019]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.114.6 op=0 op_state=1 2024-08-18T16:19:23.337 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.329+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:33]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:33]:entry[obj00000000000000000019]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000019[0] 2024-08-18T16:19:23.337 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.329+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:33]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:33]: listing bilog for incremental sync; position=00000000002.114.6 2024-08-18T16:19:23.337 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.329+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:30]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:30]: parsed entry: id=00000000001.86.6 iter->object=obj00000000000000000036 iter->instance= name=obj00000000000000000036 instance= ns= 2024-08-18T16:19:23.337 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.329+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:30]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:30]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:30/obj00000000000000000036 2024-08-18T16:19:23.337 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.329+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:30]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:30]: entry->timestamp=2024-08-18T16:19:20.013736+0000 2024-08-18T16:19:23.337 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.329+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:30]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:30]:entry[obj00000000000000000036]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.86.6 op=0 op_state=1 2024-08-18T16:19:23.337 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.329+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:30]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:30]:entry[obj00000000000000000036]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000036[0] 2024-08-18T16:19:23.337 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.329+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:30]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:30]: listing bilog for incremental sync; position=00000000001.86.6 2024-08-18T16:19:23.337 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.329+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:entry[obj00000000000000000002]: success 2024-08-18T16:19:23.337 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.329+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:17:6 marker=00000000002.116.6 timestamp=2024-08-18T16:19:19.940549+0000 2024-08-18T16:19:23.337 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.330+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:32]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:32]: parsed entry: id=00000000002.87.6 iter->object=obj00000000000000000014 iter->instance= name=obj00000000000000000014 instance= ns= 2024-08-18T16:19:23.338 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.330+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:32]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:32]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:32/obj00000000000000000014 2024-08-18T16:19:23.338 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.330+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:32]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:32]: entry->timestamp=2024-08-18T16:19:20.396690+0000 2024-08-18T16:19:23.338 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.330+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:32]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:32]:entry[obj00000000000000000014]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.87.6 op=0 op_state=1 2024-08-18T16:19:23.338 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.330+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:32]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:32]:entry[obj00000000000000000014]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000014[0] 2024-08-18T16:19:23.338 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.330+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:32]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:32]: listing bilog for incremental sync; position=00000000002.87.6 2024-08-18T16:19:23.338 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.331+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:35]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:35]: parsed entry: id=00000000002.93.6 iter->object=obj00000000000000000018 iter->instance= name=obj00000000000000000018 instance= ns= 2024-08-18T16:19:23.338 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.331+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:35]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:35]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:35/obj00000000000000000018 2024-08-18T16:19:23.338 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.331+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:35]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:35]: entry->timestamp=2024-08-18T16:19:20.284525+0000 2024-08-18T16:19:23.338 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.331+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:35]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:35]:entry[obj00000000000000000018]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.93.6 op=0 op_state=1 2024-08-18T16:19:23.338 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.331+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:35]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:35]:entry[obj00000000000000000018]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000018[0] 2024-08-18T16:19:23.338 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.331+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:35]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:35]: listing bilog for incremental sync; position=00000000002.93.6 2024-08-18T16:19:23.338 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.331+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:36]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:36]: parsed entry: id=00000000002.161.6 iter->object=obj00000000000000000008 iter->instance= name=obj00000000000000000008 instance= ns= 2024-08-18T16:19:23.338 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.331+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:36]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:36]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:36/obj00000000000000000008 2024-08-18T16:19:23.339 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.331+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:36]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:36]: entry->timestamp=2024-08-18T16:19:19.536516+0000 2024-08-18T16:19:23.339 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.331+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:36]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:36]:entry[obj00000000000000000008]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.161.6 op=0 op_state=1 2024-08-18T16:19:23.339 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.331+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:36]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:36]:entry[obj00000000000000000008]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000008[0] 2024-08-18T16:19:23.339 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.331+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:36]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:36]: listing bilog for incremental sync; position=00000000002.161.6 2024-08-18T16:19:23.339 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.332+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:entry[obj00000000000000000060]: success 2024-08-18T16:19:23.339 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.332+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:0:6 marker=00000000001.140.6 timestamp=2024-08-18T16:19:19.996680+0000 2024-08-18T16:19:23.339 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.333+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:33]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:33]:entry[obj00000000000000000019]: success 2024-08-18T16:19:23.339 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.333+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:33]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:33]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:33:6 marker=00000000002.114.6 timestamp=2024-08-18T16:19:19.884831+0000 2024-08-18T16:19:23.340 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.334+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:entry[obj00000000000000000045]: success 2024-08-18T16:19:23.340 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.334+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:19:6 marker=00000000002.106.6 timestamp=2024-08-18T16:19:20.340703+0000 2024-08-18T16:19:23.340 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.335+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:35]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:35]:entry[obj00000000000000000018]: success 2024-08-18T16:19:23.340 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.335+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:35]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:35]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:35:6 marker=00000000002.93.6 timestamp=2024-08-18T16:19:20.284525+0000 2024-08-18T16:19:23.340 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.335+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:32]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:32]:entry[obj00000000000000000014]: success 2024-08-18T16:19:23.340 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.335+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:32]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:32]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:32:6 marker=00000000002.87.6 timestamp=2024-08-18T16:19:20.396690+0000 2024-08-18T16:19:23.340 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.336+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:31]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:31]:entry[obj00000000000000000003]: success 2024-08-18T16:19:23.340 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.336+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:31]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:31]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:31:6 marker=00000000002.142.6 timestamp=2024-08-18T16:19:20.172927+0000 2024-08-18T16:19:23.341 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.336+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:36]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:36]:entry[obj00000000000000000008]: success 2024-08-18T16:19:23.341 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.336+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:36]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:36]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:36:6 marker=00000000002.161.6 timestamp=2024-08-18T16:19:19.536516+0000 2024-08-18T16:19:23.350 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.350+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: parsed entry: id=00000000002.108.6 iter->object=obj00000000000000000011 iter->instance= name=obj00000000000000000011 instance= ns= 2024-08-18T16:19:23.351 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.350+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2/obj00000000000000000011 2024-08-18T16:19:23.351 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.350+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: entry->timestamp=2024-08-18T16:19:19.775284+0000 2024-08-18T16:19:23.351 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.350+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:entry[obj00000000000000000011]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.108.6 op=0 op_state=1 2024-08-18T16:19:23.351 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.350+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:entry[obj00000000000000000011]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000011[0] 2024-08-18T16:19:23.351 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.350+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: listing bilog for incremental sync; position=00000000002.108.6 2024-08-18T16:19:23.352 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.351+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:entry[obj00000000000000000011]: success 2024-08-18T16:19:23.352 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.351+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:2:6 marker=00000000002.108.6 timestamp=2024-08-18T16:19:19.775284+0000 2024-08-18T16:19:23.355 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.355+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: parsed entry: id=00000000002.163.6 iter->object=obj00000000000000000026 iter->instance= name=obj00000000000000000026 instance= ns= 2024-08-18T16:19:23.355 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.355+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10/obj00000000000000000026 2024-08-18T16:19:23.355 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.355+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: entry->timestamp=2024-08-18T16:19:20.238134+0000 2024-08-18T16:19:23.355 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.355+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:entry[obj00000000000000000026]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.163.6 op=0 op_state=1 2024-08-18T16:19:23.355 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.355+0000 7fbf28131880 0 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:entry[obj00000000000000000026]: : injecting data sync error on key=obj00000000000000000026 2024-08-18T16:19:23.356 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.355+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: listing bilog for incremental sync; position=00000000002.163.6 2024-08-18T16:19:23.356 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.355+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:entry[obj00000000000000000026]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:19:23.356 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.355+0000 7fbf28131880 0 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:entry[obj00000000000000000026]: ERROR: failed to sync object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10/obj00000000000000000026 2024-08-18T16:19:23.356 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.356+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: parsed entry: id=00000000002.83.6 iter->object=obj00000000000000000015 iter->instance= name=obj00000000000000000015 instance= ns= 2024-08-18T16:19:23.356 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.356+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9/obj00000000000000000015 2024-08-18T16:19:23.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.356+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: entry->timestamp=2024-08-18T16:19:20.126126+0000 2024-08-18T16:19:23.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.356+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:entry[obj00000000000000000015]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.83.6 op=0 op_state=1 2024-08-18T16:19:23.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.356+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:entry[obj00000000000000000015]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000015[0] 2024-08-18T16:19:23.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.356+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: listing bilog for incremental sync; position=00000000002.83.6 2024-08-18T16:19:23.358 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.357+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:entry[obj00000000000000000015]: success 2024-08-18T16:19:23.358 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.357+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:9:6 marker=00000000002.83.6 timestamp=2024-08-18T16:19:20.126126+0000 2024-08-18T16:19:23.358 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.358+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: parsed entry: id=00000000001.112.6 iter->object=obj00000000000000000062 iter->instance= name=obj00000000000000000062 instance= ns= 2024-08-18T16:19:23.358 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.358+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12/obj00000000000000000062 2024-08-18T16:19:23.358 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.358+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: entry->timestamp=2024-08-18T16:19:20.061021+0000 2024-08-18T16:19:23.358 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.358+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:entry[obj00000000000000000062]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.112.6 op=0 op_state=1 2024-08-18T16:19:23.359 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.358+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:entry[obj00000000000000000062]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000062[0] 2024-08-18T16:19:23.359 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.358+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: listing bilog for incremental sync; position=00000000001.112.6 2024-08-18T16:19:23.363 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.363+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]: parsed entry: id=00000000002.90.6 iter->object=obj00000000000000000009 iter->instance= name=obj00000000000000000009 instance= ns= 2024-08-18T16:19:23.364 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.363+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13/obj00000000000000000009 2024-08-18T16:19:23.364 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.363+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]: entry->timestamp=2024-08-18T16:19:20.069388+0000 2024-08-18T16:19:23.364 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.363+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:entry[obj00000000000000000009]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.90.6 op=0 op_state=1 2024-08-18T16:19:23.364 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.363+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: parsed entry: id=00000000002.91.6 iter->object=obj00000000000000000012 iter->instance= name=obj00000000000000000012 instance= ns= 2024-08-18T16:19:23.364 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.363+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16/obj00000000000000000012 2024-08-18T16:19:23.364 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.363+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: entry->timestamp=2024-08-18T16:19:19.602880+0000 2024-08-18T16:19:23.364 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.363+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:entry[obj00000000000000000012]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.91.6 op=0 op_state=1 2024-08-18T16:19:23.364 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.363+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: parsed entry: id=00000000002.165.6 iter->object=obj00000000000000000040 iter->instance= name=obj00000000000000000040 instance= ns= 2024-08-18T16:19:23.364 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.363+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14/obj00000000000000000040 2024-08-18T16:19:23.364 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.363+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: entry->timestamp=2024-08-18T16:19:20.293593+0000 2024-08-18T16:19:23.364 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.363+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:entry[obj00000000000000000040]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.165.6 op=0 op_state=1 2024-08-18T16:19:23.365 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.363+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:entry[obj00000000000000000009]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000009[0] 2024-08-18T16:19:23.365 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.363+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]: listing bilog for incremental sync; position=00000000002.90.6 2024-08-18T16:19:23.365 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.363+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:entry[obj00000000000000000012]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000012[0] 2024-08-18T16:19:23.365 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.363+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: listing bilog for incremental sync; position=00000000002.91.6 2024-08-18T16:19:23.365 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.363+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:entry[obj00000000000000000040]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000040[0] 2024-08-18T16:19:23.365 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.363+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: listing bilog for incremental sync; position=00000000002.165.6 2024-08-18T16:19:23.365 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.363+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:entry[obj00000000000000000065]: success 2024-08-18T16:19:23.365 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.363+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:5:6 marker=00000000001.81.6 timestamp=2024-08-18T16:19:19.716103+0000 2024-08-18T16:19:23.366 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.364+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]: parsed entry: id=00000000002.94.6 iter->object=obj00000000000000000006 iter->instance= name=obj00000000000000000006 instance= ns= 2024-08-18T16:19:23.366 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.364+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20/obj00000000000000000006 2024-08-18T16:19:23.366 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.364+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]: entry->timestamp=2024-08-18T16:19:19.949473+0000 2024-08-18T16:19:23.366 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.364+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:entry[obj00000000000000000006]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.94.6 op=0 op_state=1 2024-08-18T16:19:23.366 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.364+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:entry[obj00000000000000000006]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000006[0] 2024-08-18T16:19:23.366 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.364+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]: listing bilog for incremental sync; position=00000000002.94.6 2024-08-18T16:19:23.366 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.365+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25]: parsed entry: id=00000000002.114.6 iter->object=obj00000000000000000010 iter->instance= name=obj00000000000000000010 instance= ns= 2024-08-18T16:19:23.366 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.365+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25/obj00000000000000000010 2024-08-18T16:19:23.366 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.365+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25]: entry->timestamp=2024-08-18T16:19:20.182223+0000 2024-08-18T16:19:23.367 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.365+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25]:entry[obj00000000000000000010]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.114.6 op=0 op_state=1 2024-08-18T16:19:23.367 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.365+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25]:entry[obj00000000000000000010]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000010[0] 2024-08-18T16:19:23.367 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.365+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25]: listing bilog for incremental sync; position=00000000002.114.6 2024-08-18T16:19:23.367 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.366+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:entry[obj00000000000000000009]: success 2024-08-18T16:19:23.367 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.366+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:13:6 marker=00000000002.90.6 timestamp=2024-08-18T16:19:20.069388+0000 2024-08-18T16:19:23.368 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.366+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:entry[obj00000000000000000040]: success 2024-08-18T16:19:23.368 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.366+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:14:6 marker=00000000002.165.6 timestamp=2024-08-18T16:19:20.293593+0000 2024-08-18T16:19:23.368 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.366+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: parsed entry: id=00000000002.95.6 iter->object=obj00000000000000000024 iter->instance= name=obj00000000000000000024 instance= ns= 2024-08-18T16:19:23.368 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.366+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15/obj00000000000000000024 2024-08-18T16:19:23.368 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.366+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: entry->timestamp=2024-08-18T16:19:19.704871+0000 2024-08-18T16:19:23.368 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.366+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:entry[obj00000000000000000024]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.95.6 op=0 op_state=1 2024-08-18T16:19:23.368 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.366+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:entry[obj00000000000000000024]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000024[0] 2024-08-18T16:19:23.368 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.366+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: listing bilog for incremental sync; position=00000000002.95.6 2024-08-18T16:19:23.369 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.368+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:entry[obj00000000000000000012]: success 2024-08-18T16:19:23.369 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.368+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:16:6 marker=00000000002.91.6 timestamp=2024-08-18T16:19:19.602880+0000 2024-08-18T16:19:23.369 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.368+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:entry[obj00000000000000000006]: success 2024-08-18T16:19:23.369 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.368+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:20:6 marker=00000000002.94.6 timestamp=2024-08-18T16:19:19.949473+0000 2024-08-18T16:19:23.370 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.368+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:entry[obj00000000000000000048]: success 2024-08-18T16:19:23.370 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.369+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:8:6 marker=00000000001.170.6 timestamp=2024-08-18T16:19:19.833816+0000 2024-08-18T16:19:23.370 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.369+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:entry[obj00000000000000000024]: success 2024-08-18T16:19:23.370 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.369+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:15:6 marker=00000000002.95.6 timestamp=2024-08-18T16:19:19.704871+0000 2024-08-18T16:19:23.370 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.369+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]: parsed entry: id=00000000002.100.6 iter->object=obj00000000000000000053 iter->instance= name=obj00000000000000000053 instance= ns= 2024-08-18T16:19:23.370 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.369+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27/obj00000000000000000053 2024-08-18T16:19:23.370 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.369+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]: entry->timestamp=2024-08-18T16:19:19.482927+0000 2024-08-18T16:19:23.370 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.369+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]:entry[obj00000000000000000053]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.100.6 op=0 op_state=1 2024-08-18T16:19:23.371 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.369+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25]:entry[obj00000000000000000010]: success 2024-08-18T16:19:23.371 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.369+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:25:6 marker=00000000002.114.6 timestamp=2024-08-18T16:19:20.182223+0000 2024-08-18T16:19:23.371 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.369+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]:entry[obj00000000000000000053]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000053[0] 2024-08-18T16:19:23.371 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.369+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]: listing bilog for incremental sync; position=00000000002.100.6 2024-08-18T16:19:23.371 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.370+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]: parsed entry: id=00000000002.97.6 iter->object=obj00000000000000000004 iter->instance= name=obj00000000000000000004 instance= ns= 2024-08-18T16:19:23.371 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.370+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29/obj00000000000000000004 2024-08-18T16:19:23.371 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.370+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]: entry->timestamp=2024-08-18T16:19:19.764610+0000 2024-08-18T16:19:23.371 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.370+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]:entry[obj00000000000000000004]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.97.6 op=0 op_state=1 2024-08-18T16:19:23.372 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.370+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]:entry[obj00000000000000000004]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000004[0] 2024-08-18T16:19:23.372 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.370+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]: listing bilog for incremental sync; position=00000000002.97.6 2024-08-18T16:19:23.372 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.372+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]:entry[obj00000000000000000004]: success 2024-08-18T16:19:23.372 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.372+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:29:6 marker=00000000002.97.6 timestamp=2024-08-18T16:19:19.764610+0000 2024-08-18T16:19:23.376 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.376+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:34]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:34]: a sync operation returned error 2024-08-18T16:19:23.376 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.376+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:34]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:34]: backing out with sync_status=-5 2024-08-18T16:19:23.376 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.376+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:34]: incremental sync on bucket failed, retcode=-5 2024-08-18T16:19:23.376 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.376+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:34]: ERROR: incremental sync failed. error: -5 2024-08-18T16:19:23.377 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.376+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:34]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:23.377 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.376+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:34]: sync status for source bucket shard: 0 2024-08-18T16:19:23.377 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.376+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:34]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:34]: listing bilog for incremental sync; position= 2024-08-18T16:19:23.378 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.378+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:30]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:30]:entry[obj00000000000000000036]: success 2024-08-18T16:19:23.378 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.378+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:30]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:30]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:30:6 marker=00000000001.86.6 timestamp=2024-08-18T16:19:20.013736+0000 2024-08-18T16:19:23.397 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.397+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: a sync operation returned error 2024-08-18T16:19:23.397 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.397+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: backing out with sync_status=-5 2024-08-18T16:19:23.397 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.397+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: incremental sync on bucket failed, retcode=-5 2024-08-18T16:19:23.397 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.397+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: ERROR: incremental sync failed. error: -5 2024-08-18T16:19:23.398 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.397+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:19:23.398 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.397+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: sync status for source bucket shard: 0 2024-08-18T16:19:23.398 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.397+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: listing bilog for incremental sync; position= 2024-08-18T16:19:23.403 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.402+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:entry[obj00000000000000000062]: success 2024-08-18T16:19:23.403 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.402+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:12:6 marker=00000000001.112.6 timestamp=2024-08-18T16:19:20.061021+0000 2024-08-18T16:19:23.415 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.415+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]:entry[obj00000000000000000053]: success 2024-08-18T16:19:23.415 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.415+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:27:6 marker=00000000002.100.6 timestamp=2024-08-18T16:19:19.482927+0000 2024-08-18T16:19:23.418 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.418+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:34]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:34]: parsed entry: id=00000000001.94.6 iter->object=obj00000000000000000057 iter->instance= name=obj00000000000000000057 instance= ns= 2024-08-18T16:19:23.418 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.418+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:34]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:34]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:34/obj00000000000000000057 2024-08-18T16:19:23.418 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.418+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:34]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:34]: entry->timestamp=2024-08-18T16:19:19.657944+0000 2024-08-18T16:19:23.418 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.418+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:34]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:34]:entry[obj00000000000000000057]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000001.94.6 op=0 op_state=1 2024-08-18T16:19:23.418 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.418+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:34]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:34]:entry[obj00000000000000000057]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000057[0] 2024-08-18T16:19:23.418 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.418+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:34]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:34]: listing bilog for incremental sync; position=00000000001.94.6 2024-08-18T16:19:23.439 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.439+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: parsed entry: id=00000000002.163.6 iter->object=obj00000000000000000026 iter->instance= name=obj00000000000000000026 instance= ns= 2024-08-18T16:19:23.439 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.439+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: syncing object: kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10/obj00000000000000000026 2024-08-18T16:19:23.440 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.439+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: entry->timestamp=2024-08-18T16:19:20.238134+0000 2024-08-18T16:19:23.440 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.439+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:entry[obj00000000000000000026]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=00000000002.163.6 op=0 op_state=1 2024-08-18T16:19:23.440 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.439+0000 7fbf28131880 5 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:entry[obj00000000000000000026]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-77[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25])/obj00000000000000000026[0] 2024-08-18T16:19:23.440 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.439+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: listing bilog for incremental sync; position=00000000002.163.6 2024-08-18T16:19:23.440 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.440+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:entry[obj00000000000000000026]: success 2024-08-18T16:19:23.440 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.440+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:10:6 marker=00000000002.163.6 timestamp=2024-08-18T16:19:20.238134+0000 2024-08-18T16:19:23.462 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.462+0000 7fbf28131880 10 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:34]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:34]:entry[obj00000000000000000057]: success 2024-08-18T16:19:23.462 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:23.462+0000 7fbf28131880 20 RGW-SYNC:bucket[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25<-kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:34]:inc_sync[kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:34]: updating marker marker_oid=bucket.sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-77:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.25:34:6 marker=00000000001.94.6 timestamp=2024-08-18T16:19:19.657944+0000 2024-08-18T16:19:23.502 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'list', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-77', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:19:23.502 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'list', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-77', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:19:23.502 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket list --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-77 --max-entries 10000 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:19:23.592 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:19:23.592 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:19:23.625 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T16:19:23.625 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.626 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000000", 2024-08-18T16:19:23.626 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.626 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.626 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:23.626 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 27 2024-08-18T16:19:23.626 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.626 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.626 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.626 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.626 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.626 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 5, 2024-08-18T16:19:23.626 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:19.893792Z", 2024-08-18T16:19:23.627 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.627 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.627 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.627 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.627 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.627 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.627 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.627 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.627 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.627 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.12694390134036672627", 2024-08-18T16:19:23.627 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.627 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.627 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.628 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.628 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.628 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000001", 2024-08-18T16:19:23.628 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.628 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.628 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:23.628 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 29 2024-08-18T16:19:23.628 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.628 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.628 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.628 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.628 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.629 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:19:23.629 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:20.116900Z", 2024-08-18T16:19:23.629 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.629 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.629 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.629 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.629 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.629 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.629 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.629 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.629 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.629 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.1775643464440672235", 2024-08-18T16:19:23.630 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.630 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.630 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.630 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.630 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.630 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000002", 2024-08-18T16:19:23.630 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.630 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.630 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:23.630 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 27 2024-08-18T16:19:23.630 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.630 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.630 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.631 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.631 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.631 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:19:23.631 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:19.940549Z", 2024-08-18T16:19:23.631 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.631 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.631 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.631 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.631 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.631 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.631 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.631 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.631 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.632 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.793237549756425130", 2024-08-18T16:19:23.632 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.632 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.632 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.632 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.632 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.632 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000003", 2024-08-18T16:19:23.632 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.632 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.632 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:23.632 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 11 2024-08-18T16:19:23.633 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.633 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.633 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.633 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.633 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.633 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 6, 2024-08-18T16:19:23.633 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:20.172927Z", 2024-08-18T16:19:23.633 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.633 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.633 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.633 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.633 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.633 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.634 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.634 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.634 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.634 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.7219353317483614029", 2024-08-18T16:19:23.634 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.634 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.634 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.634 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.634 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.634 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000004", 2024-08-18T16:19:23.634 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.634 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.635 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:23.635 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 20 2024-08-18T16:19:23.635 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.635 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.635 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.635 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.635 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.635 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-08-18T16:19:23.635 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:19.764610Z", 2024-08-18T16:19:23.635 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.635 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.635 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.636 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.636 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.636 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.636 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.636 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.636 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.636 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.9740686867748003437", 2024-08-18T16:19:23.636 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.636 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.636 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.636 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.636 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.636 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000005", 2024-08-18T16:19:23.637 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.637 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.637 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:23.637 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 17 2024-08-18T16:19:23.637 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.637 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.637 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.637 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.637 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.637 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-08-18T16:19:23.637 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:20.349953Z", 2024-08-18T16:19:23.637 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.637 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.638 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.638 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.638 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.638 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.638 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.638 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.638 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.638 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.14675195546044874488", 2024-08-18T16:19:23.638 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.638 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.638 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.638 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.639 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.639 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000006", 2024-08-18T16:19:23.639 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.639 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.639 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:23.639 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 15 2024-08-18T16:19:23.639 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.639 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.639 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.639 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.639 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.640 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 6, 2024-08-18T16:19:23.640 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:19.949473Z", 2024-08-18T16:19:23.640 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.640 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.640 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.640 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.640 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.640 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.640 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.640 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.640 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.640 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.5927078113473162591", 2024-08-18T16:19:23.640 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.640 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.641 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.641 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.641 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.641 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000007", 2024-08-18T16:19:23.641 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.641 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.641 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:23.641 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 26 2024-08-18T16:19:23.641 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.641 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.641 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.641 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.642 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.642 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:19:23.642 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:19.491062Z", 2024-08-18T16:19:23.642 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.642 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.642 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.642 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.642 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.642 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.642 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.642 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.642 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.643 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.6096458032678042161", 2024-08-18T16:19:23.643 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.643 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.643 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.643 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.643 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.643 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000008", 2024-08-18T16:19:23.643 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.643 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.643 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:23.643 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 12 2024-08-18T16:19:23.643 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.643 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.643 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.644 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.644 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.644 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 5, 2024-08-18T16:19:23.644 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:19.536516Z", 2024-08-18T16:19:23.644 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.644 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.644 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.644 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.644 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.644 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.644 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.644 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.644 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.645 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.8062311990328899558", 2024-08-18T16:19:23.645 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.645 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.645 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.645 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.645 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.645 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000009", 2024-08-18T16:19:23.645 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.645 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.645 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:23.645 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 15 2024-08-18T16:19:23.645 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.645 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.646 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.646 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.646 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.646 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 5, 2024-08-18T16:19:23.646 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:20.069388Z", 2024-08-18T16:19:23.646 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.646 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.646 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.646 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.646 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.646 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.646 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.646 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.647 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.647 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.14028552126077933249", 2024-08-18T16:19:23.647 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.647 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.647 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.647 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.647 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.647 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000010", 2024-08-18T16:19:23.647 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.647 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.647 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:23.647 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 26 2024-08-18T16:19:23.647 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.648 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.648 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.648 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.648 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.648 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 6, 2024-08-18T16:19:23.648 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:20.182223Z", 2024-08-18T16:19:23.648 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.648 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.648 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.648 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.648 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.648 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.648 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.649 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.649 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.649 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.3142064944146292633", 2024-08-18T16:19:23.649 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.649 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.649 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.649 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.649 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.649 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000011", 2024-08-18T16:19:23.649 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.649 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.649 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:23.650 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 21 2024-08-18T16:19:23.650 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.650 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.650 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.650 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.650 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.650 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:19:23.650 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:19.775284Z", 2024-08-18T16:19:23.650 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.650 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.650 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.650 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.650 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.651 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.651 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.651 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.651 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.651 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.12809818972740143366", 2024-08-18T16:19:23.651 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.651 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.651 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.651 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.651 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.651 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000012", 2024-08-18T16:19:23.651 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.651 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.652 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:23.652 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 14 2024-08-18T16:19:23.652 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.652 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.652 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.652 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.652 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.652 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-08-18T16:19:23.652 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:19.602880Z", 2024-08-18T16:19:23.652 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.652 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.652 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.652 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.652 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.652 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.653 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.653 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.653 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.653 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.17262977424700118057", 2024-08-18T16:19:23.653 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.653 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.653 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.653 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.653 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.653 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000013", 2024-08-18T16:19:23.653 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.653 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.653 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:23.654 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 22 2024-08-18T16:19:23.654 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.654 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.654 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.654 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.654 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.654 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:19:23.654 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:20.004845Z", 2024-08-18T16:19:23.654 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.654 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.654 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.654 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.655 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.655 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.655 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.655 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.655 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.655 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.10984888302709676165", 2024-08-18T16:19:23.655 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.655 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.655 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.655 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.655 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.655 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000014", 2024-08-18T16:19:23.655 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.655 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.656 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:23.656 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 11 2024-08-18T16:19:23.656 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.656 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.656 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.656 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.656 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.656 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 5, 2024-08-18T16:19:23.656 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:20.396690Z", 2024-08-18T16:19:23.656 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.656 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.656 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.656 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.656 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.657 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.657 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.657 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.657 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.657 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.6617372424373138853", 2024-08-18T16:19:23.657 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.657 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.657 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.657 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.657 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.657 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000015", 2024-08-18T16:19:23.657 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.657 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.658 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:23.658 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 19 2024-08-18T16:19:23.658 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.658 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.658 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.658 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.658 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.658 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 5, 2024-08-18T16:19:23.658 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:20.126126Z", 2024-08-18T16:19:23.658 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.658 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.658 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.659 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.659 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.659 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.659 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.659 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.659 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.659 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.11736278763128699882", 2024-08-18T16:19:23.659 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.659 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.659 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.659 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.659 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.659 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000016", 2024-08-18T16:19:23.660 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.660 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.660 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:23.660 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 15 2024-08-18T16:19:23.660 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.660 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.660 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.660 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.660 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.660 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:19:23.660 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:20.228897Z", 2024-08-18T16:19:23.660 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.660 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.660 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.661 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.661 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.661 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.661 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.661 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.661 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.661 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.5916908414622314875", 2024-08-18T16:19:23.661 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.661 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.661 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.661 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.661 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.661 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000017", 2024-08-18T16:19:23.662 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.662 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.662 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:23.662 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 5 2024-08-18T16:19:23.663 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.663 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.663 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.663 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.663 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.663 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-08-18T16:19:23.663 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:18.792258Z", 2024-08-18T16:19:23.663 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.663 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.663 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.663 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.663 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.663 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.664 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.664 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.664 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.664 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.9410209081733305536", 2024-08-18T16:19:23.664 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.664 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.664 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.664 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.664 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.664 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000018", 2024-08-18T16:19:23.664 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.664 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.664 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:23.665 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 14 2024-08-18T16:19:23.665 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.665 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.665 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.665 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.665 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.665 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 6, 2024-08-18T16:19:23.665 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:20.284525Z", 2024-08-18T16:19:23.665 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.665 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.665 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.665 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.665 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.665 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.666 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.666 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.666 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.666 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.5742324590714405983", 2024-08-18T16:19:23.666 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.666 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.666 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.666 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.666 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.666 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000019", 2024-08-18T16:19:23.666 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.666 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.666 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:23.667 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 16 2024-08-18T16:19:23.667 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.667 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.667 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.667 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.667 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.667 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:19:23.667 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:19.884831Z", 2024-08-18T16:19:23.667 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.667 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.667 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.667 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.667 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.668 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.668 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.668 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.668 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.668 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.4545522631134190464", 2024-08-18T16:19:23.668 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.668 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.668 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.668 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.668 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.668 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000020", 2024-08-18T16:19:23.668 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.668 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.669 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:23.669 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 7 2024-08-18T16:19:23.669 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.669 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.669 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.669 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.669 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.669 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 6, 2024-08-18T16:19:23.669 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:17.856631Z", 2024-08-18T16:19:23.669 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.669 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.669 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.669 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.669 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.670 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.670 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.670 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.670 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.670 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.15037230926380855934", 2024-08-18T16:19:23.670 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.670 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.670 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.670 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.670 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.670 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000021", 2024-08-18T16:19:23.670 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.670 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.671 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:23.671 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 24 2024-08-18T16:19:23.671 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.671 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.671 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.671 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.671 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.671 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 5, 2024-08-18T16:19:23.671 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:19.648761Z", 2024-08-18T16:19:23.671 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.671 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.671 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.671 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.672 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.672 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.672 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.672 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.672 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.672 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.17107778765257565441", 2024-08-18T16:19:23.672 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.672 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.672 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.672 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.672 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.672 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000022", 2024-08-18T16:19:23.672 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.672 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.673 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:23.673 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 8 2024-08-18T16:19:23.673 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.673 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.673 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.673 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.673 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.673 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 5, 2024-08-18T16:19:23.673 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:18.551431Z", 2024-08-18T16:19:23.673 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.673 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.673 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.673 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.673 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.674 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.674 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.674 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.674 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.674 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.16936946981126341225", 2024-08-18T16:19:23.674 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.674 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.674 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.674 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.674 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.674 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000023", 2024-08-18T16:19:23.674 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.675 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.675 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:23.675 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 6 2024-08-18T16:19:23.675 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.675 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.675 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.675 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.675 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.675 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:19:23.675 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:18.904686Z", 2024-08-18T16:19:23.675 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.675 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.675 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.676 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.676 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.676 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.676 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.676 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.676 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.676 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.13289786810085350536", 2024-08-18T16:19:23.676 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.676 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.676 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.676 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.676 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.676 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000024", 2024-08-18T16:19:23.676 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.677 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.677 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:23.677 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 7 2024-08-18T16:19:23.677 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.677 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.677 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.677 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.677 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.677 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-08-18T16:19:23.677 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:19.704871Z", 2024-08-18T16:19:23.677 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.677 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.677 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.678 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.678 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.678 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.678 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.678 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.678 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.678 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.9646989395737438153", 2024-08-18T16:19:23.678 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.678 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.679 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.679 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.679 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.679 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000025", 2024-08-18T16:19:23.679 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.679 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.679 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:23.679 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 12 2024-08-18T16:19:23.679 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.679 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.679 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.679 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.679 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.680 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:19:23.680 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:17.792369Z", 2024-08-18T16:19:23.680 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.680 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.680 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.680 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.680 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.680 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.680 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.680 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.680 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.680 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.5464753145558169266", 2024-08-18T16:19:23.680 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.680 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.681 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.681 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.681 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.681 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000026", 2024-08-18T16:19:23.681 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.681 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.681 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:23.681 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 28 2024-08-18T16:19:23.681 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.681 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.681 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.681 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.681 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.681 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 6, 2024-08-18T16:19:23.682 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:20.238134Z", 2024-08-18T16:19:23.682 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.682 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.682 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.682 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.682 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.682 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.682 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.682 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.682 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.682 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.14265561417317866991", 2024-08-18T16:19:23.682 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.683 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.683 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.683 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.683 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.683 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000027", 2024-08-18T16:19:23.683 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.683 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.683 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:23.683 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 15 2024-08-18T16:19:23.683 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.684 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.684 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.684 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.684 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.684 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 5, 2024-08-18T16:19:23.684 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:18.160384Z", 2024-08-18T16:19:23.684 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.684 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.684 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.684 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.684 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.684 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.684 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.685 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.685 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.685 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.6696940155157261459", 2024-08-18T16:19:23.685 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.685 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.685 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.685 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.685 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.685 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000028", 2024-08-18T16:19:23.685 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.685 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.685 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:23.685 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 19 2024-08-18T16:19:23.686 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.686 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.686 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.686 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.686 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.686 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:19:23.686 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:15.858769Z", 2024-08-18T16:19:23.686 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.686 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.686 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.686 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.686 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.686 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.687 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.687 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.687 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.687 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.8082080429937621815", 2024-08-18T16:19:23.687 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.687 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.687 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.687 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.687 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.687 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000029", 2024-08-18T16:19:23.687 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.687 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.687 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:23.688 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 13 2024-08-18T16:19:23.688 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.688 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.688 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.688 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.688 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.688 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 5, 2024-08-18T16:19:23.688 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:18.977845Z", 2024-08-18T16:19:23.688 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.688 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.688 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.688 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.689 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.689 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.689 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.689 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.689 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.689 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.2095082254158583061", 2024-08-18T16:19:23.689 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.689 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.689 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.689 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.689 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.689 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000030", 2024-08-18T16:19:23.689 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.690 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.690 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:23.690 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 9 2024-08-18T16:19:23.690 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.690 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.690 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.690 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.690 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.690 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:19:23.690 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:19.545863Z", 2024-08-18T16:19:23.690 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.691 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.691 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.691 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.691 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.691 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.691 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.691 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.691 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.691 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.6389725217193624987", 2024-08-18T16:19:23.691 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.691 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.691 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.691 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.691 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.692 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000031", 2024-08-18T16:19:23.692 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.692 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.692 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:23.692 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 14 2024-08-18T16:19:23.692 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.692 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.692 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.692 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.692 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.692 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:19:23.692 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:17.912320Z", 2024-08-18T16:19:23.692 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.693 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.693 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.693 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.693 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.693 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.693 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.693 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.693 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.693 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.1323283957345927255", 2024-08-18T16:19:23.693 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.693 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.693 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.693 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.693 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.694 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000032", 2024-08-18T16:19:23.694 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.694 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.694 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:23.694 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 17 2024-08-18T16:19:23.694 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.694 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.694 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.694 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.694 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.694 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-08-18T16:19:23.695 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:17.116603Z", 2024-08-18T16:19:23.695 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.695 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.695 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.695 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.695 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.695 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.695 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.695 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.695 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.695 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.2466765093447699000", 2024-08-18T16:19:23.695 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.695 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.695 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.696 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.696 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.696 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000033", 2024-08-18T16:19:23.696 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.696 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.696 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:23.696 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 25 2024-08-18T16:19:23.696 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.696 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.696 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.696 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.696 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.696 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 6, 2024-08-18T16:19:23.696 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:19.593080Z", 2024-08-18T16:19:23.697 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.697 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.697 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.697 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.697 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.697 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.697 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.697 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.697 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.697 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.15803267565377777934", 2024-08-18T16:19:23.697 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.697 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.697 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.698 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.698 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.698 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000036", 2024-08-18T16:19:23.698 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.698 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.698 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:23.698 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 10 2024-08-18T16:19:23.698 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.698 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.698 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.698 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.698 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.698 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:19:23.698 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:20.013736Z", 2024-08-18T16:19:23.699 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.699 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.699 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.699 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.699 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.699 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.699 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.699 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.699 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.699 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.7641068590117292671", 2024-08-18T16:19:23.699 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.699 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.699 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.699 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.700 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.700 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000037", 2024-08-18T16:19:23.700 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.700 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.700 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:23.700 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 14 2024-08-18T16:19:23.700 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.700 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.700 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.700 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.700 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.700 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:19:23.700 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:17.746196Z", 2024-08-18T16:19:23.700 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.701 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.701 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.701 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.701 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.701 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.701 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.701 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.701 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.702 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.3958915118237578826", 2024-08-18T16:19:23.702 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.702 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.702 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.702 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.702 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.702 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000039", 2024-08-18T16:19:23.702 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.702 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.702 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:23.702 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 16 2024-08-18T16:19:23.702 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.702 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.702 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.703 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.703 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.703 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 6, 2024-08-18T16:19:23.703 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:19.824786Z", 2024-08-18T16:19:23.703 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.703 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.703 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.703 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.703 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.703 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.703 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.703 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.703 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.704 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.17774398092704509620", 2024-08-18T16:19:23.704 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.704 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.704 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.704 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.704 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.704 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000040", 2024-08-18T16:19:23.704 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.704 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.704 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:23.704 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 9 2024-08-18T16:19:23.704 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.704 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.704 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.705 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.705 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.705 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 5, 2024-08-18T16:19:23.705 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:20.293593Z", 2024-08-18T16:19:23.705 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.705 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.705 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.705 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.705 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.705 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.705 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.705 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.705 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.705 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.14997593642927215052", 2024-08-18T16:19:23.706 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.706 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.706 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.706 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.706 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.706 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000042", 2024-08-18T16:19:23.706 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.706 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.706 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:23.706 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 7 2024-08-18T16:19:23.706 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.706 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.706 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.706 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.707 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.707 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:19:23.707 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:18.280618Z", 2024-08-18T16:19:23.707 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.707 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.707 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.707 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.707 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.707 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.707 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.707 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.707 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.707 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.7641885035429566441", 2024-08-18T16:19:23.708 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.708 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.708 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.708 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.708 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.708 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000045", 2024-08-18T16:19:23.708 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.708 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.708 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:23.708 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 12 2024-08-18T16:19:23.708 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.708 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.708 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.708 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.709 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.709 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-08-18T16:19:23.709 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:20.340703Z", 2024-08-18T16:19:23.709 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.709 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.709 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.709 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.709 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.709 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.709 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.709 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.709 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.710 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.10308047693451003210", 2024-08-18T16:19:23.710 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.710 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.710 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.710 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.710 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.710 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000046", 2024-08-18T16:19:23.710 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.710 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.710 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:23.710 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 5 2024-08-18T16:19:23.710 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.710 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.710 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.711 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.711 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.711 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-08-18T16:19:23.711 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:17.181145Z", 2024-08-18T16:19:23.711 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.711 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.711 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.711 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.711 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.711 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.711 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.711 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.711 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.711 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.7654529219595040550", 2024-08-18T16:19:23.712 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.712 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.712 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.712 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.712 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.712 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000048", 2024-08-18T16:19:23.712 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.712 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.712 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:23.712 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 8 2024-08-18T16:19:23.712 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.712 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.712 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.713 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.713 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.713 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:19:23.713 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:19.833816Z", 2024-08-18T16:19:23.713 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.713 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.713 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.713 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.713 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.713 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.713 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.713 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.713 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.713 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.15237095668940641587", 2024-08-18T16:19:23.714 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.714 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.714 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.714 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.714 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.714 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000049", 2024-08-18T16:19:23.714 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.714 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.714 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:23.714 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 10 2024-08-18T16:19:23.714 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.714 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.714 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.715 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.715 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.715 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:19:23.715 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:17.984613Z", 2024-08-18T16:19:23.715 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.715 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.715 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.715 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.715 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.715 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.715 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.715 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.715 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.715 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.2272497271796380264", 2024-08-18T16:19:23.716 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.716 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.716 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.716 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.716 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.716 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000050", 2024-08-18T16:19:23.716 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.716 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.716 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:23.716 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 13 2024-08-18T16:19:23.716 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.716 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.716 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.717 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.717 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.717 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:19:23.717 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:17.125461Z", 2024-08-18T16:19:23.717 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.717 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.717 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.717 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.717 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.717 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.717 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.717 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.717 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.718 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.4477381748113686257", 2024-08-18T16:19:23.718 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.718 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.718 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.718 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.718 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.718 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000053", 2024-08-18T16:19:23.718 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.718 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.718 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:23.718 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 23 2024-08-18T16:19:23.718 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.718 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.719 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.719 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.719 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.719 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:19:23.719 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:19.482927Z", 2024-08-18T16:19:23.719 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.719 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.719 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.719 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.719 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.719 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.719 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.719 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.719 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.720 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.11468292518739981541", 2024-08-18T16:19:23.720 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.720 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.720 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.720 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.720 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.720 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000055", 2024-08-18T16:19:23.720 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.720 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.720 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:23.720 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 49 2024-08-18T16:19:23.720 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.720 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.720 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.721 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.721 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.721 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 5, 2024-08-18T16:19:23.721 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:17.295995Z", 2024-08-18T16:19:23.721 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.721 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.721 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.721 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.721 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.721 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.721 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.721 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.721 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.721 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.7944227825297655296", 2024-08-18T16:19:23.722 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.722 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.722 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.722 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.722 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.722 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000056", 2024-08-18T16:19:23.722 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.722 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.722 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:23.722 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 10 2024-08-18T16:19:23.722 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.722 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.722 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.723 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.723 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.723 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 6, 2024-08-18T16:19:23.723 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:19.144248Z", 2024-08-18T16:19:23.723 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.723 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.723 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.723 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.723 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.723 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.723 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.723 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.723 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.723 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.5024964257853397233", 2024-08-18T16:19:23.724 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.724 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.724 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.724 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.724 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.724 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000057", 2024-08-18T16:19:23.724 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.724 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.724 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:23.724 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 11 2024-08-18T16:19:23.724 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.724 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.724 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.725 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.725 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.725 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-08-18T16:19:23.725 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:19.657944Z", 2024-08-18T16:19:23.725 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.725 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.725 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.725 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.725 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.725 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.725 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.725 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.725 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.725 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.10146843407256219658", 2024-08-18T16:19:23.725 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.726 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.726 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.726 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.726 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.726 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000058", 2024-08-18T16:19:23.726 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.726 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.726 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:23.726 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 11 2024-08-18T16:19:23.726 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.727 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.727 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.727 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.727 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.727 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:19:23.727 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:17.303274Z", 2024-08-18T16:19:23.727 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.727 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.727 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.727 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.727 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.727 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.727 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.728 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.728 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.728 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.1248125027812078921", 2024-08-18T16:19:23.728 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.728 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.728 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.728 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.728 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.728 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000059", 2024-08-18T16:19:23.728 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.728 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.728 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:23.728 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 9 2024-08-18T16:19:23.729 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.729 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.729 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.729 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.729 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.729 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-08-18T16:19:23.729 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:18.040381Z", 2024-08-18T16:19:23.729 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.729 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.729 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.729 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.729 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.729 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.730 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.730 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.730 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.730 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.1559192484951112292", 2024-08-18T16:19:23.730 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.730 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.730 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.730 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.730 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.730 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000060", 2024-08-18T16:19:23.730 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.730 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.730 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:23.731 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 22 2024-08-18T16:19:23.731 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.731 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.731 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.731 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.731 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.731 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:19:23.731 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:19.996680Z", 2024-08-18T16:19:23.731 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.731 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.731 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.731 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.731 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.731 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.732 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.732 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.732 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.732 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.11751167038022683473", 2024-08-18T16:19:23.732 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.732 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.732 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.732 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.732 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.732 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000061", 2024-08-18T16:19:23.732 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.732 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.732 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:23.732 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 9 2024-08-18T16:19:23.732 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.733 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.733 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.733 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.733 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.733 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-08-18T16:19:23.733 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:18.800570Z", 2024-08-18T16:19:23.733 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.733 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.733 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.733 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.733 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.733 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.733 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.733 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.734 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.734 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.18125434188322097673", 2024-08-18T16:19:23.734 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.734 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.734 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.734 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.734 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.734 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000062", 2024-08-18T16:19:23.734 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.734 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.734 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:23.734 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 7 2024-08-18T16:19:23.734 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.734 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.735 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.735 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.735 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.735 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 5, 2024-08-18T16:19:23.735 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:20.061021Z", 2024-08-18T16:19:23.735 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.735 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.735 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.735 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.735 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.735 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.735 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.735 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.736 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.736 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.8171122739445927861", 2024-08-18T16:19:23.736 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.736 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.736 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.736 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.736 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.736 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000065", 2024-08-18T16:19:23.736 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.736 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.736 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:23.736 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 11 2024-08-18T16:19:23.736 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.736 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.737 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.737 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.737 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.737 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 5, 2024-08-18T16:19:23.737 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:19.716103Z", 2024-08-18T16:19:23.737 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.737 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.737 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.737 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.737 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.737 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.737 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.737 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.737 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.738 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.14973101582796792823", 2024-08-18T16:19:23.738 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.738 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.738 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.738 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.738 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.738 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000076", 2024-08-18T16:19:23.738 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.738 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.738 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:23.738 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 10 2024-08-18T16:19:23.738 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.738 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.738 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.739 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.739 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.739 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-08-18T16:19:23.739 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:15.743831Z", 2024-08-18T16:19:23.739 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.739 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.739 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.739 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.739 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.739 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.739 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.739 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.739 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.740 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.14675623907960268010", 2024-08-18T16:19:23.740 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.740 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.740 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.740 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.740 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.740 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000133", 2024-08-18T16:19:23.740 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.740 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.740 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:23.740 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 22 2024-08-18T16:19:23.740 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.740 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.740 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.741 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.741 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.741 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 5, 2024-08-18T16:19:23.741 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:19.158889Z", 2024-08-18T16:19:23.741 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.741 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.741 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.741 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.741 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.741 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.741 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.741 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.741 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.741 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.7134523735991721996", 2024-08-18T16:19:23.741 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.742 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.742 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.742 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.742 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.742 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000162", 2024-08-18T16:19:23.742 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.742 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.742 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:23.742 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 11 2024-08-18T16:19:23.742 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.742 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.742 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.742 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.743 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.743 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:19:23.743 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:18.617486Z", 2024-08-18T16:19:23.743 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.743 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.743 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.743 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.743 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.743 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.743 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.743 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.743 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.743 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.15849648118366767075", 2024-08-18T16:19:23.743 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.744 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.744 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.744 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.744 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.744 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000271", 2024-08-18T16:19:23.744 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.744 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.744 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:23.744 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 10 2024-08-18T16:19:23.744 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.744 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.744 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.744 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.745 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.745 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 6, 2024-08-18T16:19:23.745 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:18.858093Z", 2024-08-18T16:19:23.745 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.745 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.745 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.745 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.745 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.745 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.745 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.745 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.745 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.745 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.11174342658862842768", 2024-08-18T16:19:23.745 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.746 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.746 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.746 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.746 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.746 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000283", 2024-08-18T16:19:23.746 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.746 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.746 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:23.746 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 9 2024-08-18T16:19:23.746 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.746 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.746 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.746 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.747 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.747 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 5, 2024-08-18T16:19:23.747 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:18.728571Z", 2024-08-18T16:19:23.747 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.747 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.747 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.747 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.747 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.747 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.747 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.747 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.747 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.747 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.12018170805249087996", 2024-08-18T16:19:23.748 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.748 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.749 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.749 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.749 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.749 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000317", 2024-08-18T16:19:23.749 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.749 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.749 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:23.749 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 9 2024-08-18T16:19:23.749 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.750 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.750 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.750 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.750 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.750 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-08-18T16:19:23.750 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:18.784378Z", 2024-08-18T16:19:23.750 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.750 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.750 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.750 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.750 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.750 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.751 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.751 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.751 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.751 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.5571667273815001199", 2024-08-18T16:19:23.751 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.751 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.751 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.751 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.751 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.751 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000423", 2024-08-18T16:19:23.751 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.751 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.751 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:23.752 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 12 2024-08-18T16:19:23.752 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.752 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.752 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.752 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.752 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.752 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-08-18T16:19:23.752 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:18.720502Z", 2024-08-18T16:19:23.752 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.752 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.752 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.752 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.752 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.752 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.753 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.753 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.753 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.753 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.14231938979835901774", 2024-08-18T16:19:23.753 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.753 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.753 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.753 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.753 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.753 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000467", 2024-08-18T16:19:23.753 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.753 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.753 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:19:23.754 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 5 2024-08-18T16:19:23.754 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.754 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.754 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.754 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.754 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.754 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 5, 2024-08-18T16:19:23.754 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:19.151651Z", 2024-08-18T16:19:23.754 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.754 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.754 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.754 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.754 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.755 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.755 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.755 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.755 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.755 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.13528487640187220054", 2024-08-18T16:19:23.755 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.755 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.755 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.755 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:19:23.755 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T16:19:23.756 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'list', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-77', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:19:23.756 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'list', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-77', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:19:23.756 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket list --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-77 --max-entries 10000 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:19:23.814 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:19:23.814 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:19:23.847 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T16:19:23.847 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.847 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000000", 2024-08-18T16:19:23.847 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.847 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.847 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:19:23.847 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 16 2024-08-18T16:19:23.847 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.847 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.847 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.847 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.848 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.848 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-08-18T16:19:23.848 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:19.893792Z", 2024-08-18T16:19:23.848 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.848 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.848 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.848 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.848 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.848 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.848 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.848 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.848 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.848 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_5PmQaN-90qfBHI0fOUMJz-fabPEUefb", 2024-08-18T16:19:23.848 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.849 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.849 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.849 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.849 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.849 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000001", 2024-08-18T16:19:23.849 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.849 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.849 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:19:23.849 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 17 2024-08-18T16:19:23.849 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.849 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.849 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.849 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.850 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.850 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:19:23.850 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:20.116900Z", 2024-08-18T16:19:23.850 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.850 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.850 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.850 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.850 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.850 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.850 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.850 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.850 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.851 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_pejK5qZ7vWhaF0YBpSreMEO-BNdc33U", 2024-08-18T16:19:23.851 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.851 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.851 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.851 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.851 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.851 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000002", 2024-08-18T16:19:23.851 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.851 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.851 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:19:23.851 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 10 2024-08-18T16:19:23.851 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.851 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.851 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.852 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.852 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.852 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 6, 2024-08-18T16:19:23.852 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:19.940549Z", 2024-08-18T16:19:23.852 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.852 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.852 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.852 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.852 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.852 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.852 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.852 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.852 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.853 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_6eMs6KdhqJBfTtY-ZIlzJ9HNSZeccji", 2024-08-18T16:19:23.853 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.853 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.853 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.853 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.853 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.853 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000003", 2024-08-18T16:19:23.853 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.853 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.853 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:19:23.853 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 4 2024-08-18T16:19:23.853 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.853 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.854 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.854 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.854 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.854 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:19:23.854 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:20.172927Z", 2024-08-18T16:19:23.854 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.854 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.854 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.854 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.854 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.854 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.854 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.854 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.855 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.855 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_gHycnPFw7fae-WncoqotBv6v0BSwhml", 2024-08-18T16:19:23.855 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.855 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.855 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.855 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.855 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.855 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000004", 2024-08-18T16:19:23.855 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.855 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.855 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:19:23.855 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 9 2024-08-18T16:19:23.855 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.855 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.856 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.856 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.856 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.856 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 6, 2024-08-18T16:19:23.856 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:19.764610Z", 2024-08-18T16:19:23.856 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.856 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.856 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.856 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.856 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.856 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.856 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.856 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.857 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.857 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_Ck3AcaUN6PfoY5GOMGukNxY_sEjXEJ7", 2024-08-18T16:19:23.857 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.857 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.857 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.857 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.857 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.857 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000005", 2024-08-18T16:19:23.857 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.857 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.857 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:19:23.857 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 7 2024-08-18T16:19:23.857 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.858 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.858 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.858 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.858 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.858 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:19:23.858 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:20.349953Z", 2024-08-18T16:19:23.858 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.858 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.858 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.858 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.858 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.858 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.858 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.858 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.859 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.859 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_1kwW--UdyWj2KRHZ8XxbAXVTL9DOGWr", 2024-08-18T16:19:23.859 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.859 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.859 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.859 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.859 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.859 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000006", 2024-08-18T16:19:23.859 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.859 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.859 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:19:23.859 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 6 2024-08-18T16:19:23.859 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.860 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.860 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.860 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.860 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.860 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:19:23.860 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:19.949473Z", 2024-08-18T16:19:23.860 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.860 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.860 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.860 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.860 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.860 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.861 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.861 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.861 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.861 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_VLsh3mGX8aFC9_-rYou16cB4igKq0-c", 2024-08-18T16:19:23.861 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.861 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.861 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.861 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.861 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.861 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000007", 2024-08-18T16:19:23.861 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.861 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.861 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:19:23.861 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 10 2024-08-18T16:19:23.862 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.862 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.862 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.862 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.862 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.862 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:19:23.862 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:19.491062Z", 2024-08-18T16:19:23.862 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.862 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.862 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.862 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.862 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.863 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.863 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.863 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.863 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.863 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_mbuiQgCGjQT_1ydDutv7eF6TTmiR6lx", 2024-08-18T16:19:23.863 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.863 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.863 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.863 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.863 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.863 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000008", 2024-08-18T16:19:23.863 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.863 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.864 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:19:23.864 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 4 2024-08-18T16:19:23.864 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.864 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.864 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.864 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.864 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.864 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 5, 2024-08-18T16:19:23.864 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:19.536516Z", 2024-08-18T16:19:23.864 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.864 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.864 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.864 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.865 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.865 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.865 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.865 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.865 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.865 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_pztCA_2mV5ZNxFHKhOsv0-0-QTmJE06", 2024-08-18T16:19:23.865 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.865 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.865 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.865 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.865 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.865 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000009", 2024-08-18T16:19:23.866 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.866 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.866 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:19:23.866 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 6 2024-08-18T16:19:23.866 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.866 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.866 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.866 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.866 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.866 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:19:23.866 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:20.069388Z", 2024-08-18T16:19:23.866 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.867 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.867 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.867 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.867 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.867 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.867 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.867 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.867 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.867 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_v5RqRTwW9XkQ6g8IJh6-MOxdLDfbk1f", 2024-08-18T16:19:23.867 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.867 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.867 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.867 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.868 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.868 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000010", 2024-08-18T16:19:23.868 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.868 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.868 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:19:23.868 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 7 2024-08-18T16:19:23.868 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.868 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.868 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.868 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.868 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.868 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 6, 2024-08-18T16:19:23.869 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:20.182223Z", 2024-08-18T16:19:23.869 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.869 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.869 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.869 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.869 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.869 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.869 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.869 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.869 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.869 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_cjtmHTQ1qXVddkQ9neMBdPX0MSdarfv", 2024-08-18T16:19:23.869 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.870 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.870 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.870 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.870 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.870 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000011", 2024-08-18T16:19:23.870 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.870 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.870 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:19:23.870 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 7 2024-08-18T16:19:23.870 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.870 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.870 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.870 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.870 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.871 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 5, 2024-08-18T16:19:23.871 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:19.775284Z", 2024-08-18T16:19:23.871 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.871 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.871 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.871 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.871 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.871 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.871 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.871 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.871 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.871 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_cs6uOptwdwG5O-VADsB-UnXw0K44C5n", 2024-08-18T16:19:23.871 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.872 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.872 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.872 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.872 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.872 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000012", 2024-08-18T16:19:23.872 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.872 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.872 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:19:23.872 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 6 2024-08-18T16:19:23.872 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.872 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.872 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.872 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.873 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.873 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:19:23.873 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:19.602880Z", 2024-08-18T16:19:23.873 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.873 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.873 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.873 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.873 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.873 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.873 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.873 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.873 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.873 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_FtA4a77uIgcgep9H-_IKm-rPhLFYYNA", 2024-08-18T16:19:23.874 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.874 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.874 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.874 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.874 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.874 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000013", 2024-08-18T16:19:23.874 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.874 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.874 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:19:23.874 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 8 2024-08-18T16:19:23.874 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.874 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.875 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.875 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.875 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.875 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-08-18T16:19:23.875 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:20.004845Z", 2024-08-18T16:19:23.875 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.875 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.875 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.875 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.875 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.876 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.876 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.876 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.876 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.876 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_PLC-vUaby9XzoyRXHhUCzjMPaWdvdHo", 2024-08-18T16:19:23.876 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.876 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.876 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.876 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.876 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.876 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000014", 2024-08-18T16:19:23.876 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.876 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.877 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:19:23.877 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 4 2024-08-18T16:19:23.877 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.877 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.877 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.877 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.877 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.877 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 5, 2024-08-18T16:19:23.877 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:20.396690Z", 2024-08-18T16:19:23.877 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.877 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.877 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.877 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.878 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.878 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.878 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.878 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.878 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.878 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_Bc747l5D0ADItfUfI2zGzjlWgXb3kVf", 2024-08-18T16:19:23.878 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.878 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.878 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.878 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.878 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.878 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000015", 2024-08-18T16:19:23.879 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.879 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.879 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:19:23.879 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 15 2024-08-18T16:19:23.879 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.879 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.879 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.879 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.879 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.879 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 6, 2024-08-18T16:19:23.879 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:20.126126Z", 2024-08-18T16:19:23.879 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.879 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.879 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.880 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.880 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.880 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.880 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.880 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.880 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.880 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_W6ZEND5kyLKg1nst9oNmojKIA7Z4td2", 2024-08-18T16:19:23.880 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.880 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.880 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.880 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.880 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.880 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000016", 2024-08-18T16:19:23.880 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.881 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.881 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:19:23.881 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 5 2024-08-18T16:19:23.881 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.881 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.881 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.881 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.881 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.881 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-08-18T16:19:23.881 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:20.228897Z", 2024-08-18T16:19:23.881 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.881 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.881 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.882 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.882 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.882 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.882 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.882 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.882 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.882 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_FlFqCmUDVof3eYCKDVScs2nxZouZW_7", 2024-08-18T16:19:23.882 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.882 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.882 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.882 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.882 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.882 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000017", 2024-08-18T16:19:23.882 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.883 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.883 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:19:23.883 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 3 2024-08-18T16:19:23.883 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.883 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.883 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.883 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.883 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.883 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:19:23.883 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:18.792258Z", 2024-08-18T16:19:23.883 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.883 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.883 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.883 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.884 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.884 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.884 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.884 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.884 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.884 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_p_FuQ0_57jHy6L9S8-V36u6NqmKhm2y", 2024-08-18T16:19:23.884 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.884 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.884 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.884 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.884 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.884 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000018", 2024-08-18T16:19:23.884 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.884 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.885 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:19:23.885 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 6 2024-08-18T16:19:23.885 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.885 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.885 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.885 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.885 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.885 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 5, 2024-08-18T16:19:23.885 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:20.284525Z", 2024-08-18T16:19:23.885 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.885 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.885 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.885 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.885 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.886 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.886 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.886 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.886 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.886 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_ZwETvmO1P20xJXuRQvVDEzckXO_m8w_", 2024-08-18T16:19:23.886 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.886 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.886 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.886 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.886 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.886 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000019", 2024-08-18T16:19:23.886 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.886 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.886 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:19:23.887 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 7 2024-08-18T16:19:23.887 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.887 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.887 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.887 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.887 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.887 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 5, 2024-08-18T16:19:23.887 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:19.884831Z", 2024-08-18T16:19:23.887 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.887 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.887 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.887 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.887 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.887 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.888 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.888 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.888 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.888 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_EmNB1ziWF8_hgbzX5MgB77DM7Tahswl", 2024-08-18T16:19:23.888 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.888 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.888 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.888 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.888 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.888 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000020", 2024-08-18T16:19:23.888 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.888 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.888 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:19:23.888 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 3 2024-08-18T16:19:23.889 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.889 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.889 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.889 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.889 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.889 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-08-18T16:19:23.889 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:17.856631Z", 2024-08-18T16:19:23.889 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.889 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.889 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.889 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.889 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.889 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.889 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.890 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.890 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.890 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_Hm6lN_e-JKMW5COhV1bZnJeHOKYA3Tq", 2024-08-18T16:19:23.890 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.890 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.890 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.890 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.890 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.890 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000021", 2024-08-18T16:19:23.890 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.890 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.890 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:19:23.890 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 6 2024-08-18T16:19:23.891 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.891 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.891 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.891 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.891 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.891 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:19:23.891 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:19.648761Z", 2024-08-18T16:19:23.891 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.891 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.891 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.891 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.891 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.891 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.891 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.892 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.892 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.892 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_wyK0VLmulez1J3Fzv36LzNKnmLT4FGE", 2024-08-18T16:19:23.892 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.892 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.892 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.892 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.892 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.892 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000022", 2024-08-18T16:19:23.892 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.892 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.892 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:19:23.892 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 6 2024-08-18T16:19:23.892 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.893 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.893 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.893 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.893 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.893 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 5, 2024-08-18T16:19:23.893 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:18.551431Z", 2024-08-18T16:19:23.893 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.893 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.893 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.893 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.893 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.893 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.893 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.893 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.894 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.894 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_9fFNgqXZi9F4y9sq2xf9uI_0Lg-KiNS", 2024-08-18T16:19:23.894 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.894 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.894 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.894 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.894 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.894 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000023", 2024-08-18T16:19:23.894 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.894 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.895 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:19:23.895 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 4 2024-08-18T16:19:23.895 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.895 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.895 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.895 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.895 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.895 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 6, 2024-08-18T16:19:23.895 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:18.904686Z", 2024-08-18T16:19:23.895 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.895 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.895 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.895 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.895 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.896 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.896 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.896 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.896 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.896 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_RBk5XzPvmQ9gHjPP1h_vTSsuRDDQA_I", 2024-08-18T16:19:23.896 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.896 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.896 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.896 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.896 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.896 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000024", 2024-08-18T16:19:23.896 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.896 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.896 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:19:23.897 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 5 2024-08-18T16:19:23.897 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.897 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.897 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.897 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.897 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.897 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 6, 2024-08-18T16:19:23.897 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:19.704871Z", 2024-08-18T16:19:23.897 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.897 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.897 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.897 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.897 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.897 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.898 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.898 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.898 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.898 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_qSa8VLBwAWZfkVt14JgLrdtQOSunOeV", 2024-08-18T16:19:23.898 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.898 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.898 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.898 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.898 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.898 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000025", 2024-08-18T16:19:23.898 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.898 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.898 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:19:23.899 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 14 2024-08-18T16:19:23.899 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.899 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.899 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.899 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.899 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.899 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:19:23.899 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:17.792369Z", 2024-08-18T16:19:23.899 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.899 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.899 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.899 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.899 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.900 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.900 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.900 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.900 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.900 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_OkF6NtbWk4_dqcz7f8r_vj4ZF5PkjSD", 2024-08-18T16:19:23.900 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.900 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.900 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.900 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.900 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.900 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000026", 2024-08-18T16:19:23.900 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.901 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.901 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:19:23.901 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 9 2024-08-18T16:19:23.901 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.901 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.901 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.901 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.901 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.901 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-08-18T16:19:23.901 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:20.238134Z", 2024-08-18T16:19:23.902 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.902 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.902 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.902 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.902 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.902 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.902 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.902 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.902 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.902 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_qT-o27eoUIXLYHaDOYCg485sqrzUvaL", 2024-08-18T16:19:23.902 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.902 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.902 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.903 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.903 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.903 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000027", 2024-08-18T16:19:23.903 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.903 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.903 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:19:23.903 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 8 2024-08-18T16:19:23.903 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.903 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.903 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.903 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.903 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.903 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 6, 2024-08-18T16:19:23.903 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:18.160384Z", 2024-08-18T16:19:23.904 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.904 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.904 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.904 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.904 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.904 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.904 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.904 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.904 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.904 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_HFTRm-T7wF4GgX35P55LjA-yWGoQdfW", 2024-08-18T16:19:23.904 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.904 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.904 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.905 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.905 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.905 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000028", 2024-08-18T16:19:23.905 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.905 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.905 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:19:23.905 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 11 2024-08-18T16:19:23.905 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.905 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.905 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.905 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.905 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.905 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 6, 2024-08-18T16:19:23.905 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:15.858769Z", 2024-08-18T16:19:23.906 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.906 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.906 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.906 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.906 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.906 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.906 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.906 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.906 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.906 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_qMdKeWe5GcU4kC3zl7Wd8WsSoBZ_LS-", 2024-08-18T16:19:23.906 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.906 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.906 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.906 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.907 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.907 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000029", 2024-08-18T16:19:23.907 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.907 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.907 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:19:23.907 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 8 2024-08-18T16:19:23.907 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.907 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.907 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.907 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.907 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.907 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:19:23.907 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:18.977845Z", 2024-08-18T16:19:23.907 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.908 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.908 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.908 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.908 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.908 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.908 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.908 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.908 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.908 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_ItaWDG9JMCQzRpjjV2-fW_94D4-9IxM", 2024-08-18T16:19:23.908 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.908 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.908 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.908 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.909 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.909 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000030", 2024-08-18T16:19:23.909 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.909 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.909 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:19:23.909 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 3 2024-08-18T16:19:23.909 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.909 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.909 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.909 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.909 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.909 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 5, 2024-08-18T16:19:23.909 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:19.545863Z", 2024-08-18T16:19:23.909 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.910 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.910 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.910 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.910 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.910 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.910 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.910 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.910 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.910 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_JUby4CTPWK-gPBY8xKuXIa2J_efyyQC", 2024-08-18T16:19:23.910 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.910 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.910 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.911 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.911 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.911 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000031", 2024-08-18T16:19:23.911 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.911 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.911 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:19:23.911 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 13 2024-08-18T16:19:23.911 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.911 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.911 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.911 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.911 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.911 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 5, 2024-08-18T16:19:23.911 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:17.912320Z", 2024-08-18T16:19:23.912 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.912 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.912 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.912 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.912 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.912 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.912 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.912 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.912 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.912 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_nVbqhl-rvmKckK2tyhwA_qkuKemZkEK", 2024-08-18T16:19:23.912 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.912 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.912 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.912 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.913 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.913 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000032", 2024-08-18T16:19:23.913 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.913 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.913 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:19:23.913 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 8 2024-08-18T16:19:23.913 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.913 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.913 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.913 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.913 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.913 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-08-18T16:19:23.913 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:17.116603Z", 2024-08-18T16:19:23.913 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.913 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.914 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.914 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.914 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.914 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.914 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.914 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.914 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.914 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_G3hksalfKJuXmZHbGMw4Zk8m00qGnU2", 2024-08-18T16:19:23.914 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.914 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.914 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.914 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.914 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.915 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000033", 2024-08-18T16:19:23.915 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.915 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.915 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:19:23.915 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 9 2024-08-18T16:19:23.915 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.915 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.915 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.915 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.915 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.915 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-08-18T16:19:23.915 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:19.593080Z", 2024-08-18T16:19:23.915 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.915 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.916 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.916 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.916 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.916 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.916 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.916 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.916 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.916 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_w_ZTCJZgowQUuicMPN3VAtbHsboCweX", 2024-08-18T16:19:23.916 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.916 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.916 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.916 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.916 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.916 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000036", 2024-08-18T16:19:23.917 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.917 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.917 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:19:23.917 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 4 2024-08-18T16:19:23.917 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.917 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.917 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.917 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.917 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.917 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 5, 2024-08-18T16:19:23.917 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:20.013736Z", 2024-08-18T16:19:23.917 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.917 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.917 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.918 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.918 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.918 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.918 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.918 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.918 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.918 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_VQcwqEBrAUzp1G7gdmWNhWrgftAP9cw", 2024-08-18T16:19:23.918 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.918 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.918 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.918 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.918 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.918 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000037", 2024-08-18T16:19:23.918 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.919 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.919 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:19:23.919 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 10 2024-08-18T16:19:23.919 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.919 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.919 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.919 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.919 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.919 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 6, 2024-08-18T16:19:23.919 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:17.746196Z", 2024-08-18T16:19:23.919 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.919 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.919 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.919 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.920 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.920 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.920 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.920 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.920 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.920 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_kj5zqc7YqHJDt6LKuKnwGljF7lwR1Xt", 2024-08-18T16:19:23.920 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.920 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.920 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.920 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.920 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.920 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000039", 2024-08-18T16:19:23.920 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.920 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.921 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:19:23.921 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 12 2024-08-18T16:19:23.921 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.921 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.921 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.921 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.921 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.921 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 5, 2024-08-18T16:19:23.921 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:19.824786Z", 2024-08-18T16:19:23.921 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.921 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.921 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.921 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.922 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.922 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.922 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.922 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.922 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.922 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_BmCdjnTS3EpEWCwjQ943M-XvRBpvU0c", 2024-08-18T16:19:23.922 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.922 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.922 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.922 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.922 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.922 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000040", 2024-08-18T16:19:23.922 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.922 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.923 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:19:23.923 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 5 2024-08-18T16:19:23.923 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.923 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.923 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.923 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.923 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.923 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:19:23.923 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:20.293593Z", 2024-08-18T16:19:23.923 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.923 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.923 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.923 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.923 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.924 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.924 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.924 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.924 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.924 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_BrZWm-9WlBDOPeFe9HnA-vujhcsdV-_", 2024-08-18T16:19:23.924 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.924 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.924 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.924 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.924 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.924 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000042", 2024-08-18T16:19:23.924 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.924 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.925 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:19:23.925 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 7 2024-08-18T16:19:23.925 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.925 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.925 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.925 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.925 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.925 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:19:23.925 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:18.280618Z", 2024-08-18T16:19:23.925 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.925 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.925 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.925 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.925 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.926 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.926 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.926 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.926 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.926 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_TUwf6hJ4810Zof5cZ3IK3ybLOJJpLBE", 2024-08-18T16:19:23.926 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.926 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.926 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.926 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.926 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.926 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000045", 2024-08-18T16:19:23.926 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.926 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.927 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:19:23.927 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 13 2024-08-18T16:19:23.927 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.927 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.927 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.927 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.927 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.927 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 5, 2024-08-18T16:19:23.927 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:20.340703Z", 2024-08-18T16:19:23.927 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.927 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.928 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.928 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.928 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.928 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.928 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.928 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.928 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.928 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_4-1A_Cj9P8_T6fLYvvbOo1I93ynjZDu", 2024-08-18T16:19:23.928 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.928 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.928 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.928 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.928 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.928 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000046", 2024-08-18T16:19:23.929 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.929 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.929 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:19:23.929 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 6 2024-08-18T16:19:23.929 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.929 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.929 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.929 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.929 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.929 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:19:23.929 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:17.181145Z", 2024-08-18T16:19:23.929 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.929 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.930 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.930 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.930 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.930 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.930 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.930 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.930 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.930 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_-YZLXj572qnc0L-uIkMkcJ6mBzKPYL_", 2024-08-18T16:19:23.930 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.930 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.930 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.930 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.930 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.931 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000048", 2024-08-18T16:19:23.931 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.931 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.931 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:19:23.931 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 8 2024-08-18T16:19:23.931 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.931 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.931 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.931 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.931 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.931 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:19:23.931 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:19.833816Z", 2024-08-18T16:19:23.931 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.932 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.932 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.932 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.932 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.932 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.932 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.932 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.932 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.932 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_4AFnFbM2uqMGZGp8aa1WAc4jIjkBd0i", 2024-08-18T16:19:23.932 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.932 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.932 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.932 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.932 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.932 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000049", 2024-08-18T16:19:23.933 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.933 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.933 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:19:23.933 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 6 2024-08-18T16:19:23.933 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.933 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.933 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.933 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.933 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.933 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:19:23.933 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:17.984613Z", 2024-08-18T16:19:23.933 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.933 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.934 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.934 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.934 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.934 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.934 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.934 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.934 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.934 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_GO1uqCqXJt7WTSp9-s9UvClyBbJXd87", 2024-08-18T16:19:23.934 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.934 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.934 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.934 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.934 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.935 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000050", 2024-08-18T16:19:23.935 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.935 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.935 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:19:23.935 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 8 2024-08-18T16:19:23.935 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.935 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.935 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.935 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.935 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.935 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 5, 2024-08-18T16:19:23.935 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:17.125461Z", 2024-08-18T16:19:23.935 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.935 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.936 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.936 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.936 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.936 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.936 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.936 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.936 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.936 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_ySw6szgM4ED8z-iKbMKONcR5mL7eb6Z", 2024-08-18T16:19:23.936 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.936 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.936 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.936 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.936 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.937 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000053", 2024-08-18T16:19:23.937 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.937 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.937 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:19:23.937 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 12 2024-08-18T16:19:23.937 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.937 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.937 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.937 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.937 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.937 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 5, 2024-08-18T16:19:23.937 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:19.482927Z", 2024-08-18T16:19:23.937 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.937 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.938 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.938 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.938 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.938 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.938 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.938 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.938 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.938 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_6d3daQdUZ-JLdcvtGSVL8QPAVE9xlAE", 2024-08-18T16:19:23.938 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.938 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.938 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.938 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.938 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.939 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000055", 2024-08-18T16:19:23.939 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.939 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.939 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:19:23.939 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 47 2024-08-18T16:19:23.939 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.939 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.939 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.939 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.939 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.939 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:19:23.939 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:17.295995Z", 2024-08-18T16:19:23.939 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.939 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.939 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.940 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.940 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.940 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.940 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.940 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.940 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.940 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_r2Bu7YQujenbP-gqQgt2-0DVAAUQAKI", 2024-08-18T16:19:23.940 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.940 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.940 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.940 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.940 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.940 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000056", 2024-08-18T16:19:23.940 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.941 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.941 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:19:23.941 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 10 2024-08-18T16:19:23.941 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.941 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.941 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.941 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.941 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.941 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 5, 2024-08-18T16:19:23.941 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:19.144248Z", 2024-08-18T16:19:23.941 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.941 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.941 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.941 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.942 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.942 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.942 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.942 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.942 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.942 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_nLhSe3nR24HFKA5PfaMZFOJRNDGSPZ9", 2024-08-18T16:19:23.942 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.942 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.942 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.942 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.942 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.942 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000057", 2024-08-18T16:19:23.942 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.943 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.943 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:19:23.943 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 9 2024-08-18T16:19:23.943 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.943 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.943 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.943 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.943 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.943 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 5, 2024-08-18T16:19:23.943 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:19.657944Z", 2024-08-18T16:19:23.943 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.943 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.943 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.943 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.943 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.944 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.944 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.944 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.944 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.944 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_5RimT9LcVpBDrzLhO9IZaF7JnHymobA", 2024-08-18T16:19:23.944 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.944 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.944 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.944 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.944 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.944 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000058", 2024-08-18T16:19:23.944 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.944 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.945 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:19:23.945 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 11 2024-08-18T16:19:23.945 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.945 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.945 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.945 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.945 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.945 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 6, 2024-08-18T16:19:23.945 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:17.303274Z", 2024-08-18T16:19:23.945 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.945 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.945 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.945 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.946 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.946 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.946 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.946 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.946 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.946 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_5GogsuLyQgs_qPPsYmRbEin2nsXySDZ", 2024-08-18T16:19:23.946 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.946 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.946 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.946 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.946 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.946 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000059", 2024-08-18T16:19:23.947 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.947 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.947 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:19:23.947 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 7 2024-08-18T16:19:23.947 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.947 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.947 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.947 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.947 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.947 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 5, 2024-08-18T16:19:23.947 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:18.040381Z", 2024-08-18T16:19:23.947 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.947 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.948 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.948 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.948 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.948 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.948 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.948 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.948 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.948 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_XKmMEVXRV6Vgqw8DqfCxKBwVFwt6l6N", 2024-08-18T16:19:23.948 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.948 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.948 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.948 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.948 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.948 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000060", 2024-08-18T16:19:23.949 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.949 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.949 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:19:23.949 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 9 2024-08-18T16:19:23.949 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.949 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.949 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.949 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.949 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.949 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:19:23.949 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:19.996680Z", 2024-08-18T16:19:23.949 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.949 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.950 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.950 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.950 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.950 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.950 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.950 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.950 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.950 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_BRfYtOS_a5o6jrThcyJb1dSlMje8L_-", 2024-08-18T16:19:23.950 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.950 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.950 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.950 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.951 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.951 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000061", 2024-08-18T16:19:23.951 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.951 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.951 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:19:23.951 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 9 2024-08-18T16:19:23.951 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.951 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.951 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.951 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.951 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.951 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:19:23.951 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:18.800570Z", 2024-08-18T16:19:23.951 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.952 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.952 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.952 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.952 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.952 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.952 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.952 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.952 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.952 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_GKDi2TQ2eoEC___nvwgQxVFS8Mf4BR4", 2024-08-18T16:19:23.952 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.952 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.952 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.952 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.952 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.953 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000062", 2024-08-18T16:19:23.953 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.953 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.953 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:19:23.953 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 7 2024-08-18T16:19:23.953 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.953 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.953 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.953 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.953 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.953 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:19:23.953 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:20.061021Z", 2024-08-18T16:19:23.953 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.953 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.954 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.954 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.954 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.954 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.954 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.954 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.954 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.954 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_-kJC4mTlxobMGGjdwZLd5_IRMEd-oP3", 2024-08-18T16:19:23.954 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.954 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.954 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.954 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.954 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.955 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000065", 2024-08-18T16:19:23.955 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.955 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.955 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:19:23.955 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 13 2024-08-18T16:19:23.955 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.955 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.955 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.955 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.955 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.955 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:19:23.955 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:19.716103Z", 2024-08-18T16:19:23.955 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.955 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.956 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.956 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.956 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.956 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.956 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.956 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.956 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.956 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_0CbyJ2zqtYpoOH886i9wc9xbOvbTSlO", 2024-08-18T16:19:23.956 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.956 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.956 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.956 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.956 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.957 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000076", 2024-08-18T16:19:23.957 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.957 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.957 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:19:23.957 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 12 2024-08-18T16:19:23.957 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.957 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.957 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.957 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.957 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.957 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 6, 2024-08-18T16:19:23.957 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:15.743831Z", 2024-08-18T16:19:23.957 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.958 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.958 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.958 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.958 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.958 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.958 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.958 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.958 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.958 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_PkIoefyX0MCLEA0WM20Di8qBVmdVRb9", 2024-08-18T16:19:23.958 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.958 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.958 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.959 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.959 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.959 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000133", 2024-08-18T16:19:23.959 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.959 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.959 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:19:23.959 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 11 2024-08-18T16:19:23.959 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.959 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.959 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.959 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.959 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.959 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:19:23.959 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:19.158889Z", 2024-08-18T16:19:23.960 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.960 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.960 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.960 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.960 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.960 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.960 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.960 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.960 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.960 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_D_At-Oh9-gmWTvLWSUuViyC2WtbFAbU", 2024-08-18T16:19:23.960 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.960 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.960 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.961 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.961 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.961 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000162", 2024-08-18T16:19:23.961 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.961 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.961 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:19:23.961 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 7 2024-08-18T16:19:23.961 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.961 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.961 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.961 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.961 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.961 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 6, 2024-08-18T16:19:23.961 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:18.617486Z", 2024-08-18T16:19:23.962 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.962 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.962 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.962 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.962 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.962 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.962 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.962 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.962 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.962 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_RG1XCqZjUHEuFirRSOjG8uUBXAGKwoT", 2024-08-18T16:19:23.962 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.962 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.962 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.963 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.963 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.963 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000271", 2024-08-18T16:19:23.963 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.963 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.963 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:19:23.963 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 7 2024-08-18T16:19:23.963 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.963 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.963 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.963 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.963 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.963 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:19:23.964 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:18.858093Z", 2024-08-18T16:19:23.964 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.964 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.964 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.964 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.964 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.964 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.964 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.964 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.964 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.964 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_0U4egurH_EsBFqzt8zy-4XhK844NeHe", 2024-08-18T16:19:23.964 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.964 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.964 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.965 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.965 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.965 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000283", 2024-08-18T16:19:23.965 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.965 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.965 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:19:23.965 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 5 2024-08-18T16:19:23.965 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.965 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.965 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.965 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.965 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.965 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:19:23.965 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:18.728571Z", 2024-08-18T16:19:23.966 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.966 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.966 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.966 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.966 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.966 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.966 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.966 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.966 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.966 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_sZkEk2bKeADxOv-aLHGzoEo2X8RLWPn", 2024-08-18T16:19:23.966 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.966 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.966 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.967 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.967 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.967 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000317", 2024-08-18T16:19:23.967 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.967 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.967 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:19:23.967 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 5 2024-08-18T16:19:23.967 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.967 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.967 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.967 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.967 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.967 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 5, 2024-08-18T16:19:23.967 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:18.784378Z", 2024-08-18T16:19:23.968 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.968 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.968 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.968 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.968 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.968 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.968 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.968 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.968 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.968 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_XBjBiGNiqfKOMHIjCVWfEhP_RsaKw7o", 2024-08-18T16:19:23.968 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.968 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.968 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.968 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.968 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.969 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000423", 2024-08-18T16:19:23.969 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.969 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.969 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:19:23.969 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 12 2024-08-18T16:19:23.969 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.969 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.969 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.969 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.969 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.969 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:19:23.969 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:18.720502Z", 2024-08-18T16:19:23.970 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.970 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.970 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.970 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.970 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.970 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.970 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.970 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.970 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.970 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_33wapta6j11W-sNlZlw5t0qwTt9vsLH", 2024-08-18T16:19:23.970 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.970 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.971 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.971 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.971 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:19:23.971 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj00000000000000000467", 2024-08-18T16:19:23.972 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:19:23.972 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:19:23.972 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:19:23.972 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 7 2024-08-18T16:19:23.972 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.972 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:19:23.972 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:19:23.972 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:19:23.973 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:19:23.973 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:19:23.973 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:19:19.151651Z", 2024-08-18T16:19:23.973 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:19:23.973 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:19:23.973 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:19:23.973 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:19:23.973 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:19:23.973 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:19:23.973 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:19:23.973 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:19:23.973 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:19:23.973 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_s14VtpSi_eyBGM6QVXQczKPRYdKGrdE", 2024-08-18T16:19:23.974 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:19:23.974 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:19:23.974 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:19:23.974 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:19:23.974 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T16:19:23.975 INFO:tasks.rgw.c2.client.0:Restarting daemon 2024-08-18T16:19:23.975 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2024-08-18T16:19:23.978 INFO:tasks.rgw.c2.client.0:Started 2024-08-18T16:19:23.978 DEBUG:teuthology.orchestra.run.smithi193:> curl http://smithi193.front.sepia.ceph.com:8000 2024-08-18T16:19:24.000 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-08-18T16:19:24.001 INFO:teuthology.orchestra.run.smithi193.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-08-18T16:19:24.001 INFO:teuthology.orchestra.run.smithi193.stderr: Dload Upload Total Spent Left Speed 2024-08-18T16:19:24.001 INFO:teuthology.orchestra.run.smithi193.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2024-08-18T16:19:24.002 INFO:teuthology.orchestra.run.smithi193.stderr:curl: (7) Failed to connect to smithi193.front.sepia.ceph.com port 8000: Connection refused 2024-08-18T16:19:25.003 DEBUG:teuthology.orchestra.run.smithi193:> curl http://smithi193.front.sepia.ceph.com:8000 2024-08-18T16:19:25.023 INFO:teuthology.orchestra.run.smithi193.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-08-18T16:19:25.024 INFO:teuthology.orchestra.run.smithi193.stderr: Dload Upload Total Spent Left Speed 2024-08-18T16:19:25.024 INFO:teuthology.orchestra.run.smithi193.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-08-18T16:19:25.025 INFO:teuthology.orchestra.run.smithi193.stdout:anonymous 2024-08-18T16:19:25.025 INFO:tasks.rgw.c1.client.1:Restarting daemon 2024-08-18T16:19:25.025 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2024-08-18T16:19:25.028 INFO:tasks.rgw.c1.client.1:Started 2024-08-18T16:19:25.028 DEBUG:teuthology.orchestra.run.smithi134:> curl http://smithi134.front.sepia.ceph.com:8001 2024-08-18T16:19:25.046 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-08-18T16:19:25.047 INFO:teuthology.orchestra.run.smithi134.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-08-18T16:19:25.047 INFO:teuthology.orchestra.run.smithi134.stderr: Dload Upload Total Spent Left Speed 2024-08-18T16:19:25.047 INFO:teuthology.orchestra.run.smithi134.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2024-08-18T16:19:25.047 INFO:teuthology.orchestra.run.smithi134.stderr:curl: (7) Failed to connect to smithi134.front.sepia.ceph.com port 8001: Connection refused 2024-08-18T16:19:26.048 DEBUG:teuthology.orchestra.run.smithi134:> curl http://smithi134.front.sepia.ceph.com:8001 2024-08-18T16:19:26.070 INFO:teuthology.orchestra.run.smithi134.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-08-18T16:19:26.070 INFO:teuthology.orchestra.run.smithi134.stderr: Dload Upload Total Spent Left Speed 2024-08-18T16:19:26.071 INFO:teuthology.orchestra.run.smithi134.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-08-18T16:19:26.071 INFO:teuthology.orchestra.run.smithi134.stdout:anonymous 2024-08-18T16:19:26.071 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-77', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:19:26.072 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-77', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:19:26.072 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-77 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T16:19:26.130 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:19:26.130 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:19:26.171 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:26.171+0000 7fd28b542880 1 bucket sync caught up with source: 2024-08-18T16:19:26.172 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000001.140.6, 00000000002.100.6, 00000000002.108.6, 00000000002.128.6, 00000000002.112.6, 00000000001.81.6, 00000000002.167.6, 00000000002.81.6, 00000000001.170.6, 00000000002.83.6, 00000000002.163.6, 00000000002.138.6, 00000000001.112.6, 00000000002.90.6, 00000000002.165.6, 00000000002.95.6, 00000000002.91.6, 00000000002.116.6, 00000000002.100.6, 00000000002.106.6, 00000000002.94.6, , 00000000002.115.6, 00000000002.98.6, , 00000000002.114.6, 00000000002.110.6, 00000000002.100.6, , 00000000002.97.6, 00000000001.86.6, 00000000002.142.6, 00000000002.87.6, 00000000002.114.6, 00000000001.94.6, 00000000002.93.6, 00000000002.161.6] 2024-08-18T16:19:26.172 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000001.140.6, 00000000002.100.6, 00000000002.108.6, 00000000002.128.6, 00000000002.112.6, 00000000001.81.6, 00000000002.167.6, 00000000002.81.6, 00000000001.170.6, 00000000002.83.6, 00000000002.163.6, 00000000002.138.6, 00000000001.112.6, 00000000002.90.6, 00000000002.165.6, 00000000002.95.6, 00000000002.91.6, 00000000002.116.6, 00000000002.100.6, 00000000002.106.6, 00000000002.94.6, , 00000000002.115.6, 00000000002.98.6, , 00000000002.114.6, 00000000002.110.6, 00000000002.100.6, , 00000000002.97.6, 00000000001.86.6, 00000000002.142.6, 00000000002.87.6, 00000000002.114.6, 00000000001.94.6, 00000000002.93.6, 00000000002.161.6] 2024-08-18T16:19:26.172 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:19:26.171+0000 7fd28b542880 0 bucket checkpoint complete 2024-08-18T16:19:26.186 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-77', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:19:26.186 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-77', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:19:26.187 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-77 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:19:26.241 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:19:26.241 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:19:26.280 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:26.280+0000 7ff7eac62880 1 bucket sync caught up with source: 2024-08-18T16:19:26.281 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000001.140.6, 00000000002.100.6, 00000000002.108.6, 00000000002.128.6, 00000000002.112.6, 00000000001.81.6, 00000000002.167.6, 00000000002.81.6, 00000000001.170.6, 00000000002.83.6, 00000000002.163.6, 00000000002.138.6, 00000000001.112.6, 00000000002.90.6, 00000000002.165.6, 00000000002.95.6, 00000000002.91.6, 00000000002.116.6, 00000000002.100.6, 00000000002.106.6, 00000000002.94.6, , 00000000002.115.6, 00000000002.98.6, , 00000000002.114.6, 00000000002.110.6, 00000000002.100.6, , 00000000002.97.6, 00000000001.86.6, 00000000002.142.6, 00000000002.87.6, 00000000002.114.6, 00000000001.94.6, 00000000002.93.6, 00000000002.161.6] 2024-08-18T16:19:26.281 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000001.140.6, 00000000002.100.6, 00000000002.108.6, 00000000002.128.6, 00000000002.112.6, 00000000001.81.6, 00000000002.167.6, 00000000002.81.6, 00000000001.170.6, 00000000002.83.6, 00000000002.163.6, 00000000002.138.6, 00000000001.112.6, 00000000002.90.6, 00000000002.165.6, 00000000002.95.6, 00000000002.91.6, 00000000002.116.6, 00000000002.100.6, 00000000002.106.6, 00000000002.94.6, , 00000000002.115.6, 00000000002.98.6, , 00000000002.114.6, 00000000002.110.6, 00000000002.100.6, , 00000000002.97.6, 00000000001.86.6, 00000000002.142.6, 00000000002.87.6, 00000000002.114.6, 00000000001.94.6, 00000000002.93.6, 00000000002.161.6] 2024-08-18T16:19:26.281 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:26.280+0000 7ff7eac62880 0 bucket checkpoint complete 2024-08-18T16:19:26.294 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-77', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:19:26.295 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-77', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:19:26.295 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-77 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:19:26.348 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:19:26.348 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:19:26.386 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:26.385+0000 7f78eba35880 1 bucket sync caught up with source: 2024-08-18T16:19:26.386 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000005.300.6, 00000000007.223.6, 00000000006.270.6, 00000000009.330.6, 00000000005.302.6, 00000000008.292.6, 00000000008.262.6, 00000000006.296.6, 00000000007.225.6, 00000000007.155.6, 00000000004.318.6] 2024-08-18T16:19:26.386 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000005.300.6, 00000000007.223.6, 00000000006.270.6, 00000000009.330.6, 00000000005.302.6, 00000000008.292.6, 00000000008.262.6, 00000000006.296.6, 00000000007.225.6, 00000000007.155.6, 00000000004.318.6] 2024-08-18T16:19:26.386 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:26.385+0000 7f78eba35880 0 bucket checkpoint complete 2024-08-18T16:19:26.400 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-77', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:19:26.400 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-77', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:19:26.401 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-77 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:19:26.453 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:19:26.454 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:19:26.511 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:26.510+0000 7f70e0c1f880 1 waiting for incremental sync to catch up: 2024-08-18T16:19:26.511 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000001.285.6, 00000000007.223.6, 00000000002.262.6, , 00000000001.286.6, 00000000008.292.6, 00000000003.252.6, 00000000001.272.6, 00000000001.200.6, 00000000001.143.6, 00000000003.316.6] 2024-08-18T16:19:26.511 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000005.300.6, 00000000007.223.6, 00000000006.270.6, 00000000009.330.6, 00000000005.302.6, 00000000008.292.6, 00000000008.262.6, 00000000006.296.6, 00000000007.225.6, 00000000007.155.6, 00000000004.318.6] 2024-08-18T16:19:31.512 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:31.512+0000 7f70e0c1f880 1 waiting for incremental sync to catch up: 2024-08-18T16:19:31.513 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000001.285.6, 00000000007.223.6, 00000000002.262.6, , 00000000001.286.6, 00000000008.292.6, 00000000003.252.6, 00000000001.272.6, 00000000001.200.6, 00000000001.143.6, 00000000003.316.6] 2024-08-18T16:19:31.513 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000005.300.6, 00000000007.223.6, 00000000006.270.6, 00000000009.330.6, 00000000005.302.6, 00000000008.292.6, 00000000008.262.6, 00000000006.296.6, 00000000007.225.6, 00000000007.155.6, 00000000004.318.6] 2024-08-18T16:19:36.514 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:36.513+0000 7f70e0c1f880 1 waiting for incremental sync to catch up: 2024-08-18T16:19:36.514 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000001.285.6, 00000000007.223.6, 00000000002.262.6, , 00000000001.286.6, 00000000008.292.6, 00000000003.252.6, 00000000001.272.6, 00000000001.200.6, 00000000001.143.6, 00000000003.316.6] 2024-08-18T16:19:36.514 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000005.300.6, 00000000007.223.6, 00000000006.270.6, 00000000009.330.6, 00000000005.302.6, 00000000008.292.6, 00000000008.262.6, 00000000006.296.6, 00000000007.225.6, 00000000007.155.6, 00000000004.318.6] 2024-08-18T16:19:41.515 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:41.514+0000 7f70e0c1f880 1 waiting for incremental sync to catch up: 2024-08-18T16:19:41.515 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000001.285.6, 00000000007.223.6, 00000000002.262.6, , 00000000001.286.6, 00000000008.292.6, 00000000003.252.6, 00000000001.272.6, 00000000001.200.6, 00000000001.143.6, 00000000003.316.6] 2024-08-18T16:19:41.515 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000005.300.6, 00000000007.223.6, 00000000006.270.6, 00000000009.330.6, 00000000005.302.6, 00000000008.292.6, 00000000008.262.6, 00000000006.296.6, 00000000007.225.6, 00000000007.155.6, 00000000004.318.6] 2024-08-18T16:19:46.516 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:46.516+0000 7f70e0c1f880 1 waiting for incremental sync to catch up: 2024-08-18T16:19:46.517 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000001.285.6, 00000000007.223.6, 00000000002.262.6, , 00000000001.286.6, 00000000008.292.6, 00000000003.252.6, 00000000001.272.6, 00000000001.200.6, 00000000001.143.6, 00000000003.316.6] 2024-08-18T16:19:46.517 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000005.300.6, 00000000007.223.6, 00000000006.270.6, 00000000009.330.6, 00000000005.302.6, 00000000008.292.6, 00000000008.262.6, 00000000006.296.6, 00000000007.225.6, 00000000007.155.6, 00000000004.318.6] 2024-08-18T16:19:51.518 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:51.517+0000 7f70e0c1f880 1 waiting for incremental sync to catch up: 2024-08-18T16:19:51.518 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000001.285.6, 00000000007.223.6, 00000000002.262.6, , 00000000001.286.6, 00000000008.292.6, 00000000003.252.6, 00000000001.272.6, 00000000001.200.6, 00000000001.143.6, 00000000003.316.6] 2024-08-18T16:19:51.518 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000005.300.6, 00000000007.223.6, 00000000006.270.6, 00000000009.330.6, 00000000005.302.6, 00000000008.292.6, 00000000008.262.6, 00000000006.296.6, 00000000007.225.6, 00000000007.155.6, 00000000004.318.6] 2024-08-18T16:19:56.519 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:19:56.519+0000 7f70e0c1f880 1 waiting for incremental sync to catch up: 2024-08-18T16:19:56.520 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000001.285.6, 00000000007.223.6, 00000000002.262.6, , 00000000001.286.6, 00000000008.292.6, 00000000003.252.6, 00000000001.272.6, 00000000001.200.6, 00000000001.143.6, 00000000003.316.6] 2024-08-18T16:19:56.520 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000005.300.6, 00000000007.223.6, 00000000006.270.6, 00000000009.330.6, 00000000005.302.6, 00000000008.292.6, 00000000008.262.6, 00000000006.296.6, 00000000007.225.6, 00000000007.155.6, 00000000004.318.6] 2024-08-18T16:20:01.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:20:01.520+0000 7f70e0c1f880 1 waiting for incremental sync to catch up: 2024-08-18T16:20:01.521 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000001.285.6, 00000000007.223.6, 00000000002.262.6, , 00000000001.286.6, 00000000008.292.6, 00000000003.252.6, 00000000001.272.6, 00000000001.200.6, 00000000001.143.6, 00000000003.316.6] 2024-08-18T16:20:01.521 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000005.300.6, 00000000007.223.6, 00000000006.270.6, 00000000009.330.6, 00000000005.302.6, 00000000008.292.6, 00000000008.262.6, 00000000006.296.6, 00000000007.225.6, 00000000007.155.6, 00000000004.318.6] 2024-08-18T16:20:06.522 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:20:06.522+0000 7f70e0c1f880 1 waiting for incremental sync to catch up: 2024-08-18T16:20:06.523 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000001.285.6, 00000000007.223.6, 00000000002.262.6, , 00000000001.286.6, 00000000008.292.6, 00000000003.252.6, 00000000001.272.6, 00000000001.200.6, 00000000001.143.6, 00000000003.316.6] 2024-08-18T16:20:06.523 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000005.300.6, 00000000007.223.6, 00000000006.270.6, 00000000009.330.6, 00000000005.302.6, 00000000008.292.6, 00000000008.262.6, 00000000006.296.6, 00000000007.225.6, 00000000007.155.6, 00000000004.318.6] 2024-08-18T16:20:11.524 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:20:11.523+0000 7f70e0c1f880 1 waiting for incremental sync to catch up: 2024-08-18T16:20:11.524 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000001.285.6, 00000000007.223.6, 00000000002.262.6, , 00000000001.286.6, 00000000008.292.6, 00000000003.252.6, 00000000001.272.6, 00000000001.200.6, 00000000001.143.6, 00000000003.316.6] 2024-08-18T16:20:11.524 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000005.300.6, 00000000007.223.6, 00000000006.270.6, 00000000009.330.6, 00000000005.302.6, 00000000008.292.6, 00000000008.262.6, 00000000006.296.6, 00000000007.225.6, 00000000007.155.6, 00000000004.318.6] 2024-08-18T16:20:16.525 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:20:16.525+0000 7f70e0c1f880 1 waiting for incremental sync to catch up: 2024-08-18T16:20:16.525 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000001.285.6, 00000000007.223.6, 00000000002.262.6, , 00000000001.286.6, 00000000008.292.6, 00000000003.252.6, 00000000001.272.6, 00000000001.200.6, 00000000001.143.6, 00000000003.316.6] 2024-08-18T16:20:16.526 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000005.300.6, 00000000007.223.6, 00000000006.270.6, 00000000009.330.6, 00000000005.302.6, 00000000008.292.6, 00000000008.262.6, 00000000006.296.6, 00000000007.225.6, 00000000007.155.6, 00000000004.318.6] 2024-08-18T16:20:21.527 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:20:21.526+0000 7f70e0c1f880 1 waiting for incremental sync to catch up: 2024-08-18T16:20:21.527 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000001.285.6, 00000000007.223.6, 00000000002.262.6, , 00000000001.286.6, 00000000008.292.6, 00000000003.252.6, 00000000001.272.6, 00000000001.200.6, 00000000001.143.6, 00000000003.316.6] 2024-08-18T16:20:21.527 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000005.300.6, 00000000007.223.6, 00000000006.270.6, 00000000009.330.6, 00000000005.302.6, 00000000008.292.6, 00000000008.262.6, 00000000006.296.6, 00000000007.225.6, 00000000007.155.6, 00000000004.318.6] 2024-08-18T16:20:26.528 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:20:26.527+0000 7f70e0c1f880 1 waiting for incremental sync to catch up: 2024-08-18T16:20:26.528 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000001.285.6, 00000000007.223.6, 00000000002.262.6, , 00000000001.286.6, 00000000008.292.6, 00000000003.252.6, 00000000001.272.6, 00000000001.200.6, 00000000001.143.6, 00000000003.316.6] 2024-08-18T16:20:26.528 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000005.300.6, 00000000007.223.6, 00000000006.270.6, 00000000009.330.6, 00000000005.302.6, 00000000008.292.6, 00000000008.262.6, 00000000006.296.6, 00000000007.225.6, 00000000007.155.6, 00000000004.318.6] 2024-08-18T16:20:31.529 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:20:31.529+0000 7f70e0c1f880 1 waiting for incremental sync to catch up: 2024-08-18T16:20:31.530 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000001.285.6, 00000000007.223.6, 00000000002.262.6, , 00000000001.286.6, 00000000008.292.6, 00000000003.252.6, 00000000001.272.6, 00000000001.200.6, 00000000001.143.6, 00000000003.316.6] 2024-08-18T16:20:31.530 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000005.300.6, 00000000007.223.6, 00000000006.270.6, 00000000009.330.6, 00000000005.302.6, 00000000008.292.6, 00000000008.262.6, 00000000006.296.6, 00000000007.225.6, 00000000007.155.6, 00000000004.318.6] 2024-08-18T16:20:36.531 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:20:36.531+0000 7f70e0c1f880 1 waiting for incremental sync to catch up: 2024-08-18T16:20:36.531 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000001.285.6, 00000000007.223.6, 00000000002.262.6, , 00000000001.286.6, 00000000008.292.6, 00000000003.252.6, 00000000001.272.6, 00000000001.200.6, 00000000001.143.6, 00000000003.316.6] 2024-08-18T16:20:36.532 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000005.300.6, 00000000007.223.6, 00000000006.270.6, 00000000009.330.6, 00000000005.302.6, 00000000008.292.6, 00000000008.262.6, 00000000006.296.6, 00000000007.225.6, 00000000007.155.6, 00000000004.318.6] 2024-08-18T16:20:41.533 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:20:41.532+0000 7f70e0c1f880 1 waiting for incremental sync to catch up: 2024-08-18T16:20:41.533 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000001.285.6, 00000000007.223.6, 00000000002.262.6, , 00000000001.286.6, 00000000008.292.6, 00000000003.252.6, 00000000001.272.6, 00000000001.200.6, 00000000001.143.6, 00000000003.316.6] 2024-08-18T16:20:41.533 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000005.300.6, 00000000007.223.6, 00000000006.270.6, 00000000009.330.6, 00000000005.302.6, 00000000008.292.6, 00000000008.262.6, 00000000006.296.6, 00000000007.225.6, 00000000007.155.6, 00000000004.318.6] 2024-08-18T16:20:46.534 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:20:46.533+0000 7f70e0c1f880 1 waiting for incremental sync to catch up: 2024-08-18T16:20:46.534 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000001.285.6, 00000000007.223.6, 00000000006.270.6, , 00000000001.286.6, 00000000008.292.6, 00000000008.262.6, 00000000001.272.6, 00000000001.200.6, 00000000004.149.6, 00000000004.318.6] 2024-08-18T16:20:46.534 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000005.300.6, 00000000007.223.6, 00000000006.270.6, 00000000009.330.6, 00000000005.302.6, 00000000008.292.6, 00000000008.262.6, 00000000006.296.6, 00000000007.225.6, 00000000007.155.6, 00000000004.318.6] 2024-08-18T16:20:51.535 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:20:51.535+0000 7f70e0c1f880 1 waiting for incremental sync to catch up: 2024-08-18T16:20:51.536 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000001.285.6, 00000000007.223.6, 00000000006.270.6, , 00000000001.286.6, 00000000008.292.6, 00000000008.262.6, 00000000001.272.6, 00000000001.200.6, 00000000004.149.6, 00000000004.318.6] 2024-08-18T16:20:51.536 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000005.300.6, 00000000007.223.6, 00000000006.270.6, 00000000009.330.6, 00000000005.302.6, 00000000008.292.6, 00000000008.262.6, 00000000006.296.6, 00000000007.225.6, 00000000007.155.6, 00000000004.318.6] 2024-08-18T16:20:56.537 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:20:56.536+0000 7f70e0c1f880 1 waiting for incremental sync to catch up: 2024-08-18T16:20:56.537 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000001.285.6, 00000000007.223.6, 00000000006.270.6, , 00000000001.286.6, 00000000008.292.6, 00000000008.262.6, 00000000001.272.6, 00000000001.200.6, 00000000004.149.6, 00000000004.318.6] 2024-08-18T16:20:56.537 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000005.300.6, 00000000007.223.6, 00000000006.270.6, 00000000009.330.6, 00000000005.302.6, 00000000008.292.6, 00000000008.262.6, 00000000006.296.6, 00000000007.225.6, 00000000007.155.6, 00000000004.318.6] 2024-08-18T16:21:01.538 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:21:01.537+0000 7f70e0c1f880 1 waiting for incremental sync to catch up: 2024-08-18T16:21:01.538 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000001.285.6, 00000000007.223.6, 00000000006.270.6, , 00000000001.286.6, 00000000008.292.6, 00000000008.262.6, 00000000001.272.6, 00000000001.200.6, 00000000004.149.6, 00000000004.318.6] 2024-08-18T16:21:01.539 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000005.300.6, 00000000007.223.6, 00000000006.270.6, 00000000009.330.6, 00000000005.302.6, 00000000008.292.6, 00000000008.262.6, 00000000006.296.6, 00000000007.225.6, 00000000007.155.6, 00000000004.318.6] 2024-08-18T16:21:06.540 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:21:06.539+0000 7f70e0c1f880 1 waiting for incremental sync to catch up: 2024-08-18T16:21:06.540 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000001.285.6, 00000000007.223.6, 00000000006.270.6, , 00000000001.286.6, 00000000008.292.6, 00000000008.262.6, 00000000001.272.6, 00000000001.200.6, 00000000004.149.6, 00000000004.318.6] 2024-08-18T16:21:06.540 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000005.300.6, 00000000007.223.6, 00000000006.270.6, 00000000009.330.6, 00000000005.302.6, 00000000008.292.6, 00000000008.262.6, 00000000006.296.6, 00000000007.225.6, 00000000007.155.6, 00000000004.318.6] 2024-08-18T16:21:11.541 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:21:11.540+0000 7f70e0c1f880 1 waiting for incremental sync to catch up: 2024-08-18T16:21:11.541 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000001.285.6, 00000000007.223.6, 00000000006.270.6, , 00000000001.286.6, 00000000008.292.6, 00000000008.262.6, 00000000001.272.6, 00000000001.200.6, 00000000004.149.6, 00000000004.318.6] 2024-08-18T16:21:11.541 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000005.300.6, 00000000007.223.6, 00000000006.270.6, 00000000009.330.6, 00000000005.302.6, 00000000008.292.6, 00000000008.262.6, 00000000006.296.6, 00000000007.225.6, 00000000007.155.6, 00000000004.318.6] 2024-08-18T16:21:16.542 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:21:16.542+0000 7f70e0c1f880 1 waiting for incremental sync to catch up: 2024-08-18T16:21:16.542 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000001.285.6, 00000000007.223.6, 00000000006.270.6, , 00000000001.286.6, 00000000008.292.6, 00000000008.262.6, 00000000001.272.6, 00000000001.200.6, 00000000004.149.6, 00000000004.318.6] 2024-08-18T16:21:16.543 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000005.300.6, 00000000007.223.6, 00000000006.270.6, 00000000009.330.6, 00000000005.302.6, 00000000008.292.6, 00000000008.262.6, 00000000006.296.6, 00000000007.225.6, 00000000007.155.6, 00000000004.318.6] 2024-08-18T16:21:21.543 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:21:21.543+0000 7f70e0c1f880 1 waiting for incremental sync to catch up: 2024-08-18T16:21:21.544 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000001.285.6, 00000000007.223.6, 00000000006.270.6, , 00000000001.286.6, 00000000008.292.6, 00000000008.262.6, 00000000001.272.6, 00000000001.200.6, 00000000004.149.6, 00000000004.318.6] 2024-08-18T16:21:21.544 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000005.300.6, 00000000007.223.6, 00000000006.270.6, 00000000009.330.6, 00000000005.302.6, 00000000008.292.6, 00000000008.262.6, 00000000006.296.6, 00000000007.225.6, 00000000007.155.6, 00000000004.318.6] 2024-08-18T16:21:26.544 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:21:26.544+0000 7f70e0c1f880 1 waiting for incremental sync to catch up: 2024-08-18T16:21:26.545 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000005.300.6, 00000000007.223.6, 00000000006.270.6, , 00000000002.292.6, 00000000008.292.6, 00000000008.262.6, 00000000002.282.6, 00000000007.225.6, 00000000004.149.6, 00000000004.318.6] 2024-08-18T16:21:26.545 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000005.300.6, 00000000007.223.6, 00000000006.270.6, 00000000009.330.6, 00000000005.302.6, 00000000008.292.6, 00000000008.262.6, 00000000006.296.6, 00000000007.225.6, 00000000007.155.6, 00000000004.318.6] 2024-08-18T16:21:31.546 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:21:31.545+0000 7f70e0c1f880 1 waiting for incremental sync to catch up: 2024-08-18T16:21:31.546 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000005.300.6, 00000000007.223.6, 00000000006.270.6, , 00000000002.292.6, 00000000008.292.6, 00000000008.262.6, 00000000002.282.6, 00000000007.225.6, 00000000004.149.6, 00000000004.318.6] 2024-08-18T16:21:31.547 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000005.300.6, 00000000007.223.6, 00000000006.270.6, 00000000009.330.6, 00000000005.302.6, 00000000008.292.6, 00000000008.262.6, 00000000006.296.6, 00000000007.225.6, 00000000007.155.6, 00000000004.318.6] 2024-08-18T16:21:36.548 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:21:36.548+0000 7f70e0c1f880 1 waiting for incremental sync to catch up: 2024-08-18T16:21:36.549 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000005.300.6, 00000000007.223.6, 00000000006.270.6, , 00000000002.292.6, 00000000008.292.6, 00000000008.262.6, 00000000002.282.6, 00000000007.225.6, 00000000004.149.6, 00000000004.318.6] 2024-08-18T16:21:36.549 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000005.300.6, 00000000007.223.6, 00000000006.270.6, 00000000009.330.6, 00000000005.302.6, 00000000008.292.6, 00000000008.262.6, 00000000006.296.6, 00000000007.225.6, 00000000007.155.6, 00000000004.318.6] 2024-08-18T16:21:41.550 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:21:41.549+0000 7f70e0c1f880 1 waiting for incremental sync to catch up: 2024-08-18T16:21:41.550 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000005.300.6, 00000000007.223.6, 00000000006.270.6, , 00000000002.292.6, 00000000008.292.6, 00000000008.262.6, 00000000002.282.6, 00000000007.225.6, 00000000004.149.6, 00000000004.318.6] 2024-08-18T16:21:41.550 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000005.300.6, 00000000007.223.6, 00000000006.270.6, 00000000009.330.6, 00000000005.302.6, 00000000008.292.6, 00000000008.262.6, 00000000006.296.6, 00000000007.225.6, 00000000007.155.6, 00000000004.318.6] 2024-08-18T16:21:46.551 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:21:46.550+0000 7f70e0c1f880 1 waiting for incremental sync to catch up: 2024-08-18T16:21:46.551 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000005.300.6, 00000000007.223.6, 00000000006.270.6, , 00000000002.292.6, 00000000008.292.6, 00000000008.262.6, 00000000002.282.6, 00000000007.225.6, 00000000004.149.6, 00000000004.318.6] 2024-08-18T16:21:46.551 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000005.300.6, 00000000007.223.6, 00000000006.270.6, 00000000009.330.6, 00000000005.302.6, 00000000008.292.6, 00000000008.262.6, 00000000006.296.6, 00000000007.225.6, 00000000007.155.6, 00000000004.318.6] 2024-08-18T16:21:51.552 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:21:51.552+0000 7f70e0c1f880 1 waiting for incremental sync to catch up: 2024-08-18T16:21:51.553 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000005.300.6, 00000000007.223.6, 00000000006.270.6, , 00000000002.292.6, 00000000008.292.6, 00000000008.262.6, 00000000002.282.6, 00000000007.225.6, 00000000004.149.6, 00000000004.318.6] 2024-08-18T16:21:51.553 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000005.300.6, 00000000007.223.6, 00000000006.270.6, 00000000009.330.6, 00000000005.302.6, 00000000008.292.6, 00000000008.262.6, 00000000006.296.6, 00000000007.225.6, 00000000007.155.6, 00000000004.318.6] 2024-08-18T16:21:56.554 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:21:56.553+0000 7f70e0c1f880 1 waiting for incremental sync to catch up: 2024-08-18T16:21:56.554 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000005.300.6, 00000000007.223.6, 00000000006.270.6, , 00000000002.292.6, 00000000008.292.6, 00000000008.262.6, 00000000002.282.6, 00000000007.225.6, 00000000004.149.6, 00000000004.318.6] 2024-08-18T16:21:56.554 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000005.300.6, 00000000007.223.6, 00000000006.270.6, 00000000009.330.6, 00000000005.302.6, 00000000008.292.6, 00000000008.262.6, 00000000006.296.6, 00000000007.225.6, 00000000007.155.6, 00000000004.318.6] 2024-08-18T16:22:01.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:22:01.555+0000 7f70e0c1f880 1 waiting for incremental sync to catch up: 2024-08-18T16:22:01.556 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000005.300.6, 00000000007.223.6, 00000000006.270.6, , 00000000002.292.6, 00000000008.292.6, 00000000008.262.6, 00000000002.282.6, 00000000007.225.6, 00000000004.149.6, 00000000004.318.6] 2024-08-18T16:22:01.556 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000005.300.6, 00000000007.223.6, 00000000006.270.6, 00000000009.330.6, 00000000005.302.6, 00000000008.292.6, 00000000008.262.6, 00000000006.296.6, 00000000007.225.6, 00000000007.155.6, 00000000004.318.6] 2024-08-18T16:22:06.557 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:22:06.556+0000 7f70e0c1f880 1 waiting for incremental sync to catch up: 2024-08-18T16:22:06.557 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000005.300.6, 00000000007.223.6, 00000000006.270.6, , 00000000002.292.6, 00000000008.292.6, 00000000008.262.6, 00000000002.282.6, 00000000007.225.6, 00000000004.149.6, 00000000004.318.6] 2024-08-18T16:22:06.557 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000005.300.6, 00000000007.223.6, 00000000006.270.6, 00000000009.330.6, 00000000005.302.6, 00000000008.292.6, 00000000008.262.6, 00000000006.296.6, 00000000007.225.6, 00000000007.155.6, 00000000004.318.6] 2024-08-18T16:22:11.558 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:22:11.557+0000 7f70e0c1f880 1 waiting for incremental sync to catch up: 2024-08-18T16:22:11.559 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000005.300.6, 00000000007.223.6, 00000000006.270.6, , 00000000002.292.6, 00000000008.292.6, 00000000008.262.6, 00000000002.282.6, 00000000007.225.6, 00000000004.149.6, 00000000004.318.6] 2024-08-18T16:22:11.559 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000005.300.6, 00000000007.223.6, 00000000006.270.6, 00000000009.330.6, 00000000005.302.6, 00000000008.292.6, 00000000008.262.6, 00000000006.296.6, 00000000007.225.6, 00000000007.155.6, 00000000004.318.6] 2024-08-18T16:22:16.559 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:22:16.559+0000 7f70e0c1f880 1 waiting for incremental sync to catch up: 2024-08-18T16:22:16.560 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000005.300.6, 00000000007.223.6, 00000000006.270.6, , 00000000002.292.6, 00000000008.292.6, 00000000008.262.6, 00000000002.282.6, 00000000007.225.6, 00000000004.149.6, 00000000004.318.6] 2024-08-18T16:22:16.560 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000005.300.6, 00000000007.223.6, 00000000006.270.6, 00000000009.330.6, 00000000005.302.6, 00000000008.292.6, 00000000008.262.6, 00000000006.296.6, 00000000007.225.6, 00000000007.155.6, 00000000004.318.6] 2024-08-18T16:22:21.561 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:22:21.560+0000 7f70e0c1f880 1 waiting for incremental sync to catch up: 2024-08-18T16:22:21.561 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000005.300.6, 00000000007.223.6, 00000000006.270.6, , 00000000002.292.6, 00000000008.292.6, 00000000008.262.6, 00000000002.282.6, 00000000007.225.6, 00000000004.149.6, 00000000004.318.6] 2024-08-18T16:22:21.561 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000005.300.6, 00000000007.223.6, 00000000006.270.6, 00000000009.330.6, 00000000005.302.6, 00000000008.292.6, 00000000008.262.6, 00000000006.296.6, 00000000007.225.6, 00000000007.155.6, 00000000004.318.6] 2024-08-18T16:22:26.562 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:22:26.562+0000 7f70e0c1f880 1 waiting for incremental sync to catch up: 2024-08-18T16:22:26.563 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000005.300.6, 00000000007.223.6, 00000000006.270.6, 00000000009.330.6, 00000000005.302.6, 00000000008.292.6, 00000000008.262.6, 00000000006.296.6, 00000000007.225.6, 00000000004.149.6, 00000000004.318.6] 2024-08-18T16:22:26.563 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000005.300.6, 00000000007.223.6, 00000000006.270.6, 00000000009.330.6, 00000000005.302.6, 00000000008.292.6, 00000000008.262.6, 00000000006.296.6, 00000000007.225.6, 00000000007.155.6, 00000000004.318.6] 2024-08-18T16:22:31.564 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:22:31.563+0000 7f70e0c1f880 1 waiting for incremental sync to catch up: 2024-08-18T16:22:31.564 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000005.300.6, 00000000007.223.6, 00000000006.270.6, 00000000009.330.6, 00000000005.302.6, 00000000008.292.6, 00000000008.262.6, 00000000006.296.6, 00000000007.225.6, 00000000004.149.6, 00000000004.318.6] 2024-08-18T16:22:31.564 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000005.300.6, 00000000007.223.6, 00000000006.270.6, 00000000009.330.6, 00000000005.302.6, 00000000008.292.6, 00000000008.262.6, 00000000006.296.6, 00000000007.225.6, 00000000007.155.6, 00000000004.318.6] 2024-08-18T16:22:36.565 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:22:36.565+0000 7f70e0c1f880 1 waiting for incremental sync to catch up: 2024-08-18T16:22:36.566 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000005.300.6, 00000000007.223.6, 00000000006.270.6, 00000000009.330.6, 00000000005.302.6, 00000000008.292.6, 00000000008.262.6, 00000000006.296.6, 00000000007.225.6, 00000000004.149.6, 00000000004.318.6] 2024-08-18T16:22:36.566 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000005.300.6, 00000000007.223.6, 00000000006.270.6, 00000000009.330.6, 00000000005.302.6, 00000000008.292.6, 00000000008.262.6, 00000000006.296.6, 00000000007.225.6, 00000000007.155.6, 00000000004.318.6] 2024-08-18T16:22:41.567 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:22:41.566+0000 7f70e0c1f880 1 waiting for incremental sync to catch up: 2024-08-18T16:22:41.567 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000005.300.6, 00000000007.223.6, 00000000006.270.6, 00000000009.330.6, 00000000005.302.6, 00000000008.292.6, 00000000008.262.6, 00000000006.296.6, 00000000007.225.6, 00000000004.149.6, 00000000004.318.6] 2024-08-18T16:22:41.567 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000005.300.6, 00000000007.223.6, 00000000006.270.6, 00000000009.330.6, 00000000005.302.6, 00000000008.292.6, 00000000008.262.6, 00000000006.296.6, 00000000007.225.6, 00000000007.155.6, 00000000004.318.6] 2024-08-18T16:22:46.568 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:22:46.568+0000 7f70e0c1f880 1 waiting for incremental sync to catch up: 2024-08-18T16:22:46.568 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000005.300.6, 00000000007.223.6, 00000000006.270.6, 00000000009.330.6, 00000000005.302.6, 00000000008.292.6, 00000000008.262.6, 00000000006.296.6, 00000000007.225.6, 00000000004.149.6, 00000000004.318.6] 2024-08-18T16:22:46.569 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000005.300.6, 00000000007.223.6, 00000000006.270.6, 00000000009.330.6, 00000000005.302.6, 00000000008.292.6, 00000000008.262.6, 00000000006.296.6, 00000000007.225.6, 00000000007.155.6, 00000000004.318.6] 2024-08-18T16:22:51.570 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:22:51.569+0000 7f70e0c1f880 1 waiting for incremental sync to catch up: 2024-08-18T16:22:51.570 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000005.300.6, 00000000007.223.6, 00000000006.270.6, 00000000009.330.6, 00000000005.302.6, 00000000008.292.6, 00000000008.262.6, 00000000006.296.6, 00000000007.225.6, 00000000004.149.6, 00000000004.318.6] 2024-08-18T16:22:51.570 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000005.300.6, 00000000007.223.6, 00000000006.270.6, 00000000009.330.6, 00000000005.302.6, 00000000008.292.6, 00000000008.262.6, 00000000006.296.6, 00000000007.225.6, 00000000007.155.6, 00000000004.318.6] 2024-08-18T16:22:56.571 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:22:56.570+0000 7f70e0c1f880 1 waiting for incremental sync to catch up: 2024-08-18T16:22:56.572 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000005.300.6, 00000000007.223.6, 00000000006.270.6, 00000000009.330.6, 00000000005.302.6, 00000000008.292.6, 00000000008.262.6, 00000000006.296.6, 00000000007.225.6, 00000000004.149.6, 00000000004.318.6] 2024-08-18T16:22:56.572 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000005.300.6, 00000000007.223.6, 00000000006.270.6, 00000000009.330.6, 00000000005.302.6, 00000000008.292.6, 00000000008.262.6, 00000000006.296.6, 00000000007.225.6, 00000000007.155.6, 00000000004.318.6] 2024-08-18T16:23:01.573 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:23:01.572+0000 7f70e0c1f880 1 waiting for incremental sync to catch up: 2024-08-18T16:23:01.573 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000005.300.6, 00000000007.223.6, 00000000006.270.6, 00000000009.330.6, 00000000005.302.6, 00000000008.292.6, 00000000008.262.6, 00000000006.296.6, 00000000007.225.6, 00000000004.149.6, 00000000004.318.6] 2024-08-18T16:23:01.573 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000005.300.6, 00000000007.223.6, 00000000006.270.6, 00000000009.330.6, 00000000005.302.6, 00000000008.292.6, 00000000008.262.6, 00000000006.296.6, 00000000007.225.6, 00000000007.155.6, 00000000004.318.6] 2024-08-18T16:23:06.574 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:23:06.573+0000 7f70e0c1f880 1 bucket sync caught up with source: 2024-08-18T16:23:06.574 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000005.300.6, 00000000007.223.6, 00000000006.270.6, 00000000009.330.6, 00000000005.302.6, 00000000008.292.6, 00000000008.262.6, 00000000006.296.6, 00000000007.225.6, 00000000007.155.6, 00000000004.318.6] 2024-08-18T16:23:06.574 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000005.300.6, 00000000007.223.6, 00000000006.270.6, 00000000009.330.6, 00000000005.302.6, 00000000008.292.6, 00000000008.262.6, 00000000006.296.6, 00000000007.225.6, 00000000007.155.6, 00000000004.318.6] 2024-08-18T16:23:06.574 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:23:06.573+0000 7f70e0c1f880 0 bucket checkpoint complete 2024-08-18T16:23:06.592 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-77', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:23:06.592 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-77', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:23:06.593 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-77 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:23:06.652 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:23:06.652 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:23:06.689 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:23:06.688+0000 7f1780d36880 1 bucket sync caught up with source: 2024-08-18T16:23:06.689 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000005.167.6, 00000000007.207.6, 00000000006.166.6, 00000000009.166.6, 00000000005.169.6, 00000000008.212.6, 00000000008.249.6, 00000000006.216.6, 00000000007.209.6, 00000000007.118.6, 00000000004.137.6] 2024-08-18T16:23:06.689 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000005.167.6, 00000000007.207.6, 00000000006.166.6, 00000000009.166.6, 00000000005.169.6, 00000000008.212.6, 00000000008.249.6, 00000000006.216.6, 00000000007.209.6, 00000000007.118.6, 00000000004.137.6] 2024-08-18T16:23:06.689 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:23:06.688+0000 7f1780d36880 0 bucket checkpoint complete 2024-08-18T16:23:06.704 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-77', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:23:06.704 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-77', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:23:06.704 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-77 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T16:23:06.767 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:23:06.767 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:23:06.806 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:23:06.805+0000 7f5407a19880 1 waiting for incremental sync to catch up: 2024-08-18T16:23:06.806 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000005.167.6, 00000000007.207.6, 00000000006.166.6, 00000000009.166.6, 00000000005.169.6, 00000000008.212.6, 00000000008.249.6, 00000000004.210.6, 00000000007.209.6, 00000000007.118.6, 00000000004.137.6] 2024-08-18T16:23:06.807 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000005.167.6, 00000000007.207.6, 00000000006.166.6, 00000000009.166.6, 00000000005.169.6, 00000000008.212.6, 00000000008.249.6, 00000000006.216.6, 00000000007.209.6, 00000000007.118.6, 00000000004.137.6] 2024-08-18T16:23:11.808 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:23:11.807+0000 7f5407a19880 1 waiting for incremental sync to catch up: 2024-08-18T16:23:11.808 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000005.167.6, 00000000007.207.6, 00000000006.166.6, 00000000009.166.6, 00000000005.169.6, 00000000008.212.6, 00000000008.249.6, 00000000004.210.6, 00000000007.209.6, 00000000007.118.6, 00000000004.137.6] 2024-08-18T16:23:11.808 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000005.167.6, 00000000007.207.6, 00000000006.166.6, 00000000009.166.6, 00000000005.169.6, 00000000008.212.6, 00000000008.249.6, 00000000006.216.6, 00000000007.209.6, 00000000007.118.6, 00000000004.137.6] 2024-08-18T16:23:16.809 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:23:16.808+0000 7f5407a19880 1 bucket sync caught up with source: 2024-08-18T16:23:16.809 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000005.167.6, 00000000007.207.6, 00000000006.166.6, 00000000009.166.6, 00000000005.169.6, 00000000008.212.6, 00000000008.249.6, 00000000006.216.6, 00000000007.209.6, 00000000007.118.6, 00000000004.137.6] 2024-08-18T16:23:16.809 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000005.167.6, 00000000007.207.6, 00000000006.166.6, 00000000009.166.6, 00000000005.169.6, 00000000008.212.6, 00000000008.249.6, 00000000006.216.6, 00000000007.209.6, 00000000007.118.6, 00000000004.137.6] 2024-08-18T16:23:16.809 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:23:16.808+0000 7f5407a19880 0 bucket checkpoint complete 2024-08-18T16:23:16.828 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-77 zones={test-zone2, test-zone1} 2024-08-18T16:23:16.843 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:23:16.850 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000000 2024-08-18T16:23:16.850 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000001 2024-08-18T16:23:16.850 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000002 2024-08-18T16:23:16.851 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000003 2024-08-18T16:23:16.851 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000004 2024-08-18T16:23:16.851 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000005 2024-08-18T16:23:16.851 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000006 2024-08-18T16:23:16.851 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000007 2024-08-18T16:23:16.851 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000008 2024-08-18T16:23:16.851 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000009 2024-08-18T16:23:16.851 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000010 2024-08-18T16:23:16.851 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000011 2024-08-18T16:23:16.851 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000012 2024-08-18T16:23:16.851 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000013 2024-08-18T16:23:16.851 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000014 2024-08-18T16:23:16.851 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000015 2024-08-18T16:23:16.851 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000016 2024-08-18T16:23:16.851 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000017 2024-08-18T16:23:16.852 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000018 2024-08-18T16:23:16.852 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000019 2024-08-18T16:23:16.852 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000020 2024-08-18T16:23:16.852 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000021 2024-08-18T16:23:16.852 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000022 2024-08-18T16:23:16.852 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000023 2024-08-18T16:23:16.852 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000024 2024-08-18T16:23:16.852 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000025 2024-08-18T16:23:16.852 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000026 2024-08-18T16:23:16.852 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000027 2024-08-18T16:23:16.852 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000028 2024-08-18T16:23:16.853 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000029 2024-08-18T16:23:16.853 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000030 2024-08-18T16:23:16.853 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000031 2024-08-18T16:23:16.853 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000032 2024-08-18T16:23:16.853 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000033 2024-08-18T16:23:16.853 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000036 2024-08-18T16:23:16.853 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000037 2024-08-18T16:23:16.853 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000039 2024-08-18T16:23:16.853 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000040 2024-08-18T16:23:16.853 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000042 2024-08-18T16:23:16.853 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000045 2024-08-18T16:23:16.853 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000046 2024-08-18T16:23:16.853 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000048 2024-08-18T16:23:16.853 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000049 2024-08-18T16:23:16.854 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000050 2024-08-18T16:23:16.854 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000053 2024-08-18T16:23:16.854 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000055 2024-08-18T16:23:16.854 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000056 2024-08-18T16:23:16.854 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000057 2024-08-18T16:23:16.854 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000058 2024-08-18T16:23:16.854 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000059 2024-08-18T16:23:16.854 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000060 2024-08-18T16:23:16.854 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000061 2024-08-18T16:23:16.854 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000062 2024-08-18T16:23:16.854 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000065 2024-08-18T16:23:16.854 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000076 2024-08-18T16:23:16.854 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000133 2024-08-18T16:23:16.855 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000162 2024-08-18T16:23:16.855 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000271 2024-08-18T16:23:16.855 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000283 2024-08-18T16:23:16.855 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000317 2024-08-18T16:23:16.855 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000423 2024-08-18T16:23:16.855 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000467 2024-08-18T16:23:16.855 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:23:16.862 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000000 2024-08-18T16:23:16.862 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000001 2024-08-18T16:23:16.863 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000002 2024-08-18T16:23:16.863 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000003 2024-08-18T16:23:16.863 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000004 2024-08-18T16:23:16.863 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000005 2024-08-18T16:23:16.863 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000006 2024-08-18T16:23:16.863 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000007 2024-08-18T16:23:16.863 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000008 2024-08-18T16:23:16.863 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000009 2024-08-18T16:23:16.863 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000010 2024-08-18T16:23:16.863 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000011 2024-08-18T16:23:16.863 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000012 2024-08-18T16:23:16.863 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000013 2024-08-18T16:23:16.863 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000014 2024-08-18T16:23:16.863 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000015 2024-08-18T16:23:16.864 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000016 2024-08-18T16:23:16.864 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000017 2024-08-18T16:23:16.864 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000018 2024-08-18T16:23:16.864 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000019 2024-08-18T16:23:16.864 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000020 2024-08-18T16:23:16.864 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000021 2024-08-18T16:23:16.864 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000022 2024-08-18T16:23:16.864 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000023 2024-08-18T16:23:16.864 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000024 2024-08-18T16:23:16.864 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000025 2024-08-18T16:23:16.864 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000026 2024-08-18T16:23:16.864 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000027 2024-08-18T16:23:16.864 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000028 2024-08-18T16:23:16.864 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000029 2024-08-18T16:23:16.865 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000030 2024-08-18T16:23:16.865 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000031 2024-08-18T16:23:16.865 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000032 2024-08-18T16:23:16.865 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000033 2024-08-18T16:23:16.865 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000036 2024-08-18T16:23:16.865 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000037 2024-08-18T16:23:16.865 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000039 2024-08-18T16:23:16.865 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000040 2024-08-18T16:23:16.865 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000042 2024-08-18T16:23:16.865 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000045 2024-08-18T16:23:16.865 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000046 2024-08-18T16:23:16.865 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000048 2024-08-18T16:23:16.866 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000049 2024-08-18T16:23:16.866 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000050 2024-08-18T16:23:16.866 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000053 2024-08-18T16:23:16.866 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000055 2024-08-18T16:23:16.866 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000056 2024-08-18T16:23:16.866 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000057 2024-08-18T16:23:16.866 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000058 2024-08-18T16:23:16.866 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000059 2024-08-18T16:23:16.866 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000060 2024-08-18T16:23:16.867 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000061 2024-08-18T16:23:16.867 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000062 2024-08-18T16:23:16.867 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000065 2024-08-18T16:23:16.867 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000076 2024-08-18T16:23:16.867 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000133 2024-08-18T16:23:16.867 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000162 2024-08-18T16:23:16.868 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000271 2024-08-18T16:23:16.868 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000283 2024-08-18T16:23:16.868 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000317 2024-08-18T16:23:16.868 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000423 2024-08-18T16:23:16.868 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000467 2024-08-18T16:23:16.886 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2024-08-18T16:23:16.938 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2024-08-18T16:23:17.004 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2024-08-18T16:23:17.016 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000001 2024-08-18T16:23:17.067 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000001 2024-08-18T16:23:17.082 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000001 2024-08-18T16:23:17.093 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2024-08-18T16:23:17.147 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2024-08-18T16:23:17.204 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2024-08-18T16:23:17.216 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2024-08-18T16:23:17.269 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2024-08-18T16:23:17.331 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2024-08-18T16:23:17.385 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000004 2024-08-18T16:23:17.393 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000004 2024-08-18T16:23:17.450 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000004 2024-08-18T16:23:17.506 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2024-08-18T16:23:17.513 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2024-08-18T16:23:17.574 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2024-08-18T16:23:17.584 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2024-08-18T16:23:17.635 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2024-08-18T16:23:17.650 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2024-08-18T16:23:17.701 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000007 2024-08-18T16:23:17.710 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000007 2024-08-18T16:23:17.766 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000007 2024-08-18T16:23:17.817 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000008 2024-08-18T16:23:17.824 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000008 2024-08-18T16:23:17.878 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000008 2024-08-18T16:23:17.933 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000009 2024-08-18T16:23:17.940 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000009 2024-08-18T16:23:17.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000009 2024-08-18T16:23:18.045 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2024-08-18T16:23:18.053 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2024-08-18T16:23:18.111 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2024-08-18T16:23:18.166 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2024-08-18T16:23:18.174 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2024-08-18T16:23:18.187 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2024-08-18T16:23:18.237 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2024-08-18T16:23:18.244 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2024-08-18T16:23:18.302 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2024-08-18T16:23:18.357 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2024-08-18T16:23:18.365 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2024-08-18T16:23:18.426 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2024-08-18T16:23:18.481 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000014 2024-08-18T16:23:18.488 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000014 2024-08-18T16:23:18.546 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000014 2024-08-18T16:23:18.597 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000015 2024-08-18T16:23:18.603 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000015 2024-08-18T16:23:18.657 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000015 2024-08-18T16:23:18.715 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000016 2024-08-18T16:23:18.766 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000016 2024-08-18T16:23:18.779 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000016 2024-08-18T16:23:18.834 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000017 2024-08-18T16:23:18.841 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000017 2024-08-18T16:23:18.898 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000017 2024-08-18T16:23:18.955 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000018 2024-08-18T16:23:18.963 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000018 2024-08-18T16:23:19.022 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000018 2024-08-18T16:23:19.032 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000019 2024-08-18T16:23:19.039 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000019 2024-08-18T16:23:19.095 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000019 2024-08-18T16:23:19.151 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000020 2024-08-18T16:23:19.159 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000020 2024-08-18T16:23:19.220 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000020 2024-08-18T16:23:19.275 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000021 2024-08-18T16:23:19.330 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000021 2024-08-18T16:23:19.344 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000021 2024-08-18T16:23:19.395 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000022 2024-08-18T16:23:19.403 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000022 2024-08-18T16:23:19.459 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000022 2024-08-18T16:23:19.468 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000023 2024-08-18T16:23:19.518 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000023 2024-08-18T16:23:19.532 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000023 2024-08-18T16:23:19.542 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000024 2024-08-18T16:23:19.548 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000024 2024-08-18T16:23:19.602 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000024 2024-08-18T16:23:19.653 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000025 2024-08-18T16:23:19.660 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000025 2024-08-18T16:23:19.718 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000025 2024-08-18T16:23:19.769 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000026 2024-08-18T16:23:19.776 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000026 2024-08-18T16:23:19.834 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000026 2024-08-18T16:23:19.885 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000027 2024-08-18T16:23:19.892 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000027 2024-08-18T16:23:19.946 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000027 2024-08-18T16:23:20.001 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000028 2024-08-18T16:23:20.008 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000028 2024-08-18T16:23:20.062 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000028 2024-08-18T16:23:20.113 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000029 2024-08-18T16:23:20.119 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000029 2024-08-18T16:23:20.178 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000029 2024-08-18T16:23:20.229 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000030 2024-08-18T16:23:20.236 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000030 2024-08-18T16:23:20.289 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000030 2024-08-18T16:23:20.343 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000031 2024-08-18T16:23:20.390 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000031 2024-08-18T16:23:20.405 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000031 2024-08-18T16:23:20.455 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000032 2024-08-18T16:23:20.506 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000032 2024-08-18T16:23:20.520 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000032 2024-08-18T16:23:20.530 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000033 2024-08-18T16:23:20.537 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000033 2024-08-18T16:23:20.594 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000033 2024-08-18T16:23:20.645 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000036 2024-08-18T16:23:20.652 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000036 2024-08-18T16:23:20.706 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000036 2024-08-18T16:23:20.757 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000037 2024-08-18T16:23:20.764 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000037 2024-08-18T16:23:20.819 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000037 2024-08-18T16:23:20.872 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000039 2024-08-18T16:23:20.880 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000039 2024-08-18T16:23:20.934 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000039 2024-08-18T16:23:20.985 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000040 2024-08-18T16:23:20.993 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000040 2024-08-18T16:23:21.054 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000040 2024-08-18T16:23:21.105 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000042 2024-08-18T16:23:21.113 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000042 2024-08-18T16:23:21.169 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000042 2024-08-18T16:23:21.226 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000045 2024-08-18T16:23:21.234 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000045 2024-08-18T16:23:21.294 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000045 2024-08-18T16:23:21.303 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000046 2024-08-18T16:23:21.352 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000046 2024-08-18T16:23:21.410 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000046 2024-08-18T16:23:21.420 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000048 2024-08-18T16:23:21.427 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000048 2024-08-18T16:23:21.485 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000048 2024-08-18T16:23:21.539 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000049 2024-08-18T16:23:21.594 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000049 2024-08-18T16:23:21.608 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000049 2024-08-18T16:23:21.661 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000050 2024-08-18T16:23:21.669 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000050 2024-08-18T16:23:21.730 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000050 2024-08-18T16:23:21.781 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000053 2024-08-18T16:23:21.788 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000053 2024-08-18T16:23:21.843 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000053 2024-08-18T16:23:21.897 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000055 2024-08-18T16:23:21.904 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000055 2024-08-18T16:23:21.958 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000055 2024-08-18T16:23:22.009 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000056 2024-08-18T16:23:22.016 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000056 2024-08-18T16:23:22.074 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000056 2024-08-18T16:23:22.125 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000057 2024-08-18T16:23:22.132 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000057 2024-08-18T16:23:22.186 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000057 2024-08-18T16:23:22.238 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000058 2024-08-18T16:23:22.290 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000058 2024-08-18T16:23:22.303 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000058 2024-08-18T16:23:22.357 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000059 2024-08-18T16:23:22.364 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000059 2024-08-18T16:23:22.378 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000059 2024-08-18T16:23:22.431 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000060 2024-08-18T16:23:22.438 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000060 2024-08-18T16:23:22.497 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000060 2024-08-18T16:23:22.551 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000061 2024-08-18T16:23:22.603 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000061 2024-08-18T16:23:22.662 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000061 2024-08-18T16:23:22.673 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000062 2024-08-18T16:23:22.722 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000062 2024-08-18T16:23:22.737 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000062 2024-08-18T16:23:22.790 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000065 2024-08-18T16:23:22.838 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000065 2024-08-18T16:23:22.851 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000065 2024-08-18T16:23:22.906 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000076 2024-08-18T16:23:22.915 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000076 2024-08-18T16:23:22.970 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000076 2024-08-18T16:23:22.981 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000133 2024-08-18T16:23:23.030 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000133 2024-08-18T16:23:23.044 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000133 2024-08-18T16:23:23.099 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000162 2024-08-18T16:23:23.107 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000162 2024-08-18T16:23:23.168 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000162 2024-08-18T16:23:23.223 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000271 2024-08-18T16:23:23.274 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000271 2024-08-18T16:23:23.288 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000271 2024-08-18T16:23:23.339 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000283 2024-08-18T16:23:23.394 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000283 2024-08-18T16:23:23.408 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000283 2024-08-18T16:23:23.459 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000317 2024-08-18T16:23:23.467 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000317 2024-08-18T16:23:23.525 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000317 2024-08-18T16:23:23.582 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000423 2024-08-18T16:23:23.589 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000423 2024-08-18T16:23:23.650 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000423 2024-08-18T16:23:23.701 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000467 2024-08-18T16:23:23.708 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000467 2024-08-18T16:23:23.767 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000467 2024-08-18T16:23:23.819 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-77 zones={test-zone2, test-zone1} 2024-08-18T16:23:23.819 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-77 zones={test-zone3, test-zone1} 2024-08-18T16:23:23.828 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:23:23.833 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000000 2024-08-18T16:23:23.833 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000001 2024-08-18T16:23:23.833 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000002 2024-08-18T16:23:23.833 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000003 2024-08-18T16:23:23.834 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000004 2024-08-18T16:23:23.834 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000005 2024-08-18T16:23:23.834 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000006 2024-08-18T16:23:23.834 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000007 2024-08-18T16:23:23.834 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000008 2024-08-18T16:23:23.834 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000009 2024-08-18T16:23:23.834 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000010 2024-08-18T16:23:23.834 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000011 2024-08-18T16:23:23.834 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000012 2024-08-18T16:23:23.834 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000013 2024-08-18T16:23:23.834 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000014 2024-08-18T16:23:23.834 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000015 2024-08-18T16:23:23.835 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000016 2024-08-18T16:23:23.835 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000017 2024-08-18T16:23:23.835 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000018 2024-08-18T16:23:23.835 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000019 2024-08-18T16:23:23.835 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000020 2024-08-18T16:23:23.835 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000021 2024-08-18T16:23:23.835 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000022 2024-08-18T16:23:23.835 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000023 2024-08-18T16:23:23.835 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000024 2024-08-18T16:23:23.835 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000025 2024-08-18T16:23:23.835 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000026 2024-08-18T16:23:23.835 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000027 2024-08-18T16:23:23.835 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000028 2024-08-18T16:23:23.836 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000029 2024-08-18T16:23:23.836 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000030 2024-08-18T16:23:23.836 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000031 2024-08-18T16:23:23.836 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000032 2024-08-18T16:23:23.836 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000033 2024-08-18T16:23:23.836 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000036 2024-08-18T16:23:23.836 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000037 2024-08-18T16:23:23.836 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000039 2024-08-18T16:23:23.836 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000040 2024-08-18T16:23:23.836 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000042 2024-08-18T16:23:23.836 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000045 2024-08-18T16:23:23.836 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000046 2024-08-18T16:23:23.836 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000048 2024-08-18T16:23:23.837 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000049 2024-08-18T16:23:23.837 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000050 2024-08-18T16:23:23.837 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000053 2024-08-18T16:23:23.837 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000055 2024-08-18T16:23:23.837 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000056 2024-08-18T16:23:23.837 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000057 2024-08-18T16:23:23.837 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000058 2024-08-18T16:23:23.837 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000059 2024-08-18T16:23:23.837 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000060 2024-08-18T16:23:23.837 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000061 2024-08-18T16:23:23.837 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000062 2024-08-18T16:23:23.837 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000065 2024-08-18T16:23:23.838 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000076 2024-08-18T16:23:23.838 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000133 2024-08-18T16:23:23.838 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000162 2024-08-18T16:23:23.838 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000271 2024-08-18T16:23:23.838 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000283 2024-08-18T16:23:23.838 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000317 2024-08-18T16:23:23.838 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000423 2024-08-18T16:23:23.838 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000467 2024-08-18T16:23:23.838 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:23:23.845 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000000 2024-08-18T16:23:23.845 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000001 2024-08-18T16:23:23.845 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000002 2024-08-18T16:23:23.845 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000003 2024-08-18T16:23:23.846 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000004 2024-08-18T16:23:23.846 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000005 2024-08-18T16:23:23.846 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000006 2024-08-18T16:23:23.846 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000007 2024-08-18T16:23:23.846 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000008 2024-08-18T16:23:23.846 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000009 2024-08-18T16:23:23.846 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000010 2024-08-18T16:23:23.846 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000011 2024-08-18T16:23:23.846 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000012 2024-08-18T16:23:23.846 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000013 2024-08-18T16:23:23.846 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000014 2024-08-18T16:23:23.847 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000015 2024-08-18T16:23:23.847 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000016 2024-08-18T16:23:23.847 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000017 2024-08-18T16:23:23.847 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000018 2024-08-18T16:23:23.847 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000019 2024-08-18T16:23:23.847 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000020 2024-08-18T16:23:23.847 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000021 2024-08-18T16:23:23.847 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000022 2024-08-18T16:23:23.847 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000023 2024-08-18T16:23:23.847 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000024 2024-08-18T16:23:23.847 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000025 2024-08-18T16:23:23.848 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000026 2024-08-18T16:23:23.848 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000027 2024-08-18T16:23:23.848 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000028 2024-08-18T16:23:23.848 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000029 2024-08-18T16:23:23.848 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000030 2024-08-18T16:23:23.848 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000031 2024-08-18T16:23:23.848 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000032 2024-08-18T16:23:23.848 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000033 2024-08-18T16:23:23.848 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000036 2024-08-18T16:23:23.848 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000037 2024-08-18T16:23:23.848 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000039 2024-08-18T16:23:23.848 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000040 2024-08-18T16:23:23.849 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000042 2024-08-18T16:23:23.849 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000045 2024-08-18T16:23:23.849 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000046 2024-08-18T16:23:23.849 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000048 2024-08-18T16:23:23.849 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000049 2024-08-18T16:23:23.849 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000050 2024-08-18T16:23:23.849 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000053 2024-08-18T16:23:23.849 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000055 2024-08-18T16:23:23.849 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000056 2024-08-18T16:23:23.849 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000057 2024-08-18T16:23:23.849 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000058 2024-08-18T16:23:23.849 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000059 2024-08-18T16:23:23.849 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000060 2024-08-18T16:23:23.850 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000061 2024-08-18T16:23:23.850 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000062 2024-08-18T16:23:23.850 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000065 2024-08-18T16:23:23.850 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000076 2024-08-18T16:23:23.850 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000133 2024-08-18T16:23:23.850 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000162 2024-08-18T16:23:23.850 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000271 2024-08-18T16:23:23.850 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000283 2024-08-18T16:23:23.850 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000317 2024-08-18T16:23:23.850 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000423 2024-08-18T16:23:23.850 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000467 2024-08-18T16:23:23.862 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2024-08-18T16:23:23.914 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2024-08-18T16:23:23.928 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2024-08-18T16:23:23.982 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000001 2024-08-18T16:23:24.034 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000001 2024-08-18T16:23:24.046 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000001 2024-08-18T16:23:24.099 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2024-08-18T16:23:24.150 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2024-08-18T16:23:24.162 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2024-08-18T16:23:24.214 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2024-08-18T16:23:24.221 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2024-08-18T16:23:24.273 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2024-08-18T16:23:24.329 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000004 2024-08-18T16:23:24.334 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000004 2024-08-18T16:23:24.388 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000004 2024-08-18T16:23:24.443 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2024-08-18T16:23:24.449 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2024-08-18T16:23:24.461 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2024-08-18T16:23:24.515 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2024-08-18T16:23:24.521 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2024-08-18T16:23:24.533 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2024-08-18T16:23:24.582 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000007 2024-08-18T16:23:24.589 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000007 2024-08-18T16:23:24.601 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000007 2024-08-18T16:23:24.651 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000008 2024-08-18T16:23:24.657 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000008 2024-08-18T16:23:24.712 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000008 2024-08-18T16:23:24.763 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000009 2024-08-18T16:23:24.769 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000009 2024-08-18T16:23:24.822 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000009 2024-08-18T16:23:24.872 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2024-08-18T16:23:24.878 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2024-08-18T16:23:24.932 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2024-08-18T16:23:24.983 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2024-08-18T16:23:24.990 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2024-08-18T16:23:25.001 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2024-08-18T16:23:25.052 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2024-08-18T16:23:25.058 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2024-08-18T16:23:25.070 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2024-08-18T16:23:25.122 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2024-08-18T16:23:25.128 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2024-08-18T16:23:25.186 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2024-08-18T16:23:25.195 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000014 2024-08-18T16:23:25.201 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000014 2024-08-18T16:23:25.256 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000014 2024-08-18T16:23:25.307 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000015 2024-08-18T16:23:25.314 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000015 2024-08-18T16:23:25.374 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000015 2024-08-18T16:23:25.384 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000016 2024-08-18T16:23:25.435 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000016 2024-08-18T16:23:25.447 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000016 2024-08-18T16:23:25.501 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000017 2024-08-18T16:23:25.507 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000017 2024-08-18T16:23:25.562 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000017 2024-08-18T16:23:25.571 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000018 2024-08-18T16:23:25.620 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000018 2024-08-18T16:23:25.632 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000018 2024-08-18T16:23:25.684 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000019 2024-08-18T16:23:25.711 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000019 2024-08-18T16:23:25.765 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000019 2024-08-18T16:23:25.819 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000020 2024-08-18T16:23:25.870 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000020 2024-08-18T16:23:25.881 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000020 2024-08-18T16:23:25.940 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000021 2024-08-18T16:23:25.990 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000021 2024-08-18T16:23:26.002 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000021 2024-08-18T16:23:26.011 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000022 2024-08-18T16:23:26.060 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000022 2024-08-18T16:23:26.118 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000022 2024-08-18T16:23:26.127 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000023 2024-08-18T16:23:26.136 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000023 2024-08-18T16:23:26.192 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000023 2024-08-18T16:23:26.201 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000024 2024-08-18T16:23:26.208 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000024 2024-08-18T16:23:26.261 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000024 2024-08-18T16:23:26.310 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000025 2024-08-18T16:23:26.317 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000025 2024-08-18T16:23:26.329 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000025 2024-08-18T16:23:26.383 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000026 2024-08-18T16:23:26.430 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000026 2024-08-18T16:23:26.442 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000026 2024-08-18T16:23:26.451 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000027 2024-08-18T16:23:26.503 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000027 2024-08-18T16:23:26.560 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000027 2024-08-18T16:23:26.581 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000028 2024-08-18T16:23:26.634 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000028 2024-08-18T16:23:26.646 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000028 2024-08-18T16:23:26.702 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000029 2024-08-18T16:23:26.708 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000029 2024-08-18T16:23:26.765 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000029 2024-08-18T16:23:26.815 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000030 2024-08-18T16:23:26.866 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000030 2024-08-18T16:23:26.878 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000030 2024-08-18T16:23:26.887 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000031 2024-08-18T16:23:26.936 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000031 2024-08-18T16:23:26.990 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000031 2024-08-18T16:23:26.999 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000032 2024-08-18T16:23:27.005 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000032 2024-08-18T16:23:27.058 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000032 2024-08-18T16:23:27.108 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000033 2024-08-18T16:23:27.115 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000033 2024-08-18T16:23:27.172 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000033 2024-08-18T16:23:27.181 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000036 2024-08-18T16:23:27.230 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000036 2024-08-18T16:23:27.242 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000036 2024-08-18T16:23:27.251 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000037 2024-08-18T16:23:27.299 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000037 2024-08-18T16:23:27.351 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000037 2024-08-18T16:23:27.407 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000039 2024-08-18T16:23:27.414 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000039 2024-08-18T16:23:27.426 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000039 2024-08-18T16:23:27.477 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000040 2024-08-18T16:23:27.483 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000040 2024-08-18T16:23:27.539 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000040 2024-08-18T16:23:27.588 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000042 2024-08-18T16:23:27.595 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000042 2024-08-18T16:23:27.649 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000042 2024-08-18T16:23:27.699 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000045 2024-08-18T16:23:27.751 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000045 2024-08-18T16:23:27.806 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000045 2024-08-18T16:23:27.815 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000046 2024-08-18T16:23:27.821 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000046 2024-08-18T16:23:27.876 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000046 2024-08-18T16:23:27.930 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000048 2024-08-18T16:23:27.936 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000048 2024-08-18T16:23:27.990 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000048 2024-08-18T16:23:27.999 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000049 2024-08-18T16:23:28.051 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000049 2024-08-18T16:23:28.063 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000049 2024-08-18T16:23:28.113 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000050 2024-08-18T16:23:28.119 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000050 2024-08-18T16:23:28.172 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000050 2024-08-18T16:23:28.223 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000053 2024-08-18T16:23:28.229 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000053 2024-08-18T16:23:28.241 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000053 2024-08-18T16:23:28.294 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000055 2024-08-18T16:23:28.300 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000055 2024-08-18T16:23:28.311 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000055 2024-08-18T16:23:28.367 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000056 2024-08-18T16:23:28.373 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000056 2024-08-18T16:23:28.385 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000056 2024-08-18T16:23:28.442 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000057 2024-08-18T16:23:28.449 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000057 2024-08-18T16:23:28.510 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000057 2024-08-18T16:23:28.521 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000058 2024-08-18T16:23:28.571 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000058 2024-08-18T16:23:28.583 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000058 2024-08-18T16:23:28.637 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000059 2024-08-18T16:23:28.643 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000059 2024-08-18T16:23:28.700 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000059 2024-08-18T16:23:28.755 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000060 2024-08-18T16:23:28.762 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000060 2024-08-18T16:23:28.818 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000060 2024-08-18T16:23:28.872 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000061 2024-08-18T16:23:28.879 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000061 2024-08-18T16:23:28.931 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000061 2024-08-18T16:23:28.987 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000062 2024-08-18T16:23:28.993 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000062 2024-08-18T16:23:29.005 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000062 2024-08-18T16:23:29.058 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000065 2024-08-18T16:23:29.065 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000065 2024-08-18T16:23:29.077 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000065 2024-08-18T16:23:29.131 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000076 2024-08-18T16:23:29.138 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000076 2024-08-18T16:23:29.150 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000076 2024-08-18T16:23:29.200 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000133 2024-08-18T16:23:29.207 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000133 2024-08-18T16:23:29.260 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000133 2024-08-18T16:23:29.311 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000162 2024-08-18T16:23:29.318 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000162 2024-08-18T16:23:29.330 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000162 2024-08-18T16:23:29.383 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000271 2024-08-18T16:23:29.389 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000271 2024-08-18T16:23:29.449 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000271 2024-08-18T16:23:29.458 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000283 2024-08-18T16:23:29.507 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000283 2024-08-18T16:23:29.559 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000283 2024-08-18T16:23:29.611 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000317 2024-08-18T16:23:29.619 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000317 2024-08-18T16:23:29.674 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000317 2024-08-18T16:23:29.684 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000423 2024-08-18T16:23:29.690 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000423 2024-08-18T16:23:29.743 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000423 2024-08-18T16:23:29.795 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000467 2024-08-18T16:23:29.801 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000467 2024-08-18T16:23:29.813 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000467 2024-08-18T16:23:29.863 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-77 zones={test-zone3, test-zone1} 2024-08-18T16:23:29.863 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-77 zones={test-zone3, test-zone2} 2024-08-18T16:23:29.868 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:23:29.873 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000000 2024-08-18T16:23:29.873 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000001 2024-08-18T16:23:29.873 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000002 2024-08-18T16:23:29.873 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000003 2024-08-18T16:23:29.873 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000004 2024-08-18T16:23:29.873 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000005 2024-08-18T16:23:29.873 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000006 2024-08-18T16:23:29.874 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000007 2024-08-18T16:23:29.874 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000008 2024-08-18T16:23:29.874 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000009 2024-08-18T16:23:29.874 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000010 2024-08-18T16:23:29.874 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000011 2024-08-18T16:23:29.874 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000012 2024-08-18T16:23:29.874 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000013 2024-08-18T16:23:29.874 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000014 2024-08-18T16:23:29.874 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000015 2024-08-18T16:23:29.874 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000016 2024-08-18T16:23:29.874 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000017 2024-08-18T16:23:29.874 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000018 2024-08-18T16:23:29.874 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000019 2024-08-18T16:23:29.874 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000020 2024-08-18T16:23:29.874 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000021 2024-08-18T16:23:29.875 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000022 2024-08-18T16:23:29.875 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000023 2024-08-18T16:23:29.875 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000024 2024-08-18T16:23:29.875 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000025 2024-08-18T16:23:29.875 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000026 2024-08-18T16:23:29.875 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000027 2024-08-18T16:23:29.875 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000028 2024-08-18T16:23:29.875 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000029 2024-08-18T16:23:29.875 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000030 2024-08-18T16:23:29.875 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000031 2024-08-18T16:23:29.875 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000032 2024-08-18T16:23:29.875 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000033 2024-08-18T16:23:29.875 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000036 2024-08-18T16:23:29.875 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000037 2024-08-18T16:23:29.875 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000039 2024-08-18T16:23:29.875 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000040 2024-08-18T16:23:29.876 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000042 2024-08-18T16:23:29.876 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000045 2024-08-18T16:23:29.876 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000046 2024-08-18T16:23:29.876 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000048 2024-08-18T16:23:29.876 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000049 2024-08-18T16:23:29.876 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000050 2024-08-18T16:23:29.876 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000053 2024-08-18T16:23:29.876 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000055 2024-08-18T16:23:29.876 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000056 2024-08-18T16:23:29.876 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000057 2024-08-18T16:23:29.876 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000058 2024-08-18T16:23:29.876 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000059 2024-08-18T16:23:29.876 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000060 2024-08-18T16:23:29.876 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000061 2024-08-18T16:23:29.876 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000062 2024-08-18T16:23:29.877 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000065 2024-08-18T16:23:29.877 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000076 2024-08-18T16:23:29.877 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000133 2024-08-18T16:23:29.877 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000162 2024-08-18T16:23:29.877 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000271 2024-08-18T16:23:29.877 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000283 2024-08-18T16:23:29.877 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000317 2024-08-18T16:23:29.877 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000423 2024-08-18T16:23:29.877 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000467 2024-08-18T16:23:29.877 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:23:29.883 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000000 2024-08-18T16:23:29.883 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000001 2024-08-18T16:23:29.883 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000002 2024-08-18T16:23:29.883 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000003 2024-08-18T16:23:29.883 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000004 2024-08-18T16:23:29.883 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000005 2024-08-18T16:23:29.883 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000006 2024-08-18T16:23:29.883 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000007 2024-08-18T16:23:29.883 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000008 2024-08-18T16:23:29.883 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000009 2024-08-18T16:23:29.883 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000010 2024-08-18T16:23:29.883 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000011 2024-08-18T16:23:29.884 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000012 2024-08-18T16:23:29.884 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000013 2024-08-18T16:23:29.884 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000014 2024-08-18T16:23:29.884 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000015 2024-08-18T16:23:29.884 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000016 2024-08-18T16:23:29.884 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000017 2024-08-18T16:23:29.884 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000018 2024-08-18T16:23:29.884 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000019 2024-08-18T16:23:29.884 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000020 2024-08-18T16:23:29.884 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000021 2024-08-18T16:23:29.884 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000022 2024-08-18T16:23:29.884 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000023 2024-08-18T16:23:29.884 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000024 2024-08-18T16:23:29.884 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000025 2024-08-18T16:23:29.885 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000026 2024-08-18T16:23:29.885 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000027 2024-08-18T16:23:29.885 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000028 2024-08-18T16:23:29.885 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000029 2024-08-18T16:23:29.885 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000030 2024-08-18T16:23:29.885 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000031 2024-08-18T16:23:29.885 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000032 2024-08-18T16:23:29.885 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000033 2024-08-18T16:23:29.885 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000036 2024-08-18T16:23:29.885 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000037 2024-08-18T16:23:29.885 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000039 2024-08-18T16:23:29.885 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000040 2024-08-18T16:23:29.885 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000042 2024-08-18T16:23:29.885 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000045 2024-08-18T16:23:29.885 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000046 2024-08-18T16:23:29.886 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000048 2024-08-18T16:23:29.886 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000049 2024-08-18T16:23:29.886 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000050 2024-08-18T16:23:29.886 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000053 2024-08-18T16:23:29.886 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000055 2024-08-18T16:23:29.886 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000056 2024-08-18T16:23:29.886 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000057 2024-08-18T16:23:29.886 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000058 2024-08-18T16:23:29.886 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000059 2024-08-18T16:23:29.886 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000060 2024-08-18T16:23:29.886 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000061 2024-08-18T16:23:29.886 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000062 2024-08-18T16:23:29.886 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000065 2024-08-18T16:23:29.886 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000076 2024-08-18T16:23:29.886 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000133 2024-08-18T16:23:29.887 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000162 2024-08-18T16:23:29.887 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000271 2024-08-18T16:23:29.887 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000283 2024-08-18T16:23:29.887 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000317 2024-08-18T16:23:29.887 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000423 2024-08-18T16:23:29.887 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000467 2024-08-18T16:23:29.898 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2024-08-18T16:23:29.946 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2024-08-18T16:23:29.960 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2024-08-18T16:23:30.011 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000001 2024-08-18T16:23:30.018 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000001 2024-08-18T16:23:30.030 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000001 2024-08-18T16:23:30.080 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2024-08-18T16:23:30.087 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2024-08-18T16:23:30.145 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2024-08-18T16:23:30.197 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2024-08-18T16:23:30.205 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2024-08-18T16:23:30.260 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2024-08-18T16:23:30.313 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000004 2024-08-18T16:23:30.320 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000004 2024-08-18T16:23:30.373 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000004 2024-08-18T16:23:30.431 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2024-08-18T16:23:30.486 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2024-08-18T16:23:30.498 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2024-08-18T16:23:30.548 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2024-08-18T16:23:30.555 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2024-08-18T16:23:30.611 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2024-08-18T16:23:30.664 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000007 2024-08-18T16:23:30.671 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000007 2024-08-18T16:23:30.723 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000007 2024-08-18T16:23:30.772 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000008 2024-08-18T16:23:30.780 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000008 2024-08-18T16:23:30.833 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000008 2024-08-18T16:23:30.882 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000009 2024-08-18T16:23:30.889 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000009 2024-08-18T16:23:30.945 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000009 2024-08-18T16:23:30.998 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2024-08-18T16:23:31.005 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2024-08-18T16:23:31.067 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2024-08-18T16:23:31.124 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2024-08-18T16:23:31.131 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2024-08-18T16:23:31.191 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2024-08-18T16:23:31.245 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2024-08-18T16:23:31.253 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2024-08-18T16:23:31.313 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2024-08-18T16:23:31.366 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2024-08-18T16:23:31.373 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2024-08-18T16:23:31.430 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2024-08-18T16:23:31.486 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000014 2024-08-18T16:23:31.493 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000014 2024-08-18T16:23:31.506 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000014 2024-08-18T16:23:31.561 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000015 2024-08-18T16:23:31.569 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000015 2024-08-18T16:23:31.629 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000015 2024-08-18T16:23:31.681 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000016 2024-08-18T16:23:31.692 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000016 2024-08-18T16:23:31.746 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000016 2024-08-18T16:23:31.801 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000017 2024-08-18T16:23:31.809 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000017 2024-08-18T16:23:31.869 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000017 2024-08-18T16:23:31.922 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000018 2024-08-18T16:23:31.929 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000018 2024-08-18T16:23:31.989 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000018 2024-08-18T16:23:32.041 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000019 2024-08-18T16:23:32.049 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000019 2024-08-18T16:23:32.105 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000019 2024-08-18T16:23:32.114 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000020 2024-08-18T16:23:32.164 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000020 2024-08-18T16:23:32.178 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000020 2024-08-18T16:23:32.229 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000021 2024-08-18T16:23:32.236 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000021 2024-08-18T16:23:32.288 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000021 2024-08-18T16:23:32.338 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000022 2024-08-18T16:23:32.345 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000022 2024-08-18T16:23:32.402 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000022 2024-08-18T16:23:32.411 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000023 2024-08-18T16:23:32.466 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000023 2024-08-18T16:23:32.478 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000023 2024-08-18T16:23:32.527 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000024 2024-08-18T16:23:32.534 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000024 2024-08-18T16:23:32.587 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000024 2024-08-18T16:23:32.635 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000025 2024-08-18T16:23:32.643 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000025 2024-08-18T16:23:32.696 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000025 2024-08-18T16:23:32.749 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000026 2024-08-18T16:23:32.757 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000026 2024-08-18T16:23:32.817 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000026 2024-08-18T16:23:32.874 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000027 2024-08-18T16:23:32.881 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000027 2024-08-18T16:23:32.942 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000027 2024-08-18T16:23:32.951 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000028 2024-08-18T16:23:33.000 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000028 2024-08-18T16:23:33.013 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000028 2024-08-18T16:23:33.070 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000029 2024-08-18T16:23:33.077 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000029 2024-08-18T16:23:33.132 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000029 2024-08-18T16:23:33.185 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000030 2024-08-18T16:23:33.234 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000030 2024-08-18T16:23:33.247 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000030 2024-08-18T16:23:33.303 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000031 2024-08-18T16:23:33.311 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000031 2024-08-18T16:23:33.369 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000031 2024-08-18T16:23:33.421 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000032 2024-08-18T16:23:33.428 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000032 2024-08-18T16:23:33.487 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000032 2024-08-18T16:23:33.540 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000033 2024-08-18T16:23:33.547 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000033 2024-08-18T16:23:33.608 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000033 2024-08-18T16:23:33.659 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000036 2024-08-18T16:23:33.667 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000036 2024-08-18T16:23:33.723 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000036 2024-08-18T16:23:33.779 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000037 2024-08-18T16:23:33.787 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000037 2024-08-18T16:23:33.843 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000037 2024-08-18T16:23:33.896 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000039 2024-08-18T16:23:33.904 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000039 2024-08-18T16:23:33.917 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000039 2024-08-18T16:23:33.970 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000040 2024-08-18T16:23:33.978 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000040 2024-08-18T16:23:33.991 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000040 2024-08-18T16:23:34.040 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000042 2024-08-18T16:23:34.047 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000042 2024-08-18T16:23:34.104 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000042 2024-08-18T16:23:34.155 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000045 2024-08-18T16:23:34.162 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000045 2024-08-18T16:23:34.223 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000045 2024-08-18T16:23:34.275 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000046 2024-08-18T16:23:34.283 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000046 2024-08-18T16:23:34.340 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000046 2024-08-18T16:23:34.391 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000048 2024-08-18T16:23:34.399 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000048 2024-08-18T16:23:34.456 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000048 2024-08-18T16:23:34.465 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000049 2024-08-18T16:23:34.516 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000049 2024-08-18T16:23:34.529 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000049 2024-08-18T16:23:34.585 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000050 2024-08-18T16:23:34.593 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000050 2024-08-18T16:23:34.651 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000050 2024-08-18T16:23:34.709 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000053 2024-08-18T16:23:34.717 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000053 2024-08-18T16:23:34.774 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000053 2024-08-18T16:23:34.783 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000055 2024-08-18T16:23:34.790 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000055 2024-08-18T16:23:34.842 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000055 2024-08-18T16:23:34.895 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000056 2024-08-18T16:23:34.902 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000056 2024-08-18T16:23:34.915 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000056 2024-08-18T16:23:34.964 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000057 2024-08-18T16:23:34.971 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000057 2024-08-18T16:23:35.027 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000057 2024-08-18T16:23:35.080 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000058 2024-08-18T16:23:35.088 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000058 2024-08-18T16:23:35.143 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000058 2024-08-18T16:23:35.191 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000059 2024-08-18T16:23:35.199 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000059 2024-08-18T16:23:35.255 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000059 2024-08-18T16:23:35.308 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000060 2024-08-18T16:23:35.316 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000060 2024-08-18T16:23:35.328 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000060 2024-08-18T16:23:35.337 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000061 2024-08-18T16:23:35.344 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000061 2024-08-18T16:23:35.401 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000061 2024-08-18T16:23:35.457 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000062 2024-08-18T16:23:35.465 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000062 2024-08-18T16:23:35.521 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000062 2024-08-18T16:23:35.623 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000065 2024-08-18T16:23:35.631 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000065 2024-08-18T16:23:35.688 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000065 2024-08-18T16:23:35.739 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000076 2024-08-18T16:23:35.748 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000076 2024-08-18T16:23:35.761 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000076 2024-08-18T16:23:35.818 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000133 2024-08-18T16:23:35.826 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000133 2024-08-18T16:23:35.886 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000133 2024-08-18T16:23:35.894 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000162 2024-08-18T16:23:35.944 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000162 2024-08-18T16:23:36.001 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000162 2024-08-18T16:23:36.055 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000271 2024-08-18T16:23:36.107 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000271 2024-08-18T16:23:36.162 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000271 2024-08-18T16:23:36.172 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000283 2024-08-18T16:23:36.226 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000283 2024-08-18T16:23:36.239 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000283 2024-08-18T16:23:36.292 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000317 2024-08-18T16:23:36.299 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000317 2024-08-18T16:23:36.355 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000317 2024-08-18T16:23:36.408 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000423 2024-08-18T16:23:36.415 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000423 2024-08-18T16:23:36.475 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000423 2024-08-18T16:23:36.528 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000467 2024-08-18T16:23:36.535 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000467 2024-08-18T16:23:36.596 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000467 2024-08-18T16:23:36.605 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-77 zones={test-zone3, test-zone2} 2024-08-18T16:23:36.605 INFO:tasks.rgw_multisite_tests:Create several generations of objects, then run bucket sync ... ok 2024-08-18T16:23:36.625 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:23:36.625 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:23:36.625 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:23:36.677 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:23:36.677 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:23:36.710 INFO:teuthology.orchestra.run.smithi134.stderr:No --period given, using current period=01050e30-179b-43be-bdbb-3bb23cf37ae8 2024-08-18T16:23:36.726 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T16:23:36.727 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:23:36.727 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T16:23:36.727 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T16:23:36.727 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:23:36.727 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:23:36.727 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T16:23:36.727 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T16:23:36.727 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:23:36.727 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:23:36.727 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T16:23:36.727 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T16:23:36.727 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:23:36.728 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:23:36.728 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723998216.623520_951.1", 2024-08-18T16:23:36.728 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:23:36.623520Z" 2024-08-18T16:23:36.728 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:23:36.728 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T16:23:36.729 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-08-18T16:23:36.729 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:23:36.729 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:23:36.729 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:23:36.784 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:23:36.784 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:23:36.817 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:23:36.816+0000 7f797e926880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:23:36.817 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:23:36.817+0000 7f797e926880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:23:36.832 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T16:23:36.832 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T16:23:36.832 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T16:23:36.832 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T16:23:36.832 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T16:23:36.832 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T16:23:36.832 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T16:23:36.832 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:23:36.832 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T16:23:36.833 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:23:36.833 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T16:23:36.833 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:23:36.833 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:23:36.833 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T16:23:36.833 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:23:36.833 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T16:23:36.833 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:23:36.833 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T16:23:36.833 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T16:23:36.833 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:23:36.833 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:23:36.833 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:23:36.834 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T16:23:36.834 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:23:36.834 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:23:36.834 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T16:23:36.834 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:23:36.834 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:23:36.834 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:23:36.834 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T16:23:36.834 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T16:23:36.834 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:23:36.834 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:23:36.834 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:23:36.834 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T16:23:36.835 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:23:36.835 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:23:36.835 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T16:23:36.835 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:23:36.835 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:23:36.835 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:23:36.835 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T16:23:36.835 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T16:23:36.835 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:23:36.835 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:23:36.835 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:23:36.836 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T16:23:36.836 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:23:36.836 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:23:36.836 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723998216.623520_951.1", 2024-08-18T16:23:36.836 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:23:36.836 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:23:36.836 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:23:36.836 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:23:36.623520Z", 2024-08-18T16:23:36.836 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T16:23:36.836 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:23:36.836 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:23:36.836 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T16:23:36.836 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:23:36.837 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T16:23:36.837 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T16:23:36.837 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T16:23:36.837 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:23:36.837 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T16:23:36Z" 2024-08-18T16:23:36.837 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T16:23:36.838 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-08-18T16:23:36.838 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-08-18T16:23:36.838 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:23:36.838 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:23:36.839 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:23:36.887 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:23:36.887 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:23:36.916 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:23:36.915+0000 7f3396226880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:23:36.917 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:23:36.916+0000 7f3396226880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:23:36.929 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T16:23:36.930 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T16:23:36.930 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T16:23:36.930 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T16:23:36.930 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T16:23:36.930 INFO:teuthology.orchestra.run.smithi134.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T16:23:36.930 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T16:23:36.930 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:23:36.930 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T16:23:36.930 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:23:36.930 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T16:23:36.931 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:23:36.931 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T16:23:36.931 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T16:23:36.931 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:23:36.931 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 1, 2024-08-18T16:23:36.931 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:23:36.931 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T16:23:36.931 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T16:23:36.931 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:23:36.931 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:23:36.931 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:23:36.931 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T16:23:36.931 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:23:36.932 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T16:23:36.932 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T16:23:36.932 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:23:36.932 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:23:36.932 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:23:36.932 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T16:23:36.932 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T16:23:36.932 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:23:36.932 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:23:36.932 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:23:36.932 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T16:23:36.932 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:23:36.932 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T16:23:36.933 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T16:23:36.933 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:23:36.933 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:23:36.933 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:23:36.933 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T16:23:36.933 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T16:23:36.933 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:23:36.933 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:23:36.933 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:23:36.933 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T16:23:36.933 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:23:36.933 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T16:23:36.933 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723997959.328272_896.1", 2024-08-18T16:23:36.933 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:23:36.934 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:23:36.934 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:23:36.934 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:19:19.328272Z", 2024-08-18T16:23:36.934 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T16:23:36.934 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:23:36.934 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:23:36.934 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T16:23:36.934 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:23:36.934 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T16:23:36.934 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 1, 2024-08-18T16:23:36.934 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 1 2024-08-18T16:23:36.934 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:23:36.935 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T16:23:36Z" 2024-08-18T16:23:36.935 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T16:23:36.935 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1723998216.623520_951.1 target=1_1723997959.328272_896.1 2024-08-18T16:23:41.936 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:23:41.937 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:23:41.937 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:23:41.992 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:23:41.993 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:23:42.025 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:23:42.025+0000 7fc56e523880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:23:42.026 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:23:42.026+0000 7fc56e523880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:23:42.040 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T16:23:42.040 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T16:23:42.040 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T16:23:42.040 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T16:23:42.041 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T16:23:42.041 INFO:teuthology.orchestra.run.smithi134.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T16:23:42.041 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T16:23:42.041 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:23:42.041 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T16:23:42.041 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:23:42.041 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T16:23:42.041 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:23:42.041 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T16:23:42.041 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T16:23:42.041 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:23:42.041 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 1, 2024-08-18T16:23:42.041 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:23:42.042 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T16:23:42.042 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T16:23:42.042 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:23:42.042 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:23:42.042 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:23:42.042 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T16:23:42.042 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:23:42.042 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T16:23:42.042 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T16:23:42.042 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:23:42.042 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:23:42.042 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:23:42.042 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T16:23:42.043 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T16:23:42.043 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:23:42.043 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:23:42.043 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:23:42.043 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T16:23:42.043 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:23:42.043 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T16:23:42.043 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994494.952005_96.1", 2024-08-18T16:23:42.043 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:23:42.043 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:23:42.043 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:23:42.043 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:21:34.952005Z", 2024-08-18T16:23:42.043 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T16:23:42.044 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:23:42.044 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:23:42.044 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:23:42.044 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T16:23:42.044 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:23:42.044 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T16:23:42.044 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723998216.623520_951.1", 2024-08-18T16:23:42.044 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:23:42.044 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:23:42.044 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:23:42.044 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:23:36.623520Z", 2024-08-18T16:23:42.044 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T16:23:42.044 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:23:42.044 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:23:42.044 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T16:23:42.045 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:23:42.045 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T16:23:42.045 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 1, 2024-08-18T16:23:42.045 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 1 2024-08-18T16:23:42.045 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:23:42.045 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T16:23:42Z" 2024-08-18T16:23:42.045 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T16:23:42.046 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-08-18T16:23:42.123 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:23:42.124 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:23:42.124 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-78 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T16:23:42.184 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:23:42.184 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:23:42.223 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:23:42.222+0000 7fa321233880 1 waiting to reach incremental sync.. 2024-08-18T16:23:47.224 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:23:47.224+0000 7fa321233880 1 bucket sync caught up with source: 2024-08-18T16:23:47.225 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , 00000000001.93.6, 00000000002.99.6, 00000000002.99.6] 2024-08-18T16:23:47.225 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , , , 00000000001.93.6, 00000000002.99.6, 00000000002.99.6] 2024-08-18T16:23:47.225 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:23:47.224+0000 7fa321233880 0 bucket checkpoint complete 2024-08-18T16:23:47.243 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:23:47.243 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:23:47.244 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-78 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:23:47.303 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:23:47.303 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:23:47.340 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:23:47.340+0000 7f124d60f880 1 waiting to reach incremental sync.. 2024-08-18T16:23:52.342 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:23:52.342+0000 7f124d60f880 1 bucket sync caught up with source: 2024-08-18T16:23:52.343 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , , , 00000000001.93.6, 00000000002.99.6, 00000000002.99.6] 2024-08-18T16:23:52.343 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , , , , 00000000001.93.6, 00000000002.99.6, 00000000002.99.6] 2024-08-18T16:23:52.343 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:23:52.342+0000 7f124d60f880 0 bucket checkpoint complete 2024-08-18T16:23:52.360 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:23:52.361 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:23:52.361 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-78 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:23:52.420 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:23:52.420 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:23:52.458 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:23:52.457+0000 7f3055815880 1 waiting to reach incremental sync.. 2024-08-18T16:23:57.462 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:23:57.458+0000 7f3055815880 1 waiting to reach incremental sync.. 2024-08-18T16:24:02.459 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:24:02.458+0000 7f3055815880 1 waiting to reach incremental sync.. 2024-08-18T16:24:07.461 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:24:07.460+0000 7f3055815880 1 bucket sync caught up with source: 2024-08-18T16:24:07.461 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , , , 00000000001.265.6, 00000000002.305.6, 00000000002.255.6] 2024-08-18T16:24:07.461 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , , , , 00000000001.265.6, 00000000002.305.6, 00000000002.255.6] 2024-08-18T16:24:07.461 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:24:07.460+0000 7f3055815880 0 bucket checkpoint complete 2024-08-18T16:24:07.479 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:24:07.479 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:24:07.479 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-78 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:24:07.536 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:24:07.537 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:24:07.574 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:24:07.573+0000 7f4c6e40f880 1 bucket sync caught up with source: 2024-08-18T16:24:07.574 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , , , 00000000001.265.6, 00000000002.305.6, 00000000002.255.6] 2024-08-18T16:24:07.574 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , , , , 00000000001.265.6, 00000000002.305.6, 00000000002.255.6] 2024-08-18T16:24:07.574 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:24:07.573+0000 7f4c6e40f880 0 bucket checkpoint complete 2024-08-18T16:24:07.587 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:24:07.588 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:24:07.588 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-78 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:24:07.641 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:24:07.642 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:24:07.678 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:24:07.678+0000 7fc0c1724880 1 bucket sync caught up with source: 2024-08-18T16:24:07.679 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , , , 00000000001.169.6, 00000000002.175.6, 00000000002.131.6] 2024-08-18T16:24:07.679 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , , , , 00000000001.169.6, 00000000002.175.6, 00000000002.131.6] 2024-08-18T16:24:07.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:24:07.678+0000 7fc0c1724880 0 bucket checkpoint complete 2024-08-18T16:24:07.693 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:24:07.693 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:24:07.694 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-78 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T16:24:07.755 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:24:07.756 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:24:07.794 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:24:07.793+0000 7f404f732880 1 bucket sync caught up with source: 2024-08-18T16:24:07.795 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , 00000000001.169.6, 00000000002.175.6, 00000000002.131.6] 2024-08-18T16:24:07.795 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , , , 00000000001.169.6, 00000000002.175.6, 00000000002.131.6] 2024-08-18T16:24:07.795 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:24:07.793+0000 7f404f732880 0 bucket checkpoint complete 2024-08-18T16:24:07.810 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-78 zones={test-zone2, test-zone1} 2024-08-18T16:24:07.815 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:24:07.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-08-18T16:24:07.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-08-18T16:24:07.819 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-08-18T16:24:07.819 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-08-18T16:24:07.819 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-08-18T16:24:07.819 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:24:07.822 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-08-18T16:24:07.822 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-08-18T16:24:07.822 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-08-18T16:24:07.822 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-08-18T16:24:07.822 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-08-18T16:24:07.827 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T16:24:07.874 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T16:24:07.886 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T16:24:07.895 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T16:24:07.946 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T16:24:07.957 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T16:24:08.013 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T16:24:08.020 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T16:24:08.073 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T16:24:08.129 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T16:24:08.136 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T16:24:08.189 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T16:24:08.241 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T16:24:08.248 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T16:24:08.300 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T16:24:08.354 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-78 zones={test-zone2, test-zone1} 2024-08-18T16:24:08.354 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-78 zones={test-zone3, test-zone1} 2024-08-18T16:24:08.358 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:24:08.361 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-08-18T16:24:08.361 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-08-18T16:24:08.361 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-08-18T16:24:08.361 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-08-18T16:24:08.361 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-08-18T16:24:08.361 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:24:08.364 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-08-18T16:24:08.364 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-08-18T16:24:08.364 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-08-18T16:24:08.364 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-08-18T16:24:08.364 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-08-18T16:24:08.369 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T16:24:08.419 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T16:24:08.474 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T16:24:08.526 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T16:24:08.576 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T16:24:08.632 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T16:24:08.685 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T16:24:08.736 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T16:24:08.788 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T16:24:08.837 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T16:24:08.892 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T16:24:08.948 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T16:24:08.997 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T16:24:09.048 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T16:24:09.100 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T16:24:09.149 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-78 zones={test-zone3, test-zone1} 2024-08-18T16:24:09.150 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-78 zones={test-zone3, test-zone2} 2024-08-18T16:24:09.154 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:24:09.157 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-08-18T16:24:09.157 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-08-18T16:24:09.157 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-08-18T16:24:09.157 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-08-18T16:24:09.157 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-08-18T16:24:09.157 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:24:09.160 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-08-18T16:24:09.160 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-08-18T16:24:09.160 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-08-18T16:24:09.160 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-08-18T16:24:09.160 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-08-18T16:24:09.166 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T16:24:09.220 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T16:24:09.317 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T16:24:09.413 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T16:24:09.465 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T16:24:09.521 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T16:24:09.574 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T16:24:09.624 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T16:24:09.677 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T16:24:09.774 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T16:24:09.824 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T16:24:09.878 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T16:24:09.974 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T16:24:10.024 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T16:24:10.081 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T16:24:10.135 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-78 zones={test-zone3, test-zone2} 2024-08-18T16:24:10.135 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:24:10.136 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:24:10.136 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-78 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:24:10.195 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:24:10.196 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:24:10.242 INFO:teuthology.orchestra.run.smithi134.stdout:tenant: 2024-08-18T16:24:10.242 INFO:teuthology.orchestra.run.smithi134.stdout:bucket name: kibgmb-78 2024-08-18T16:24:10.245 INFO:teuthology.orchestra.run.smithi134.stdout:total entries: 5 2024-08-18T16:24:10.361 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:24:10.361 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:24:10.361 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-78 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T16:24:10.423 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:24:10.423 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:24:10.462 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:24:10.461+0000 7f6540119880 1 waiting to reach latest gen 1, on 0.. 2024-08-18T16:24:15.463 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:24:15.462+0000 7f6540119880 1 waiting to reach latest gen 1, on 0.. 2024-08-18T16:24:20.463 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:24:20.463+0000 7f6540119880 1 waiting to reach latest gen 1, on 0.. 2024-08-18T16:24:25.465 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:24:25.465+0000 7f6540119880 1 waiting for incremental sync to catch up: 2024-08-18T16:24:25.466 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , , , , , , , , , ] 2024-08-18T16:24:25.466 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000001.97.6, , , , , , 00000000001.119.6, , 00000000001.92.6, , , , 00000000001.61.6, 00000000001.173.6, , , ] 2024-08-18T16:24:30.467 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:24:30.466+0000 7f6540119880 1 waiting for incremental sync to catch up: 2024-08-18T16:24:30.467 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , , , , , , , , , ] 2024-08-18T16:24:30.467 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000001.97.6, , , , , , 00000000001.119.6, , 00000000001.92.6, , , , 00000000001.61.6, 00000000001.173.6, , , ] 2024-08-18T16:24:35.468 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:24:35.468+0000 7f6540119880 1 waiting for incremental sync to catch up: 2024-08-18T16:24:35.469 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , , , , , , , , , ] 2024-08-18T16:24:35.469 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000001.97.6, , , , , , 00000000001.119.6, , 00000000001.92.6, , , , 00000000001.61.6, 00000000001.173.6, , , ] 2024-08-18T16:24:40.470 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:24:40.469+0000 7f6540119880 1 waiting for incremental sync to catch up: 2024-08-18T16:24:40.470 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , , , , , , , , , ] 2024-08-18T16:24:40.470 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000001.97.6, , , , , , 00000000001.119.6, , 00000000001.92.6, , , , 00000000001.61.6, 00000000001.173.6, , , ] 2024-08-18T16:24:45.471 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:24:45.470+0000 7f6540119880 1 waiting for incremental sync to catch up: 2024-08-18T16:24:45.471 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , , , , , , , , , ] 2024-08-18T16:24:45.472 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000001.97.6, , , , , , 00000000001.119.6, , 00000000001.92.6, , , , 00000000001.61.6, 00000000001.173.6, , , ] 2024-08-18T16:24:50.473 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:24:50.472+0000 7f6540119880 1 waiting for incremental sync to catch up: 2024-08-18T16:24:50.473 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , , , , , , , , , ] 2024-08-18T16:24:50.473 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000001.97.6, , , , , , 00000000001.119.6, , 00000000001.92.6, , , , 00000000001.61.6, 00000000001.173.6, , , ] 2024-08-18T16:24:55.474 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:24:55.473+0000 7f6540119880 1 waiting for incremental sync to catch up: 2024-08-18T16:24:55.475 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000001.97.6, , , , , , , , 00000000001.92.6, , , , 00000000001.61.6, , , , ] 2024-08-18T16:24:55.475 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000001.97.6, , , , , , 00000000001.119.6, , 00000000001.92.6, , , , 00000000001.61.6, 00000000001.173.6, , , ] 2024-08-18T16:25:00.476 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:25:00.475+0000 7f6540119880 1 waiting for incremental sync to catch up: 2024-08-18T16:25:00.476 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000001.97.6, , , , , , , , 00000000001.92.6, , , , 00000000001.61.6, , , , ] 2024-08-18T16:25:00.476 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000001.97.6, , , , , , 00000000001.119.6, , 00000000001.92.6, , , , 00000000001.61.6, 00000000001.173.6, , , ] 2024-08-18T16:25:05.477 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:25:05.477+0000 7f6540119880 1 waiting for incremental sync to catch up: 2024-08-18T16:25:05.477 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000001.97.6, , , , , , , , 00000000001.92.6, , , , 00000000001.61.6, , , , ] 2024-08-18T16:25:05.477 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000001.97.6, , , , , , 00000000001.119.6, , 00000000001.92.6, , , , 00000000001.61.6, 00000000001.173.6, , , ] 2024-08-18T16:25:10.479 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:25:10.477+0000 7f6540119880 1 waiting for incremental sync to catch up: 2024-08-18T16:25:10.479 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000001.97.6, , , , , , , , 00000000001.92.6, , , , 00000000001.61.6, , , , ] 2024-08-18T16:25:10.479 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000001.97.6, , , , , , 00000000001.119.6, , 00000000001.92.6, , , , 00000000001.61.6, 00000000001.173.6, , , ] 2024-08-18T16:25:15.480 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:25:15.478+0000 7f6540119880 1 waiting for incremental sync to catch up: 2024-08-18T16:25:15.480 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000001.97.6, , , , , , 00000000001.119.6, , 00000000001.92.6, , , , 00000000001.61.6, , , , ] 2024-08-18T16:25:15.480 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000001.97.6, , , , , , 00000000001.119.6, , 00000000001.92.6, , , , 00000000001.61.6, 00000000001.173.6, , , ] 2024-08-18T16:25:20.482 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:25:20.481+0000 7f6540119880 1 waiting for incremental sync to catch up: 2024-08-18T16:25:20.482 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000001.97.6, , , , , , 00000000001.119.6, , 00000000001.92.6, , , , 00000000001.61.6, , , , ] 2024-08-18T16:25:20.482 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000001.97.6, , , , , , 00000000001.119.6, , 00000000001.92.6, , , , 00000000001.61.6, 00000000001.173.6, , , ] 2024-08-18T16:25:25.483 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:25:25.482+0000 7f6540119880 1 bucket sync caught up with source: 2024-08-18T16:25:25.483 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000001.97.6, , , , , , 00000000001.119.6, , 00000000001.92.6, , , , 00000000001.61.6, 00000000001.173.6, , , ] 2024-08-18T16:25:25.483 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000001.97.6, , , , , , 00000000001.119.6, , 00000000001.92.6, , , , 00000000001.61.6, 00000000001.173.6, , , ] 2024-08-18T16:25:25.484 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:25:25.482+0000 7f6540119880 0 bucket checkpoint complete 2024-08-18T16:25:25.501 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:25:25.501 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:25:25.501 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-78 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:25:25.563 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:25:25.563 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:25:25.602 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:25:25.602+0000 7f9473142880 1 waiting for incremental sync to catch up: 2024-08-18T16:25:25.602 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000001.97.6, , , , , , 00000000001.119.6, , 00000000001.92.6, , , , 00000000001.61.6, , , , ] 2024-08-18T16:25:25.603 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000001.97.6, , , , , , 00000000001.119.6, , 00000000001.92.6, , , , 00000000001.61.6, 00000000001.173.6, , , ] 2024-08-18T16:25:30.604 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:25:30.603+0000 7f9473142880 1 waiting for incremental sync to catch up: 2024-08-18T16:25:30.604 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000001.97.6, , , , , , 00000000001.119.6, , 00000000001.92.6, , , , 00000000001.61.6, , , , ] 2024-08-18T16:25:30.604 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000001.97.6, , , , , , 00000000001.119.6, , 00000000001.92.6, , , , 00000000001.61.6, 00000000001.173.6, , , ] 2024-08-18T16:25:35.605 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:25:35.605+0000 7f9473142880 1 waiting for incremental sync to catch up: 2024-08-18T16:25:35.606 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000001.97.6, , , , , , 00000000001.119.6, , 00000000001.92.6, , , , 00000000001.61.6, , , , ] 2024-08-18T16:25:35.606 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000001.97.6, , , , , , 00000000001.119.6, , 00000000001.92.6, , , , 00000000001.61.6, 00000000001.173.6, , , ] 2024-08-18T16:25:40.607 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:25:40.606+0000 7f9473142880 1 waiting for incremental sync to catch up: 2024-08-18T16:25:40.607 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000001.97.6, , , , , , 00000000001.119.6, , 00000000001.92.6, , , , 00000000001.61.6, , , , ] 2024-08-18T16:25:40.607 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000001.97.6, , , , , , 00000000001.119.6, , 00000000001.92.6, , , , 00000000001.61.6, 00000000001.173.6, , , ] 2024-08-18T16:25:45.609 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:25:45.608+0000 7f9473142880 1 waiting for incremental sync to catch up: 2024-08-18T16:25:45.609 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000001.97.6, , , , , , 00000000001.119.6, , 00000000001.92.6, , , , 00000000001.61.6, , , , ] 2024-08-18T16:25:45.609 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000001.97.6, , , , , , 00000000001.119.6, , 00000000001.92.6, , , , 00000000001.61.6, 00000000001.173.6, , , ] 2024-08-18T16:25:50.610 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:25:50.609+0000 7f9473142880 1 bucket sync caught up with source: 2024-08-18T16:25:50.610 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000001.97.6, , , , , , 00000000001.119.6, , 00000000001.92.6, , , , 00000000001.61.6, 00000000001.173.6, , , ] 2024-08-18T16:25:50.611 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000001.97.6, , , , , , 00000000001.119.6, , 00000000001.92.6, , , , 00000000001.61.6, 00000000001.173.6, , , ] 2024-08-18T16:25:50.611 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:25:50.609+0000 7f9473142880 0 bucket checkpoint complete 2024-08-18T16:25:50.628 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:25:50.628 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:25:50.629 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-78 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:25:50.689 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:25:50.689 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:25:50.729 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:25:50.728+0000 7f1b1e698880 1 bucket sync caught up with source: 2024-08-18T16:25:50.729 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000001.315.6, 00000000001.228.6, , , 00000000001.185.6, , , 00000000001.265.6, 00000000003.317.6, 00000000003.261.6] 2024-08-18T16:25:50.729 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000001.315.6, 00000000001.228.6, , , 00000000001.185.6, , , 00000000001.265.6, 00000000003.317.6, 00000000003.261.6] 2024-08-18T16:25:50.729 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:25:50.728+0000 7f1b1e698880 0 bucket checkpoint complete 2024-08-18T16:25:50.744 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:25:50.744 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:25:50.745 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-78 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:25:50.798 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:25:50.798 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:25:50.833 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:25:50.833+0000 7f041d1af880 1 bucket sync caught up with source: 2024-08-18T16:25:50.834 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000001.315.6, 00000000001.228.6, , , 00000000001.185.6, , , 00000000001.265.6, 00000000003.317.6, 00000000003.261.6] 2024-08-18T16:25:50.834 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000001.315.6, 00000000001.228.6, , , 00000000001.185.6, , , 00000000001.265.6, 00000000003.317.6, 00000000003.261.6] 2024-08-18T16:25:50.834 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:25:50.833+0000 7f041d1af880 0 bucket checkpoint complete 2024-08-18T16:25:50.847 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:25:50.847 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:25:50.847 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-78 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:25:50.900 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:25:50.900 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:25:50.935 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:25:50.934+0000 7fe6fde0f880 1 bucket sync caught up with source: 2024-08-18T16:25:50.935 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000001.183.6, 00000000001.472.6, , , 00000000001.130.6, , , 00000000001.169.6, 00000000003.182.6, 00000000003.135.6] 2024-08-18T16:25:50.935 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000001.183.6, 00000000001.472.6, , , 00000000001.130.6, , , 00000000001.169.6, 00000000003.182.6, 00000000003.135.6] 2024-08-18T16:25:50.935 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:25:50.934+0000 7fe6fde0f880 0 bucket checkpoint complete 2024-08-18T16:25:50.949 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:25:50.950 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:25:50.950 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-78 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T16:25:51.011 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:25:51.012 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:25:51.051 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:25:51.051+0000 7f46c1b42880 1 bucket sync caught up with source: 2024-08-18T16:25:51.051 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, 00000000001.183.6, 00000000001.472.6, , , 00000000001.130.6, , , 00000000001.169.6, 00000000003.182.6, 00000000003.135.6] 2024-08-18T16:25:51.051 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, 00000000001.183.6, 00000000001.472.6, , , 00000000001.130.6, , , 00000000001.169.6, 00000000003.182.6, 00000000003.135.6] 2024-08-18T16:25:51.051 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:25:51.051+0000 7f46c1b42880 0 bucket checkpoint complete 2024-08-18T16:25:51.066 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-78 zones={test-zone2, test-zone1} 2024-08-18T16:25:51.078 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:25:51.081 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-08-18T16:25:51.081 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-08-18T16:25:51.081 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-08-18T16:25:51.081 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-08-18T16:25:51.081 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-08-18T16:25:51.081 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-08-18T16:25:51.082 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-08-18T16:25:51.082 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-08-18T16:25:51.082 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-08-18T16:25:51.082 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-08-18T16:25:51.082 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:25:51.085 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-08-18T16:25:51.085 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-08-18T16:25:51.085 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-08-18T16:25:51.085 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-08-18T16:25:51.086 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-08-18T16:25:51.086 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-08-18T16:25:51.086 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-08-18T16:25:51.086 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-08-18T16:25:51.086 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-08-18T16:25:51.086 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-08-18T16:25:51.092 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T16:25:51.098 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T16:25:51.109 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T16:25:51.117 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-08-18T16:25:51.123 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-08-18T16:25:51.176 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-08-18T16:25:51.225 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T16:25:51.232 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T16:25:51.284 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T16:25:51.338 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T16:25:51.344 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T16:25:51.355 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T16:25:51.411 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-08-18T16:25:51.418 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-08-18T16:25:51.475 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-08-18T16:25:51.527 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T16:25:51.533 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T16:25:51.586 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T16:25:51.640 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-08-18T16:25:51.647 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-08-18T16:25:51.703 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-08-18T16:25:51.751 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T16:25:51.758 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T16:25:51.815 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T16:25:51.863 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-08-18T16:25:51.870 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-08-18T16:25:51.922 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-08-18T16:25:51.930 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-08-18T16:25:51.984 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-08-18T16:25:52.080 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-08-18T16:25:52.173 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-78 zones={test-zone2, test-zone1} 2024-08-18T16:25:52.174 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-78 zones={test-zone3, test-zone1} 2024-08-18T16:25:52.181 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:25:52.183 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-08-18T16:25:52.184 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-08-18T16:25:52.184 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-08-18T16:25:52.184 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-08-18T16:25:52.184 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-08-18T16:25:52.184 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-08-18T16:25:52.184 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-08-18T16:25:52.184 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-08-18T16:25:52.184 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-08-18T16:25:52.184 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-08-18T16:25:52.184 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:25:52.187 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-08-18T16:25:52.187 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-08-18T16:25:52.187 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-08-18T16:25:52.187 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-08-18T16:25:52.187 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-08-18T16:25:52.187 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-08-18T16:25:52.187 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-08-18T16:25:52.188 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-08-18T16:25:52.188 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-08-18T16:25:52.188 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-08-18T16:25:52.193 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T16:25:52.242 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T16:25:52.296 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T16:25:52.349 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-08-18T16:25:52.400 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-08-18T16:25:52.452 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-08-18T16:25:52.546 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T16:25:52.596 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T16:25:52.693 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T16:25:52.794 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T16:25:52.847 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T16:25:52.952 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T16:25:53.046 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-08-18T16:25:53.096 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-08-18T16:25:53.190 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-08-18T16:25:53.241 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T16:25:53.292 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T16:25:53.348 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T16:25:53.397 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-08-18T16:25:53.452 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-08-18T16:25:53.504 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-08-18T16:25:53.553 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T16:25:53.604 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T16:25:53.656 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T16:25:53.709 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-08-18T16:25:53.764 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-08-18T16:25:53.820 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-08-18T16:25:53.869 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-08-18T16:25:53.876 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-08-18T16:25:53.934 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-08-18T16:25:53.942 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-78 zones={test-zone3, test-zone1} 2024-08-18T16:25:53.943 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-78 zones={test-zone3, test-zone2} 2024-08-18T16:25:53.948 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:25:53.950 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-08-18T16:25:53.950 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-08-18T16:25:53.951 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-08-18T16:25:53.951 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-08-18T16:25:53.951 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-08-18T16:25:53.951 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-08-18T16:25:53.951 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-08-18T16:25:53.951 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-08-18T16:25:53.951 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-08-18T16:25:53.951 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-08-18T16:25:53.951 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:25:53.954 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-08-18T16:25:53.954 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-08-18T16:25:53.955 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-08-18T16:25:53.955 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-08-18T16:25:53.955 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-08-18T16:25:53.955 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-08-18T16:25:53.955 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-08-18T16:25:53.955 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-08-18T16:25:53.955 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-08-18T16:25:53.955 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-08-18T16:25:53.962 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T16:25:54.016 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T16:25:54.070 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T16:25:54.121 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-08-18T16:25:54.176 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-08-18T16:25:54.229 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-08-18T16:25:54.281 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T16:25:54.332 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T16:25:54.437 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T16:25:54.533 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T16:25:54.584 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T16:25:54.641 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T16:25:54.694 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-08-18T16:25:54.745 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-08-18T16:25:54.801 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-08-18T16:25:54.854 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T16:25:54.904 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T16:25:54.957 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T16:25:55.014 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-08-18T16:25:55.064 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-08-18T16:25:55.126 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-08-18T16:25:55.230 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T16:25:55.280 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T16:25:55.338 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T16:25:55.434 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-08-18T16:25:55.484 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-08-18T16:25:55.581 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-08-18T16:25:55.677 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-08-18T16:25:55.685 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-08-18T16:25:55.741 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-08-18T16:25:55.797 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-78 zones={test-zone3, test-zone2} 2024-08-18T16:25:55.797 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:25:55.797 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:25:55.798 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-78 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:25:55.855 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:25:55.855 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:25:55.903 INFO:teuthology.orchestra.run.smithi134.stdout:tenant: 2024-08-18T16:25:55.904 INFO:teuthology.orchestra.run.smithi134.stdout:bucket name: kibgmb-78 2024-08-18T16:25:55.910 INFO:teuthology.orchestra.run.smithi134.stdout:total entries: 10 2024-08-18T16:25:56.028 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:25:56.029 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:25:56.029 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-78 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T16:25:56.089 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:25:56.090 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:25:56.128 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:25:56.127+0000 7ff785342880 1 waiting to reach latest gen 2, on 1.. 2024-08-18T16:26:01.128 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:26:01.128+0000 7ff785342880 1 waiting to reach latest gen 2, on 1.. 2024-08-18T16:26:06.129 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:26:06.128+0000 7ff785342880 1 waiting to reach latest gen 2, on 1.. 2024-08-18T16:26:11.129 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:26:11.128+0000 7ff785342880 1 waiting to reach latest gen 2, on 1.. 2024-08-18T16:26:16.131 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:26:16.129+0000 7ff785342880 1 waiting for incremental sync to catch up: 2024-08-18T16:26:16.131 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , , , , , , , , , , , ] 2024-08-18T16:26:16.131 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , 00000000002.147.6, 00000000004.130.6, , , , , , , , , , , ] 2024-08-18T16:26:21.132 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:26:21.131+0000 7ff785342880 1 waiting for incremental sync to catch up: 2024-08-18T16:26:21.132 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , , , , , , , , , , , ] 2024-08-18T16:26:21.132 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , 00000000002.147.6, 00000000004.130.6, , , , , , , , , , , ] 2024-08-18T16:26:26.134 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:26:26.133+0000 7ff785342880 1 waiting for incremental sync to catch up: 2024-08-18T16:26:26.134 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , , , , , , , , , , , ] 2024-08-18T16:26:26.134 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , 00000000002.147.6, 00000000004.130.6, , , , , , , , , , , ] 2024-08-18T16:26:31.135 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:26:31.134+0000 7ff785342880 1 waiting for incremental sync to catch up: 2024-08-18T16:26:31.135 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , , , , , , , , , , , ] 2024-08-18T16:26:31.135 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , 00000000002.147.6, 00000000004.130.6, , , , , , , , , , , ] 2024-08-18T16:26:36.137 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:26:36.136+0000 7ff785342880 1 waiting for incremental sync to catch up: 2024-08-18T16:26:36.137 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , , , , , , , , , , , ] 2024-08-18T16:26:36.137 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , 00000000002.147.6, 00000000004.130.6, , , , , , , , , , , ] 2024-08-18T16:26:41.139 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:26:41.138+0000 7ff785342880 1 waiting for incremental sync to catch up: 2024-08-18T16:26:41.139 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , , , , , , , , , , , ] 2024-08-18T16:26:41.139 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , 00000000002.147.6, 00000000004.130.6, , , , , , , , , , , ] 2024-08-18T16:26:46.140 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:26:46.139+0000 7ff785342880 1 waiting for incremental sync to catch up: 2024-08-18T16:26:46.140 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , , , , , , , , , , , ] 2024-08-18T16:26:46.140 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , 00000000002.147.6, 00000000004.130.6, , , , , , , , , , , ] 2024-08-18T16:26:51.142 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:26:51.141+0000 7ff785342880 1 waiting for incremental sync to catch up: 2024-08-18T16:26:51.142 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , , , , , , , , , , , ] 2024-08-18T16:26:51.142 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , 00000000002.147.6, 00000000004.130.6, , , , , , , , , , , ] 2024-08-18T16:26:56.143 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:26:56.142+0000 7ff785342880 1 waiting for incremental sync to catch up: 2024-08-18T16:26:56.143 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , 00000000004.130.6, , , , , , , , , , , ] 2024-08-18T16:26:56.143 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , 00000000002.147.6, 00000000004.130.6, , , , , , , , , , , ] 2024-08-18T16:27:01.144 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:27:01.144+0000 7ff785342880 1 waiting for incremental sync to catch up: 2024-08-18T16:27:01.145 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , 00000000004.130.6, , , , , , , , , , , ] 2024-08-18T16:27:01.145 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , 00000000002.147.6, 00000000004.130.6, , , , , , , , , , , ] 2024-08-18T16:27:06.146 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:27:06.145+0000 7ff785342880 1 waiting for incremental sync to catch up: 2024-08-18T16:27:06.146 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , 00000000004.130.6, , , , , , , , , , , ] 2024-08-18T16:27:06.146 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , 00000000002.147.6, 00000000004.130.6, , , , , , , , , , , ] 2024-08-18T16:27:11.148 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:27:11.147+0000 7ff785342880 1 waiting for incremental sync to catch up: 2024-08-18T16:27:11.148 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , 00000000004.130.6, , , , , , , , , , , ] 2024-08-18T16:27:11.148 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , 00000000002.147.6, 00000000004.130.6, , , , , , , , , , , ] 2024-08-18T16:27:16.150 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:27:16.148+0000 7ff785342880 1 bucket sync caught up with source: 2024-08-18T16:27:16.150 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , 00000000002.147.6, 00000000004.130.6, , , , , , , , , , , ] 2024-08-18T16:27:16.150 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , 00000000002.147.6, 00000000004.130.6, , , , , , , , , , , ] 2024-08-18T16:27:16.150 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:27:16.148+0000 7ff785342880 0 bucket checkpoint complete 2024-08-18T16:27:16.169 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:27:16.169 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:27:16.169 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-78 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:27:16.231 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:27:16.231 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:27:16.269 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:27:16.269+0000 7f91e3f22880 1 bucket sync caught up with source: 2024-08-18T16:27:16.270 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , 00000000002.147.6, 00000000004.130.6, , , , , , , , , , , ] 2024-08-18T16:27:16.270 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , , 00000000002.147.6, 00000000004.130.6, , , , , , , , , , , ] 2024-08-18T16:27:16.270 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:27:16.269+0000 7f91e3f22880 0 bucket checkpoint complete 2024-08-18T16:27:16.285 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:27:16.285 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:27:16.285 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-78 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:27:16.340 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:27:16.340 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:27:16.376 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:27:16.375+0000 7f23d5f2e880 1 bucket sync caught up with source: 2024-08-18T16:27:16.376 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000002.323.6, 00000000001.228.6, 00000000001.362.6, 00000000001.212.6, 00000000001.185.6, , , 00000000002.273.6, 00000000004.325.6, 00000000003.261.6] 2024-08-18T16:27:16.376 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000002.323.6, 00000000001.228.6, 00000000001.362.6, 00000000001.212.6, 00000000001.185.6, , , 00000000002.273.6, 00000000004.325.6, 00000000003.261.6] 2024-08-18T16:27:16.376 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:27:16.375+0000 7f23d5f2e880 0 bucket checkpoint complete 2024-08-18T16:27:16.390 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:27:16.391 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:27:16.391 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-78 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:27:16.444 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:27:16.444 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:27:16.480 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:27:16.479+0000 7ff505c0f880 1 bucket sync caught up with source: 2024-08-18T16:27:16.480 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000002.323.6, 00000000001.228.6, 00000000001.362.6, 00000000001.212.6, 00000000001.185.6, , , 00000000002.273.6, 00000000004.325.6, 00000000003.261.6] 2024-08-18T16:27:16.480 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000002.323.6, 00000000001.228.6, 00000000001.362.6, 00000000001.212.6, 00000000001.185.6, , , 00000000002.273.6, 00000000004.325.6, 00000000003.261.6] 2024-08-18T16:27:16.480 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:27:16.479+0000 7ff505c0f880 0 bucket checkpoint complete 2024-08-18T16:27:16.493 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:27:16.494 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:27:16.494 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-78 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:27:16.547 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:27:16.547 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:27:16.581 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:27:16.581+0000 7fc7abd42880 1 bucket sync caught up with source: 2024-08-18T16:27:16.582 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000002.191.6, 00000000001.472.6, 00000000001.243.6, 00000000001.269.6, 00000000001.130.6, , , 00000000002.176.6, 00000000004.193.6, 00000000003.135.6] 2024-08-18T16:27:16.582 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000002.191.6, 00000000001.472.6, 00000000001.243.6, 00000000001.269.6, 00000000001.130.6, , , 00000000002.176.6, 00000000004.193.6, 00000000003.135.6] 2024-08-18T16:27:16.582 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:27:16.581+0000 7fc7abd42880 0 bucket checkpoint complete 2024-08-18T16:27:16.596 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:27:16.596 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:27:16.597 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-78 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T16:27:16.658 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:27:16.658 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:27:16.697 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:27:16.695+0000 7f99cff20880 1 bucket sync caught up with source: 2024-08-18T16:27:16.697 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, 00000000002.191.6, 00000000001.472.6, 00000000001.243.6, 00000000001.269.6, 00000000001.130.6, , , 00000000002.176.6, 00000000004.193.6, 00000000003.135.6] 2024-08-18T16:27:16.697 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, 00000000002.191.6, 00000000001.472.6, 00000000001.243.6, 00000000001.269.6, 00000000001.130.6, , , 00000000002.176.6, 00000000004.193.6, 00000000003.135.6] 2024-08-18T16:27:16.697 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:27:16.695+0000 7f99cff20880 0 bucket checkpoint complete 2024-08-18T16:27:16.712 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-78 zones={test-zone2, test-zone1} 2024-08-18T16:27:16.722 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:27:16.726 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-08-18T16:27:16.726 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-08-18T16:27:16.726 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-08-18T16:27:16.726 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-08-18T16:27:16.727 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-08-18T16:27:16.727 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-08-18T16:27:16.727 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-08-18T16:27:16.727 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-08-18T16:27:16.727 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-08-18T16:27:16.727 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-08-18T16:27:16.727 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-08-18T16:27:16.727 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-08-18T16:27:16.727 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-08-18T16:27:16.727 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-08-18T16:27:16.727 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-08-18T16:27:16.727 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:27:16.731 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-08-18T16:27:16.732 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-08-18T16:27:16.732 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-08-18T16:27:16.732 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-08-18T16:27:16.732 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-08-18T16:27:16.732 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-08-18T16:27:16.732 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-08-18T16:27:16.732 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-08-18T16:27:16.732 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-08-18T16:27:16.732 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-08-18T16:27:16.732 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-08-18T16:27:16.732 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-08-18T16:27:16.732 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-08-18T16:27:16.732 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-08-18T16:27:16.733 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-08-18T16:27:16.739 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T16:27:16.745 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T16:27:16.760 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T16:27:16.810 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-08-18T16:27:16.816 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-08-18T16:27:16.872 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-08-18T16:27:16.927 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-08-18T16:27:16.934 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-08-18T16:27:16.945 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-08-18T16:27:16.996 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T16:27:17.004 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T16:27:17.062 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T16:27:17.070 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T16:27:17.120 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T16:27:17.132 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T16:27:17.140 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-08-18T16:27:17.194 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-08-18T16:27:17.206 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-08-18T16:27:17.214 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-08-18T16:27:17.263 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-08-18T16:27:17.317 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-08-18T16:27:17.373 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T16:27:17.380 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T16:27:17.437 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T16:27:17.490 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-08-18T16:27:17.496 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-08-18T16:27:17.548 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-08-18T16:27:17.597 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T16:27:17.604 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T16:27:17.657 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T16:27:17.706 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-08-18T16:27:17.755 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-08-18T16:27:17.809 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-08-18T16:27:17.865 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-08-18T16:27:17.872 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-08-18T16:27:17.924 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-08-18T16:27:17.979 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-08-18T16:27:17.990 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-08-18T16:27:18.001 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-08-18T16:27:18.053 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-08-18T16:27:18.060 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-08-18T16:27:18.113 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-08-18T16:27:18.122 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-08-18T16:27:18.172 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-08-18T16:27:18.183 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-08-18T16:27:18.236 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-78 zones={test-zone2, test-zone1} 2024-08-18T16:27:18.236 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-78 zones={test-zone3, test-zone1} 2024-08-18T16:27:18.242 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:27:18.244 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-08-18T16:27:18.245 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-08-18T16:27:18.245 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-08-18T16:27:18.245 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-08-18T16:27:18.245 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-08-18T16:27:18.245 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-08-18T16:27:18.245 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-08-18T16:27:18.245 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-08-18T16:27:18.245 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-08-18T16:27:18.245 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-08-18T16:27:18.245 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-08-18T16:27:18.245 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-08-18T16:27:18.245 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-08-18T16:27:18.245 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-08-18T16:27:18.245 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-08-18T16:27:18.246 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:27:18.249 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-08-18T16:27:18.249 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-08-18T16:27:18.249 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-08-18T16:27:18.249 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-08-18T16:27:18.249 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-08-18T16:27:18.249 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-08-18T16:27:18.249 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-08-18T16:27:18.249 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-08-18T16:27:18.249 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-08-18T16:27:18.249 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-08-18T16:27:18.250 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-08-18T16:27:18.250 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-08-18T16:27:18.250 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-08-18T16:27:18.250 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-08-18T16:27:18.250 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-08-18T16:27:18.256 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T16:27:18.306 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T16:27:18.361 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T16:27:18.413 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-08-18T16:27:18.463 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-08-18T16:27:18.521 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-08-18T16:27:18.573 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-08-18T16:27:18.580 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-08-18T16:27:18.633 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-08-18T16:27:18.686 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T16:27:18.736 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T16:27:18.793 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T16:27:18.846 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T16:27:18.900 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T16:27:18.953 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T16:27:19.009 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-08-18T16:27:19.060 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-08-18T16:27:19.114 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-08-18T16:27:19.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-08-18T16:27:19.223 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-08-18T16:27:19.284 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-08-18T16:27:19.335 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T16:27:19.384 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T16:27:19.485 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T16:27:19.538 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-08-18T16:27:19.588 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-08-18T16:27:19.689 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-08-18T16:27:19.747 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T16:27:19.796 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T16:27:19.897 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T16:27:19.951 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-08-18T16:27:20.001 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-08-18T16:27:20.058 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-08-18T16:27:20.067 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-08-18T16:27:20.116 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-08-18T16:27:20.214 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-08-18T16:27:20.266 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-08-18T16:27:20.322 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-08-18T16:27:20.334 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-08-18T16:27:20.386 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-08-18T16:27:20.393 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-08-18T16:27:20.449 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-08-18T16:27:20.500 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-08-18T16:27:20.508 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-08-18T16:27:20.521 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-08-18T16:27:20.571 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-78 zones={test-zone3, test-zone1} 2024-08-18T16:27:20.571 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-78 zones={test-zone3, test-zone2} 2024-08-18T16:27:20.575 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:27:20.579 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-08-18T16:27:20.579 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-08-18T16:27:20.579 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-08-18T16:27:20.579 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-08-18T16:27:20.579 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-08-18T16:27:20.579 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-08-18T16:27:20.579 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-08-18T16:27:20.579 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-08-18T16:27:20.579 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-08-18T16:27:20.580 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-08-18T16:27:20.580 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-08-18T16:27:20.580 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-08-18T16:27:20.580 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-08-18T16:27:20.580 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-08-18T16:27:20.580 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-08-18T16:27:20.580 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:27:20.583 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-08-18T16:27:20.583 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-08-18T16:27:20.583 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-08-18T16:27:20.583 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-08-18T16:27:20.584 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-08-18T16:27:20.584 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-08-18T16:27:20.584 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-08-18T16:27:20.584 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-08-18T16:27:20.584 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-08-18T16:27:20.584 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-08-18T16:27:20.584 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-08-18T16:27:20.584 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-08-18T16:27:20.584 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-08-18T16:27:20.584 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-08-18T16:27:20.584 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-08-18T16:27:20.591 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T16:27:20.646 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T16:27:20.749 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T16:27:20.845 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-08-18T16:27:20.898 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-08-18T16:27:20.996 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-08-18T16:27:21.089 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-08-18T16:27:21.097 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-08-18T16:27:21.153 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-08-18T16:27:21.205 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T16:27:21.261 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T16:27:21.321 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T16:27:21.419 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T16:27:21.470 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T16:27:21.525 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T16:27:21.625 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-08-18T16:27:21.676 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-08-18T16:27:21.773 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-08-18T16:27:21.869 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-08-18T16:27:21.878 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-08-18T16:27:21.938 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-08-18T16:27:21.948 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T16:27:21.996 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T16:27:22.049 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T16:27:22.101 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-08-18T16:27:22.152 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-08-18T16:27:22.209 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-08-18T16:27:22.305 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T16:27:22.360 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T16:27:22.461 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T16:27:22.561 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-08-18T16:27:22.569 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-08-18T16:27:22.624 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-08-18T16:27:22.674 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-08-18T16:27:22.724 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-08-18T16:27:22.777 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-08-18T16:27:22.834 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-08-18T16:27:22.841 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-08-18T16:27:22.899 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-08-18T16:27:22.954 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-08-18T16:27:22.962 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-08-18T16:27:23.017 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-08-18T16:27:23.070 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-08-18T16:27:23.078 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-08-18T16:27:23.135 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-08-18T16:27:23.191 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-78 zones={test-zone3, test-zone2} 2024-08-18T16:27:23.192 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:27:23.192 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:27:23.192 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-78 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:27:23.252 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:27:23.252 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:27:23.300 INFO:teuthology.orchestra.run.smithi134.stdout:tenant: 2024-08-18T16:27:23.300 INFO:teuthology.orchestra.run.smithi134.stdout:bucket name: kibgmb-78 2024-08-18T16:27:23.307 INFO:teuthology.orchestra.run.smithi134.stdout:total entries: 15 2024-08-18T16:27:23.469 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:27:23.470 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:27:23.470 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-78 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T16:27:23.531 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:27:23.532 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:27:23.571 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:27:23.571+0000 7ffb45323880 1 waiting to reach latest gen 3, on 2.. 2024-08-18T16:27:28.572 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:27:28.571+0000 7ffb45323880 1 waiting to reach latest gen 3, on 2.. 2024-08-18T16:27:33.573 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:27:33.572+0000 7ffb45323880 1 waiting to reach latest gen 3, on 2.. 2024-08-18T16:27:38.575 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:27:38.575+0000 7ffb45323880 1 waiting for incremental sync to catch up: 2024-08-18T16:27:38.576 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , 00000000002.108.6, , , , , , , , 00000000001.91.6, , , , , , ] 2024-08-18T16:27:38.576 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000001.170.6, , , 00000000002.108.6, , 00000000002.87.6, , , , , , 00000000001.91.6, , 00000000001.121.6, , , , ] 2024-08-18T16:27:43.577 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:27:43.576+0000 7ffb45323880 1 waiting for incremental sync to catch up: 2024-08-18T16:27:43.577 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , 00000000002.108.6, , , , , , , , 00000000001.91.6, , , , , , ] 2024-08-18T16:27:43.577 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000001.170.6, , , 00000000002.108.6, , 00000000002.87.6, , , , , , 00000000001.91.6, , 00000000001.121.6, , , , ] 2024-08-18T16:27:48.579 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:27:48.578+0000 7ffb45323880 1 waiting for incremental sync to catch up: 2024-08-18T16:27:48.580 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , 00000000002.108.6, , , , , , , , 00000000001.91.6, , , , , , ] 2024-08-18T16:27:48.580 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000001.170.6, , , 00000000002.108.6, , 00000000002.87.6, , , , , , 00000000001.91.6, , 00000000001.121.6, , , , ] 2024-08-18T16:27:53.581 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:27:53.580+0000 7ffb45323880 1 waiting for incremental sync to catch up: 2024-08-18T16:27:53.582 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , 00000000002.108.6, , , , , , , , 00000000001.91.6, , , , , , ] 2024-08-18T16:27:53.582 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000001.170.6, , , 00000000002.108.6, , 00000000002.87.6, , , , , , 00000000001.91.6, , 00000000001.121.6, , , , ] 2024-08-18T16:27:58.583 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:27:58.582+0000 7ffb45323880 1 waiting for incremental sync to catch up: 2024-08-18T16:27:58.583 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , 00000000002.108.6, , , , , , , , 00000000001.91.6, , , , , , ] 2024-08-18T16:27:58.583 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000001.170.6, , , 00000000002.108.6, , 00000000002.87.6, , , , , , 00000000001.91.6, , 00000000001.121.6, , , , ] 2024-08-18T16:28:03.584 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:28:03.584+0000 7ffb45323880 1 waiting for incremental sync to catch up: 2024-08-18T16:28:03.585 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , 00000000002.108.6, , , , , , , , 00000000001.91.6, , , , , , ] 2024-08-18T16:28:03.585 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000001.170.6, , , 00000000002.108.6, , 00000000002.87.6, , , , , , 00000000001.91.6, , 00000000001.121.6, , , , ] 2024-08-18T16:28:08.586 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:28:08.585+0000 7ffb45323880 1 waiting for incremental sync to catch up: 2024-08-18T16:28:08.586 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , 00000000002.108.6, , , , , , , , 00000000001.91.6, , , , , , ] 2024-08-18T16:28:08.586 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000001.170.6, , , 00000000002.108.6, , 00000000002.87.6, , , , , , 00000000001.91.6, , 00000000001.121.6, , , , ] 2024-08-18T16:28:13.588 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:28:13.588+0000 7ffb45323880 1 waiting for incremental sync to catch up: 2024-08-18T16:28:13.589 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , 00000000002.108.6, , , , , , , , 00000000001.91.6, , , , , , ] 2024-08-18T16:28:13.589 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000001.170.6, , , 00000000002.108.6, , 00000000002.87.6, , , , , , 00000000001.91.6, , 00000000001.121.6, , , , ] 2024-08-18T16:28:18.590 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:28:18.588+0000 7ffb45323880 1 waiting for incremental sync to catch up: 2024-08-18T16:28:18.590 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , 00000000002.108.6, , , , , , , , 00000000001.91.6, , , , , , ] 2024-08-18T16:28:18.590 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000001.170.6, , , 00000000002.108.6, , 00000000002.87.6, , , , , , 00000000001.91.6, , 00000000001.121.6, , , , ] 2024-08-18T16:28:23.592 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:28:23.590+0000 7ffb45323880 1 waiting for incremental sync to catch up: 2024-08-18T16:28:23.592 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , 00000000002.108.6, , , , , , , , 00000000001.91.6, , , , , , ] 2024-08-18T16:28:23.592 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000001.170.6, , , 00000000002.108.6, , 00000000002.87.6, , , , , , 00000000001.91.6, , 00000000001.121.6, , , , ] 2024-08-18T16:28:28.594 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:28:28.593+0000 7ffb45323880 1 waiting for incremental sync to catch up: 2024-08-18T16:28:28.594 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000001.170.6, , , 00000000002.108.6, , , , , , , , 00000000001.91.6, , , , , , ] 2024-08-18T16:28:28.594 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000001.170.6, , , 00000000002.108.6, , 00000000002.87.6, , , , , , 00000000001.91.6, , 00000000001.121.6, , , , ] 2024-08-18T16:28:33.596 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:28:33.595+0000 7ffb45323880 1 waiting for incremental sync to catch up: 2024-08-18T16:28:33.596 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000001.170.6, , , 00000000002.108.6, , , , , , , , 00000000001.91.6, , , , , , ] 2024-08-18T16:28:33.596 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000001.170.6, , , 00000000002.108.6, , 00000000002.87.6, , , , , , 00000000001.91.6, , 00000000001.121.6, , , , ] 2024-08-18T16:28:38.598 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:28:38.597+0000 7ffb45323880 1 bucket sync caught up with source: 2024-08-18T16:28:38.598 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000001.170.6, , , 00000000002.108.6, , 00000000002.87.6, , , , , , 00000000001.91.6, , 00000000001.121.6, , , , ] 2024-08-18T16:28:38.598 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000001.170.6, , , 00000000002.108.6, , 00000000002.87.6, , , , , , 00000000001.91.6, , 00000000001.121.6, , , , ] 2024-08-18T16:28:38.598 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:28:38.597+0000 7ffb45323880 0 bucket checkpoint complete 2024-08-18T16:28:38.617 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:28:38.618 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:28:38.618 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-78 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:28:38.680 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:28:38.680 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:28:38.719 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:28:38.718+0000 7ff7a941f880 1 waiting for incremental sync to catch up: 2024-08-18T16:28:38.719 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , , , , , 00000000002.87.6, , , , , , , , 00000000001.121.6, , , , ] 2024-08-18T16:28:38.719 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , 00000000001.170.6, , , 00000000002.108.6, , 00000000002.87.6, , , , , , 00000000001.91.6, , 00000000001.121.6, , , , ] 2024-08-18T16:28:43.721 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:28:43.720+0000 7ff7a941f880 1 waiting for incremental sync to catch up: 2024-08-18T16:28:43.721 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , , , , , 00000000002.87.6, , , , , , , , 00000000001.121.6, , , , ] 2024-08-18T16:28:43.721 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , 00000000001.170.6, , , 00000000002.108.6, , 00000000002.87.6, , , , , , 00000000001.91.6, , 00000000001.121.6, , , , ] 2024-08-18T16:28:48.724 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:28:48.721+0000 7ff7a941f880 1 waiting for incremental sync to catch up: 2024-08-18T16:28:48.724 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , , , 00000000002.108.6, , 00000000002.87.6, , , , , , 00000000001.91.6, , 00000000001.121.6, , , , ] 2024-08-18T16:28:48.724 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , 00000000001.170.6, , , 00000000002.108.6, , 00000000002.87.6, , , , , , 00000000001.91.6, , 00000000001.121.6, , , , ] 2024-08-18T16:28:53.724 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:28:53.723+0000 7ff7a941f880 1 waiting for incremental sync to catch up: 2024-08-18T16:28:53.724 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , , , 00000000002.108.6, , 00000000002.87.6, , , , , , 00000000001.91.6, , 00000000001.121.6, , , , ] 2024-08-18T16:28:53.724 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , 00000000001.170.6, , , 00000000002.108.6, , 00000000002.87.6, , , , , , 00000000001.91.6, , 00000000001.121.6, , , , ] 2024-08-18T16:28:58.726 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:28:58.725+0000 7ff7a941f880 1 waiting for incremental sync to catch up: 2024-08-18T16:28:58.726 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , , , 00000000002.108.6, , 00000000002.87.6, , , , , , 00000000001.91.6, , 00000000001.121.6, , , , ] 2024-08-18T16:28:58.726 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , 00000000001.170.6, , , 00000000002.108.6, , 00000000002.87.6, , , , , , 00000000001.91.6, , 00000000001.121.6, , , , ] 2024-08-18T16:29:03.728 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:29:03.727+0000 7ff7a941f880 1 waiting for incremental sync to catch up: 2024-08-18T16:29:03.728 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , , , 00000000002.108.6, , 00000000002.87.6, , , , , , 00000000001.91.6, , 00000000001.121.6, , , , ] 2024-08-18T16:29:03.728 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , 00000000001.170.6, , , 00000000002.108.6, , 00000000002.87.6, , , , , , 00000000001.91.6, , 00000000001.121.6, , , , ] 2024-08-18T16:29:08.729 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:29:08.729+0000 7ff7a941f880 1 waiting for incremental sync to catch up: 2024-08-18T16:29:08.730 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , , , 00000000002.108.6, , 00000000002.87.6, , , , , , 00000000001.91.6, , 00000000001.121.6, , , , ] 2024-08-18T16:29:08.730 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , 00000000001.170.6, , , 00000000002.108.6, , 00000000002.87.6, , , , , , 00000000001.91.6, , 00000000001.121.6, , , , ] 2024-08-18T16:29:13.732 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:29:13.731+0000 7ff7a941f880 1 waiting for incremental sync to catch up: 2024-08-18T16:29:13.732 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , , , 00000000002.108.6, , 00000000002.87.6, , , , , , 00000000001.91.6, , 00000000001.121.6, , , , ] 2024-08-18T16:29:13.732 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , 00000000001.170.6, , , 00000000002.108.6, , 00000000002.87.6, , , , , , 00000000001.91.6, , 00000000001.121.6, , , , ] 2024-08-18T16:29:18.734 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:29:18.733+0000 7ff7a941f880 1 waiting for incremental sync to catch up: 2024-08-18T16:29:18.734 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , , , 00000000002.108.6, , 00000000002.87.6, , , , , , 00000000001.91.6, , 00000000001.121.6, , , , ] 2024-08-18T16:29:18.734 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , 00000000001.170.6, , , 00000000002.108.6, , 00000000002.87.6, , , , , , 00000000001.91.6, , 00000000001.121.6, , , , ] 2024-08-18T16:29:23.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:29:23.734+0000 7ff7a941f880 1 waiting for incremental sync to catch up: 2024-08-18T16:29:23.736 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , , , 00000000002.108.6, , 00000000002.87.6, , , , , , 00000000001.91.6, , 00000000001.121.6, , , , ] 2024-08-18T16:29:23.736 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , 00000000001.170.6, , , 00000000002.108.6, , 00000000002.87.6, , , , , , 00000000001.91.6, , 00000000001.121.6, , , , ] 2024-08-18T16:29:28.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:29:28.737+0000 7ff7a941f880 1 waiting for incremental sync to catch up: 2024-08-18T16:29:28.738 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , , , 00000000002.108.6, , 00000000002.87.6, , , , , , 00000000001.91.6, , 00000000001.121.6, , , , ] 2024-08-18T16:29:28.738 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , 00000000001.170.6, , , 00000000002.108.6, , 00000000002.87.6, , , , , , 00000000001.91.6, , 00000000001.121.6, , , , ] 2024-08-18T16:29:33.740 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:29:33.739+0000 7ff7a941f880 1 waiting for incremental sync to catch up: 2024-08-18T16:29:33.740 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , , , 00000000002.108.6, , 00000000002.87.6, , , , , , 00000000001.91.6, , 00000000001.121.6, , , , ] 2024-08-18T16:29:33.740 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , 00000000001.170.6, , , 00000000002.108.6, , 00000000002.87.6, , , , , , 00000000001.91.6, , 00000000001.121.6, , , , ] 2024-08-18T16:29:38.742 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:29:38.741+0000 7ff7a941f880 1 waiting for incremental sync to catch up: 2024-08-18T16:29:38.742 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , , , 00000000002.108.6, , 00000000002.87.6, , , , , , 00000000001.91.6, , 00000000001.121.6, , , , ] 2024-08-18T16:29:38.742 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , 00000000001.170.6, , , 00000000002.108.6, , 00000000002.87.6, , , , , , 00000000001.91.6, , 00000000001.121.6, , , , ] 2024-08-18T16:29:43.743 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:29:43.743+0000 7ff7a941f880 1 waiting for incremental sync to catch up: 2024-08-18T16:29:43.744 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , , , 00000000002.108.6, , 00000000002.87.6, , , , , , 00000000001.91.6, , 00000000001.121.6, , , , ] 2024-08-18T16:29:43.744 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , 00000000001.170.6, , , 00000000002.108.6, , 00000000002.87.6, , , , , , 00000000001.91.6, , 00000000001.121.6, , , , ] 2024-08-18T16:29:48.745 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:29:48.744+0000 7ff7a941f880 1 waiting for incremental sync to catch up: 2024-08-18T16:29:48.745 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , , , 00000000002.108.6, , 00000000002.87.6, , , , , , 00000000001.91.6, , 00000000001.121.6, , , , ] 2024-08-18T16:29:48.745 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , 00000000001.170.6, , , 00000000002.108.6, , 00000000002.87.6, , , , , , 00000000001.91.6, , 00000000001.121.6, , , , ] 2024-08-18T16:29:53.747 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:29:53.746+0000 7ff7a941f880 1 waiting for incremental sync to catch up: 2024-08-18T16:29:53.747 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , , , 00000000002.108.6, , 00000000002.87.6, , , , , , 00000000001.91.6, , 00000000001.121.6, , , , ] 2024-08-18T16:29:53.747 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , 00000000001.170.6, , , 00000000002.108.6, , 00000000002.87.6, , , , , , 00000000001.91.6, , 00000000001.121.6, , , , ] 2024-08-18T16:29:58.748 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:29:58.748+0000 7ff7a941f880 1 waiting for incremental sync to catch up: 2024-08-18T16:29:58.749 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , , , 00000000002.108.6, , 00000000002.87.6, , , , , , 00000000001.91.6, , 00000000001.121.6, , , , ] 2024-08-18T16:29:58.749 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , 00000000001.170.6, , , 00000000002.108.6, , 00000000002.87.6, , , , , , 00000000001.91.6, , 00000000001.121.6, , , , ] 2024-08-18T16:30:03.750 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:30:03.749+0000 7ff7a941f880 1 waiting for incremental sync to catch up: 2024-08-18T16:30:03.751 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , , , 00000000002.108.6, , 00000000002.87.6, , , , , , 00000000001.91.6, , 00000000001.121.6, , , , ] 2024-08-18T16:30:03.751 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , 00000000001.170.6, , , 00000000002.108.6, , 00000000002.87.6, , , , , , 00000000001.91.6, , 00000000001.121.6, , , , ] 2024-08-18T16:30:08.752 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:30:08.751+0000 7ff7a941f880 1 waiting for incremental sync to catch up: 2024-08-18T16:30:08.752 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , , , 00000000002.108.6, , 00000000002.87.6, , , , , , 00000000001.91.6, , 00000000001.121.6, , , , ] 2024-08-18T16:30:08.752 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , 00000000001.170.6, , , 00000000002.108.6, , 00000000002.87.6, , , , , , 00000000001.91.6, , 00000000001.121.6, , , , ] 2024-08-18T16:30:13.754 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:30:13.753+0000 7ff7a941f880 1 waiting for incremental sync to catch up: 2024-08-18T16:30:13.754 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , , , 00000000002.108.6, , 00000000002.87.6, , , , , , 00000000001.91.6, , 00000000001.121.6, , , , ] 2024-08-18T16:30:13.754 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , 00000000001.170.6, , , 00000000002.108.6, , 00000000002.87.6, , , , , , 00000000001.91.6, , 00000000001.121.6, , , , ] 2024-08-18T16:30:18.756 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:30:18.755+0000 7ff7a941f880 1 waiting for incremental sync to catch up: 2024-08-18T16:30:18.756 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , , , 00000000002.108.6, , 00000000002.87.6, , , , , , 00000000001.91.6, , 00000000001.121.6, , , , ] 2024-08-18T16:30:18.756 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , 00000000001.170.6, , , 00000000002.108.6, , 00000000002.87.6, , , , , , 00000000001.91.6, , 00000000001.121.6, , , , ] 2024-08-18T16:30:23.758 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:30:23.757+0000 7ff7a941f880 1 waiting for incremental sync to catch up: 2024-08-18T16:30:23.758 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , , , 00000000002.108.6, , 00000000002.87.6, , , , , , 00000000001.91.6, , 00000000001.121.6, , , , ] 2024-08-18T16:30:23.758 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , 00000000001.170.6, , , 00000000002.108.6, , 00000000002.87.6, , , , , , 00000000001.91.6, , 00000000001.121.6, , , , ] 2024-08-18T16:30:28.760 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:30:28.759+0000 7ff7a941f880 1 waiting for incremental sync to catch up: 2024-08-18T16:30:28.760 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , , , 00000000002.108.6, , 00000000002.87.6, , , , , , 00000000001.91.6, , 00000000001.121.6, , , , ] 2024-08-18T16:30:28.760 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , 00000000001.170.6, , , 00000000002.108.6, , 00000000002.87.6, , , , , , 00000000001.91.6, , 00000000001.121.6, , , , ] 2024-08-18T16:30:33.761 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:30:33.760+0000 7ff7a941f880 1 waiting for incremental sync to catch up: 2024-08-18T16:30:33.762 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , , , 00000000002.108.6, , 00000000002.87.6, , , , , , 00000000001.91.6, , 00000000001.121.6, , , , ] 2024-08-18T16:30:33.762 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , 00000000001.170.6, , , 00000000002.108.6, , 00000000002.87.6, , , , , , 00000000001.91.6, , 00000000001.121.6, , , , ] 2024-08-18T16:30:38.763 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:30:38.762+0000 7ff7a941f880 1 waiting for incremental sync to catch up: 2024-08-18T16:30:38.763 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , , , 00000000002.108.6, , 00000000002.87.6, , , , , , 00000000001.91.6, , 00000000001.121.6, , , , ] 2024-08-18T16:30:38.763 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , 00000000001.170.6, , , 00000000002.108.6, , 00000000002.87.6, , , , , , 00000000001.91.6, , 00000000001.121.6, , , , ] 2024-08-18T16:30:43.765 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:30:43.764+0000 7ff7a941f880 1 waiting for incremental sync to catch up: 2024-08-18T16:30:43.765 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , , , 00000000002.108.6, , 00000000002.87.6, , , , , , 00000000001.91.6, , 00000000001.121.6, , , , ] 2024-08-18T16:30:43.765 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , 00000000001.170.6, , , 00000000002.108.6, , 00000000002.87.6, , , , , , 00000000001.91.6, , 00000000001.121.6, , , , ] 2024-08-18T16:30:48.767 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:30:48.766+0000 7ff7a941f880 1 bucket sync caught up with source: 2024-08-18T16:30:48.767 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , 00000000001.170.6, , , 00000000002.108.6, , 00000000002.87.6, , , , , , 00000000001.91.6, , 00000000001.121.6, , , , ] 2024-08-18T16:30:48.767 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , 00000000001.170.6, , , 00000000002.108.6, , 00000000002.87.6, , , , , , 00000000001.91.6, , 00000000001.121.6, , , , ] 2024-08-18T16:30:48.767 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:30:48.766+0000 7ff7a941f880 0 bucket checkpoint complete 2024-08-18T16:30:48.782 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:30:48.782 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:30:48.783 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-78 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:30:48.838 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:30:48.838 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:30:48.877 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:30:48.876+0000 7f8f2f435880 1 bucket sync caught up with source: 2024-08-18T16:30:48.877 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000002.323.6, 00000000002.238.6, 00000000001.362.6, 00000000002.216.6, 00000000002.191.6, , , 00000000002.273.6, 00000000005.331.6, 00000000004.267.6] 2024-08-18T16:30:48.878 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000002.323.6, 00000000002.238.6, 00000000001.362.6, 00000000002.216.6, 00000000002.191.6, , , 00000000002.273.6, 00000000005.331.6, 00000000004.267.6] 2024-08-18T16:30:48.878 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:30:48.876+0000 7f8f2f435880 0 bucket checkpoint complete 2024-08-18T16:30:48.890 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:30:48.891 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:30:48.891 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-78 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:30:48.943 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:30:48.943 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:30:48.979 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:30:48.978+0000 7fe88c210880 1 bucket sync caught up with source: 2024-08-18T16:30:48.979 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000002.323.6, 00000000002.238.6, 00000000001.362.6, 00000000002.216.6, 00000000002.191.6, , , 00000000002.273.6, 00000000005.331.6, 00000000004.267.6] 2024-08-18T16:30:48.979 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000002.323.6, 00000000002.238.6, 00000000001.362.6, 00000000002.216.6, 00000000002.191.6, , , 00000000002.273.6, 00000000005.331.6, 00000000004.267.6] 2024-08-18T16:30:48.979 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:30:48.978+0000 7fe88c210880 0 bucket checkpoint complete 2024-08-18T16:30:48.992 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:30:48.992 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:30:48.993 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-78 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:30:49.045 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:30:49.046 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:30:49.081 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:30:49.080+0000 7faf9dc0f880 1 bucket sync caught up with source: 2024-08-18T16:30:49.081 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000002.191.6, 00000000002.490.6, 00000000001.243.6, 00000000002.276.6, 00000000002.140.6, , , 00000000002.176.6, 00000000005.205.6, 00000000004.145.6] 2024-08-18T16:30:49.081 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000002.191.6, 00000000002.490.6, 00000000001.243.6, 00000000002.276.6, 00000000002.140.6, , , 00000000002.176.6, 00000000005.205.6, 00000000004.145.6] 2024-08-18T16:30:49.081 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:30:49.080+0000 7faf9dc0f880 0 bucket checkpoint complete 2024-08-18T16:30:49.095 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:30:49.095 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:30:49.095 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-78 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T16:30:49.156 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:30:49.156 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:30:49.196 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:30:49.196+0000 7fafc6e2f880 1 bucket sync caught up with source: 2024-08-18T16:30:49.197 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, 00000000002.191.6, 00000000002.490.6, 00000000001.243.6, 00000000002.276.6, 00000000002.140.6, , , 00000000002.176.6, 00000000005.205.6, 00000000004.145.6] 2024-08-18T16:30:49.197 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, 00000000002.191.6, 00000000002.490.6, 00000000001.243.6, 00000000002.276.6, 00000000002.140.6, , , 00000000002.176.6, 00000000005.205.6, 00000000004.145.6] 2024-08-18T16:30:49.197 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:30:49.196+0000 7fafc6e2f880 0 bucket checkpoint complete 2024-08-18T16:30:49.212 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-78 zones={test-zone2, test-zone1} 2024-08-18T16:30:49.222 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:30:49.226 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-08-18T16:30:49.226 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2024-08-18T16:30:49.226 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-08-18T16:30:49.226 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-08-18T16:30:49.227 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-08-18T16:30:49.227 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2024-08-18T16:30:49.227 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-08-18T16:30:49.227 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-08-18T16:30:49.227 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-08-18T16:30:49.227 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-08-18T16:30:49.227 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2024-08-18T16:30:49.227 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-08-18T16:30:49.227 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-08-18T16:30:49.227 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-08-18T16:30:49.227 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-08-18T16:30:49.227 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2024-08-18T16:30:49.227 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-08-18T16:30:49.227 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-08-18T16:30:49.227 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2024-08-18T16:30:49.227 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-08-18T16:30:49.228 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:30:49.231 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-08-18T16:30:49.231 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2024-08-18T16:30:49.232 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-08-18T16:30:49.232 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-08-18T16:30:49.232 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-08-18T16:30:49.232 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2024-08-18T16:30:49.232 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-08-18T16:30:49.232 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-08-18T16:30:49.232 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-08-18T16:30:49.232 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-08-18T16:30:49.232 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2024-08-18T16:30:49.232 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-08-18T16:30:49.232 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-08-18T16:30:49.232 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-08-18T16:30:49.232 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-08-18T16:30:49.233 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2024-08-18T16:30:49.233 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-08-18T16:30:49.233 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-08-18T16:30:49.233 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2024-08-18T16:30:49.233 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-08-18T16:30:49.239 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T16:30:49.246 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T16:30:49.299 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T16:30:49.353 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-08-18T16:30:49.360 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-08-18T16:30:49.372 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-08-18T16:30:49.423 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-08-18T16:30:49.430 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-08-18T16:30:49.441 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-08-18T16:30:49.493 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-08-18T16:30:49.500 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-08-18T16:30:49.554 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-08-18T16:30:49.565 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T16:30:49.618 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T16:30:49.630 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T16:30:49.681 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-08-18T16:30:49.688 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-08-18T16:30:49.742 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-08-18T16:30:49.750 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T16:30:49.804 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T16:30:49.858 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T16:30:49.867 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-08-18T16:30:49.874 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-08-18T16:30:49.928 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-08-18T16:30:49.983 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-08-18T16:30:49.990 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-08-18T16:30:50.089 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-08-18T16:30:50.142 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T16:30:50.192 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T16:30:50.247 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T16:30:50.255 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-08-18T16:30:50.304 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-08-18T16:30:50.315 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-08-18T16:30:50.372 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-08-18T16:30:50.381 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-08-18T16:30:50.435 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-08-18T16:30:50.489 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T16:30:50.496 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T16:30:50.553 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T16:30:50.602 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-08-18T16:30:50.608 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-08-18T16:30:50.619 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-08-18T16:30:50.671 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-08-18T16:30:50.678 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-08-18T16:30:50.737 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-08-18T16:30:50.786 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-08-18T16:30:50.794 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-08-18T16:30:50.805 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-08-18T16:30:50.857 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-08-18T16:30:50.864 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-08-18T16:30:50.920 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-08-18T16:30:50.969 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-08-18T16:30:50.976 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-08-18T16:30:51.028 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-08-18T16:30:51.038 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-08-18T16:30:51.094 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-08-18T16:30:51.150 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-08-18T16:30:51.158 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-08-18T16:30:51.208 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-08-18T16:30:51.219 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-08-18T16:30:51.227 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-78 zones={test-zone2, test-zone1} 2024-08-18T16:30:51.227 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-78 zones={test-zone3, test-zone1} 2024-08-18T16:30:51.234 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:30:51.237 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-08-18T16:30:51.237 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2024-08-18T16:30:51.238 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-08-18T16:30:51.238 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-08-18T16:30:51.238 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-08-18T16:30:51.238 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2024-08-18T16:30:51.238 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-08-18T16:30:51.238 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-08-18T16:30:51.238 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-08-18T16:30:51.238 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-08-18T16:30:51.238 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2024-08-18T16:30:51.238 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-08-18T16:30:51.238 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-08-18T16:30:51.238 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-08-18T16:30:51.238 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-08-18T16:30:51.239 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2024-08-18T16:30:51.239 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-08-18T16:30:51.239 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-08-18T16:30:51.239 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2024-08-18T16:30:51.239 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-08-18T16:30:51.239 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:30:51.243 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-08-18T16:30:51.243 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2024-08-18T16:30:51.243 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-08-18T16:30:51.243 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-08-18T16:30:51.243 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-08-18T16:30:51.243 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2024-08-18T16:30:51.243 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-08-18T16:30:51.243 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-08-18T16:30:51.243 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-08-18T16:30:51.243 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-08-18T16:30:51.243 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2024-08-18T16:30:51.243 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-08-18T16:30:51.244 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-08-18T16:30:51.244 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-08-18T16:30:51.244 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-08-18T16:30:51.244 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2024-08-18T16:30:51.244 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-08-18T16:30:51.244 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-08-18T16:30:51.244 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2024-08-18T16:30:51.244 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-08-18T16:30:51.251 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T16:30:51.300 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T16:30:51.353 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T16:30:51.408 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-08-18T16:30:51.415 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-08-18T16:30:51.467 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-08-18T16:30:51.516 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-08-18T16:30:51.523 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-08-18T16:30:51.584 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-08-18T16:30:51.594 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-08-18T16:30:51.601 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-08-18T16:30:51.614 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-08-18T16:30:51.669 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T16:30:51.724 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T16:30:51.821 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T16:30:51.874 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-08-18T16:30:51.923 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-08-18T16:30:51.980 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-08-18T16:30:52.029 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T16:30:52.079 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T16:30:52.173 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T16:30:52.229 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-08-18T16:30:52.280 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-08-18T16:30:52.332 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-08-18T16:30:52.426 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-08-18T16:30:52.435 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-08-18T16:30:52.447 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-08-18T16:30:52.496 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T16:30:52.543 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T16:30:52.596 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T16:30:52.694 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-08-18T16:30:52.700 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-08-18T16:30:52.711 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-08-18T16:30:52.760 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-08-18T16:30:52.811 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-08-18T16:30:52.864 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-08-18T16:30:52.917 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T16:30:52.971 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T16:30:53.065 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T16:30:53.114 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-08-18T16:30:53.167 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-08-18T16:30:53.179 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-08-18T16:30:53.228 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-08-18T16:30:53.280 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-08-18T16:30:53.377 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-08-18T16:30:53.429 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-08-18T16:30:53.480 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-08-18T16:30:53.536 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-08-18T16:30:53.586 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-08-18T16:30:53.594 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-08-18T16:30:53.606 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-08-18T16:30:53.660 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-08-18T16:30:53.669 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-08-18T16:30:53.725 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-08-18T16:30:53.777 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-08-18T16:30:53.783 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-08-18T16:30:53.842 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-08-18T16:30:53.850 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-08-18T16:30:53.899 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-08-18T16:30:53.912 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-08-18T16:30:53.968 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-78 zones={test-zone3, test-zone1} 2024-08-18T16:30:53.968 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-78 zones={test-zone3, test-zone2} 2024-08-18T16:30:53.973 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:30:53.976 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-08-18T16:30:53.976 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2024-08-18T16:30:53.976 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-08-18T16:30:53.976 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-08-18T16:30:53.976 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-08-18T16:30:53.976 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2024-08-18T16:30:53.976 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-08-18T16:30:53.976 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-08-18T16:30:53.976 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-08-18T16:30:53.976 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-08-18T16:30:53.977 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2024-08-18T16:30:53.977 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-08-18T16:30:53.977 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-08-18T16:30:53.977 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-08-18T16:30:53.977 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-08-18T16:30:53.977 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2024-08-18T16:30:53.977 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-08-18T16:30:53.977 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-08-18T16:30:53.977 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2024-08-18T16:30:53.977 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-08-18T16:30:53.977 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:30:53.980 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-08-18T16:30:53.981 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2024-08-18T16:30:53.981 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-08-18T16:30:53.981 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-08-18T16:30:53.981 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-08-18T16:30:53.981 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2024-08-18T16:30:53.981 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-08-18T16:30:53.981 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-08-18T16:30:53.981 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-08-18T16:30:53.981 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-08-18T16:30:53.981 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2024-08-18T16:30:53.981 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-08-18T16:30:53.981 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-08-18T16:30:53.981 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-08-18T16:30:53.981 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-08-18T16:30:53.981 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2024-08-18T16:30:53.982 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-08-18T16:30:53.982 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-08-18T16:30:53.982 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2024-08-18T16:30:53.982 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-08-18T16:30:53.988 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T16:30:54.042 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T16:30:54.141 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T16:30:54.238 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-08-18T16:30:54.245 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-08-18T16:30:54.345 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-08-18T16:30:54.437 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-08-18T16:30:54.488 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-08-18T16:30:54.590 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-08-18T16:30:54.641 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-08-18T16:30:54.692 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-08-18T16:30:54.704 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-08-18T16:30:54.712 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T16:30:54.765 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T16:30:54.821 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T16:30:54.873 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-08-18T16:30:54.924 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-08-18T16:30:54.977 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-08-18T16:30:55.030 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T16:30:55.084 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T16:30:55.141 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T16:30:55.238 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-08-18T16:30:55.289 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-08-18T16:30:55.389 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-08-18T16:30:55.485 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-08-18T16:30:55.494 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-08-18T16:30:55.506 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-08-18T16:30:55.560 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T16:30:55.608 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T16:30:55.665 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T16:30:55.718 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-08-18T16:30:55.725 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-08-18T16:30:55.781 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-08-18T16:30:55.833 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-08-18T16:30:55.885 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-08-18T16:30:55.945 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-08-18T16:30:56.042 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T16:30:56.097 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T16:30:56.154 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T16:30:56.254 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-08-18T16:30:56.274 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-08-18T16:30:56.285 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-08-18T16:30:56.341 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-08-18T16:30:56.393 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-08-18T16:30:56.445 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-08-18T16:30:56.498 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-08-18T16:30:56.549 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-08-18T16:30:56.606 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-08-18T16:30:56.702 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-08-18T16:30:56.710 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-08-18T16:30:56.770 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-08-18T16:30:56.779 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-08-18T16:30:56.835 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-08-18T16:30:56.894 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-08-18T16:30:56.947 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-08-18T16:30:56.954 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-08-18T16:30:56.966 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-08-18T16:30:57.016 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-08-18T16:30:57.023 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-08-18T16:30:57.080 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-08-18T16:30:57.131 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-78 zones={test-zone3, test-zone2} 2024-08-18T16:30:57.131 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:30:57.131 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:30:57.131 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-78 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:30:57.191 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:30:57.191 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:30:57.243 INFO:teuthology.orchestra.run.smithi134.stdout:tenant: 2024-08-18T16:30:57.243 INFO:teuthology.orchestra.run.smithi134.stdout:bucket name: kibgmb-78 2024-08-18T16:30:57.252 INFO:teuthology.orchestra.run.smithi134.stdout:total entries: 20 2024-08-18T16:30:57.382 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:30:57.382 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:30:57.382 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-78 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T16:30:57.442 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:30:57.442 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:30:57.480 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:30:57.480+0000 7ff52a610880 1 waiting to reach latest gen 4, on 3.. 2024-08-18T16:31:02.481 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:31:02.480+0000 7ff52a610880 1 waiting to reach latest gen 4, on 3.. 2024-08-18T16:31:07.482 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:31:07.481+0000 7ff52a610880 1 waiting to reach latest gen 4, on 3.. 2024-08-18T16:31:12.483 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:31:12.482+0000 7ff52a610880 1 waiting to reach latest gen 4, on 3.. 2024-08-18T16:31:17.485 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:31:17.484+0000 7ff52a610880 1 waiting for incremental sync to catch up: 2024-08-18T16:31:17.485 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2024-08-18T16:31:17.486 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000002.123.6, 00000000001.105.6, , , , , 00000000001.114.6, , , , , , , , , , , 00000000002.175.6, 00000000001.182.6, , , , , , , , , , ] 2024-08-18T16:31:22.487 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:31:22.486+0000 7ff52a610880 1 waiting for incremental sync to catch up: 2024-08-18T16:31:22.488 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2024-08-18T16:31:22.488 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000002.123.6, 00000000001.105.6, , , , , 00000000001.114.6, , , , , , , , , , , 00000000002.175.6, 00000000001.182.6, , , , , , , , , , ] 2024-08-18T16:31:27.489 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:31:27.487+0000 7ff52a610880 1 waiting for incremental sync to catch up: 2024-08-18T16:31:27.489 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2024-08-18T16:31:27.489 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000002.123.6, 00000000001.105.6, , , , , 00000000001.114.6, , , , , , , , , , , 00000000002.175.6, 00000000001.182.6, , , , , , , , , , ] 2024-08-18T16:31:32.491 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:31:32.490+0000 7ff52a610880 1 waiting for incremental sync to catch up: 2024-08-18T16:31:32.491 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2024-08-18T16:31:32.491 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000002.123.6, 00000000001.105.6, , , , , 00000000001.114.6, , , , , , , , , , , 00000000002.175.6, 00000000001.182.6, , , , , , , , , , ] 2024-08-18T16:31:37.493 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:31:37.492+0000 7ff52a610880 1 waiting for incremental sync to catch up: 2024-08-18T16:31:37.493 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000002.123.6, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2024-08-18T16:31:37.493 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000002.123.6, 00000000001.105.6, , , , , 00000000001.114.6, , , , , , , , , , , 00000000002.175.6, 00000000001.182.6, , , , , , , , , , ] 2024-08-18T16:31:42.495 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:31:42.494+0000 7ff52a610880 1 waiting for incremental sync to catch up: 2024-08-18T16:31:42.495 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000002.123.6, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2024-08-18T16:31:42.495 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000002.123.6, 00000000001.105.6, , , , , 00000000001.114.6, , , , , , , , , , , 00000000002.175.6, 00000000001.182.6, , , , , , , , , , ] 2024-08-18T16:31:47.497 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:31:47.496+0000 7ff52a610880 1 waiting for incremental sync to catch up: 2024-08-18T16:31:47.497 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000002.123.6, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2024-08-18T16:31:47.497 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000002.123.6, 00000000001.105.6, , , , , 00000000001.114.6, , , , , , , , , , , 00000000002.175.6, 00000000001.182.6, , , , , , , , , , ] 2024-08-18T16:31:52.499 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:31:52.498+0000 7ff52a610880 1 waiting for incremental sync to catch up: 2024-08-18T16:31:52.499 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000002.123.6, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2024-08-18T16:31:52.499 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000002.123.6, 00000000001.105.6, , , , , 00000000001.114.6, , , , , , , , , , , 00000000002.175.6, 00000000001.182.6, , , , , , , , , , ] 2024-08-18T16:31:57.501 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:31:57.500+0000 7ff52a610880 1 waiting for incremental sync to catch up: 2024-08-18T16:31:57.501 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000002.123.6, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2024-08-18T16:31:57.501 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000002.123.6, 00000000001.105.6, , , , , 00000000001.114.6, , , , , , , , , , , 00000000002.175.6, 00000000001.182.6, , , , , , , , , , ] 2024-08-18T16:32:02.503 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:32:02.502+0000 7ff52a610880 1 waiting for incremental sync to catch up: 2024-08-18T16:32:02.504 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000002.123.6, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2024-08-18T16:32:02.504 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000002.123.6, 00000000001.105.6, , , , , 00000000001.114.6, , , , , , , , , , , 00000000002.175.6, 00000000001.182.6, , , , , , , , , , ] 2024-08-18T16:32:07.506 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:32:07.505+0000 7ff52a610880 1 waiting for incremental sync to catch up: 2024-08-18T16:32:07.506 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000002.123.6, 00000000001.105.6, , , , , , , , , , , , , , , , 00000000002.175.6, , , , , , , , , , , ] 2024-08-18T16:32:07.506 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000002.123.6, 00000000001.105.6, , , , , 00000000001.114.6, , , , , , , , , , , 00000000002.175.6, 00000000001.182.6, , , , , , , , , , ] 2024-08-18T16:32:12.508 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:32:12.507+0000 7ff52a610880 1 waiting for incremental sync to catch up: 2024-08-18T16:32:12.508 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000002.123.6, 00000000001.105.6, , , , , , , , , , , , , , , , 00000000002.175.6, , , , , , , , , , , ] 2024-08-18T16:32:12.508 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000002.123.6, 00000000001.105.6, , , , , 00000000001.114.6, , , , , , , , , , , 00000000002.175.6, 00000000001.182.6, , , , , , , , , , ] 2024-08-18T16:32:17.510 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:32:17.509+0000 7ff52a610880 1 waiting for incremental sync to catch up: 2024-08-18T16:32:17.510 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000002.123.6, 00000000001.105.6, , , , , , , , , , , , , , , , 00000000002.175.6, 00000000001.182.6, , , , , , , , , , ] 2024-08-18T16:32:17.510 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000002.123.6, 00000000001.105.6, , , , , 00000000001.114.6, , , , , , , , , , , 00000000002.175.6, 00000000001.182.6, , , , , , , , , , ] 2024-08-18T16:32:22.512 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:32:22.511+0000 7ff52a610880 1 waiting for incremental sync to catch up: 2024-08-18T16:32:22.512 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000002.123.6, 00000000001.105.6, , , , , , , , , , , , , , , , 00000000002.175.6, 00000000001.182.6, , , , , , , , , , ] 2024-08-18T16:32:22.512 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000002.123.6, 00000000001.105.6, , , , , 00000000001.114.6, , , , , , , , , , , 00000000002.175.6, 00000000001.182.6, , , , , , , , , , ] 2024-08-18T16:32:27.514 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:32:27.513+0000 7ff52a610880 1 waiting for incremental sync to catch up: 2024-08-18T16:32:27.517 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000002.123.6, 00000000001.105.6, , , , , , , , , , , , , , , , 00000000002.175.6, 00000000001.182.6, , , , , , , , , , ] 2024-08-18T16:32:27.517 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000002.123.6, 00000000001.105.6, , , , , 00000000001.114.6, , , , , , , , , , , 00000000002.175.6, 00000000001.182.6, , , , , , , , , , ] 2024-08-18T16:32:32.516 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:32:32.514+0000 7ff52a610880 1 waiting for incremental sync to catch up: 2024-08-18T16:32:32.517 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000002.123.6, 00000000001.105.6, , , , , , , , , , , , , , , , 00000000002.175.6, 00000000001.182.6, , , , , , , , , , ] 2024-08-18T16:32:32.517 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000002.123.6, 00000000001.105.6, , , , , 00000000001.114.6, , , , , , , , , , , 00000000002.175.6, 00000000001.182.6, , , , , , , , , , ] 2024-08-18T16:32:37.518 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:32:37.518+0000 7ff52a610880 1 waiting for incremental sync to catch up: 2024-08-18T16:32:37.519 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000002.123.6, 00000000001.105.6, , , , , , , , , , , , , , , , 00000000002.175.6, 00000000001.182.6, , , , , , , , , , ] 2024-08-18T16:32:37.519 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000002.123.6, 00000000001.105.6, , , , , 00000000001.114.6, , , , , , , , , , , 00000000002.175.6, 00000000001.182.6, , , , , , , , , , ] 2024-08-18T16:32:42.521 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:32:42.520+0000 7ff52a610880 1 waiting for incremental sync to catch up: 2024-08-18T16:32:42.521 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000002.123.6, 00000000001.105.6, , , , , , , , , , , , , , , , 00000000002.175.6, 00000000001.182.6, , , , , , , , , , ] 2024-08-18T16:32:42.521 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000002.123.6, 00000000001.105.6, , , , , 00000000001.114.6, , , , , , , , , , , 00000000002.175.6, 00000000001.182.6, , , , , , , , , , ] 2024-08-18T16:32:47.523 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:32:47.522+0000 7ff52a610880 1 waiting for incremental sync to catch up: 2024-08-18T16:32:47.523 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000002.123.6, 00000000001.105.6, , , , , , , , , , , , , , , , 00000000002.175.6, 00000000001.182.6, , , , , , , , , , ] 2024-08-18T16:32:47.523 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000002.123.6, 00000000001.105.6, , , , , 00000000001.114.6, , , , , , , , , , , 00000000002.175.6, 00000000001.182.6, , , , , , , , , , ] 2024-08-18T16:32:52.525 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:32:52.524+0000 7ff52a610880 1 waiting for incremental sync to catch up: 2024-08-18T16:32:52.525 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000002.123.6, 00000000001.105.6, , , , , , , , , , , , , , , , 00000000002.175.6, 00000000001.182.6, , , , , , , , , , ] 2024-08-18T16:32:52.525 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000002.123.6, 00000000001.105.6, , , , , 00000000001.114.6, , , , , , , , , , , 00000000002.175.6, 00000000001.182.6, , , , , , , , , , ] 2024-08-18T16:32:57.527 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:32:57.527+0000 7ff52a610880 1 waiting for incremental sync to catch up: 2024-08-18T16:32:57.527 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000002.123.6, 00000000001.105.6, , , , , , , , , , , , , , , , 00000000002.175.6, 00000000001.182.6, , , , , , , , , , ] 2024-08-18T16:32:57.528 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000002.123.6, 00000000001.105.6, , , , , 00000000001.114.6, , , , , , , , , , , 00000000002.175.6, 00000000001.182.6, , , , , , , , , , ] 2024-08-18T16:33:02.530 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:33:02.529+0000 7ff52a610880 1 waiting for incremental sync to catch up: 2024-08-18T16:33:02.530 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000002.123.6, 00000000001.105.6, , , , , , , , , , , , , , , , 00000000002.175.6, 00000000001.182.6, , , , , , , , , , ] 2024-08-18T16:33:02.530 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000002.123.6, 00000000001.105.6, , , , , 00000000001.114.6, , , , , , , , , , , 00000000002.175.6, 00000000001.182.6, , , , , , , , , , ] 2024-08-18T16:33:07.532 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:33:07.531+0000 7ff52a610880 1 waiting for incremental sync to catch up: 2024-08-18T16:33:07.532 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000002.123.6, 00000000001.105.6, , , , , , , , , , , , , , , , 00000000002.175.6, 00000000001.182.6, , , , , , , , , , ] 2024-08-18T16:33:07.532 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000002.123.6, 00000000001.105.6, , , , , 00000000001.114.6, , , , , , , , , , , 00000000002.175.6, 00000000001.182.6, , , , , , , , , , ] 2024-08-18T16:33:12.534 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:33:12.533+0000 7ff52a610880 1 waiting for incremental sync to catch up: 2024-08-18T16:33:12.534 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000002.123.6, 00000000001.105.6, , , , , , , , , , , , , , , , 00000000002.175.6, 00000000001.182.6, , , , , , , , , , ] 2024-08-18T16:33:12.535 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000002.123.6, 00000000001.105.6, , , , , 00000000001.114.6, , , , , , , , , , , 00000000002.175.6, 00000000001.182.6, , , , , , , , , , ] 2024-08-18T16:33:17.536 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:33:17.535+0000 7ff52a610880 1 bucket sync caught up with source: 2024-08-18T16:33:17.537 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [00000000002.123.6, 00000000001.105.6, , , , , 00000000001.114.6, , , , , , , , , , , 00000000002.175.6, 00000000001.182.6, , , , , , , , , , ] 2024-08-18T16:33:17.537 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [00000000002.123.6, 00000000001.105.6, , , , , 00000000001.114.6, , , , , , , , , , , 00000000002.175.6, 00000000001.182.6, , , , , , , , , , ] 2024-08-18T16:33:17.537 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:33:17.535+0000 7ff52a610880 0 bucket checkpoint complete 2024-08-18T16:33:17.556 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:33:17.556 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:33:17.556 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-78 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:33:17.617 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:33:17.617 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:33:17.657 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:33:17.656+0000 7f2a94d40880 1 bucket sync caught up with source: 2024-08-18T16:33:17.657 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [00000000002.123.6, 00000000001.105.6, , , , , 00000000001.114.6, , , , , , , , , , , 00000000002.175.6, 00000000001.182.6, , , , , , , , , , ] 2024-08-18T16:33:17.658 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [00000000002.123.6, 00000000001.105.6, , , , , 00000000001.114.6, , , , , , , , , , , 00000000002.175.6, 00000000001.182.6, , , , , , , , , , ] 2024-08-18T16:33:17.658 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:33:17.656+0000 7f2a94d40880 0 bucket checkpoint complete 2024-08-18T16:33:17.671 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:33:17.671 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:33:17.672 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-78 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:33:17.725 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:33:17.725 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:33:17.761 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:33:17.760+0000 7f0561b26880 1 bucket sync caught up with source: 2024-08-18T16:33:17.761 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000002.323.6, 00000000002.238.6, 00000000001.362.6, 00000000003.220.6, 00000000004.197.6, 00000000001.404.6, 00000000001.244.6, 00000000002.273.6, 00000000005.331.6, 00000000004.267.6] 2024-08-18T16:33:17.761 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000002.323.6, 00000000002.238.6, 00000000001.362.6, 00000000003.220.6, 00000000004.197.6, 00000000001.404.6, 00000000001.244.6, 00000000002.273.6, 00000000005.331.6, 00000000004.267.6] 2024-08-18T16:33:17.761 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:33:17.760+0000 7f0561b26880 0 bucket checkpoint complete 2024-08-18T16:33:17.775 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:33:17.776 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:33:17.776 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-78 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:33:17.829 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:33:17.830 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:33:17.865 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:33:17.864+0000 7f75d2213880 1 bucket sync caught up with source: 2024-08-18T16:33:17.865 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000002.323.6, 00000000002.238.6, 00000000001.362.6, 00000000003.220.6, 00000000004.197.6, 00000000001.404.6, 00000000001.244.6, 00000000002.273.6, 00000000005.331.6, 00000000004.267.6] 2024-08-18T16:33:17.865 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000002.323.6, 00000000002.238.6, 00000000001.362.6, 00000000003.220.6, 00000000004.197.6, 00000000001.404.6, 00000000001.244.6, 00000000002.273.6, 00000000005.331.6, 00000000004.267.6] 2024-08-18T16:33:17.865 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:33:17.864+0000 7f75d2213880 0 bucket checkpoint complete 2024-08-18T16:33:17.879 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:33:17.879 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:33:17.879 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-78 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:33:17.932 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:33:17.932 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:33:17.967 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:33:17.966+0000 7fa654b17880 1 bucket sync caught up with source: 2024-08-18T16:33:17.967 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000002.191.6, 00000000002.490.6, 00000000001.243.6, 00000000003.280.6, 00000000004.146.6, 00000000001.526.6, 00000000001.496.6, 00000000002.176.6, 00000000005.205.6, 00000000004.145.6] 2024-08-18T16:33:17.967 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000002.191.6, 00000000002.490.6, 00000000001.243.6, 00000000003.280.6, 00000000004.146.6, 00000000001.526.6, 00000000001.496.6, 00000000002.176.6, 00000000005.205.6, 00000000004.145.6] 2024-08-18T16:33:17.967 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:33:17.966+0000 7fa654b17880 0 bucket checkpoint complete 2024-08-18T16:33:17.982 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:33:17.982 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:33:17.982 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-78 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T16:33:18.044 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:33:18.044 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:33:18.086 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:33:18.086+0000 7f323e435880 1 bucket sync caught up with source: 2024-08-18T16:33:18.087 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, 00000000002.191.6, 00000000002.490.6, 00000000001.243.6, 00000000003.280.6, 00000000004.146.6, 00000000001.526.6, 00000000001.496.6, 00000000002.176.6, 00000000005.205.6, 00000000004.145.6] 2024-08-18T16:33:18.087 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, 00000000002.191.6, 00000000002.490.6, 00000000001.243.6, 00000000003.280.6, 00000000004.146.6, 00000000001.526.6, 00000000001.496.6, 00000000002.176.6, 00000000005.205.6, 00000000004.145.6] 2024-08-18T16:33:18.087 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:33:18.086+0000 7f323e435880 0 bucket checkpoint complete 2024-08-18T16:33:18.102 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-78 zones={test-zone2, test-zone1} 2024-08-18T16:33:18.114 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:33:18.118 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-08-18T16:33:18.118 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2024-08-18T16:33:18.118 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2024-08-18T16:33:18.118 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2024-08-18T16:33:18.118 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-08-18T16:33:18.119 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-08-18T16:33:18.119 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-08-18T16:33:18.119 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2024-08-18T16:33:18.119 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2024-08-18T16:33:18.119 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-08-18T16:33:18.119 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-08-18T16:33:18.119 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-08-18T16:33:18.119 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-08-18T16:33:18.119 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2024-08-18T16:33:18.119 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-08-18T16:33:18.119 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-08-18T16:33:18.119 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-08-18T16:33:18.119 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2024-08-18T16:33:18.119 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-08-18T16:33:18.119 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2024-08-18T16:33:18.120 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-08-18T16:33:18.120 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-08-18T16:33:18.120 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2024-08-18T16:33:18.120 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2024-08-18T16:33:18.120 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-08-18T16:33:18.120 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:33:18.124 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-08-18T16:33:18.124 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2024-08-18T16:33:18.124 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2024-08-18T16:33:18.124 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2024-08-18T16:33:18.125 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-08-18T16:33:18.125 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-08-18T16:33:18.125 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-08-18T16:33:18.125 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2024-08-18T16:33:18.125 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2024-08-18T16:33:18.125 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-08-18T16:33:18.125 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-08-18T16:33:18.125 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-08-18T16:33:18.125 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-08-18T16:33:18.125 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2024-08-18T16:33:18.125 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-08-18T16:33:18.125 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-08-18T16:33:18.125 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-08-18T16:33:18.125 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2024-08-18T16:33:18.126 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-08-18T16:33:18.126 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2024-08-18T16:33:18.126 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-08-18T16:33:18.126 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-08-18T16:33:18.126 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2024-08-18T16:33:18.126 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2024-08-18T16:33:18.126 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-08-18T16:33:18.134 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T16:33:18.186 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T16:33:18.198 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T16:33:18.248 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-08-18T16:33:18.255 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-08-18T16:33:18.309 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-08-18T16:33:18.358 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-08-18T16:33:18.365 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-08-18T16:33:18.377 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-08-18T16:33:18.430 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-08-18T16:33:18.436 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-08-18T16:33:18.533 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-08-18T16:33:18.586 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-08-18T16:33:18.636 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-08-18T16:33:18.648 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-08-18T16:33:18.657 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-08-18T16:33:18.706 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-08-18T16:33:18.719 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-08-18T16:33:18.768 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T16:33:18.775 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T16:33:18.827 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T16:33:18.885 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-08-18T16:33:18.891 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-08-18T16:33:18.945 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-08-18T16:33:18.997 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-08-18T16:33:19.004 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-08-18T16:33:19.057 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-08-18T16:33:19.114 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T16:33:19.166 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T16:33:19.178 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T16:33:19.230 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-08-18T16:33:19.237 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-08-18T16:33:19.249 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-08-18T16:33:19.298 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-08-18T16:33:19.304 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-08-18T16:33:19.362 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-08-18T16:33:19.371 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T16:33:19.420 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T16:33:19.431 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T16:33:19.488 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-08-18T16:33:19.494 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-08-18T16:33:19.548 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-08-18T16:33:19.557 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-08-18T16:33:19.611 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-08-18T16:33:19.667 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-08-18T16:33:19.719 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T16:33:19.726 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T16:33:19.738 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T16:33:19.789 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-08-18T16:33:19.796 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-08-18T16:33:19.807 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-08-18T16:33:19.816 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-08-18T16:33:19.866 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-08-18T16:33:19.879 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-08-18T16:33:19.932 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-08-18T16:33:19.938 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-08-18T16:33:19.949 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-08-18T16:33:20.002 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-08-18T16:33:20.009 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-08-18T16:33:20.061 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-08-18T16:33:20.110 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-08-18T16:33:20.117 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-08-18T16:33:20.129 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-08-18T16:33:20.182 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-08-18T16:33:20.188 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-08-18T16:33:20.241 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-08-18T16:33:20.290 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-08-18T16:33:20.297 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-08-18T16:33:20.349 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-08-18T16:33:20.358 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-08-18T16:33:20.408 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-08-18T16:33:20.420 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-08-18T16:33:20.429 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-08-18T16:33:20.436 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-08-18T16:33:20.448 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-08-18T16:33:20.458 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-78 zones={test-zone2, test-zone1} 2024-08-18T16:33:20.458 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-78 zones={test-zone3, test-zone1} 2024-08-18T16:33:20.465 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:33:20.468 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-08-18T16:33:20.468 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2024-08-18T16:33:20.468 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2024-08-18T16:33:20.468 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2024-08-18T16:33:20.469 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-08-18T16:33:20.469 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-08-18T16:33:20.469 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-08-18T16:33:20.469 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2024-08-18T16:33:20.469 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2024-08-18T16:33:20.469 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-08-18T16:33:20.469 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-08-18T16:33:20.469 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-08-18T16:33:20.469 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-08-18T16:33:20.469 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2024-08-18T16:33:20.469 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-08-18T16:33:20.469 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-08-18T16:33:20.469 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-08-18T16:33:20.469 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2024-08-18T16:33:20.470 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-08-18T16:33:20.470 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2024-08-18T16:33:20.470 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-08-18T16:33:20.470 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-08-18T16:33:20.470 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2024-08-18T16:33:20.470 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2024-08-18T16:33:20.470 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-08-18T16:33:20.470 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:33:20.475 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-08-18T16:33:20.475 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2024-08-18T16:33:20.475 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2024-08-18T16:33:20.475 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2024-08-18T16:33:20.475 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-08-18T16:33:20.475 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-08-18T16:33:20.475 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-08-18T16:33:20.475 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2024-08-18T16:33:20.475 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2024-08-18T16:33:20.475 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-08-18T16:33:20.475 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-08-18T16:33:20.475 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-08-18T16:33:20.475 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-08-18T16:33:20.476 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2024-08-18T16:33:20.476 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-08-18T16:33:20.476 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-08-18T16:33:20.476 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-08-18T16:33:20.476 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2024-08-18T16:33:20.476 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-08-18T16:33:20.476 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2024-08-18T16:33:20.476 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-08-18T16:33:20.476 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-08-18T16:33:20.476 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2024-08-18T16:33:20.476 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2024-08-18T16:33:20.476 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-08-18T16:33:20.483 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T16:33:20.531 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T16:33:20.584 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T16:33:20.634 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-08-18T16:33:20.640 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-08-18T16:33:20.693 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-08-18T16:33:20.701 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-08-18T16:33:20.750 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-08-18T16:33:20.761 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-08-18T16:33:20.814 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-08-18T16:33:20.820 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-08-18T16:33:20.872 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-08-18T16:33:20.881 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-08-18T16:33:20.887 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-08-18T16:33:20.941 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-08-18T16:33:20.949 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-08-18T16:33:20.956 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-08-18T16:33:21.008 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-08-18T16:33:21.017 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T16:33:21.022 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T16:33:21.034 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T16:33:21.084 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-08-18T16:33:21.090 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-08-18T16:33:21.143 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-08-18T16:33:21.234 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-08-18T16:33:21.240 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-08-18T16:33:21.298 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-08-18T16:33:21.349 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T16:33:21.400 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T16:33:21.501 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T16:33:21.555 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-08-18T16:33:21.604 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-08-18T16:33:21.657 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-08-18T16:33:21.710 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-08-18T16:33:21.717 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-08-18T16:33:21.769 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-08-18T16:33:21.777 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T16:33:21.828 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T16:33:21.880 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T16:33:21.930 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-08-18T16:33:21.936 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-08-18T16:33:21.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-08-18T16:33:22.003 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-08-18T16:33:22.051 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-08-18T16:33:22.109 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-08-18T16:33:22.162 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T16:33:22.211 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T16:33:22.265 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T16:33:22.362 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-08-18T16:33:22.411 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-08-18T16:33:22.466 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-08-18T16:33:22.475 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-08-18T16:33:22.482 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-08-18T16:33:22.495 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-08-18T16:33:22.549 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-08-18T16:33:22.600 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-08-18T16:33:22.653 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-08-18T16:33:22.701 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-08-18T16:33:22.752 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-08-18T16:33:22.849 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-08-18T16:33:22.902 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-08-18T16:33:22.908 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-08-18T16:33:22.961 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-08-18T16:33:23.014 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-08-18T16:33:23.020 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-08-18T16:33:23.031 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-08-18T16:33:23.080 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-08-18T16:33:23.086 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-08-18T16:33:23.139 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-08-18T16:33:23.192 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-08-18T16:33:23.199 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-08-18T16:33:23.251 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-08-18T16:33:23.304 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-08-18T16:33:23.310 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-08-18T16:33:23.321 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-08-18T16:33:23.378 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-78 zones={test-zone3, test-zone1} 2024-08-18T16:33:23.378 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-78 zones={test-zone3, test-zone2} 2024-08-18T16:33:23.382 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:33:23.386 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-08-18T16:33:23.386 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2024-08-18T16:33:23.386 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2024-08-18T16:33:23.386 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2024-08-18T16:33:23.386 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-08-18T16:33:23.386 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-08-18T16:33:23.386 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-08-18T16:33:23.386 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2024-08-18T16:33:23.386 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2024-08-18T16:33:23.386 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-08-18T16:33:23.386 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-08-18T16:33:23.386 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-08-18T16:33:23.387 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-08-18T16:33:23.387 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2024-08-18T16:33:23.387 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-08-18T16:33:23.387 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-08-18T16:33:23.387 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-08-18T16:33:23.387 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2024-08-18T16:33:23.387 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-08-18T16:33:23.387 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2024-08-18T16:33:23.387 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-08-18T16:33:23.387 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-08-18T16:33:23.387 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2024-08-18T16:33:23.387 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2024-08-18T16:33:23.387 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-08-18T16:33:23.387 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:33:23.392 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-08-18T16:33:23.392 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2024-08-18T16:33:23.392 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2024-08-18T16:33:23.392 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2024-08-18T16:33:23.392 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-08-18T16:33:23.392 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-08-18T16:33:23.392 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-08-18T16:33:23.392 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2024-08-18T16:33:23.392 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2024-08-18T16:33:23.392 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-08-18T16:33:23.392 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-08-18T16:33:23.392 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-08-18T16:33:23.392 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-08-18T16:33:23.392 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2024-08-18T16:33:23.392 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-08-18T16:33:23.393 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-08-18T16:33:23.393 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-08-18T16:33:23.393 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2024-08-18T16:33:23.393 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-08-18T16:33:23.393 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2024-08-18T16:33:23.393 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-08-18T16:33:23.393 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-08-18T16:33:23.393 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2024-08-18T16:33:23.393 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2024-08-18T16:33:23.393 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-08-18T16:33:23.400 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T16:33:23.494 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T16:33:23.549 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T16:33:23.650 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-08-18T16:33:23.659 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-08-18T16:33:23.713 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-08-18T16:33:23.766 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-08-18T16:33:23.820 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-08-18T16:33:23.877 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-08-18T16:33:23.929 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-08-18T16:33:23.937 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-08-18T16:33:23.997 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-08-18T16:33:24.049 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-08-18T16:33:24.104 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-08-18T16:33:24.205 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-08-18T16:33:24.310 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-08-18T16:33:24.319 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-08-18T16:33:24.376 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-08-18T16:33:24.386 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T16:33:24.436 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T16:33:24.489 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T16:33:24.541 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-08-18T16:33:24.592 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-08-18T16:33:24.645 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-08-18T16:33:24.698 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-08-18T16:33:24.704 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-08-18T16:33:24.811 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-08-18T16:33:24.871 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T16:33:24.923 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T16:33:25.021 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T16:33:25.118 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-08-18T16:33:25.169 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-08-18T16:33:25.221 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-08-18T16:33:25.321 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-08-18T16:33:25.328 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-08-18T16:33:25.386 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-08-18T16:33:25.397 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T16:33:25.448 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T16:33:25.501 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T16:33:25.553 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-08-18T16:33:25.561 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-08-18T16:33:25.617 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-08-18T16:33:25.669 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-08-18T16:33:25.724 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-08-18T16:33:25.829 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-08-18T16:33:25.933 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T16:33:25.989 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T16:33:26.050 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T16:33:26.153 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-08-18T16:33:26.162 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-08-18T16:33:26.174 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-08-18T16:33:26.183 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-08-18T16:33:26.191 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-08-18T16:33:26.247 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-08-18T16:33:26.301 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-08-18T16:33:26.352 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-08-18T16:33:26.409 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-08-18T16:33:26.506 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-08-18T16:33:26.556 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-08-18T16:33:26.653 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-08-18T16:33:26.749 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-08-18T16:33:26.757 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-08-18T16:33:26.813 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-08-18T16:33:26.866 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-08-18T16:33:26.874 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-08-18T16:33:26.887 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-08-18T16:33:26.936 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-08-18T16:33:26.944 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-08-18T16:33:26.998 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-08-18T16:33:27.007 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-08-18T16:33:27.060 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-08-18T16:33:27.072 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-08-18T16:33:27.080 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-08-18T16:33:27.134 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-08-18T16:33:27.189 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-08-18T16:33:27.242 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-78 zones={test-zone3, test-zone2} 2024-08-18T16:33:27.242 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:33:27.242 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:33:27.242 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-78 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:33:27.298 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:33:27.299 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:33:27.350 INFO:teuthology.orchestra.run.smithi134.stdout:tenant: 2024-08-18T16:33:27.350 INFO:teuthology.orchestra.run.smithi134.stdout:bucket name: kibgmb-78 2024-08-18T16:33:27.360 INFO:teuthology.orchestra.run.smithi134.stdout:total entries: 25 2024-08-18T16:33:27.543 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:33:27.543 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:33:27.543 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-78 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T16:33:27.605 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:33:27.605 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:33:27.644 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:33:27.644+0000 7f7a83d29880 1 waiting to reach latest gen 5, on 4.. 2024-08-18T16:33:32.645 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:33:32.644+0000 7f7a83d29880 1 waiting to reach latest gen 5, on 4.. 2024-08-18T16:33:37.646 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:33:37.645+0000 7f7a83d29880 1 waiting to reach latest gen 5, on 4.. 2024-08-18T16:33:42.647 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:33:42.646+0000 7f7a83d29880 1 waiting to reach latest gen 5, on 4.. 2024-08-18T16:33:47.650 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:33:47.649+0000 7f7a83d29880 1 waiting for incremental sync to catch up: 2024-08-18T16:33:47.650 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2024-08-18T16:33:47.650 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , 00000000002.136.6, , 00000000001.186.6, , 00000000001.134.6, , , , , , , , , , 00000000001.117.6, , , , , , , , , , , , , , 00000000001.108.6] 2024-08-18T16:33:52.652 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:33:52.651+0000 7f7a83d29880 1 waiting for incremental sync to catch up: 2024-08-18T16:33:52.652 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2024-08-18T16:33:52.652 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , 00000000002.136.6, , 00000000001.186.6, , 00000000001.134.6, , , , , , , , , , 00000000001.117.6, , , , , , , , , , , , , , 00000000001.108.6] 2024-08-18T16:33:57.654 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:33:57.653+0000 7f7a83d29880 1 waiting for incremental sync to catch up: 2024-08-18T16:33:57.654 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2024-08-18T16:33:57.654 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , 00000000002.136.6, , 00000000001.186.6, , 00000000001.134.6, , , , , , , , , , 00000000001.117.6, , , , , , , , , , , , , , 00000000001.108.6] 2024-08-18T16:34:02.656 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:34:02.655+0000 7f7a83d29880 1 waiting for incremental sync to catch up: 2024-08-18T16:34:02.656 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2024-08-18T16:34:02.656 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , 00000000002.136.6, , 00000000001.186.6, , 00000000001.134.6, , , , , , , , , , 00000000001.117.6, , , , , , , , , , , , , , 00000000001.108.6] 2024-08-18T16:34:07.658 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:34:07.657+0000 7f7a83d29880 1 waiting for incremental sync to catch up: 2024-08-18T16:34:07.659 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2024-08-18T16:34:07.659 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , 00000000002.136.6, , 00000000001.186.6, , 00000000001.134.6, , , , , , , , , , 00000000001.117.6, , , , , , , , , , , , , , 00000000001.108.6] 2024-08-18T16:34:12.661 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:34:12.660+0000 7f7a83d29880 1 waiting for incremental sync to catch up: 2024-08-18T16:34:12.661 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2024-08-18T16:34:12.661 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , 00000000002.136.6, , 00000000001.186.6, , 00000000001.134.6, , , , , , , , , , 00000000001.117.6, , , , , , , , , , , , , , 00000000001.108.6] 2024-08-18T16:34:17.663 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:34:17.662+0000 7f7a83d29880 1 waiting for incremental sync to catch up: 2024-08-18T16:34:17.663 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , 00000000002.136.6, , , , 00000000001.134.6, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.108.6] 2024-08-18T16:34:17.663 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , 00000000002.136.6, , 00000000001.186.6, , 00000000001.134.6, , , , , , , , , , 00000000001.117.6, , , , , , , , , , , , , , 00000000001.108.6] 2024-08-18T16:34:22.666 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:34:22.665+0000 7f7a83d29880 1 waiting for incremental sync to catch up: 2024-08-18T16:34:22.666 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , 00000000002.136.6, , , , 00000000001.134.6, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.108.6] 2024-08-18T16:34:22.666 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , 00000000002.136.6, , 00000000001.186.6, , 00000000001.134.6, , , , , , , , , , 00000000001.117.6, , , , , , , , , , , , , , 00000000001.108.6] 2024-08-18T16:34:27.668 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:34:27.667+0000 7f7a83d29880 1 waiting for incremental sync to catch up: 2024-08-18T16:34:27.668 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , 00000000002.136.6, , , , 00000000001.134.6, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.108.6] 2024-08-18T16:34:27.668 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , 00000000002.136.6, , 00000000001.186.6, , 00000000001.134.6, , , , , , , , , , 00000000001.117.6, , , , , , , , , , , , , , 00000000001.108.6] 2024-08-18T16:34:32.670 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:34:32.669+0000 7f7a83d29880 1 waiting for incremental sync to catch up: 2024-08-18T16:34:32.670 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , 00000000002.136.6, , , , 00000000001.134.6, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.108.6] 2024-08-18T16:34:32.670 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , 00000000002.136.6, , 00000000001.186.6, , 00000000001.134.6, , , , , , , , , , 00000000001.117.6, , , , , , , , , , , , , , 00000000001.108.6] 2024-08-18T16:34:37.672 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:34:37.671+0000 7f7a83d29880 1 bucket sync caught up with source: 2024-08-18T16:34:37.673 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , 00000000002.136.6, , 00000000001.186.6, , 00000000001.134.6, , , , , , , , , , 00000000001.117.6, , , , , , , , , , , , , , 00000000001.108.6] 2024-08-18T16:34:37.673 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , 00000000002.136.6, , 00000000001.186.6, , 00000000001.134.6, , , , , , , , , , 00000000001.117.6, , , , , , , , , , , , , , 00000000001.108.6] 2024-08-18T16:34:37.673 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:34:37.671+0000 7f7a83d29880 0 bucket checkpoint complete 2024-08-18T16:34:37.692 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:34:37.692 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:34:37.692 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-78 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:34:37.754 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:34:37.754 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:34:37.794 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:34:37.793+0000 7f4ef8234880 1 waiting for incremental sync to catch up: 2024-08-18T16:34:37.794 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , 00000000002.136.6, , 00000000001.186.6, , 00000000001.134.6, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.108.6] 2024-08-18T16:34:37.795 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , 00000000002.136.6, , 00000000001.186.6, , 00000000001.134.6, , , , , , , , , , 00000000001.117.6, , , , , , , , , , , , , , 00000000001.108.6] 2024-08-18T16:34:42.796 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:34:42.796+0000 7f4ef8234880 1 waiting for incremental sync to catch up: 2024-08-18T16:34:42.797 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , 00000000002.136.6, , 00000000001.186.6, , 00000000001.134.6, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.108.6] 2024-08-18T16:34:42.797 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , 00000000002.136.6, , 00000000001.186.6, , 00000000001.134.6, , , , , , , , , , 00000000001.117.6, , , , , , , , , , , , , , 00000000001.108.6] 2024-08-18T16:34:47.799 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:34:47.798+0000 7f4ef8234880 1 waiting for incremental sync to catch up: 2024-08-18T16:34:47.799 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , 00000000002.136.6, , 00000000001.186.6, , 00000000001.134.6, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.108.6] 2024-08-18T16:34:47.799 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , 00000000002.136.6, , 00000000001.186.6, , 00000000001.134.6, , , , , , , , , , 00000000001.117.6, , , , , , , , , , , , , , 00000000001.108.6] 2024-08-18T16:34:52.801 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:34:52.800+0000 7f4ef8234880 1 waiting for incremental sync to catch up: 2024-08-18T16:34:52.801 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , 00000000002.136.6, , 00000000001.186.6, , 00000000001.134.6, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.108.6] 2024-08-18T16:34:52.801 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , 00000000002.136.6, , 00000000001.186.6, , 00000000001.134.6, , , , , , , , , , 00000000001.117.6, , , , , , , , , , , , , , 00000000001.108.6] 2024-08-18T16:34:57.804 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:34:57.803+0000 7f4ef8234880 1 waiting for incremental sync to catch up: 2024-08-18T16:34:57.804 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , 00000000002.136.6, , 00000000001.186.6, , 00000000001.134.6, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.108.6] 2024-08-18T16:34:57.804 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , 00000000002.136.6, , 00000000001.186.6, , 00000000001.134.6, , , , , , , , , , 00000000001.117.6, , , , , , , , , , , , , , 00000000001.108.6] 2024-08-18T16:35:02.806 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:35:02.805+0000 7f4ef8234880 1 waiting for incremental sync to catch up: 2024-08-18T16:35:02.806 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , 00000000002.136.6, , 00000000001.186.6, , 00000000001.134.6, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.108.6] 2024-08-18T16:35:02.807 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , 00000000002.136.6, , 00000000001.186.6, , 00000000001.134.6, , , , , , , , , , 00000000001.117.6, , , , , , , , , , , , , , 00000000001.108.6] 2024-08-18T16:35:07.808 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:35:07.806+0000 7f4ef8234880 1 waiting for incremental sync to catch up: 2024-08-18T16:35:07.809 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , 00000000002.136.6, , 00000000001.186.6, , 00000000001.134.6, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.108.6] 2024-08-18T16:35:07.809 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , 00000000002.136.6, , 00000000001.186.6, , 00000000001.134.6, , , , , , , , , , 00000000001.117.6, , , , , , , , , , , , , , 00000000001.108.6] 2024-08-18T16:35:12.811 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:35:12.810+0000 7f4ef8234880 1 waiting for incremental sync to catch up: 2024-08-18T16:35:12.811 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , 00000000002.136.6, , 00000000001.186.6, , 00000000001.134.6, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.108.6] 2024-08-18T16:35:12.811 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , 00000000002.136.6, , 00000000001.186.6, , 00000000001.134.6, , , , , , , , , , 00000000001.117.6, , , , , , , , , , , , , , 00000000001.108.6] 2024-08-18T16:35:17.813 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:35:17.812+0000 7f4ef8234880 1 waiting for incremental sync to catch up: 2024-08-18T16:35:17.814 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , 00000000002.136.6, , 00000000001.186.6, , 00000000001.134.6, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.108.6] 2024-08-18T16:35:17.814 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , 00000000002.136.6, , 00000000001.186.6, , 00000000001.134.6, , , , , , , , , , 00000000001.117.6, , , , , , , , , , , , , , 00000000001.108.6] 2024-08-18T16:35:22.816 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:35:22.815+0000 7f4ef8234880 1 waiting for incremental sync to catch up: 2024-08-18T16:35:22.816 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , 00000000002.136.6, , 00000000001.186.6, , 00000000001.134.6, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.108.6] 2024-08-18T16:35:22.816 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , 00000000002.136.6, , 00000000001.186.6, , 00000000001.134.6, , , , , , , , , , 00000000001.117.6, , , , , , , , , , , , , , 00000000001.108.6] 2024-08-18T16:35:27.818 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:35:27.817+0000 7f4ef8234880 1 waiting for incremental sync to catch up: 2024-08-18T16:35:27.818 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , 00000000002.136.6, , 00000000001.186.6, , 00000000001.134.6, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.108.6] 2024-08-18T16:35:27.819 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , 00000000002.136.6, , 00000000001.186.6, , 00000000001.134.6, , , , , , , , , , 00000000001.117.6, , , , , , , , , , , , , , 00000000001.108.6] 2024-08-18T16:35:32.821 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:35:32.820+0000 7f4ef8234880 1 waiting for incremental sync to catch up: 2024-08-18T16:35:32.821 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , 00000000002.136.6, , 00000000001.186.6, , 00000000001.134.6, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.108.6] 2024-08-18T16:35:32.821 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , 00000000002.136.6, , 00000000001.186.6, , 00000000001.134.6, , , , , , , , , , 00000000001.117.6, , , , , , , , , , , , , , 00000000001.108.6] 2024-08-18T16:35:37.823 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:35:37.822+0000 7f4ef8234880 1 waiting for incremental sync to catch up: 2024-08-18T16:35:37.823 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , 00000000002.136.6, , 00000000001.186.6, , 00000000001.134.6, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.108.6] 2024-08-18T16:35:37.823 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , 00000000002.136.6, , 00000000001.186.6, , 00000000001.134.6, , , , , , , , , , 00000000001.117.6, , , , , , , , , , , , , , 00000000001.108.6] 2024-08-18T16:35:42.825 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:35:42.825+0000 7f4ef8234880 1 waiting for incremental sync to catch up: 2024-08-18T16:35:42.826 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , 00000000002.136.6, , 00000000001.186.6, , 00000000001.134.6, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.108.6] 2024-08-18T16:35:42.826 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , 00000000002.136.6, , 00000000001.186.6, , 00000000001.134.6, , , , , , , , , , 00000000001.117.6, , , , , , , , , , , , , , 00000000001.108.6] 2024-08-18T16:35:47.828 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:35:47.827+0000 7f4ef8234880 1 waiting for incremental sync to catch up: 2024-08-18T16:35:47.828 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , 00000000002.136.6, , 00000000001.186.6, , 00000000001.134.6, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.108.6] 2024-08-18T16:35:47.828 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , 00000000002.136.6, , 00000000001.186.6, , 00000000001.134.6, , , , , , , , , , 00000000001.117.6, , , , , , , , , , , , , , 00000000001.108.6] 2024-08-18T16:35:52.830 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:35:52.829+0000 7f4ef8234880 1 bucket sync caught up with source: 2024-08-18T16:35:52.831 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , 00000000002.136.6, , 00000000001.186.6, , 00000000001.134.6, , , , , , , , , , 00000000001.117.6, , , , , , , , , , , , , , 00000000001.108.6] 2024-08-18T16:35:52.831 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , 00000000002.136.6, , 00000000001.186.6, , 00000000001.134.6, , , , , , , , , , 00000000001.117.6, , , , , , , , , , , , , , 00000000001.108.6] 2024-08-18T16:35:52.831 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:35:52.829+0000 7f4ef8234880 0 bucket checkpoint complete 2024-08-18T16:35:52.849 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:35:52.849 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:35:52.850 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-78 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:35:52.909 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:35:52.909 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:35:52.948 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:35:52.947+0000 7f75e3f32880 1 bucket sync caught up with source: 2024-08-18T16:35:52.948 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000003.343.6, 00000000002.238.6, 00000000002.370.6, 00000000004.224.6, 00000000004.197.6, 00000000001.404.6, 00000000001.244.6, 00000000003.281.6, 00000000006.342.6, 00000000004.267.6] 2024-08-18T16:35:52.948 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000003.343.6, 00000000002.238.6, 00000000002.370.6, 00000000004.224.6, 00000000004.197.6, 00000000001.404.6, 00000000001.244.6, 00000000003.281.6, 00000000006.342.6, 00000000004.267.6] 2024-08-18T16:35:52.948 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:35:52.947+0000 7f75e3f32880 0 bucket checkpoint complete 2024-08-18T16:35:52.963 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:35:52.963 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:35:52.963 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-78 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:35:53.017 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:35:53.017 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:35:53.052 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:35:53.051+0000 7f415e619880 1 bucket sync caught up with source: 2024-08-18T16:35:53.052 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000003.343.6, 00000000002.238.6, 00000000002.370.6, 00000000004.224.6, 00000000004.197.6, 00000000001.404.6, 00000000001.244.6, 00000000003.281.6, 00000000006.342.6, 00000000004.267.6] 2024-08-18T16:35:53.052 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000003.343.6, 00000000002.238.6, 00000000002.370.6, 00000000004.224.6, 00000000004.197.6, 00000000001.404.6, 00000000001.244.6, 00000000003.281.6, 00000000006.342.6, 00000000004.267.6] 2024-08-18T16:35:53.052 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:35:53.051+0000 7f415e619880 0 bucket checkpoint complete 2024-08-18T16:35:53.066 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:35:53.066 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:35:53.066 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-78 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:35:53.119 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:35:53.119 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:35:53.154 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:35:53.153+0000 7f1397698880 1 bucket sync caught up with source: 2024-08-18T16:35:53.154 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000003.221.6, 00000000002.490.6, 00000000002.256.6, 00000000004.286.6, 00000000004.146.6, 00000000001.526.6, 00000000001.496.6, 00000000003.189.6, 00000000006.220.6, 00000000004.145.6] 2024-08-18T16:35:53.154 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000003.221.6, 00000000002.490.6, 00000000002.256.6, 00000000004.286.6, 00000000004.146.6, 00000000001.526.6, 00000000001.496.6, 00000000003.189.6, 00000000006.220.6, 00000000004.145.6] 2024-08-18T16:35:53.154 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:35:53.153+0000 7f1397698880 0 bucket checkpoint complete 2024-08-18T16:35:53.170 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:35:53.170 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:35:53.170 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-78 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T16:35:53.232 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:35:53.232 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:35:53.274 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:35:53.273+0000 7ff345c11880 1 bucket sync caught up with source: 2024-08-18T16:35:53.274 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, 00000000003.221.6, 00000000002.490.6, 00000000002.256.6, 00000000004.286.6, 00000000004.146.6, 00000000001.526.6, 00000000001.496.6, 00000000003.189.6, 00000000006.220.6, 00000000004.145.6] 2024-08-18T16:35:53.274 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, 00000000003.221.6, 00000000002.490.6, 00000000002.256.6, 00000000004.286.6, 00000000004.146.6, 00000000001.526.6, 00000000001.496.6, 00000000003.189.6, 00000000006.220.6, 00000000004.145.6] 2024-08-18T16:35:53.274 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:35:53.273+0000 7ff345c11880 0 bucket checkpoint complete 2024-08-18T16:35:53.290 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-78 zones={test-zone2, test-zone1} 2024-08-18T16:35:53.300 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:35:53.305 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-08-18T16:35:53.305 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2024-08-18T16:35:53.305 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2024-08-18T16:35:53.305 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2024-08-18T16:35:53.305 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2024-08-18T16:35:53.305 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2024-08-18T16:35:53.305 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-08-18T16:35:53.305 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-08-18T16:35:53.305 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-08-18T16:35:53.305 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2024-08-18T16:35:53.305 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2024-08-18T16:35:53.305 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2024-08-18T16:35:53.305 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-08-18T16:35:53.306 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-08-18T16:35:53.306 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-08-18T16:35:53.306 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-08-18T16:35:53.306 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2024-08-18T16:35:53.306 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-08-18T16:35:53.306 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-08-18T16:35:53.306 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-08-18T16:35:53.306 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2024-08-18T16:35:53.306 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2024-08-18T16:35:53.306 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-08-18T16:35:53.306 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2024-08-18T16:35:53.306 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-08-18T16:35:53.307 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-08-18T16:35:53.307 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2024-08-18T16:35:53.307 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2024-08-18T16:35:53.307 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2024-08-18T16:35:53.307 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-08-18T16:35:53.307 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:35:53.311 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-08-18T16:35:53.311 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2024-08-18T16:35:53.311 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2024-08-18T16:35:53.311 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2024-08-18T16:35:53.311 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2024-08-18T16:35:53.312 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2024-08-18T16:35:53.312 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-08-18T16:35:53.312 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-08-18T16:35:53.312 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-08-18T16:35:53.312 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2024-08-18T16:35:53.312 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2024-08-18T16:35:53.312 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2024-08-18T16:35:53.312 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-08-18T16:35:53.312 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-08-18T16:35:53.312 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-08-18T16:35:53.312 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-08-18T16:35:53.312 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2024-08-18T16:35:53.312 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-08-18T16:35:53.313 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-08-18T16:35:53.313 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-08-18T16:35:53.313 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2024-08-18T16:35:53.313 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2024-08-18T16:35:53.313 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-08-18T16:35:53.313 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2024-08-18T16:35:53.313 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-08-18T16:35:53.313 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-08-18T16:35:53.313 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2024-08-18T16:35:53.313 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2024-08-18T16:35:53.313 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2024-08-18T16:35:53.313 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-08-18T16:35:53.321 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T16:35:53.371 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T16:35:53.384 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T16:35:53.437 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-08-18T16:35:53.443 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-08-18T16:35:53.496 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-08-18T16:35:53.550 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-08-18T16:35:53.557 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-08-18T16:35:53.570 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-08-18T16:35:53.622 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2024-08-18T16:35:53.629 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2024-08-18T16:35:53.683 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2024-08-18T16:35:53.696 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-08-18T16:35:53.768 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-08-18T16:35:53.822 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-08-18T16:35:53.879 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2024-08-18T16:35:53.928 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2024-08-18T16:35:53.981 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2024-08-18T16:35:54.033 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-08-18T16:35:54.040 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-08-18T16:35:54.097 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-08-18T16:35:54.150 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-08-18T16:35:54.157 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-08-18T16:35:54.209 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-08-18T16:35:54.266 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T16:35:54.273 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T16:35:54.285 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T16:35:54.343 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-08-18T16:35:54.394 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-08-18T16:35:54.406 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-08-18T16:35:54.459 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-08-18T16:35:54.514 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-08-18T16:35:54.527 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-08-18T16:35:54.576 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2024-08-18T16:35:54.583 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2024-08-18T16:35:54.636 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2024-08-18T16:35:54.686 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T16:35:54.692 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T16:35:54.745 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T16:35:54.798 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-08-18T16:35:54.805 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-08-18T16:35:54.857 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-08-18T16:35:54.866 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-08-18T16:35:54.916 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-08-18T16:35:54.970 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-08-18T16:35:54.979 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T16:35:54.985 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T16:35:55.040 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T16:35:55.091 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-08-18T16:35:55.097 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-08-18T16:35:55.154 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-08-18T16:35:55.162 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-08-18T16:35:55.210 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-08-18T16:35:55.223 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-08-18T16:35:55.274 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T16:35:55.280 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T16:35:55.337 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T16:35:55.439 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-08-18T16:35:55.446 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-08-18T16:35:55.457 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-08-18T16:35:55.510 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-08-18T16:35:55.519 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-08-18T16:35:55.579 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-08-18T16:35:55.635 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2024-08-18T16:35:55.642 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2024-08-18T16:35:55.654 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2024-08-18T16:35:55.710 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-08-18T16:35:55.717 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-08-18T16:35:55.729 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-08-18T16:35:55.782 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-08-18T16:35:55.790 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-08-18T16:35:55.802 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-08-18T16:35:55.811 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-08-18T16:35:55.859 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-08-18T16:35:55.914 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-08-18T16:35:55.923 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-08-18T16:35:55.931 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-08-18T16:35:55.988 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-08-18T16:35:55.997 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-08-18T16:35:56.004 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-08-18T16:35:56.057 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-08-18T16:35:56.065 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-08-18T16:35:56.072 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-08-18T16:35:56.124 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-08-18T16:35:56.134 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2024-08-18T16:35:56.140 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2024-08-18T16:35:56.193 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2024-08-18T16:35:56.242 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-08-18T16:35:56.249 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-08-18T16:35:56.261 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-08-18T16:35:56.314 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-78 zones={test-zone2, test-zone1} 2024-08-18T16:35:56.314 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-78 zones={test-zone3, test-zone1} 2024-08-18T16:35:56.321 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:35:56.324 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-08-18T16:35:56.325 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2024-08-18T16:35:56.325 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2024-08-18T16:35:56.325 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2024-08-18T16:35:56.325 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2024-08-18T16:35:56.325 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2024-08-18T16:35:56.325 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-08-18T16:35:56.325 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-08-18T16:35:56.325 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-08-18T16:35:56.325 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2024-08-18T16:35:56.325 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2024-08-18T16:35:56.325 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2024-08-18T16:35:56.325 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-08-18T16:35:56.325 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-08-18T16:35:56.326 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-08-18T16:35:56.326 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-08-18T16:35:56.326 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2024-08-18T16:35:56.326 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-08-18T16:35:56.326 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-08-18T16:35:56.326 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-08-18T16:35:56.326 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2024-08-18T16:35:56.326 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2024-08-18T16:35:56.326 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-08-18T16:35:56.326 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2024-08-18T16:35:56.326 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-08-18T16:35:56.326 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-08-18T16:35:56.326 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2024-08-18T16:35:56.326 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2024-08-18T16:35:56.327 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2024-08-18T16:35:56.327 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-08-18T16:35:56.327 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:35:56.331 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-08-18T16:35:56.331 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2024-08-18T16:35:56.331 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2024-08-18T16:35:56.331 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2024-08-18T16:35:56.332 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2024-08-18T16:35:56.332 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2024-08-18T16:35:56.332 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-08-18T16:35:56.332 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-08-18T16:35:56.332 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-08-18T16:35:56.332 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2024-08-18T16:35:56.332 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2024-08-18T16:35:56.332 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2024-08-18T16:35:56.332 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-08-18T16:35:56.332 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-08-18T16:35:56.332 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-08-18T16:35:56.332 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-08-18T16:35:56.332 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2024-08-18T16:35:56.332 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-08-18T16:35:56.332 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-08-18T16:35:56.333 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-08-18T16:35:56.333 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2024-08-18T16:35:56.333 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2024-08-18T16:35:56.333 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-08-18T16:35:56.333 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2024-08-18T16:35:56.333 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-08-18T16:35:56.333 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-08-18T16:35:56.333 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2024-08-18T16:35:56.333 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2024-08-18T16:35:56.333 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2024-08-18T16:35:56.333 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-08-18T16:35:56.342 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T16:35:56.394 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T16:35:56.497 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T16:35:56.506 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-08-18T16:35:56.560 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-08-18T16:35:56.571 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-08-18T16:35:56.624 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-08-18T16:35:56.631 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-08-18T16:35:56.643 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-08-18T16:35:56.693 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2024-08-18T16:35:56.699 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2024-08-18T16:35:56.754 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2024-08-18T16:35:56.764 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-08-18T16:35:56.814 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-08-18T16:35:56.825 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-08-18T16:35:56.878 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2024-08-18T16:35:56.885 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2024-08-18T16:35:56.896 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2024-08-18T16:35:56.946 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-08-18T16:35:56.952 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-08-18T16:35:57.010 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-08-18T16:35:57.018 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-08-18T16:35:57.067 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-08-18T16:35:57.079 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-08-18T16:35:57.128 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T16:35:57.134 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T16:35:57.192 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T16:35:57.200 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-08-18T16:35:57.208 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-08-18T16:35:57.263 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-08-18T16:35:57.271 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-08-18T16:35:57.278 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-08-18T16:35:57.331 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-08-18T16:35:57.384 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2024-08-18T16:35:57.391 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2024-08-18T16:35:57.444 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2024-08-18T16:35:57.494 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T16:35:57.501 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T16:35:57.553 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T16:35:57.606 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-08-18T16:35:57.655 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-08-18T16:35:57.709 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-08-18T16:35:57.761 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-08-18T16:35:57.767 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-08-18T16:35:57.822 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-08-18T16:35:57.830 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T16:35:57.880 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T16:35:57.932 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T16:35:57.983 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-08-18T16:35:57.989 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-08-18T16:35:58.045 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-08-18T16:35:58.054 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-08-18T16:35:58.103 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-08-18T16:35:58.156 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-08-18T16:35:58.251 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T16:35:58.304 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T16:35:58.409 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T16:35:58.462 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-08-18T16:35:58.511 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-08-18T16:35:58.523 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-08-18T16:35:58.575 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-08-18T16:35:58.582 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-08-18T16:35:58.593 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-08-18T16:35:58.646 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2024-08-18T16:35:58.653 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2024-08-18T16:35:58.711 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2024-08-18T16:35:58.763 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-08-18T16:35:58.811 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-08-18T16:35:58.867 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-08-18T16:35:58.963 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-08-18T16:35:59.016 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-08-18T16:35:59.109 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-08-18T16:35:59.162 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-08-18T16:35:59.212 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-08-18T16:35:59.223 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-08-18T16:35:59.276 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-08-18T16:35:59.283 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-08-18T16:35:59.335 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-08-18T16:35:59.388 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-08-18T16:35:59.395 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-08-18T16:35:59.452 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-08-18T16:35:59.507 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-08-18T16:35:59.513 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-08-18T16:35:59.525 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-08-18T16:35:59.582 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2024-08-18T16:35:59.589 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2024-08-18T16:35:59.646 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2024-08-18T16:35:59.697 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-08-18T16:35:59.703 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-08-18T16:35:59.759 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-08-18T16:35:59.808 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-78 zones={test-zone3, test-zone1} 2024-08-18T16:35:59.809 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-78 zones={test-zone3, test-zone2} 2024-08-18T16:35:59.813 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:35:59.817 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-08-18T16:35:59.817 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2024-08-18T16:35:59.817 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2024-08-18T16:35:59.817 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2024-08-18T16:35:59.817 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2024-08-18T16:35:59.817 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2024-08-18T16:35:59.817 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-08-18T16:35:59.817 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-08-18T16:35:59.817 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-08-18T16:35:59.817 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2024-08-18T16:35:59.817 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2024-08-18T16:35:59.817 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2024-08-18T16:35:59.817 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-08-18T16:35:59.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-08-18T16:35:59.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-08-18T16:35:59.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-08-18T16:35:59.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2024-08-18T16:35:59.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-08-18T16:35:59.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-08-18T16:35:59.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-08-18T16:35:59.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2024-08-18T16:35:59.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2024-08-18T16:35:59.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-08-18T16:35:59.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2024-08-18T16:35:59.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-08-18T16:35:59.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-08-18T16:35:59.819 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2024-08-18T16:35:59.819 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2024-08-18T16:35:59.819 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2024-08-18T16:35:59.819 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-08-18T16:35:59.819 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:35:59.823 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-08-18T16:35:59.823 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2024-08-18T16:35:59.823 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2024-08-18T16:35:59.823 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2024-08-18T16:35:59.823 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2024-08-18T16:35:59.823 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2024-08-18T16:35:59.823 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-08-18T16:35:59.823 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-08-18T16:35:59.823 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-08-18T16:35:59.823 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2024-08-18T16:35:59.823 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2024-08-18T16:35:59.823 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2024-08-18T16:35:59.823 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-08-18T16:35:59.824 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-08-18T16:35:59.824 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-08-18T16:35:59.824 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-08-18T16:35:59.824 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2024-08-18T16:35:59.824 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-08-18T16:35:59.824 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-08-18T16:35:59.824 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-08-18T16:35:59.824 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2024-08-18T16:35:59.824 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2024-08-18T16:35:59.824 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-08-18T16:35:59.824 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2024-08-18T16:35:59.824 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-08-18T16:35:59.824 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-08-18T16:35:59.824 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2024-08-18T16:35:59.824 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2024-08-18T16:35:59.825 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2024-08-18T16:35:59.825 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-08-18T16:35:59.832 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T16:35:59.884 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T16:35:59.937 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T16:35:59.990 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-08-18T16:35:59.997 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-08-18T16:36:00.097 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-08-18T16:36:00.201 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-08-18T16:36:00.252 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-08-18T16:36:00.309 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-08-18T16:36:00.406 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2024-08-18T16:36:00.462 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2024-08-18T16:36:00.516 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2024-08-18T16:36:00.565 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-08-18T16:36:00.573 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-08-18T16:36:00.630 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-08-18T16:36:00.638 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2024-08-18T16:36:00.688 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2024-08-18T16:36:00.701 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2024-08-18T16:36:00.754 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-08-18T16:36:00.804 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-08-18T16:36:00.856 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-08-18T16:36:00.958 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-08-18T16:36:00.965 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-08-18T16:36:01.021 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-08-18T16:36:01.073 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T16:36:01.124 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T16:36:01.177 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T16:36:01.233 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-08-18T16:36:01.284 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-08-18T16:36:01.341 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-08-18T16:36:01.442 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-08-18T16:36:01.449 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-08-18T16:36:01.462 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-08-18T16:36:01.514 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2024-08-18T16:36:01.521 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2024-08-18T16:36:01.577 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2024-08-18T16:36:01.633 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T16:36:01.688 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T16:36:01.793 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T16:36:01.894 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-08-18T16:36:01.944 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-08-18T16:36:02.049 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-08-18T16:36:02.145 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-08-18T16:36:02.153 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-08-18T16:36:02.213 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-08-18T16:36:02.266 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T16:36:02.320 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T16:36:02.425 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T16:36:02.525 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-08-18T16:36:02.533 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-08-18T16:36:02.593 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-08-18T16:36:02.645 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-08-18T16:36:02.700 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-08-18T16:36:02.753 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-08-18T16:36:02.853 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T16:36:02.905 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T16:36:02.962 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T16:36:03.061 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-08-18T16:36:03.070 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-08-18T16:36:03.084 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-08-18T16:36:03.140 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-08-18T16:36:03.148 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-08-18T16:36:03.202 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-08-18T16:36:03.211 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2024-08-18T16:36:03.260 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2024-08-18T16:36:03.272 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2024-08-18T16:36:03.326 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-08-18T16:36:03.376 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-08-18T16:36:03.434 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-08-18T16:36:03.529 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-08-18T16:36:03.581 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-08-18T16:36:03.638 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-08-18T16:36:03.737 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-08-18T16:36:03.787 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-08-18T16:36:03.843 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-08-18T16:36:03.896 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-08-18T16:36:03.954 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-08-18T16:36:04.012 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-08-18T16:36:04.069 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-08-18T16:36:04.119 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-08-18T16:36:04.176 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-08-18T16:36:04.275 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-08-18T16:36:04.326 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-08-18T16:36:04.431 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-08-18T16:36:04.483 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2024-08-18T16:36:04.535 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2024-08-18T16:36:04.635 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2024-08-18T16:36:04.735 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-08-18T16:36:04.790 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-08-18T16:36:04.890 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-08-18T16:36:04.940 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-78 zones={test-zone3, test-zone2} 2024-08-18T16:36:04.940 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:36:04.940 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:36:04.940 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-78 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:36:04.999 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:36:04.999 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:36:05.053 INFO:teuthology.orchestra.run.smithi134.stdout:tenant: 2024-08-18T16:36:05.054 INFO:teuthology.orchestra.run.smithi134.stdout:bucket name: kibgmb-78 2024-08-18T16:36:05.066 INFO:teuthology.orchestra.run.smithi134.stdout:total entries: 30 2024-08-18T16:36:05.247 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:36:05.247 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:36:05.248 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-78 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T16:36:05.310 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:36:05.310 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:36:05.350 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:36:05.349+0000 7fce67542880 1 waiting to reach latest gen 6, on 5.. 2024-08-18T16:36:10.351 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:36:10.350+0000 7fce67542880 1 waiting to reach latest gen 6, on 5.. 2024-08-18T16:36:15.351 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:36:15.350+0000 7fce67542880 1 waiting to reach latest gen 6, on 5.. 2024-08-18T16:36:20.355 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:36:20.353+0000 7fce67542880 1 waiting for incremental sync to catch up: 2024-08-18T16:36:20.355 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , 00000000002.82.6, , , , , , 00000000002.191.6, , , , , , , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.120.6] 2024-08-18T16:36:20.355 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , 00000000002.82.6, , , , , , 00000000002.191.6, , , , , 00000000002.107.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.120.6] 2024-08-18T16:36:25.356 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:36:25.355+0000 7fce67542880 1 waiting for incremental sync to catch up: 2024-08-18T16:36:25.356 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , 00000000002.82.6, , , , , , 00000000002.191.6, , , , , , , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.120.6] 2024-08-18T16:36:25.357 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , 00000000002.82.6, , , , , , 00000000002.191.6, , , , , 00000000002.107.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.120.6] 2024-08-18T16:36:30.359 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:36:30.358+0000 7fce67542880 1 waiting for incremental sync to catch up: 2024-08-18T16:36:30.359 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , 00000000002.82.6, , , , , , 00000000002.191.6, , , , , , , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.120.6] 2024-08-18T16:36:30.359 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , 00000000002.82.6, , , , , , 00000000002.191.6, , , , , 00000000002.107.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.120.6] 2024-08-18T16:36:35.362 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:36:35.361+0000 7fce67542880 1 waiting for incremental sync to catch up: 2024-08-18T16:36:35.362 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , 00000000002.82.6, , , , , , 00000000002.191.6, , , , , , , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.120.6] 2024-08-18T16:36:35.362 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , 00000000002.82.6, , , , , , 00000000002.191.6, , , , , 00000000002.107.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.120.6] 2024-08-18T16:36:40.364 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:36:40.362+0000 7fce67542880 1 waiting for incremental sync to catch up: 2024-08-18T16:36:40.364 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , 00000000002.82.6, , , , , , 00000000002.191.6, , , , , , , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.120.6] 2024-08-18T16:36:40.365 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , 00000000002.82.6, , , , , , 00000000002.191.6, , , , , 00000000002.107.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.120.6] 2024-08-18T16:36:45.367 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:36:45.366+0000 7fce67542880 1 waiting for incremental sync to catch up: 2024-08-18T16:36:45.367 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , 00000000002.82.6, , , , , , 00000000002.191.6, , , , , , , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.120.6] 2024-08-18T16:36:45.367 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , 00000000002.82.6, , , , , , 00000000002.191.6, , , , , 00000000002.107.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.120.6] 2024-08-18T16:36:50.370 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:36:50.369+0000 7fce67542880 1 waiting for incremental sync to catch up: 2024-08-18T16:36:50.370 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , 00000000002.82.6, , , , , , 00000000002.191.6, , , , , , , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.120.6] 2024-08-18T16:36:50.370 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , 00000000002.82.6, , , , , , 00000000002.191.6, , , , , 00000000002.107.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.120.6] 2024-08-18T16:36:55.372 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:36:55.371+0000 7fce67542880 1 waiting for incremental sync to catch up: 2024-08-18T16:36:55.372 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , 00000000002.82.6, , , , , , 00000000002.191.6, , , , , , , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.120.6] 2024-08-18T16:36:55.373 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , 00000000002.82.6, , , , , , 00000000002.191.6, , , , , 00000000002.107.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.120.6] 2024-08-18T16:37:00.375 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:37:00.374+0000 7fce67542880 1 waiting for incremental sync to catch up: 2024-08-18T16:37:00.375 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , 00000000002.82.6, , , , , , 00000000002.191.6, , , , , , , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.120.6] 2024-08-18T16:37:00.375 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , 00000000002.82.6, , , , , , 00000000002.191.6, , , , , 00000000002.107.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.120.6] 2024-08-18T16:37:05.378 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:37:05.377+0000 7fce67542880 1 waiting for incremental sync to catch up: 2024-08-18T16:37:05.378 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , 00000000002.82.6, , , , , , 00000000002.191.6, , , , , , , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.120.6] 2024-08-18T16:37:05.378 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , 00000000002.82.6, , , , , , 00000000002.191.6, , , , , 00000000002.107.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.120.6] 2024-08-18T16:37:10.381 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:37:10.380+0000 7fce67542880 1 waiting for incremental sync to catch up: 2024-08-18T16:37:10.381 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , 00000000002.82.6, , , , , , 00000000002.191.6, , , , , , , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.120.6] 2024-08-18T16:37:10.381 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , 00000000002.82.6, , , , , , 00000000002.191.6, , , , , 00000000002.107.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.120.6] 2024-08-18T16:37:15.384 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:37:15.382+0000 7fce67542880 1 waiting for incremental sync to catch up: 2024-08-18T16:37:15.384 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , 00000000002.82.6, , , , , , 00000000002.191.6, , , , , , , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.120.6] 2024-08-18T16:37:15.384 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , 00000000002.82.6, , , , , , 00000000002.191.6, , , , , 00000000002.107.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.120.6] 2024-08-18T16:37:20.385 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:37:20.384+0000 7fce67542880 1 waiting for incremental sync to catch up: 2024-08-18T16:37:20.385 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , 00000000002.82.6, , , , , , 00000000002.191.6, , , , , , , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.120.6] 2024-08-18T16:37:20.385 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , 00000000002.82.6, , , , , , 00000000002.191.6, , , , , 00000000002.107.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.120.6] 2024-08-18T16:37:25.388 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:37:25.387+0000 7fce67542880 1 waiting for incremental sync to catch up: 2024-08-18T16:37:25.388 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , 00000000002.82.6, , , , , , 00000000002.191.6, , , , , , , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.120.6] 2024-08-18T16:37:25.388 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , 00000000002.82.6, , , , , , 00000000002.191.6, , , , , 00000000002.107.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.120.6] 2024-08-18T16:37:30.390 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:37:30.389+0000 7fce67542880 1 bucket sync caught up with source: 2024-08-18T16:37:30.391 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , 00000000002.82.6, , , , , , 00000000002.191.6, , , , , 00000000002.107.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.120.6] 2024-08-18T16:37:30.391 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , 00000000002.82.6, , , , , , 00000000002.191.6, , , , , 00000000002.107.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.120.6] 2024-08-18T16:37:30.391 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:37:30.389+0000 7fce67542880 0 bucket checkpoint complete 2024-08-18T16:37:30.409 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:37:30.410 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:37:30.410 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-78 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:37:30.472 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:37:30.472 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:37:30.512 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:37:30.512+0000 7f5102c35880 1 waiting for incremental sync to catch up: 2024-08-18T16:37:30.513 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , , , , , , , 00000000002.191.6, , , , , , , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.120.6] 2024-08-18T16:37:30.513 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , , 00000000002.82.6, , , , , , 00000000002.191.6, , , , , 00000000002.107.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.120.6] 2024-08-18T16:37:35.517 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:37:35.514+0000 7f5102c35880 1 bucket sync caught up with source: 2024-08-18T16:37:35.518 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , 00000000002.82.6, , , , , , 00000000002.191.6, , , , , 00000000002.107.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.120.6] 2024-08-18T16:37:35.518 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , , 00000000002.82.6, , , , , , 00000000002.191.6, , , , , 00000000002.107.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.120.6] 2024-08-18T16:37:35.518 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:37:35.514+0000 7f5102c35880 0 bucket checkpoint complete 2024-08-18T16:37:35.533 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:37:35.534 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:37:35.534 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-78 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:37:35.592 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:37:35.592 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:37:35.631 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:37:35.631+0000 7fa012f36880 1 bucket sync caught up with source: 2024-08-18T16:37:35.632 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000003.343.6, 00000000002.238.6, 00000000002.370.6, 00000000005.228.6, 00000000004.197.6, 00000000001.404.6, 00000000001.244.6, 00000000004.285.6, 00000000007.349.6, 00000000006.277.6] 2024-08-18T16:37:35.632 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000003.343.6, 00000000002.238.6, 00000000002.370.6, 00000000005.228.6, 00000000004.197.6, 00000000001.404.6, 00000000001.244.6, 00000000004.285.6, 00000000007.349.6, 00000000006.277.6] 2024-08-18T16:37:35.632 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:37:35.631+0000 7fa012f36880 0 bucket checkpoint complete 2024-08-18T16:37:35.646 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:37:35.646 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:37:35.647 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-78 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:37:35.701 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:37:35.701 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:37:35.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:37:35.736+0000 7fc267262880 1 waiting for incremental sync to catch up: 2024-08-18T16:37:35.738 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000003.343.6, 00000000002.238.6, 00000000002.370.6, 00000000005.228.6, 00000000004.197.6, 00000000001.404.6, 00000000001.244.6, 00000000004.285.6, 00000000006.342.6, 00000000006.277.6] 2024-08-18T16:37:35.738 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000003.343.6, 00000000002.238.6, 00000000002.370.6, 00000000005.228.6, 00000000004.197.6, 00000000001.404.6, 00000000001.244.6, 00000000004.285.6, 00000000007.349.6, 00000000006.277.6] 2024-08-18T16:37:40.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:37:40.737+0000 7fc267262880 1 waiting for incremental sync to catch up: 2024-08-18T16:37:40.738 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000003.343.6, 00000000002.238.6, 00000000002.370.6, 00000000005.228.6, 00000000004.197.6, 00000000001.404.6, 00000000001.244.6, 00000000004.285.6, 00000000006.342.6, 00000000006.277.6] 2024-08-18T16:37:40.739 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000003.343.6, 00000000002.238.6, 00000000002.370.6, 00000000005.228.6, 00000000004.197.6, 00000000001.404.6, 00000000001.244.6, 00000000004.285.6, 00000000007.349.6, 00000000006.277.6] 2024-08-18T16:37:45.740 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:37:45.739+0000 7fc267262880 1 waiting for incremental sync to catch up: 2024-08-18T16:37:45.740 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000003.343.6, 00000000002.238.6, 00000000002.370.6, 00000000005.228.6, 00000000004.197.6, 00000000001.404.6, 00000000001.244.6, 00000000004.285.6, 00000000006.342.6, 00000000006.277.6] 2024-08-18T16:37:45.740 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000003.343.6, 00000000002.238.6, 00000000002.370.6, 00000000005.228.6, 00000000004.197.6, 00000000001.404.6, 00000000001.244.6, 00000000004.285.6, 00000000007.349.6, 00000000006.277.6] 2024-08-18T16:37:50.741 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:37:50.740+0000 7fc267262880 1 waiting for incremental sync to catch up: 2024-08-18T16:37:50.741 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000003.343.6, 00000000002.238.6, 00000000002.370.6, 00000000005.228.6, 00000000004.197.6, 00000000001.404.6, 00000000001.244.6, 00000000004.285.6, 00000000006.342.6, 00000000006.277.6] 2024-08-18T16:37:50.741 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000003.343.6, 00000000002.238.6, 00000000002.370.6, 00000000005.228.6, 00000000004.197.6, 00000000001.404.6, 00000000001.244.6, 00000000004.285.6, 00000000007.349.6, 00000000006.277.6] 2024-08-18T16:37:55.742 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:37:55.741+0000 7fc267262880 1 waiting for incremental sync to catch up: 2024-08-18T16:37:55.742 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000003.343.6, 00000000002.238.6, 00000000002.370.6, 00000000005.228.6, 00000000004.197.6, 00000000001.404.6, 00000000001.244.6, 00000000004.285.6, 00000000006.342.6, 00000000006.277.6] 2024-08-18T16:37:55.742 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000003.343.6, 00000000002.238.6, 00000000002.370.6, 00000000005.228.6, 00000000004.197.6, 00000000001.404.6, 00000000001.244.6, 00000000004.285.6, 00000000007.349.6, 00000000006.277.6] 2024-08-18T16:38:00.743 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:38:00.742+0000 7fc267262880 1 waiting for incremental sync to catch up: 2024-08-18T16:38:00.743 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000003.343.6, 00000000002.238.6, 00000000002.370.6, 00000000005.228.6, 00000000004.197.6, 00000000001.404.6, 00000000001.244.6, 00000000004.285.6, 00000000006.342.6, 00000000006.277.6] 2024-08-18T16:38:00.744 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000003.343.6, 00000000002.238.6, 00000000002.370.6, 00000000005.228.6, 00000000004.197.6, 00000000001.404.6, 00000000001.244.6, 00000000004.285.6, 00000000007.349.6, 00000000006.277.6] 2024-08-18T16:38:05.745 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:38:05.744+0000 7fc267262880 1 waiting for incremental sync to catch up: 2024-08-18T16:38:05.745 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000003.343.6, 00000000002.238.6, 00000000002.370.6, 00000000005.228.6, 00000000004.197.6, 00000000001.404.6, 00000000001.244.6, 00000000004.285.6, 00000000006.342.6, 00000000006.277.6] 2024-08-18T16:38:05.745 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000003.343.6, 00000000002.238.6, 00000000002.370.6, 00000000005.228.6, 00000000004.197.6, 00000000001.404.6, 00000000001.244.6, 00000000004.285.6, 00000000007.349.6, 00000000006.277.6] 2024-08-18T16:38:10.746 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:38:10.745+0000 7fc267262880 1 waiting for incremental sync to catch up: 2024-08-18T16:38:10.746 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000003.343.6, 00000000002.238.6, 00000000002.370.6, 00000000005.228.6, 00000000004.197.6, 00000000001.404.6, 00000000001.244.6, 00000000004.285.6, 00000000006.342.6, 00000000006.277.6] 2024-08-18T16:38:10.746 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000003.343.6, 00000000002.238.6, 00000000002.370.6, 00000000005.228.6, 00000000004.197.6, 00000000001.404.6, 00000000001.244.6, 00000000004.285.6, 00000000007.349.6, 00000000006.277.6] 2024-08-18T16:38:15.747 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:38:15.746+0000 7fc267262880 1 waiting for incremental sync to catch up: 2024-08-18T16:38:15.747 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000003.343.6, 00000000002.238.6, 00000000002.370.6, 00000000005.228.6, 00000000004.197.6, 00000000001.404.6, 00000000001.244.6, 00000000004.285.6, 00000000006.342.6, 00000000006.277.6] 2024-08-18T16:38:15.748 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000003.343.6, 00000000002.238.6, 00000000002.370.6, 00000000005.228.6, 00000000004.197.6, 00000000001.404.6, 00000000001.244.6, 00000000004.285.6, 00000000007.349.6, 00000000006.277.6] 2024-08-18T16:38:20.749 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:38:20.748+0000 7fc267262880 1 waiting for incremental sync to catch up: 2024-08-18T16:38:20.749 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000003.343.6, 00000000002.238.6, 00000000002.370.6, 00000000005.228.6, 00000000004.197.6, 00000000001.404.6, 00000000001.244.6, 00000000004.285.6, 00000000006.342.6, 00000000006.277.6] 2024-08-18T16:38:20.749 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000003.343.6, 00000000002.238.6, 00000000002.370.6, 00000000005.228.6, 00000000004.197.6, 00000000001.404.6, 00000000001.244.6, 00000000004.285.6, 00000000007.349.6, 00000000006.277.6] 2024-08-18T16:38:25.750 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:38:25.749+0000 7fc267262880 1 waiting for incremental sync to catch up: 2024-08-18T16:38:25.750 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000003.343.6, 00000000002.238.6, 00000000002.370.6, 00000000005.228.6, 00000000004.197.6, 00000000001.404.6, 00000000001.244.6, 00000000004.285.6, 00000000006.342.6, 00000000006.277.6] 2024-08-18T16:38:25.750 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000003.343.6, 00000000002.238.6, 00000000002.370.6, 00000000005.228.6, 00000000004.197.6, 00000000001.404.6, 00000000001.244.6, 00000000004.285.6, 00000000007.349.6, 00000000006.277.6] 2024-08-18T16:38:30.751 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:38:30.750+0000 7fc267262880 1 waiting for incremental sync to catch up: 2024-08-18T16:38:30.751 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000003.343.6, 00000000002.238.6, 00000000002.370.6, 00000000005.228.6, 00000000004.197.6, 00000000001.404.6, 00000000001.244.6, 00000000004.285.6, 00000000006.342.6, 00000000006.277.6] 2024-08-18T16:38:30.752 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000003.343.6, 00000000002.238.6, 00000000002.370.6, 00000000005.228.6, 00000000004.197.6, 00000000001.404.6, 00000000001.244.6, 00000000004.285.6, 00000000007.349.6, 00000000006.277.6] 2024-08-18T16:38:35.752 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:38:35.751+0000 7fc267262880 1 waiting for incremental sync to catch up: 2024-08-18T16:38:35.753 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000003.343.6, 00000000002.238.6, 00000000002.370.6, 00000000005.228.6, 00000000004.197.6, 00000000001.404.6, 00000000001.244.6, 00000000004.285.6, 00000000006.342.6, 00000000006.277.6] 2024-08-18T16:38:35.753 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000003.343.6, 00000000002.238.6, 00000000002.370.6, 00000000005.228.6, 00000000004.197.6, 00000000001.404.6, 00000000001.244.6, 00000000004.285.6, 00000000007.349.6, 00000000006.277.6] 2024-08-18T16:38:40.754 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:38:40.753+0000 7fc267262880 1 waiting for incremental sync to catch up: 2024-08-18T16:38:40.754 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000003.343.6, 00000000002.238.6, 00000000002.370.6, 00000000005.228.6, 00000000004.197.6, 00000000001.404.6, 00000000001.244.6, 00000000004.285.6, 00000000006.342.6, 00000000006.277.6] 2024-08-18T16:38:40.754 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000003.343.6, 00000000002.238.6, 00000000002.370.6, 00000000005.228.6, 00000000004.197.6, 00000000001.404.6, 00000000001.244.6, 00000000004.285.6, 00000000007.349.6, 00000000006.277.6] 2024-08-18T16:38:45.755 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:38:45.754+0000 7fc267262880 1 waiting for incremental sync to catch up: 2024-08-18T16:38:45.755 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000003.343.6, 00000000002.238.6, 00000000002.370.6, 00000000005.228.6, 00000000004.197.6, 00000000001.404.6, 00000000001.244.6, 00000000004.285.6, 00000000006.342.6, 00000000006.277.6] 2024-08-18T16:38:45.755 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000003.343.6, 00000000002.238.6, 00000000002.370.6, 00000000005.228.6, 00000000004.197.6, 00000000001.404.6, 00000000001.244.6, 00000000004.285.6, 00000000007.349.6, 00000000006.277.6] 2024-08-18T16:38:50.757 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:38:50.756+0000 7fc267262880 1 waiting for incremental sync to catch up: 2024-08-18T16:38:50.757 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000003.343.6, 00000000002.238.6, 00000000002.370.6, 00000000005.228.6, 00000000004.197.6, 00000000001.404.6, 00000000001.244.6, 00000000004.285.6, 00000000006.342.6, 00000000006.277.6] 2024-08-18T16:38:50.757 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000003.343.6, 00000000002.238.6, 00000000002.370.6, 00000000005.228.6, 00000000004.197.6, 00000000001.404.6, 00000000001.244.6, 00000000004.285.6, 00000000007.349.6, 00000000006.277.6] 2024-08-18T16:38:55.758 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:38:55.757+0000 7fc267262880 1 bucket sync caught up with source: 2024-08-18T16:38:55.758 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000003.343.6, 00000000002.238.6, 00000000002.370.6, 00000000005.228.6, 00000000004.197.6, 00000000001.404.6, 00000000001.244.6, 00000000004.285.6, 00000000007.349.6, 00000000006.277.6] 2024-08-18T16:38:55.758 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000003.343.6, 00000000002.238.6, 00000000002.370.6, 00000000005.228.6, 00000000004.197.6, 00000000001.404.6, 00000000001.244.6, 00000000004.285.6, 00000000007.349.6, 00000000006.277.6] 2024-08-18T16:38:55.758 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:38:55.757+0000 7fc267262880 0 bucket checkpoint complete 2024-08-18T16:38:55.776 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:38:55.777 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:38:55.777 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-78 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:38:55.837 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:38:55.837 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:38:55.874 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:38:55.874+0000 7f93bfa17880 1 bucket sync caught up with source: 2024-08-18T16:38:55.875 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000003.221.6, 00000000002.490.6, 00000000002.256.6, 00000000005.292.6, 00000000004.146.6, 00000000001.526.6, 00000000001.496.6, 00000000004.195.6, 00000000007.231.6, 00000000006.159.6] 2024-08-18T16:38:55.875 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000003.221.6, 00000000002.490.6, 00000000002.256.6, 00000000005.292.6, 00000000004.146.6, 00000000001.526.6, 00000000001.496.6, 00000000004.195.6, 00000000007.231.6, 00000000006.159.6] 2024-08-18T16:38:55.875 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:38:55.874+0000 7f93bfa17880 0 bucket checkpoint complete 2024-08-18T16:38:55.890 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:38:55.890 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:38:55.891 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-78 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T16:38:55.952 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:38:55.952 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:38:55.991 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:38:55.991+0000 7f473681f880 1 bucket sync caught up with source: 2024-08-18T16:38:55.992 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, 00000000003.221.6, 00000000002.490.6, 00000000002.256.6, 00000000005.292.6, 00000000004.146.6, 00000000001.526.6, 00000000001.496.6, 00000000004.195.6, 00000000007.231.6, 00000000006.159.6] 2024-08-18T16:38:55.992 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, 00000000003.221.6, 00000000002.490.6, 00000000002.256.6, 00000000005.292.6, 00000000004.146.6, 00000000001.526.6, 00000000001.496.6, 00000000004.195.6, 00000000007.231.6, 00000000006.159.6] 2024-08-18T16:38:55.992 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:38:55.991+0000 7f473681f880 0 bucket checkpoint complete 2024-08-18T16:38:56.013 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-78 zones={test-zone2, test-zone1} 2024-08-18T16:38:56.028 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:38:56.033 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-08-18T16:38:56.033 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2024-08-18T16:38:56.033 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2024-08-18T16:38:56.033 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2024-08-18T16:38:56.033 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2024-08-18T16:38:56.033 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2024-08-18T16:38:56.033 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2024-08-18T16:38:56.033 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2024-08-18T16:38:56.034 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-08-18T16:38:56.034 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2024-08-18T16:38:56.034 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-08-18T16:38:56.034 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-08-18T16:38:56.034 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2024-08-18T16:38:56.034 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2024-08-18T16:38:56.034 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2024-08-18T16:38:56.034 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-08-18T16:38:56.034 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-08-18T16:38:56.034 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2024-08-18T16:38:56.034 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-08-18T16:38:56.034 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-08-18T16:38:56.034 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2024-08-18T16:38:56.034 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-08-18T16:38:56.035 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-08-18T16:38:56.035 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-08-18T16:38:56.035 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2024-08-18T16:38:56.035 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2024-08-18T16:38:56.035 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-08-18T16:38:56.035 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2024-08-18T16:38:56.035 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2024-08-18T16:38:56.035 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-08-18T16:38:56.035 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-08-18T16:38:56.035 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2024-08-18T16:38:56.035 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2024-08-18T16:38:56.035 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2024-08-18T16:38:56.035 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-08-18T16:38:56.035 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:38:56.041 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-08-18T16:38:56.041 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2024-08-18T16:38:56.041 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2024-08-18T16:38:56.041 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2024-08-18T16:38:56.041 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2024-08-18T16:38:56.041 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2024-08-18T16:38:56.041 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2024-08-18T16:38:56.041 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2024-08-18T16:38:56.041 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-08-18T16:38:56.041 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2024-08-18T16:38:56.041 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-08-18T16:38:56.041 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-08-18T16:38:56.042 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2024-08-18T16:38:56.042 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2024-08-18T16:38:56.042 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2024-08-18T16:38:56.042 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-08-18T16:38:56.042 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-08-18T16:38:56.042 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2024-08-18T16:38:56.042 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-08-18T16:38:56.042 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-08-18T16:38:56.042 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2024-08-18T16:38:56.042 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-08-18T16:38:56.042 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-08-18T16:38:56.042 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-08-18T16:38:56.042 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2024-08-18T16:38:56.042 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2024-08-18T16:38:56.043 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-08-18T16:38:56.043 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2024-08-18T16:38:56.043 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2024-08-18T16:38:56.043 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-08-18T16:38:56.043 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-08-18T16:38:56.043 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2024-08-18T16:38:56.043 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2024-08-18T16:38:56.043 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2024-08-18T16:38:56.043 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-08-18T16:38:56.052 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T16:38:56.106 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T16:38:56.120 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T16:38:56.177 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2024-08-18T16:38:56.183 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2024-08-18T16:38:56.242 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2024-08-18T16:38:56.298 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-08-18T16:38:56.306 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-08-18T16:38:56.363 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-08-18T16:38:56.372 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-08-18T16:38:56.423 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-08-18T16:38:56.479 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-08-18T16:38:56.488 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2024-08-18T16:38:56.538 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2024-08-18T16:38:56.553 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2024-08-18T16:38:56.607 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-08-18T16:38:56.613 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-08-18T16:38:56.710 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-08-18T16:38:56.763 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2024-08-18T16:38:56.813 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2024-08-18T16:38:56.870 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2024-08-18T16:38:56.922 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2024-08-18T16:38:56.972 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2024-08-18T16:38:56.986 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2024-08-18T16:38:57.038 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-08-18T16:38:57.046 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-08-18T16:38:57.059 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-08-18T16:38:57.115 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2024-08-18T16:38:57.122 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2024-08-18T16:38:57.135 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2024-08-18T16:38:57.191 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-08-18T16:38:57.199 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-08-18T16:38:57.259 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-08-18T16:38:57.311 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T16:38:57.363 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T16:38:57.418 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T16:38:57.429 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-08-18T16:38:57.436 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-08-18T16:38:57.490 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-08-18T16:38:57.541 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-08-18T16:38:57.548 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-08-18T16:38:57.603 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-08-18T16:38:57.657 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2024-08-18T16:38:57.665 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2024-08-18T16:38:57.723 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2024-08-18T16:38:57.774 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T16:38:57.782 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T16:38:57.839 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T16:38:57.891 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-08-18T16:38:57.942 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-08-18T16:38:57.955 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-08-18T16:38:58.012 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2024-08-18T16:38:58.062 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2024-08-18T16:38:58.076 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2024-08-18T16:38:58.131 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-08-18T16:38:58.178 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-08-18T16:38:58.191 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-08-18T16:38:58.249 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T16:38:58.259 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T16:38:58.314 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T16:38:58.371 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-08-18T16:38:58.379 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-08-18T16:38:58.434 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-08-18T16:38:58.493 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-08-18T16:38:58.501 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-08-18T16:38:58.562 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-08-18T16:38:58.617 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T16:38:58.624 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T16:38:58.679 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T16:38:58.732 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-08-18T16:38:58.740 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-08-18T16:38:58.794 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-08-18T16:38:58.849 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-08-18T16:38:58.856 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-08-18T16:38:58.911 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-08-18T16:38:58.968 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2024-08-18T16:38:58.976 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2024-08-18T16:38:59.030 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2024-08-18T16:38:59.082 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-08-18T16:38:59.089 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-08-18T16:38:59.150 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-08-18T16:38:59.160 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-08-18T16:38:59.210 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-08-18T16:38:59.225 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-08-18T16:38:59.278 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2024-08-18T16:38:59.330 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2024-08-18T16:38:59.344 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2024-08-18T16:38:59.354 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-08-18T16:38:59.362 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-08-18T16:38:59.417 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-08-18T16:38:59.470 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-08-18T16:38:59.478 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-08-18T16:38:59.493 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-08-18T16:38:59.503 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-08-18T16:38:59.556 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-08-18T16:38:59.570 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-08-18T16:38:59.581 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-08-18T16:38:59.630 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-08-18T16:38:59.643 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-08-18T16:38:59.695 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2024-08-18T16:38:59.702 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2024-08-18T16:38:59.757 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2024-08-18T16:38:59.815 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-08-18T16:38:59.866 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-08-18T16:38:59.879 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-08-18T16:38:59.932 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-78 zones={test-zone2, test-zone1} 2024-08-18T16:38:59.932 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-78 zones={test-zone3, test-zone1} 2024-08-18T16:38:59.941 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:38:59.945 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-08-18T16:38:59.945 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2024-08-18T16:38:59.945 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2024-08-18T16:38:59.945 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2024-08-18T16:38:59.945 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2024-08-18T16:38:59.945 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2024-08-18T16:38:59.945 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2024-08-18T16:38:59.945 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2024-08-18T16:38:59.945 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-08-18T16:38:59.945 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2024-08-18T16:38:59.946 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-08-18T16:38:59.946 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-08-18T16:38:59.946 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2024-08-18T16:38:59.946 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2024-08-18T16:38:59.946 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2024-08-18T16:38:59.946 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-08-18T16:38:59.946 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-08-18T16:38:59.946 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2024-08-18T16:38:59.946 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-08-18T16:38:59.947 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-08-18T16:38:59.947 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2024-08-18T16:38:59.947 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-08-18T16:38:59.947 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-08-18T16:38:59.947 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-08-18T16:38:59.947 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2024-08-18T16:38:59.947 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2024-08-18T16:38:59.947 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-08-18T16:38:59.947 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2024-08-18T16:38:59.947 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2024-08-18T16:38:59.947 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-08-18T16:38:59.947 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-08-18T16:38:59.947 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2024-08-18T16:38:59.947 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2024-08-18T16:38:59.947 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2024-08-18T16:38:59.948 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-08-18T16:38:59.948 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:38:59.953 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-08-18T16:38:59.953 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2024-08-18T16:38:59.953 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2024-08-18T16:38:59.953 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2024-08-18T16:38:59.953 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2024-08-18T16:38:59.954 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2024-08-18T16:38:59.954 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2024-08-18T16:38:59.954 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2024-08-18T16:38:59.954 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-08-18T16:38:59.954 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2024-08-18T16:38:59.954 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-08-18T16:38:59.954 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-08-18T16:38:59.954 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2024-08-18T16:38:59.954 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2024-08-18T16:38:59.955 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2024-08-18T16:38:59.955 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-08-18T16:38:59.955 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-08-18T16:38:59.955 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2024-08-18T16:38:59.955 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-08-18T16:38:59.955 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-08-18T16:38:59.955 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2024-08-18T16:38:59.955 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-08-18T16:38:59.955 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-08-18T16:38:59.955 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-08-18T16:38:59.955 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2024-08-18T16:38:59.956 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2024-08-18T16:38:59.956 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-08-18T16:38:59.956 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2024-08-18T16:38:59.956 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2024-08-18T16:38:59.956 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-08-18T16:38:59.956 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-08-18T16:38:59.956 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2024-08-18T16:38:59.956 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2024-08-18T16:38:59.956 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2024-08-18T16:38:59.956 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-08-18T16:38:59.965 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T16:39:00.013 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T16:39:00.069 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T16:39:00.122 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2024-08-18T16:39:00.129 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2024-08-18T16:39:00.234 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2024-08-18T16:39:00.291 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-08-18T16:39:00.344 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-08-18T16:39:00.398 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-08-18T16:39:00.407 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-08-18T16:39:00.414 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-08-18T16:39:00.467 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-08-18T16:39:00.519 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2024-08-18T16:39:00.525 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2024-08-18T16:39:00.581 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2024-08-18T16:39:00.591 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-08-18T16:39:00.643 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-08-18T16:39:00.656 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-08-18T16:39:00.709 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2024-08-18T16:39:00.715 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2024-08-18T16:39:00.767 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2024-08-18T16:39:00.819 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2024-08-18T16:39:00.825 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2024-08-18T16:39:00.877 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2024-08-18T16:39:00.932 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-08-18T16:39:00.939 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-08-18T16:39:00.991 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-08-18T16:39:01.045 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2024-08-18T16:39:01.051 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2024-08-18T16:39:01.110 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2024-08-18T16:39:01.119 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-08-18T16:39:01.167 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-08-18T16:39:01.222 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-08-18T16:39:01.230 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T16:39:01.237 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T16:39:01.249 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T16:39:01.257 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-08-18T16:39:01.306 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-08-18T16:39:01.318 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-08-18T16:39:01.370 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-08-18T16:39:01.376 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-08-18T16:39:01.435 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-08-18T16:39:01.445 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2024-08-18T16:39:01.494 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2024-08-18T16:39:01.505 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2024-08-18T16:39:01.563 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T16:39:01.610 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T16:39:01.621 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T16:39:01.679 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-08-18T16:39:01.730 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-08-18T16:39:01.742 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-08-18T16:39:01.799 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2024-08-18T16:39:01.805 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2024-08-18T16:39:01.865 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2024-08-18T16:39:01.874 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-08-18T16:39:01.927 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-08-18T16:39:01.939 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-08-18T16:39:01.989 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T16:39:01.995 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T16:39:02.051 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T16:39:02.107 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-08-18T16:39:02.113 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-08-18T16:39:02.170 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-08-18T16:39:02.220 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-08-18T16:39:02.227 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-08-18T16:39:02.283 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-08-18T16:39:02.294 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T16:39:02.300 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T16:39:02.354 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T16:39:02.365 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-08-18T16:39:02.416 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-08-18T16:39:02.428 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-08-18T16:39:02.481 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-08-18T16:39:02.487 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-08-18T16:39:02.540 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-08-18T16:39:02.599 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2024-08-18T16:39:02.652 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2024-08-18T16:39:02.870 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2024-08-18T16:39:02.927 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-08-18T16:39:02.933 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-08-18T16:39:02.990 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-08-18T16:39:02.999 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-08-18T16:39:03.050 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-08-18T16:39:03.061 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-08-18T16:39:03.116 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2024-08-18T16:39:03.122 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2024-08-18T16:39:03.177 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2024-08-18T16:39:03.231 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-08-18T16:39:03.237 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-08-18T16:39:03.250 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-08-18T16:39:03.304 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-08-18T16:39:03.311 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-08-18T16:39:03.369 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-08-18T16:39:03.418 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-08-18T16:39:03.425 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-08-18T16:39:03.437 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-08-18T16:39:03.491 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-08-18T16:39:03.498 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-08-18T16:39:03.510 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-08-18T16:39:03.566 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2024-08-18T16:39:03.572 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2024-08-18T16:39:03.629 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2024-08-18T16:39:03.679 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-08-18T16:39:03.686 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-08-18T16:39:03.698 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-08-18T16:39:03.749 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-78 zones={test-zone3, test-zone1} 2024-08-18T16:39:03.749 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-78 zones={test-zone3, test-zone2} 2024-08-18T16:39:03.754 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:39:03.757 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-08-18T16:39:03.758 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2024-08-18T16:39:03.758 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2024-08-18T16:39:03.758 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2024-08-18T16:39:03.758 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2024-08-18T16:39:03.758 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2024-08-18T16:39:03.758 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2024-08-18T16:39:03.758 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2024-08-18T16:39:03.758 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-08-18T16:39:03.758 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2024-08-18T16:39:03.758 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-08-18T16:39:03.758 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-08-18T16:39:03.758 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2024-08-18T16:39:03.758 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2024-08-18T16:39:03.759 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2024-08-18T16:39:03.759 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-08-18T16:39:03.759 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-08-18T16:39:03.759 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2024-08-18T16:39:03.759 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-08-18T16:39:03.759 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-08-18T16:39:03.759 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2024-08-18T16:39:03.759 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-08-18T16:39:03.759 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-08-18T16:39:03.759 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-08-18T16:39:03.759 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2024-08-18T16:39:03.759 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2024-08-18T16:39:03.759 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-08-18T16:39:03.760 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2024-08-18T16:39:03.760 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2024-08-18T16:39:03.760 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-08-18T16:39:03.760 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-08-18T16:39:03.760 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2024-08-18T16:39:03.760 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2024-08-18T16:39:03.760 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2024-08-18T16:39:03.760 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-08-18T16:39:03.760 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:39:03.765 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-08-18T16:39:03.765 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2024-08-18T16:39:03.766 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2024-08-18T16:39:03.766 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2024-08-18T16:39:03.766 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2024-08-18T16:39:03.766 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2024-08-18T16:39:03.766 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2024-08-18T16:39:03.766 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2024-08-18T16:39:03.766 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-08-18T16:39:03.766 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2024-08-18T16:39:03.766 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-08-18T16:39:03.766 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-08-18T16:39:03.766 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2024-08-18T16:39:03.766 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2024-08-18T16:39:03.766 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2024-08-18T16:39:03.767 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-08-18T16:39:03.767 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-08-18T16:39:03.767 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2024-08-18T16:39:03.767 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-08-18T16:39:03.767 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-08-18T16:39:03.767 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2024-08-18T16:39:03.767 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-08-18T16:39:03.767 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-08-18T16:39:03.767 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-08-18T16:39:03.767 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2024-08-18T16:39:03.767 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2024-08-18T16:39:03.767 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-08-18T16:39:03.767 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2024-08-18T16:39:03.767 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2024-08-18T16:39:03.767 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-08-18T16:39:03.768 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-08-18T16:39:03.768 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2024-08-18T16:39:03.768 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2024-08-18T16:39:03.768 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2024-08-18T16:39:03.768 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-08-18T16:39:03.777 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T16:39:03.829 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T16:39:03.885 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T16:39:03.938 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2024-08-18T16:39:03.945 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2024-08-18T16:39:04.004 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2024-08-18T16:39:04.098 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-08-18T16:39:04.152 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-08-18T16:39:04.253 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-08-18T16:39:04.354 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-08-18T16:39:04.407 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-08-18T16:39:04.461 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-08-18T16:39:04.562 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2024-08-18T16:39:04.616 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2024-08-18T16:39:04.717 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2024-08-18T16:39:04.813 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-08-18T16:39:04.820 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-08-18T16:39:04.877 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-08-18T16:39:04.929 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2024-08-18T16:39:04.980 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2024-08-18T16:39:05.083 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2024-08-18T16:39:05.138 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2024-08-18T16:39:05.192 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2024-08-18T16:39:05.204 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2024-08-18T16:39:05.212 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-08-18T16:39:05.260 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-08-18T16:39:05.353 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-08-18T16:39:05.449 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2024-08-18T16:39:05.500 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2024-08-18T16:39:05.601 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2024-08-18T16:39:05.701 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-08-18T16:39:05.709 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-08-18T16:39:05.769 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-08-18T16:39:05.821 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T16:39:05.872 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T16:39:05.973 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T16:39:06.070 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-08-18T16:39:06.120 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-08-18T16:39:06.182 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-08-18T16:39:06.277 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-08-18T16:39:06.286 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-08-18T16:39:06.346 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-08-18T16:39:06.357 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2024-08-18T16:39:06.366 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2024-08-18T16:39:06.426 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2024-08-18T16:39:06.435 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T16:39:06.489 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T16:39:06.549 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T16:39:06.601 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-08-18T16:39:06.652 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-08-18T16:39:06.706 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-08-18T16:39:06.806 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2024-08-18T16:39:06.814 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2024-08-18T16:39:06.827 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2024-08-18T16:39:06.884 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-08-18T16:39:06.892 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-08-18T16:39:06.950 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-08-18T16:39:06.959 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T16:39:07.010 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T16:39:07.066 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T16:39:07.170 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-08-18T16:39:07.178 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-08-18T16:39:07.239 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-08-18T16:39:07.295 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-08-18T16:39:07.348 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-08-18T16:39:07.449 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-08-18T16:39:07.546 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T16:39:07.596 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T16:39:07.654 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T16:39:07.753 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-08-18T16:39:07.761 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-08-18T16:39:07.818 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-08-18T16:39:07.827 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-08-18T16:39:07.835 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-08-18T16:39:07.887 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-08-18T16:39:07.940 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2024-08-18T16:39:07.947 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2024-08-18T16:39:08.003 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2024-08-18T16:39:08.056 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-08-18T16:39:08.109 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-08-18T16:39:08.166 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-08-18T16:39:08.262 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-08-18T16:39:08.319 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-08-18T16:39:08.421 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-08-18T16:39:08.518 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2024-08-18T16:39:08.569 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2024-08-18T16:39:08.630 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2024-08-18T16:39:08.726 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-08-18T16:39:08.735 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-08-18T16:39:08.796 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-08-18T16:39:08.806 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-08-18T16:39:08.815 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-08-18T16:39:08.872 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-08-18T16:39:08.882 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-08-18T16:39:08.936 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-08-18T16:39:08.950 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-08-18T16:39:09.003 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-08-18T16:39:09.055 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-08-18T16:39:09.067 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-08-18T16:39:09.127 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2024-08-18T16:39:09.177 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2024-08-18T16:39:09.233 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2024-08-18T16:39:09.290 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-08-18T16:39:09.298 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-08-18T16:39:09.310 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-08-18T16:39:09.366 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-78 zones={test-zone3, test-zone2} 2024-08-18T16:39:09.367 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2024-08-18T16:39:09.367 INFO:teuthology.orchestra.run:waiting for 300 2024-08-18T16:39:09.368 INFO:tasks.rgw.c2.client.0.smithi193.stdout:2024-08-18T16:39:09.367+0000 7f93333d1640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 74147) UID: 0 2024-08-18T16:39:09.368 INFO:tasks.rgw.c2.client.0.smithi193.stdout:2024-08-18T16:39:09.367+0000 7f933980b880 -1 shutting down 2024-08-18T16:39:14.868 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2024-08-18T16:39:15.468 INFO:tasks.rgw.c2.client.0:Stopped 2024-08-18T16:39:15.468 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'list', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--max-entries', '10000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:39:15.468 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'list', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--max-entries', '10000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:39:15.468 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket list --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-78 --max-entries 10000 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:39:15.526 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:39:15.527 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:39:15.563 INFO:teuthology.orchestra.run.smithi193.stdout:[ 2024-08-18T16:39:15.563 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:39:15.563 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj11", 2024-08-18T16:39:15.563 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:39:15.563 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:39:15.563 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:39:15.563 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 14 2024-08-18T16:39:15.564 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.564 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:39:15.564 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:39:15.564 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:39:15.564 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:39:15.564 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 5, 2024-08-18T16:39:15.564 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:23:42.051163Z", 2024-08-18T16:39:15.564 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:15.564 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:39:15.564 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:15.565 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:15.565 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:15.565 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:39:15.565 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:39:15.565 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:39:15.565 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.565 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "__jah6kOV7youlYh9oXWeJgx3jIbkGkG", 2024-08-18T16:39:15.565 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:39:15.565 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:39:15.565 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:39:15.566 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.566 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:39:15.566 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj111", 2024-08-18T16:39:15.566 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:39:15.566 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:39:15.566 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:39:15.566 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 27 2024-08-18T16:39:15.566 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.566 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:39:15.566 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:39:15.566 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:39:15.567 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:39:15.567 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 6, 2024-08-18T16:39:15.567 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:36:05.189316Z", 2024-08-18T16:39:15.567 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:15.567 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:39:15.567 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:15.567 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:15.567 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:15.567 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:39:15.567 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:39:15.567 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:39:15.567 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.568 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_iShrVDKmGV6srVkm7dNOX_JlNT9NXqf", 2024-08-18T16:39:15.568 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:39:15.568 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:39:15.568 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:39:15.568 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.568 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:39:15.568 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj115", 2024-08-18T16:39:15.568 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:39:15.568 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:39:15.568 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:39:15.568 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 3 2024-08-18T16:39:15.569 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.569 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:39:15.569 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:39:15.569 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:39:15.569 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:39:15.569 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 16, 2024-08-18T16:39:15.569 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:27:23.466808Z", 2024-08-18T16:39:15.569 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:15.569 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:39:15.569 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:15.570 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:15.570 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:15.570 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:39:15.570 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:39:15.570 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:39:15.570 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.570 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_w_dourmKUWsoTGg3Qp7yAHV-boQMSbT", 2024-08-18T16:39:15.570 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:39:15.571 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:39:15.571 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:39:15.571 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.571 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:39:15.571 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj116", 2024-08-18T16:39:15.571 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:39:15.571 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:39:15.571 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:39:15.571 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 9 2024-08-18T16:39:15.571 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.571 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:39:15.571 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:39:15.572 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:39:15.572 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:39:15.572 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 5, 2024-08-18T16:39:15.572 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:30:57.371585Z", 2024-08-18T16:39:15.572 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:15.572 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:39:15.572 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:15.572 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:15.572 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:15.572 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:39:15.573 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:39:15.573 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:39:15.573 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.573 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_NUmk7gIHpNnIrFFLFEXkCkUx8VJVLyt", 2024-08-18T16:39:15.573 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:39:15.573 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:39:15.573 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:39:15.573 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.573 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:39:15.573 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj124", 2024-08-18T16:39:15.573 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:39:15.574 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:39:15.574 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:39:15.574 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 13 2024-08-18T16:39:15.574 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.574 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:39:15.574 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:39:15.574 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:39:15.574 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:39:15.574 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 6, 2024-08-18T16:39:15.574 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:33:27.492061Z", 2024-08-18T16:39:15.575 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:15.575 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:39:15.575 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:15.575 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:15.575 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:15.575 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:39:15.575 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:39:15.575 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:39:15.575 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.575 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_vgYcgYoxFX5LtpBKwXx3bPkcg8ji1HG", 2024-08-18T16:39:15.576 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:39:15.576 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:39:15.576 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:39:15.576 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.576 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:39:15.576 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj145", 2024-08-18T16:39:15.576 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:39:15.576 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:39:15.576 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:39:15.576 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 17 2024-08-18T16:39:15.577 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.577 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:39:15.577 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:39:15.577 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:39:15.577 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:39:15.577 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 16, 2024-08-18T16:39:15.577 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:30:57.379450Z", 2024-08-18T16:39:15.577 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:15.577 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:39:15.577 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:15.577 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:15.578 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:15.578 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:39:15.578 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:39:15.578 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:39:15.578 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.578 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_gqKYFtq60XgE0ERPcWPYfzlO9G9QO26", 2024-08-18T16:39:15.578 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:39:15.578 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:39:15.578 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:39:15.579 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.579 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:39:15.579 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj148", 2024-08-18T16:39:15.579 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:39:15.579 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:39:15.579 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:39:15.579 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 8 2024-08-18T16:39:15.579 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.579 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:39:15.579 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:39:15.579 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:39:15.580 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:39:15.580 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 16, 2024-08-18T16:39:15.580 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:36:05.197180Z", 2024-08-18T16:39:15.580 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:15.580 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:39:15.580 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:15.580 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:15.580 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:15.580 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:39:15.580 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:39:15.580 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:39:15.581 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.581 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_5bnIT6MvoyjpHYsVe3RpMheScgMYrea", 2024-08-18T16:39:15.581 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:39:15.581 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:39:15.581 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:39:15.581 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.581 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:39:15.581 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj155", 2024-08-18T16:39:15.581 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:39:15.581 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:39:15.581 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:39:15.582 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 9 2024-08-18T16:39:15.582 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.582 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:39:15.582 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:39:15.582 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:39:15.582 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:39:15.582 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 5, 2024-08-18T16:39:15.582 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:33:27.500403Z", 2024-08-18T16:39:15.582 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:15.582 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:39:15.582 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:15.583 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:15.583 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:15.583 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:39:15.583 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:39:15.583 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:39:15.583 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.583 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_-JT4x_tuOMlr-T6NhMajjxmnNR1fWTD", 2024-08-18T16:39:15.583 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:39:15.583 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:39:15.583 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:39:15.583 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.583 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:39:15.584 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj17", 2024-08-18T16:39:15.584 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:39:15.584 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:39:15.584 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:39:15.584 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 7 2024-08-18T16:39:15.584 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.584 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:39:15.584 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:39:15.584 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:39:15.584 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:39:15.584 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 16, 2024-08-18T16:39:15.585 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:24:10.291690Z", 2024-08-18T16:39:15.585 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:15.585 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:39:15.585 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:15.585 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:15.585 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:15.585 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:39:15.585 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:39:15.585 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:39:15.585 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.586 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_iYelNrbhTXOeG-cxaTwetREqjQLBICs", 2024-08-18T16:39:15.586 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:39:15.586 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:39:15.586 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:39:15.586 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.586 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:39:15.586 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj185", 2024-08-18T16:39:15.586 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:39:15.586 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:39:15.586 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:39:15.587 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 12 2024-08-18T16:39:15.587 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.587 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:39:15.587 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:39:15.587 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:39:15.587 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:39:15.587 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 3, 2024-08-18T16:39:15.587 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:36:05.244081Z", 2024-08-18T16:39:15.587 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:15.587 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:39:15.587 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:15.587 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:15.588 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:15.588 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:39:15.588 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:39:15.588 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:39:15.588 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.588 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_H5Xr_j8hBBhd2bPkDQGvrSRFLPp1DKX", 2024-08-18T16:39:15.588 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:39:15.588 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:39:15.588 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:39:15.588 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.588 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:39:15.588 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj19", 2024-08-18T16:39:15.588 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:39:15.589 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:39:15.589 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:39:15.589 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 6 2024-08-18T16:39:15.589 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.589 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:39:15.589 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:39:15.589 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:39:15.589 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:39:15.589 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 5, 2024-08-18T16:39:15.589 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:25:55.960750Z", 2024-08-18T16:39:15.589 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:15.589 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:39:15.590 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:15.590 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:15.590 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:15.590 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:39:15.590 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:39:15.590 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:39:15.590 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.590 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_gs-NrH4uqvFc47dFFTuSY8K5cBokXlt", 2024-08-18T16:39:15.590 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:39:15.591 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:39:15.591 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:39:15.591 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.591 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:39:15.591 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj22", 2024-08-18T16:39:15.591 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:39:15.591 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:39:15.591 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:39:15.591 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 48 2024-08-18T16:39:15.591 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.592 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:39:15.592 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:39:15.592 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:39:15.592 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:39:15.592 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 5, 2024-08-18T16:39:15.592 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:23:42.058759Z", 2024-08-18T16:39:15.592 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:15.592 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:39:15.592 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:15.592 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:15.593 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:15.593 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:39:15.593 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:39:15.593 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:39:15.593 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.593 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_sy55vvepZphq61l4FxWFUArnhd0CMYJ", 2024-08-18T16:39:15.593 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:39:15.593 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:39:15.593 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:39:15.593 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.593 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:39:15.593 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj23", 2024-08-18T16:39:15.593 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:39:15.594 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:39:15.594 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:39:15.594 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 6 2024-08-18T16:39:15.594 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.594 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:39:15.594 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:39:15.594 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:39:15.594 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:39:15.594 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 5, 2024-08-18T16:39:15.594 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:27:23.361495Z", 2024-08-18T16:39:15.594 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:15.595 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:39:15.595 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:15.595 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:15.595 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:15.595 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:39:15.595 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:39:15.595 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:39:15.595 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.595 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_ZRG8Frk41H_reFnzA87TLzhbEHLYU8j", 2024-08-18T16:39:15.595 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:39:15.595 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:39:15.595 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:39:15.596 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.596 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:39:15.596 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj29", 2024-08-18T16:39:15.596 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:39:15.596 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:39:15.596 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:39:15.596 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 5 2024-08-18T16:39:15.596 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.596 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:39:15.596 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:39:15.596 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:39:15.596 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:39:15.597 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 5, 2024-08-18T16:39:15.597 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:30:57.311630Z", 2024-08-18T16:39:15.597 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:15.597 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:39:15.597 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:15.597 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:15.597 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:15.597 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:39:15.597 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:39:15.597 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:39:15.597 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.597 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_Z5gRJKZdgrHZOMxbmoKBuKOUBlSMdS3", 2024-08-18T16:39:15.597 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:39:15.598 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:39:15.598 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:39:15.598 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.598 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:39:15.598 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj31", 2024-08-18T16:39:15.598 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:39:15.598 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:39:15.598 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:39:15.598 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 8 2024-08-18T16:39:15.599 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.599 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:39:15.599 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:39:15.599 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:39:15.599 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:39:15.599 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 16, 2024-08-18T16:39:15.599 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:33:27.426326Z", 2024-08-18T16:39:15.599 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:15.599 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:39:15.599 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:15.599 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:15.599 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:15.600 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:39:15.600 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:39:15.600 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:39:15.600 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.600 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_5J7_r7TaK4jpoTjl6Xw2ukR3tQpY3g9", 2024-08-18T16:39:15.600 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:39:15.600 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:39:15.600 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:39:15.600 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.600 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:39:15.600 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj33", 2024-08-18T16:39:15.601 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:39:15.601 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:39:15.601 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:39:15.601 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 6 2024-08-18T16:39:15.601 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.601 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:39:15.601 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:39:15.601 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:39:15.601 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:39:15.601 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 5, 2024-08-18T16:39:15.601 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:23:42.064530Z", 2024-08-18T16:39:15.601 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:15.602 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:39:15.602 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:15.602 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:15.602 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:15.602 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:39:15.602 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:39:15.602 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:39:15.602 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.602 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_TyH8ekdICG-vxze62q0bltW0vCTsNjW", 2024-08-18T16:39:15.602 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:39:15.602 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:39:15.602 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:39:15.603 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.603 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:39:15.603 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj34", 2024-08-18T16:39:15.603 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:39:15.603 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:39:15.603 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:39:15.603 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 13 2024-08-18T16:39:15.603 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.603 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:39:15.603 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:39:15.603 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:39:15.604 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:39:15.604 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 16, 2024-08-18T16:39:15.604 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:24:10.298744Z", 2024-08-18T16:39:15.604 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:15.604 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:39:15.604 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:15.604 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:15.604 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:15.604 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:39:15.604 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:39:15.604 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:39:15.604 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.605 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_18CiyrJiHpkIZ0QW7NWmuYy9k1z9DuH", 2024-08-18T16:39:15.605 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:39:15.605 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:39:15.605 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:39:15.605 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.605 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:39:15.605 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj37", 2024-08-18T16:39:15.605 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:39:15.605 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:39:15.605 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:39:15.605 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 14 2024-08-18T16:39:15.606 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.606 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:39:15.606 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:39:15.606 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:39:15.606 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:39:15.606 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 16, 2024-08-18T16:39:15.606 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:36:05.133754Z", 2024-08-18T16:39:15.606 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:15.606 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:39:15.606 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:15.606 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:15.606 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:15.607 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:39:15.607 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:39:15.607 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:39:15.607 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.607 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_s-xHrjDGmU19IYzrIC9zVvT1k0vup-o", 2024-08-18T16:39:15.607 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:39:15.607 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:39:15.607 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:39:15.607 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.607 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:39:15.607 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj38", 2024-08-18T16:39:15.608 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:39:15.608 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:39:15.608 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:39:15.608 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 8 2024-08-18T16:39:15.608 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.608 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:39:15.608 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:39:15.608 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:39:15.608 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:39:15.608 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 3, 2024-08-18T16:39:15.608 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:25:55.967916Z", 2024-08-18T16:39:15.609 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:15.609 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:39:15.609 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:15.609 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:15.609 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:15.609 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:39:15.609 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:39:15.609 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:39:15.609 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.609 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_bmXEO5X0Ap3peS_F2MXkSbSbnxGDU6z", 2024-08-18T16:39:15.610 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:39:15.610 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:39:15.610 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:39:15.610 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.610 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:39:15.610 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj44", 2024-08-18T16:39:15.610 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:39:15.610 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:39:15.610 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:39:15.610 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 6 2024-08-18T16:39:15.611 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.611 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:39:15.611 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:39:15.611 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:39:15.611 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:39:15.611 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 16, 2024-08-18T16:39:15.611 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:23:42.115387Z", 2024-08-18T16:39:15.611 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:15.611 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:39:15.611 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:15.611 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:15.612 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:15.612 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:39:15.612 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:39:15.612 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:39:15.612 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.612 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_4IZV5IkeIxHjxO4fB_0CWHBRekESE23", 2024-08-18T16:39:15.612 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:39:15.612 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:39:15.612 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:39:15.612 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.612 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:39:15.612 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj46", 2024-08-18T16:39:15.613 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:39:15.613 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:39:15.613 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:39:15.613 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 7 2024-08-18T16:39:15.613 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.613 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:39:15.613 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:39:15.613 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:39:15.613 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:39:15.613 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 3, 2024-08-18T16:39:15.613 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:27:23.407342Z", 2024-08-18T16:39:15.614 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:15.614 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:39:15.614 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:15.614 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:15.614 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:15.614 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:39:15.614 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:39:15.614 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:39:15.614 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.614 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_inOfco2fxZUVrDZ7oDXEh3Vz9h7kb1X", 2024-08-18T16:39:15.615 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:39:15.615 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:39:15.615 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:39:15.615 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.615 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:39:15.615 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj51", 2024-08-18T16:39:15.615 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:39:15.615 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:39:15.615 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:39:15.615 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 5 2024-08-18T16:39:15.616 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.616 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:39:15.616 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:39:15.616 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:39:15.616 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:39:15.616 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 16, 2024-08-18T16:39:15.616 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:24:10.305333Z", 2024-08-18T16:39:15.616 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:15.616 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:39:15.616 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:15.616 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:15.617 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:15.617 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:39:15.617 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:39:15.617 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:39:15.617 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.617 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_OuFonFuv4wnDWH1TugiMuwSFgVCb1zD", 2024-08-18T16:39:15.617 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:39:15.617 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:39:15.617 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:39:15.617 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.618 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:39:15.618 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj55", 2024-08-18T16:39:15.618 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:39:15.618 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:39:15.618 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:39:15.618 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 5 2024-08-18T16:39:15.618 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.618 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:39:15.618 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:39:15.618 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:39:15.618 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:39:15.619 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 16, 2024-08-18T16:39:15.619 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:23:42.122245Z", 2024-08-18T16:39:15.619 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:15.619 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:39:15.619 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:15.619 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:15.619 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:15.619 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:39:15.619 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:39:15.619 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:39:15.619 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.620 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_AMWkWoqUuYgSuwyi-ue4093jO9W4yoX", 2024-08-18T16:39:15.620 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:39:15.620 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:39:15.620 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:39:15.620 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.620 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:39:15.620 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj57", 2024-08-18T16:39:15.620 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:39:15.620 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:39:15.620 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:39:15.620 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 11 2024-08-18T16:39:15.621 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.621 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:39:15.621 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:39:15.621 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:39:15.621 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:39:15.621 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 5, 2024-08-18T16:39:15.621 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:25:55.973669Z", 2024-08-18T16:39:15.621 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:15.621 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:39:15.621 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:15.622 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:15.622 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:15.622 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:39:15.622 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:39:15.622 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:39:15.622 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.622 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_hwWprOuCIo1KF6-Hp26Rb03taMF1btw", 2024-08-18T16:39:15.622 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:39:15.622 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:39:15.622 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:39:15.622 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.623 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:39:15.623 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj58", 2024-08-18T16:39:15.623 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:39:15.623 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:39:15.623 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:39:15.623 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 7 2024-08-18T16:39:15.623 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.623 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:39:15.623 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:39:15.623 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:39:15.624 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:39:15.624 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 16, 2024-08-18T16:39:15.624 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:30:57.319413Z", 2024-08-18T16:39:15.624 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:15.624 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:39:15.624 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:15.624 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:15.624 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:15.624 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:39:15.624 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:39:15.625 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:39:15.625 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.625 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "__zHzZtWL19AaGRSEKEh7oQoGr2BG5uU", 2024-08-18T16:39:15.625 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:39:15.625 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:39:15.625 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:39:15.625 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.625 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:39:15.625 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj62", 2024-08-18T16:39:15.625 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:39:15.626 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:39:15.626 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:39:15.626 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 49 2024-08-18T16:39:15.626 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.626 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:39:15.626 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:39:15.626 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:39:15.626 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:39:15.626 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 5, 2024-08-18T16:39:15.626 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:33:27.434791Z", 2024-08-18T16:39:15.627 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:15.627 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:39:15.627 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:15.627 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:15.627 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:15.627 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:39:15.627 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:39:15.627 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:39:15.627 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.627 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_FNL_VcPvOB_2SxBaSe5GlUbwB214_7P", 2024-08-18T16:39:15.628 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:39:15.628 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:39:15.628 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:39:15.628 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.628 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:39:15.628 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj68", 2024-08-18T16:39:15.628 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:39:15.628 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:39:15.628 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:39:15.628 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 11 2024-08-18T16:39:15.628 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.629 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:39:15.629 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:39:15.629 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:39:15.629 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:39:15.629 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 3, 2024-08-18T16:39:15.629 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:24:10.351074Z", 2024-08-18T16:39:15.629 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:15.629 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:39:15.629 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:15.629 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:15.630 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:15.630 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:39:15.630 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:39:15.630 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:39:15.630 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.630 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_Bt80tg29jpey44FyjageMkySPkvxPTb", 2024-08-18T16:39:15.630 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:39:15.630 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:39:15.630 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:39:15.630 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.630 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:39:15.630 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj69", 2024-08-18T16:39:15.631 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:39:15.631 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:39:15.631 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:39:15.631 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 12 2024-08-18T16:39:15.631 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.631 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:39:15.631 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:39:15.631 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:39:15.631 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:39:15.631 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 6, 2024-08-18T16:39:15.631 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:27:23.414590Z", 2024-08-18T16:39:15.632 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:15.632 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:39:15.632 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:15.632 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:15.632 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:15.632 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:39:15.632 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:39:15.632 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:39:15.632 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.633 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_X9AbJBfXA8Zrel8t4zBIKc4ZpxchfAX", 2024-08-18T16:39:15.633 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:39:15.633 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:39:15.633 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:39:15.633 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.633 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:39:15.633 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj74", 2024-08-18T16:39:15.633 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:39:15.633 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:39:15.633 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:39:15.634 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 6 2024-08-18T16:39:15.634 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.634 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:39:15.634 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:39:15.634 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:39:15.634 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:39:15.634 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 5, 2024-08-18T16:39:15.634 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:36:05.141727Z", 2024-08-18T16:39:15.635 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:15.635 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:39:15.635 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:15.635 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:15.635 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:15.635 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:39:15.635 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:39:15.635 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:39:15.635 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.635 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_eddNh3T1xLnPdJ9mRck9Qjvm6T-DdHt", 2024-08-18T16:39:15.635 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:39:15.635 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:39:15.636 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:39:15.636 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.636 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:39:15.636 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj76", 2024-08-18T16:39:15.636 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:39:15.636 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:39:15.636 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:39:15.636 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 7 2024-08-18T16:39:15.636 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.636 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:39:15.636 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:39:15.637 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:39:15.637 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:39:15.637 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 3, 2024-08-18T16:39:15.637 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:25:56.019500Z", 2024-08-18T16:39:15.637 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:15.637 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:39:15.637 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:15.637 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:15.637 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:15.637 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:39:15.638 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:39:15.638 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:39:15.638 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.638 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_4_8K-waED3leqaBY4QJFjLYhxnh3pxV", 2024-08-18T16:39:15.638 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:39:15.638 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:39:15.638 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:39:15.638 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.638 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:39:15.638 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj85", 2024-08-18T16:39:15.638 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:39:15.639 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:39:15.639 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:39:15.639 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 10 2024-08-18T16:39:15.639 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.639 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:39:15.639 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:39:15.639 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:39:15.639 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:39:15.639 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 16, 2024-08-18T16:39:15.639 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:24:10.358501Z", 2024-08-18T16:39:15.640 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:15.640 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:39:15.640 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:15.640 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:15.640 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:15.640 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:39:15.640 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:39:15.640 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:39:15.640 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.640 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_O8w_vb_JzdnCLZIKxAPw-YiSxAbjGvp", 2024-08-18T16:39:15.640 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:39:15.641 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:39:15.641 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:39:15.641 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.641 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:39:15.641 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj87", 2024-08-18T16:39:15.641 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:39:15.641 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:39:15.641 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:39:15.641 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 5 2024-08-18T16:39:15.641 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.641 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:39:15.642 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:39:15.642 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:39:15.643 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:39:15.643 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 16, 2024-08-18T16:39:15.643 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:30:57.327145Z", 2024-08-18T16:39:15.643 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:15.643 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:39:15.643 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:15.643 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:15.644 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:15.644 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:39:15.644 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:39:15.644 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:39:15.644 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.644 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_H80Es59q7nK4hX3RFFZP-t1HQ0J4JkQ", 2024-08-18T16:39:15.644 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:39:15.644 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:39:15.644 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:39:15.644 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.644 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:39:15.645 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj92", 2024-08-18T16:39:15.645 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:39:15.645 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:39:15.645 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:39:15.645 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 16 2024-08-18T16:39:15.645 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.645 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:39:15.645 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:39:15.645 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:39:15.645 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:39:15.645 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 16, 2024-08-18T16:39:15.646 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:27:23.459526Z", 2024-08-18T16:39:15.646 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:15.646 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:39:15.646 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:15.646 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:15.646 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:15.646 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:39:15.646 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:39:15.646 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:39:15.646 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.646 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_6ZngAo9SW0VeoN_vV5K9nCf8LkVIjDS", 2024-08-18T16:39:15.647 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:39:15.647 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:39:15.647 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:39:15.647 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.647 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:39:15.647 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj93", 2024-08-18T16:39:15.647 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:39:15.647 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:39:15.647 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:39:15.647 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 7 2024-08-18T16:39:15.648 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.648 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:39:15.648 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:39:15.648 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:39:15.648 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:39:15.648 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 6, 2024-08-18T16:39:15.648 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:33:27.484033Z", 2024-08-18T16:39:15.648 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:15.648 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:39:15.648 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:15.648 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:15.648 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:15.649 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:39:15.649 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:39:15.649 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:39:15.649 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.649 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_bGloLpl61pAEoKmZtxVoMCnuriWI94y", 2024-08-18T16:39:15.649 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:39:15.649 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:39:15.649 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:39:15.649 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.649 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:39:15.649 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj95", 2024-08-18T16:39:15.649 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:39:15.650 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:39:15.650 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:39:15.650 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 12 2024-08-18T16:39:15.650 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.650 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:39:15.650 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:39:15.650 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:39:15.650 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:39:15.650 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 16, 2024-08-18T16:39:15.650 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:25:56.027064Z", 2024-08-18T16:39:15.650 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:15.651 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:39:15.651 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:15.651 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:15.651 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:15.651 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:39:15.651 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:39:15.651 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:39:15.651 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:15.651 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "__Xzey2c-D_m4nFlipMVz8DOOjnhSd5m", 2024-08-18T16:39:15.651 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:39:15.651 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:39:15.651 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:39:15.652 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:39:15.652 INFO:teuthology.orchestra.run.smithi193.stdout:] 2024-08-18T16:39:15.652 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:15.652 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:15.653 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-78 --object obj11 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-08-18T16:39:15.703 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:39:15.703 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:39:15.796 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj111', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:15.796 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj111', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:15.796 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-78 --object obj111 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-08-18T16:39:15.844 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:39:15.845 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:39:15.937 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj115', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:15.937 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj115', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:15.937 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-78 --object obj115 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-08-18T16:39:15.986 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:39:15.986 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:39:16.081 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj116', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:16.081 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj116', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:16.081 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-78 --object obj116 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-08-18T16:39:16.131 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:39:16.131 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:39:16.225 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj124', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:16.225 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj124', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:16.225 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-78 --object obj124 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-08-18T16:39:16.274 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:39:16.274 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:39:16.368 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj145', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:16.369 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj145', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:16.369 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-78 --object obj145 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-08-18T16:39:16.418 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:39:16.418 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:39:16.511 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj148', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:16.511 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj148', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:16.511 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-78 --object obj148 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-08-18T16:39:16.560 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:39:16.560 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:39:16.652 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj155', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:16.653 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj155', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:16.653 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-78 --object obj155 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-08-18T16:39:16.702 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:39:16.702 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:39:16.792 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:16.792 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:16.792 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-78 --object obj17 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-08-18T16:39:16.841 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:39:16.841 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:39:16.933 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj185', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:16.933 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj185', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:16.933 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-78 --object obj185 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-08-18T16:39:16.982 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:39:16.983 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:39:17.074 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:17.074 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:17.074 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-78 --object obj19 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-08-18T16:39:17.123 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:39:17.124 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:39:17.215 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:17.215 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:17.216 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-78 --object obj22 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-08-18T16:39:17.265 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:39:17.266 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:39:17.357 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:17.357 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:17.357 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-78 --object obj23 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-08-18T16:39:17.406 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:39:17.406 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:39:17.499 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj29', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:17.499 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj29', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:17.500 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-78 --object obj29 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-08-18T16:39:17.548 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:39:17.548 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:39:17.643 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj31', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:17.643 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj31', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:17.643 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-78 --object obj31 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-08-18T16:39:17.692 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:39:17.693 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:39:17.788 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj33', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:17.788 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj33', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:17.788 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-78 --object obj33 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-08-18T16:39:17.836 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:39:17.837 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:39:17.928 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj34', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:17.929 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj34', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:17.929 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-78 --object obj34 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-08-18T16:39:17.977 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:39:17.977 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:39:18.072 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj37', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:18.073 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj37', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:18.073 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-78 --object obj37 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-08-18T16:39:18.123 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:39:18.124 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:39:18.217 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj38', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:18.218 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj38', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:18.218 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-78 --object obj38 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-08-18T16:39:18.267 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:39:18.267 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:39:18.360 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj44', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:18.360 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj44', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:18.360 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-78 --object obj44 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-08-18T16:39:18.409 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:39:18.410 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:39:18.503 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj46', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:18.563 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj46', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:18.564 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-78 --object obj46 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-08-18T16:39:18.619 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:39:18.620 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:39:18.712 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj51', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:18.712 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj51', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:18.713 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-78 --object obj51 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-08-18T16:39:18.761 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:39:18.761 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:39:18.853 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj55', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:18.853 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj55', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:18.853 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-78 --object obj55 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-08-18T16:39:18.902 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:39:18.902 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:39:18.995 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj57', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:18.995 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj57', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:18.995 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-78 --object obj57 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-08-18T16:39:19.044 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:39:19.045 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:39:19.137 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj58', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:19.138 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj58', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:19.138 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-78 --object obj58 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-08-18T16:39:19.190 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:39:19.190 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:39:19.287 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:19.287 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:19.288 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-78 --object obj62 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-08-18T16:39:19.336 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:39:19.336 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:39:19.428 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj68', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:19.428 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj68', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:19.428 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-78 --object obj68 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-08-18T16:39:19.477 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:39:19.477 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:39:19.571 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:19.571 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:19.571 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-78 --object obj69 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-08-18T16:39:19.620 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:39:19.620 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:39:19.711 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj74', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:19.712 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj74', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:19.712 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-78 --object obj74 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-08-18T16:39:19.760 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:39:19.761 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:39:19.853 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj76', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:19.853 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj76', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:19.854 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-78 --object obj76 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-08-18T16:39:19.903 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:39:19.903 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:39:19.996 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:19.997 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:19.997 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-78 --object obj85 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-08-18T16:39:20.045 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:39:20.045 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:39:20.139 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj87', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:20.139 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj87', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:20.139 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-78 --object obj87 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-08-18T16:39:20.188 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:39:20.188 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:39:20.280 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj92', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:20.280 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj92', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:20.281 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-78 --object obj92 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-08-18T16:39:20.329 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:39:20.330 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:39:20.422 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj93', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:20.422 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj93', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:20.422 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-78 --object obj93 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-08-18T16:39:20.471 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:39:20.471 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:39:20.473 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2024-08-18T16:39:20.565 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj95', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:20.565 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj95', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:20.565 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-78 --object obj95 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-08-18T16:39:20.614 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:39:20.614 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:39:20.705 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'trim', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:20.705 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'trim', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:20.705 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog trim --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-78 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-08-18T16:39:20.752 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:39:20.752 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:39:20.803 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'init', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:20.803 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'init', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:20.803 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync init --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-78 --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-08-18T16:39:20.853 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:39:20.853 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:39:20.900 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'run', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:20.901 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'run', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:20.901 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync run --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-78 --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-08-18T16:39:20.950 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:39:20.951 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:39:21.025 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.024+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: sync status for source bucket: init. lease is: not taken. stop indications is: 0 2024-08-18T16:39:21.025 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.024+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: waiting on bucket lease 2024-08-18T16:39:21.031 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.030+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: status after acquiring the lock is: init 2024-08-18T16:39:21.043 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.043+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: listing bucket for full sync 2024-08-18T16:39:21.051 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.048+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj11 2024-08-18T16:39:21.052 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.048+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj11]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj11 op=0 op_state=1 2024-08-18T16:39:21.052 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.048+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj11]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj11[0] 2024-08-18T16:39:21.052 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.048+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj111 2024-08-18T16:39:21.052 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.048+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj111]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj111 op=0 op_state=1 2024-08-18T16:39:21.052 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.048+0000 7f818ca10880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj111]: : injecting data sync error on key=obj111 2024-08-18T16:39:21.052 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.048+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj111]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:39:21.052 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.048+0000 7f818ca10880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj111]: ERROR: failed to sync object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj111 2024-08-18T16:39:21.052 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.048+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj115 2024-08-18T16:39:21.052 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.048+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj115]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj115 op=0 op_state=1 2024-08-18T16:39:21.052 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.048+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj115]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj115[0] 2024-08-18T16:39:21.052 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.049+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj116 2024-08-18T16:39:21.053 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.049+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj116]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj116 op=0 op_state=1 2024-08-18T16:39:21.053 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.049+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj116]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj116[0] 2024-08-18T16:39:21.053 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.049+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj124 2024-08-18T16:39:21.053 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.049+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj124]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj124 op=0 op_state=1 2024-08-18T16:39:21.053 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.049+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj124]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj124[0] 2024-08-18T16:39:21.053 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.049+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj145 2024-08-18T16:39:21.053 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.049+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj145]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj145 op=0 op_state=1 2024-08-18T16:39:21.053 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.049+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj145]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj145[0] 2024-08-18T16:39:21.053 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.049+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj148 2024-08-18T16:39:21.053 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.049+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj148]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj148 op=0 op_state=1 2024-08-18T16:39:21.053 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.049+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj148]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj148[0] 2024-08-18T16:39:21.053 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.049+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj155 2024-08-18T16:39:21.054 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.049+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj155]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj155 op=0 op_state=1 2024-08-18T16:39:21.054 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.049+0000 7f818ca10880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj155]: : injecting data sync error on key=obj155 2024-08-18T16:39:21.054 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.049+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj155]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:39:21.054 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.049+0000 7f818ca10880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj155]: ERROR: failed to sync object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj155 2024-08-18T16:39:21.054 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.049+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj17 2024-08-18T16:39:21.054 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.049+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj17]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj17 op=0 op_state=1 2024-08-18T16:39:21.054 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.049+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj17]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj17[0] 2024-08-18T16:39:21.054 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.049+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj185 2024-08-18T16:39:21.054 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.049+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj185]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj185 op=0 op_state=1 2024-08-18T16:39:21.054 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.049+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj185]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj185[0] 2024-08-18T16:39:21.054 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.049+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj19 2024-08-18T16:39:21.054 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.049+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj19]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj19 op=0 op_state=1 2024-08-18T16:39:21.055 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.049+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj19]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj19[0] 2024-08-18T16:39:21.055 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.049+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj22 2024-08-18T16:39:21.055 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.049+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj22]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj22 op=0 op_state=1 2024-08-18T16:39:21.055 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.049+0000 7f818ca10880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj22]: : injecting data sync error on key=obj22 2024-08-18T16:39:21.056 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.049+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj22]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:39:21.056 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.049+0000 7f818ca10880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj22]: ERROR: failed to sync object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj22 2024-08-18T16:39:21.056 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.049+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj23 2024-08-18T16:39:21.057 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.049+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj23]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj23 op=0 op_state=1 2024-08-18T16:39:21.057 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.049+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj23]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj23[0] 2024-08-18T16:39:21.057 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.049+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj29 2024-08-18T16:39:21.057 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.049+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj29]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj29 op=0 op_state=1 2024-08-18T16:39:21.057 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.049+0000 7f818ca10880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj29]: : injecting data sync error on key=obj29 2024-08-18T16:39:21.057 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.049+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj29]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:39:21.057 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.049+0000 7f818ca10880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj29]: ERROR: failed to sync object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj29 2024-08-18T16:39:21.057 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.049+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj31 2024-08-18T16:39:21.057 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.049+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj31]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj31 op=0 op_state=1 2024-08-18T16:39:21.057 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.049+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj31]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj31[0] 2024-08-18T16:39:21.057 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.049+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj33 2024-08-18T16:39:21.057 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.049+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj33]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj33 op=0 op_state=1 2024-08-18T16:39:21.057 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.049+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj33]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj33[0] 2024-08-18T16:39:21.058 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.050+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj34 2024-08-18T16:39:21.058 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.050+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj34]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj34 op=0 op_state=1 2024-08-18T16:39:21.058 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.050+0000 7f818ca10880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj34]: : injecting data sync error on key=obj34 2024-08-18T16:39:21.058 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.050+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj34]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:39:21.058 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.050+0000 7f818ca10880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj34]: ERROR: failed to sync object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj34 2024-08-18T16:39:21.058 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.050+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj37 2024-08-18T16:39:21.058 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.050+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj37]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj37 op=0 op_state=1 2024-08-18T16:39:21.058 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.050+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj37]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj37[0] 2024-08-18T16:39:21.058 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.050+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj38 2024-08-18T16:39:21.058 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.050+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj38]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj38 op=0 op_state=1 2024-08-18T16:39:21.058 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.050+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj38]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj38[0] 2024-08-18T16:39:21.058 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.051+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj44 2024-08-18T16:39:21.059 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.051+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj44]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj44 op=0 op_state=1 2024-08-18T16:39:21.059 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.051+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj44]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj44[0] 2024-08-18T16:39:21.059 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.051+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj46 2024-08-18T16:39:21.059 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.051+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj46]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj46 op=0 op_state=1 2024-08-18T16:39:21.059 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.051+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj46]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj46[0] 2024-08-18T16:39:21.059 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.052+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: a sync operation returned error 2024-08-18T16:39:21.059 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.052+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj51 2024-08-18T16:39:21.059 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.052+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj51]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj51 op=0 op_state=1 2024-08-18T16:39:21.059 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.052+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj51]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj51[0] 2024-08-18T16:39:21.059 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.052+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: a sync operation returned error 2024-08-18T16:39:21.060 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.052+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj55 2024-08-18T16:39:21.060 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.052+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj55]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj55 op=0 op_state=1 2024-08-18T16:39:21.060 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.052+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj55]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj55[0] 2024-08-18T16:39:21.060 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.052+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: a sync operation returned error 2024-08-18T16:39:21.060 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.052+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj57 2024-08-18T16:39:21.060 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.052+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj57]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj57 op=0 op_state=1 2024-08-18T16:39:21.060 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.052+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj57]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj57[0] 2024-08-18T16:39:21.060 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.052+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: a sync operation returned error 2024-08-18T16:39:21.060 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.052+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj58 2024-08-18T16:39:21.060 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.052+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj58]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj58 op=0 op_state=1 2024-08-18T16:39:21.060 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.052+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj58]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj58[0] 2024-08-18T16:39:21.060 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.052+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: a sync operation returned error 2024-08-18T16:39:21.061 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.052+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj62 2024-08-18T16:39:21.061 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.052+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj62]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj62 op=0 op_state=1 2024-08-18T16:39:21.061 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.052+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj62]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj62[0] 2024-08-18T16:39:21.074 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.072+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj17]: success 2024-08-18T16:39:21.074 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.072+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj68 2024-08-18T16:39:21.074 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.072+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj68]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj68 op=0 op_state=1 2024-08-18T16:39:21.074 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.072+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj68]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj68[0] 2024-08-18T16:39:21.074 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.072+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj116]: success 2024-08-18T16:39:21.074 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.072+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj69 2024-08-18T16:39:21.074 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.072+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj69]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj69 op=0 op_state=1 2024-08-18T16:39:21.074 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.072+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj69]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj69[0] 2024-08-18T16:39:21.074 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.072+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj23]: success 2024-08-18T16:39:21.074 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.072+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj74 2024-08-18T16:39:21.074 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.072+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj74]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj74 op=0 op_state=1 2024-08-18T16:39:21.075 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.072+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj74]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj74[0] 2024-08-18T16:39:21.076 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.073+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj11]: success 2024-08-18T16:39:21.076 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.073+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj115]: success 2024-08-18T16:39:21.076 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.073+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj51]: success 2024-08-18T16:39:21.076 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.073+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj19]: success 2024-08-18T16:39:21.076 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.073+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj55]: success 2024-08-18T16:39:21.076 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.073+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj33]: success 2024-08-18T16:39:21.077 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.073+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj76 2024-08-18T16:39:21.077 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.074+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj76]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj76 op=0 op_state=1 2024-08-18T16:39:21.077 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.074+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj76]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj76[0] 2024-08-18T16:39:21.077 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.074+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj85 2024-08-18T16:39:21.077 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.074+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj85]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj85 op=0 op_state=1 2024-08-18T16:39:21.077 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.074+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj85]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj85[0] 2024-08-18T16:39:21.077 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.074+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj87 2024-08-18T16:39:21.077 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.074+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj87]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj87 op=0 op_state=1 2024-08-18T16:39:21.077 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.074+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj87]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj87[0] 2024-08-18T16:39:21.077 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.074+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj92 2024-08-18T16:39:21.077 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.074+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj92]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj92 op=0 op_state=1 2024-08-18T16:39:21.077 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.074+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj92]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj92[0] 2024-08-18T16:39:21.077 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.074+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj93 2024-08-18T16:39:21.077 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.074+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj93]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj93 op=0 op_state=1 2024-08-18T16:39:21.078 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.074+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj93]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj93[0] 2024-08-18T16:39:21.078 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.074+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj95 2024-08-18T16:39:21.078 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.074+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj95]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj95 op=0 op_state=1 2024-08-18T16:39:21.078 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.074+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj95]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj95[0] 2024-08-18T16:39:21.078 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.074+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj38]: success 2024-08-18T16:39:21.078 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.075+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj37]: success 2024-08-18T16:39:21.079 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.076+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj148]: success 2024-08-18T16:39:21.080 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.076+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj44]: success 2024-08-18T16:39:21.080 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.076+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj62]: success 2024-08-18T16:39:21.080 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.076+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj58]: success 2024-08-18T16:39:21.080 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.077+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj145]: success 2024-08-18T16:39:21.080 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.077+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj185]: success 2024-08-18T16:39:21.080 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.078+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj46]: success 2024-08-18T16:39:21.080 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.078+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj31]: success 2024-08-18T16:39:21.080 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.078+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj57]: success 2024-08-18T16:39:21.080 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.079+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj124]: success 2024-08-18T16:39:21.081 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.080+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj74]: success 2024-08-18T16:39:21.082 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.081+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj95]: success 2024-08-18T16:39:21.082 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.081+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj93]: success 2024-08-18T16:39:21.082 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.081+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj69]: success 2024-08-18T16:39:21.082 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.081+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj68]: success 2024-08-18T16:39:21.082 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.082+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj76]: success 2024-08-18T16:39:21.122 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.121+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj87]: success 2024-08-18T16:39:21.122 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.121+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj85]: success 2024-08-18T16:39:21.122 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.121+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj92]: success 2024-08-18T16:39:21.123 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.121+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: updating marker oid=bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26 marker=obj11 2024-08-18T16:39:21.123 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.123+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: backing out with sync_status=-5 2024-08-18T16:39:21.123 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.123+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: ERROR: full sync failed. error: -5 2024-08-18T16:39:21.125 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.124+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: sync status for source bucket: full. lease is: not taken. stop indications is: 0 2024-08-18T16:39:21.125 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.124+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: waiting on bucket lease 2024-08-18T16:39:21.130 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.129+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: status after acquiring the lock is: full 2024-08-18T16:39:21.142 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.141+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: listing bucket for full sync 2024-08-18T16:39:21.150 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.147+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj11 2024-08-18T16:39:21.150 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.147+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj11]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj11 op=0 op_state=1 2024-08-18T16:39:21.151 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.147+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj11]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj11[0] 2024-08-18T16:39:21.151 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.147+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj111 2024-08-18T16:39:21.151 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.147+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj111]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj111 op=0 op_state=1 2024-08-18T16:39:21.151 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.147+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj111]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj111[0] 2024-08-18T16:39:21.151 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.147+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj115 2024-08-18T16:39:21.151 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.147+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj115]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj115 op=0 op_state=1 2024-08-18T16:39:21.151 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.147+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj115]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj115[0] 2024-08-18T16:39:21.151 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.147+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj116 2024-08-18T16:39:21.151 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.147+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj116]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj116 op=0 op_state=1 2024-08-18T16:39:21.151 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.147+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj116]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj116[0] 2024-08-18T16:39:21.151 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.147+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj124 2024-08-18T16:39:21.151 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.147+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj124]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj124 op=0 op_state=1 2024-08-18T16:39:21.151 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.147+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj124]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj124[0] 2024-08-18T16:39:21.152 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.147+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj145 2024-08-18T16:39:21.152 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.147+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj145]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj145 op=0 op_state=1 2024-08-18T16:39:21.152 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.147+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj145]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj145[0] 2024-08-18T16:39:21.152 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.147+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj148 2024-08-18T16:39:21.152 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.147+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj148]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj148 op=0 op_state=1 2024-08-18T16:39:21.152 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.147+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj148]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj148[0] 2024-08-18T16:39:21.152 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.147+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj155 2024-08-18T16:39:21.152 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.147+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj155]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj155 op=0 op_state=1 2024-08-18T16:39:21.152 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.147+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj155]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj155[0] 2024-08-18T16:39:21.152 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.147+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj17 2024-08-18T16:39:21.152 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.147+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj17]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj17 op=0 op_state=1 2024-08-18T16:39:21.152 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.147+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj17]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj17[0] 2024-08-18T16:39:21.153 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.147+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj185 2024-08-18T16:39:21.153 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.147+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj185]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj185 op=0 op_state=1 2024-08-18T16:39:21.153 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.147+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj185]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj185[0] 2024-08-18T16:39:21.153 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.147+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj19 2024-08-18T16:39:21.153 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.147+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj19]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj19 op=0 op_state=1 2024-08-18T16:39:21.153 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.147+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj19]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj19[0] 2024-08-18T16:39:21.153 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.147+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj22 2024-08-18T16:39:21.153 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.147+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj22]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj22 op=0 op_state=1 2024-08-18T16:39:21.153 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.147+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj22]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj22[0] 2024-08-18T16:39:21.153 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.147+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj23 2024-08-18T16:39:21.153 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.147+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj23]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj23 op=0 op_state=1 2024-08-18T16:39:21.153 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.147+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj23]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj23[0] 2024-08-18T16:39:21.153 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.147+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj29 2024-08-18T16:39:21.154 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.147+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj29]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj29 op=0 op_state=1 2024-08-18T16:39:21.154 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.147+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj29]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj29[0] 2024-08-18T16:39:21.154 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.147+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj31 2024-08-18T16:39:21.154 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.147+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj31]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj31 op=0 op_state=1 2024-08-18T16:39:21.154 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.147+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj31]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj31[0] 2024-08-18T16:39:21.154 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.147+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj33 2024-08-18T16:39:21.154 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.147+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj33]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj33 op=0 op_state=1 2024-08-18T16:39:21.154 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.147+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj33]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj33[0] 2024-08-18T16:39:21.154 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.147+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj34 2024-08-18T16:39:21.154 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.147+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj34]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj34 op=0 op_state=1 2024-08-18T16:39:21.154 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.148+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj34]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj34[0] 2024-08-18T16:39:21.154 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.148+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj37 2024-08-18T16:39:21.155 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.148+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj37]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj37 op=0 op_state=1 2024-08-18T16:39:21.155 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.148+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj37]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj37[0] 2024-08-18T16:39:21.155 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.148+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj38 2024-08-18T16:39:21.155 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.148+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj38]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj38 op=0 op_state=1 2024-08-18T16:39:21.155 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.148+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj38]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj38[0] 2024-08-18T16:39:21.155 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.148+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj44 2024-08-18T16:39:21.155 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.148+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj44]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj44 op=0 op_state=1 2024-08-18T16:39:21.155 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.148+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj44]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj44[0] 2024-08-18T16:39:21.155 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.148+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj46 2024-08-18T16:39:21.155 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.148+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj46]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj46 op=0 op_state=1 2024-08-18T16:39:21.155 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.148+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj46]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj46[0] 2024-08-18T16:39:21.155 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.149+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj115]: success 2024-08-18T16:39:21.156 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.149+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj51 2024-08-18T16:39:21.156 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.149+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj51]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj51 op=0 op_state=1 2024-08-18T16:39:21.156 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.149+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj51]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj51[0] 2024-08-18T16:39:21.160 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.150+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj11]: success 2024-08-18T16:39:21.160 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.150+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj55 2024-08-18T16:39:21.160 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.150+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj55]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj55 op=0 op_state=1 2024-08-18T16:39:21.160 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.150+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj55]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj55[0] 2024-08-18T16:39:21.160 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.151+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj116]: success 2024-08-18T16:39:21.160 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.151+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj57 2024-08-18T16:39:21.160 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.151+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj57]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj57 op=0 op_state=1 2024-08-18T16:39:21.160 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.151+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj57]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj57[0] 2024-08-18T16:39:21.160 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.152+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj17]: success 2024-08-18T16:39:21.160 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.152+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj58 2024-08-18T16:39:21.160 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.152+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj58]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj58 op=0 op_state=1 2024-08-18T16:39:21.160 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.152+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj58]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj58[0] 2024-08-18T16:39:21.161 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.153+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj44]: success 2024-08-18T16:39:21.161 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.153+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj62 2024-08-18T16:39:21.161 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.153+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj62]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj62 op=0 op_state=1 2024-08-18T16:39:21.161 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.153+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj62]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj62[0] 2024-08-18T16:39:21.161 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.153+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj23]: success 2024-08-18T16:39:21.161 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.153+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj68 2024-08-18T16:39:21.161 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.153+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj68]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj68 op=0 op_state=1 2024-08-18T16:39:21.161 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.153+0000 7f818ca10880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj68]: : injecting data sync error on key=obj68 2024-08-18T16:39:21.161 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.153+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj68]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:39:21.161 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.153+0000 7f818ca10880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj68]: ERROR: failed to sync object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj68 2024-08-18T16:39:21.161 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.153+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj148]: success 2024-08-18T16:39:21.161 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.153+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj51]: success 2024-08-18T16:39:21.161 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.153+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj69 2024-08-18T16:39:21.162 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.153+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj69]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj69 op=0 op_state=1 2024-08-18T16:39:21.162 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.153+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj69]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj69[0] 2024-08-18T16:39:21.162 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.153+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj74 2024-08-18T16:39:21.162 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.153+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj74]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj74 op=0 op_state=1 2024-08-18T16:39:21.162 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.153+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj74]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj74[0] 2024-08-18T16:39:21.162 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.153+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj124]: success 2024-08-18T16:39:21.162 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.153+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj76 2024-08-18T16:39:21.162 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.153+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj76]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj76 op=0 op_state=1 2024-08-18T16:39:21.162 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.153+0000 7f818ca10880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj76]: : injecting data sync error on key=obj76 2024-08-18T16:39:21.162 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.153+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj76]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:39:21.163 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.153+0000 7f818ca10880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj76]: ERROR: failed to sync object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj76 2024-08-18T16:39:21.163 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.154+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj31]: success 2024-08-18T16:39:21.163 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.154+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj85 2024-08-18T16:39:21.163 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.154+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj85]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj85 op=0 op_state=1 2024-08-18T16:39:21.163 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.154+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj85]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj85[0] 2024-08-18T16:39:21.163 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.154+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj37]: success 2024-08-18T16:39:21.163 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.154+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj87 2024-08-18T16:39:21.163 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.154+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj87]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj87 op=0 op_state=1 2024-08-18T16:39:21.163 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.154+0000 7f818ca10880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj87]: : injecting data sync error on key=obj87 2024-08-18T16:39:21.163 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.154+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj87]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:39:21.163 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.154+0000 7f818ca10880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj87]: ERROR: failed to sync object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj87 2024-08-18T16:39:21.164 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.154+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj55]: success 2024-08-18T16:39:21.164 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.154+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj92 2024-08-18T16:39:21.164 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.154+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj92]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj92 op=0 op_state=1 2024-08-18T16:39:21.164 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.154+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj92]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj92[0] 2024-08-18T16:39:21.164 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.154+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj38]: success 2024-08-18T16:39:21.164 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.154+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj93 2024-08-18T16:39:21.164 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.154+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj93]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj93 op=0 op_state=1 2024-08-18T16:39:21.164 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.154+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj93]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj93[0] 2024-08-18T16:39:21.164 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.154+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj33]: success 2024-08-18T16:39:21.164 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.154+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj95 2024-08-18T16:39:21.164 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.154+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj95]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj95 op=0 op_state=1 2024-08-18T16:39:21.164 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.154+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj95]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj95[0] 2024-08-18T16:39:21.165 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.155+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: a sync operation returned error 2024-08-18T16:39:21.165 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.155+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj19]: success 2024-08-18T16:39:21.165 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.155+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj145]: success 2024-08-18T16:39:21.165 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.155+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj185]: success 2024-08-18T16:39:21.165 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.155+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj57]: success 2024-08-18T16:39:21.165 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.155+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj58]: success 2024-08-18T16:39:21.165 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.155+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj46]: success 2024-08-18T16:39:21.165 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.156+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj62]: success 2024-08-18T16:39:21.165 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.156+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj74]: success 2024-08-18T16:39:21.165 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.156+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: a sync operation returned error 2024-08-18T16:39:21.165 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.156+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj69]: success 2024-08-18T16:39:21.165 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.156+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: a sync operation returned error 2024-08-18T16:39:21.165 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.157+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj93]: success 2024-08-18T16:39:21.166 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.157+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj92]: success 2024-08-18T16:39:21.166 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.157+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj95]: success 2024-08-18T16:39:21.166 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.157+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj85]: success 2024-08-18T16:39:21.166 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.158+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj22]: success 2024-08-18T16:39:21.166 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.159+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj155]: success 2024-08-18T16:39:21.166 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.159+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj34]: success 2024-08-18T16:39:21.166 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.159+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj29]: success 2024-08-18T16:39:21.194 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.193+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj111]: success 2024-08-18T16:39:21.194 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.193+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: updating marker oid=bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26 marker=obj62 2024-08-18T16:39:21.195 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.194+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: backing out with sync_status=-5 2024-08-18T16:39:21.195 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.194+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: ERROR: full sync failed. error: -5 2024-08-18T16:39:21.196 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.196+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: sync status for source bucket: full. lease is: not taken. stop indications is: 0 2024-08-18T16:39:21.196 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.196+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: waiting on bucket lease 2024-08-18T16:39:21.201 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.200+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: status after acquiring the lock is: full 2024-08-18T16:39:21.213 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.212+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: listing bucket for full sync 2024-08-18T16:39:21.221 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.217+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj11 2024-08-18T16:39:21.221 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.217+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj11]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj11 op=0 op_state=1 2024-08-18T16:39:21.221 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.217+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj11]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj11[0] 2024-08-18T16:39:21.221 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.217+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj111 2024-08-18T16:39:21.221 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.217+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj111]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj111 op=0 op_state=1 2024-08-18T16:39:21.221 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.217+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj111]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj111[0] 2024-08-18T16:39:21.221 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.217+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj115 2024-08-18T16:39:21.221 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.218+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj115]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj115 op=0 op_state=1 2024-08-18T16:39:21.221 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.218+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj115]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj115[0] 2024-08-18T16:39:21.221 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.218+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj116 2024-08-18T16:39:21.222 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.218+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj116]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj116 op=0 op_state=1 2024-08-18T16:39:21.222 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.218+0000 7f818ca10880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj116]: : injecting data sync error on key=obj116 2024-08-18T16:39:21.222 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.218+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj116]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:39:21.222 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.218+0000 7f818ca10880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj116]: ERROR: failed to sync object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj116 2024-08-18T16:39:21.222 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.218+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj124 2024-08-18T16:39:21.222 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.218+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj124]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj124 op=0 op_state=1 2024-08-18T16:39:21.222 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.218+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj124]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj124[0] 2024-08-18T16:39:21.222 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.218+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj145 2024-08-18T16:39:21.222 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.218+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj145]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj145 op=0 op_state=1 2024-08-18T16:39:21.222 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.218+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj145]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj145[0] 2024-08-18T16:39:21.222 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.218+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj148 2024-08-18T16:39:21.223 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.218+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj148]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj148 op=0 op_state=1 2024-08-18T16:39:21.223 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.218+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj148]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj148[0] 2024-08-18T16:39:21.223 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.218+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj155 2024-08-18T16:39:21.223 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.218+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj155]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj155 op=0 op_state=1 2024-08-18T16:39:21.223 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.218+0000 7f818ca10880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj155]: : injecting data sync error on key=obj155 2024-08-18T16:39:21.223 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.218+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj155]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:39:21.223 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.218+0000 7f818ca10880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj155]: ERROR: failed to sync object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj155 2024-08-18T16:39:21.223 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.218+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj17 2024-08-18T16:39:21.223 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.218+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj17]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj17 op=0 op_state=1 2024-08-18T16:39:21.223 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.218+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj17]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj17[0] 2024-08-18T16:39:21.223 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.218+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj185 2024-08-18T16:39:21.223 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.218+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj185]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj185 op=0 op_state=1 2024-08-18T16:39:21.223 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.218+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj185]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj185[0] 2024-08-18T16:39:21.224 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.218+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj19 2024-08-18T16:39:21.224 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.218+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj19]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj19 op=0 op_state=1 2024-08-18T16:39:21.224 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.218+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj19]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj19[0] 2024-08-18T16:39:21.224 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.218+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj22 2024-08-18T16:39:21.224 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.218+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj22]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj22 op=0 op_state=1 2024-08-18T16:39:21.224 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.218+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj22]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj22[0] 2024-08-18T16:39:21.224 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.218+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj23 2024-08-18T16:39:21.224 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.218+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj23]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj23 op=0 op_state=1 2024-08-18T16:39:21.224 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.218+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj23]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj23[0] 2024-08-18T16:39:21.224 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.218+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj29 2024-08-18T16:39:21.224 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.218+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj29]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj29 op=0 op_state=1 2024-08-18T16:39:21.225 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.218+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj29]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj29[0] 2024-08-18T16:39:21.225 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.218+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj31 2024-08-18T16:39:21.225 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.218+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj31]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj31 op=0 op_state=1 2024-08-18T16:39:21.225 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.218+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj31]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj31[0] 2024-08-18T16:39:21.225 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.218+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj33 2024-08-18T16:39:21.225 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.218+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj33]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj33 op=0 op_state=1 2024-08-18T16:39:21.225 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.218+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj33]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj33[0] 2024-08-18T16:39:21.225 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.218+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj34 2024-08-18T16:39:21.225 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.218+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj34]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj34 op=0 op_state=1 2024-08-18T16:39:21.225 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.218+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj34]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj34[0] 2024-08-18T16:39:21.225 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.218+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj37 2024-08-18T16:39:21.225 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.218+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj37]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj37 op=0 op_state=1 2024-08-18T16:39:21.226 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.218+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj37]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj37[0] 2024-08-18T16:39:21.226 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.218+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj38 2024-08-18T16:39:21.226 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.218+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj38]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj38 op=0 op_state=1 2024-08-18T16:39:21.226 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.218+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj38]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj38[0] 2024-08-18T16:39:21.226 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.218+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj44 2024-08-18T16:39:21.226 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.218+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj44]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj44 op=0 op_state=1 2024-08-18T16:39:21.226 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.218+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj44]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj44[0] 2024-08-18T16:39:21.226 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.218+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj46 2024-08-18T16:39:21.226 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.218+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj46]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj46 op=0 op_state=1 2024-08-18T16:39:21.226 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.218+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj46]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj46[0] 2024-08-18T16:39:21.226 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.220+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj115]: success 2024-08-18T16:39:21.227 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.220+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj51 2024-08-18T16:39:21.227 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.220+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj51]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj51 op=0 op_state=1 2024-08-18T16:39:21.227 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.220+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj51]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj51[0] 2024-08-18T16:39:21.232 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.220+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: a sync operation returned error 2024-08-18T16:39:21.232 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.220+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj55 2024-08-18T16:39:21.232 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.220+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj55]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj55 op=0 op_state=1 2024-08-18T16:39:21.232 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.220+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj55]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj55[0] 2024-08-18T16:39:21.232 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.220+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj111]: success 2024-08-18T16:39:21.232 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.220+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj57 2024-08-18T16:39:21.232 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.220+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj57]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj57 op=0 op_state=1 2024-08-18T16:39:21.232 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.220+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj57]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj57[0] 2024-08-18T16:39:21.233 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.220+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj124]: success 2024-08-18T16:39:21.233 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.220+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj58 2024-08-18T16:39:21.233 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.220+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj58]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj58 op=0 op_state=1 2024-08-18T16:39:21.233 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.220+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj58]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj58[0] 2024-08-18T16:39:21.233 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.221+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: a sync operation returned error 2024-08-18T16:39:21.233 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.221+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj62 2024-08-18T16:39:21.233 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.221+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj62]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj62 op=0 op_state=1 2024-08-18T16:39:21.233 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.221+0000 7f818ca10880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj62]: : injecting data sync error on key=obj62 2024-08-18T16:39:21.233 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.221+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj62]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:39:21.233 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.221+0000 7f818ca10880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj62]: ERROR: failed to sync object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj62 2024-08-18T16:39:21.233 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.221+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj11]: success 2024-08-18T16:39:21.233 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.221+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj68 2024-08-18T16:39:21.233 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.221+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj68]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj68 op=0 op_state=1 2024-08-18T16:39:21.234 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.221+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj68]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj68[0] 2024-08-18T16:39:21.234 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.222+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: a sync operation returned error 2024-08-18T16:39:21.234 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.222+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj69 2024-08-18T16:39:21.234 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.222+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj69]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj69 op=0 op_state=1 2024-08-18T16:39:21.234 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.222+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj69]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj69[0] 2024-08-18T16:39:21.234 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.222+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj17]: success 2024-08-18T16:39:21.234 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.222+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj74 2024-08-18T16:39:21.234 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.222+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj74]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj74 op=0 op_state=1 2024-08-18T16:39:21.234 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.222+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj74]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj74[0] 2024-08-18T16:39:21.234 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.222+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj22]: success 2024-08-18T16:39:21.235 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.222+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj76 2024-08-18T16:39:21.235 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.222+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj76]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj76 op=0 op_state=1 2024-08-18T16:39:21.235 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.222+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj76]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj76[0] 2024-08-18T16:39:21.235 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.222+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj185]: success 2024-08-18T16:39:21.235 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.222+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj85 2024-08-18T16:39:21.235 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.222+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj85]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj85 op=0 op_state=1 2024-08-18T16:39:21.235 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.222+0000 7f818ca10880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj85]: : injecting data sync error on key=obj85 2024-08-18T16:39:21.235 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.223+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj85]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:39:21.235 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.223+0000 7f818ca10880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj85]: ERROR: failed to sync object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj85 2024-08-18T16:39:21.235 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.223+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj23]: success 2024-08-18T16:39:21.235 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.223+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj87 2024-08-18T16:39:21.235 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.223+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj87]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj87 op=0 op_state=1 2024-08-18T16:39:21.235 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.223+0000 7f818ca10880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj87]: : injecting data sync error on key=obj87 2024-08-18T16:39:21.236 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.223+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj87]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:39:21.236 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.223+0000 7f818ca10880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj87]: ERROR: failed to sync object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj87 2024-08-18T16:39:21.236 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.223+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj34]: success 2024-08-18T16:39:21.236 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.223+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj92 2024-08-18T16:39:21.236 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.223+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj92]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj92 op=0 op_state=1 2024-08-18T16:39:21.236 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.223+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj92]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj92[0] 2024-08-18T16:39:21.236 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.223+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj33]: success 2024-08-18T16:39:21.236 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.223+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj37]: success 2024-08-18T16:39:21.236 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.223+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj93 2024-08-18T16:39:21.236 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.223+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj93]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj93 op=0 op_state=1 2024-08-18T16:39:21.236 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.223+0000 7f818ca10880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj93]: : injecting data sync error on key=obj93 2024-08-18T16:39:21.236 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.223+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj93]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:39:21.237 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.223+0000 7f818ca10880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj93]: ERROR: failed to sync object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj93 2024-08-18T16:39:21.237 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.223+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj95 2024-08-18T16:39:21.237 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.223+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj95]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj95 op=0 op_state=1 2024-08-18T16:39:21.237 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.223+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj95]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj95[0] 2024-08-18T16:39:21.237 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.223+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj31]: success 2024-08-18T16:39:21.237 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.223+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj38]: success 2024-08-18T16:39:21.237 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.223+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj19]: success 2024-08-18T16:39:21.237 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.224+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj55]: success 2024-08-18T16:39:21.237 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.224+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj51]: success 2024-08-18T16:39:21.237 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.224+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj58]: success 2024-08-18T16:39:21.237 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.224+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj29]: success 2024-08-18T16:39:21.237 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.224+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj46]: success 2024-08-18T16:39:21.238 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.224+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj145]: success 2024-08-18T16:39:21.238 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.224+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj57]: success 2024-08-18T16:39:21.238 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.224+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: a sync operation returned error 2024-08-18T16:39:21.238 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.225+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj44]: success 2024-08-18T16:39:21.238 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.225+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj74]: success 2024-08-18T16:39:21.238 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.225+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: a sync operation returned error 2024-08-18T16:39:21.238 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.225+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj148]: success 2024-08-18T16:39:21.238 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.225+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj76]: success 2024-08-18T16:39:21.238 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.225+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: a sync operation returned error 2024-08-18T16:39:21.238 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.225+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj68]: success 2024-08-18T16:39:21.238 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.226+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj92]: success 2024-08-18T16:39:21.238 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.226+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj69]: success 2024-08-18T16:39:21.239 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.226+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj95]: success 2024-08-18T16:39:21.239 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.226+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: updating marker oid=bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26 marker=obj115 2024-08-18T16:39:21.239 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.227+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: backing out with sync_status=-5 2024-08-18T16:39:21.239 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.227+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: ERROR: full sync failed. error: -5 2024-08-18T16:39:21.239 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.229+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: sync status for source bucket: full. lease is: not taken. stop indications is: 0 2024-08-18T16:39:21.239 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.229+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: waiting on bucket lease 2024-08-18T16:39:21.274 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.273+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: status after acquiring the lock is: full 2024-08-18T16:39:21.286 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.285+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: listing bucket for full sync 2024-08-18T16:39:21.294 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.290+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj11 2024-08-18T16:39:21.294 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.290+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj11]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj11 op=0 op_state=1 2024-08-18T16:39:21.294 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.290+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj11]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj11[0] 2024-08-18T16:39:21.295 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.290+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj111 2024-08-18T16:39:21.295 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.290+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj111]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj111 op=0 op_state=1 2024-08-18T16:39:21.295 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.290+0000 7f818ca10880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj111]: : injecting data sync error on key=obj111 2024-08-18T16:39:21.295 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.290+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj111]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:39:21.295 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.290+0000 7f818ca10880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj111]: ERROR: failed to sync object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj111 2024-08-18T16:39:21.295 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.290+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj115 2024-08-18T16:39:21.295 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.290+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj115]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj115 op=0 op_state=1 2024-08-18T16:39:21.295 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.290+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj115]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj115[0] 2024-08-18T16:39:21.295 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.291+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj116 2024-08-18T16:39:21.295 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.291+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj116]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj116 op=0 op_state=1 2024-08-18T16:39:21.295 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.291+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj116]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj116[0] 2024-08-18T16:39:21.295 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.291+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj124 2024-08-18T16:39:21.296 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.291+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj124]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj124 op=0 op_state=1 2024-08-18T16:39:21.296 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.291+0000 7f818ca10880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj124]: : injecting data sync error on key=obj124 2024-08-18T16:39:21.296 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.291+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj124]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:39:21.296 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.291+0000 7f818ca10880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj124]: ERROR: failed to sync object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj124 2024-08-18T16:39:21.296 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.291+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj145 2024-08-18T16:39:21.296 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.291+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj145]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj145 op=0 op_state=1 2024-08-18T16:39:21.296 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.291+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj145]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj145[0] 2024-08-18T16:39:21.296 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.291+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj148 2024-08-18T16:39:21.296 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.291+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj148]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj148 op=0 op_state=1 2024-08-18T16:39:21.296 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.291+0000 7f818ca10880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj148]: : injecting data sync error on key=obj148 2024-08-18T16:39:21.297 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.291+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj148]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:39:21.297 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.291+0000 7f818ca10880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj148]: ERROR: failed to sync object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj148 2024-08-18T16:39:21.297 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.291+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj155 2024-08-18T16:39:21.297 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.291+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj155]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj155 op=0 op_state=1 2024-08-18T16:39:21.297 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.291+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj155]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj155[0] 2024-08-18T16:39:21.297 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.291+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj17 2024-08-18T16:39:21.297 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.291+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj17]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj17 op=0 op_state=1 2024-08-18T16:39:21.297 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.291+0000 7f818ca10880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj17]: : injecting data sync error on key=obj17 2024-08-18T16:39:21.297 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.291+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj17]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:39:21.297 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.291+0000 7f818ca10880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj17]: ERROR: failed to sync object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj17 2024-08-18T16:39:21.297 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.291+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj185 2024-08-18T16:39:21.298 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.291+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj185]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj185 op=0 op_state=1 2024-08-18T16:39:21.298 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.291+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj185]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj185[0] 2024-08-18T16:39:21.298 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.291+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj19 2024-08-18T16:39:21.298 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.291+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj19]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj19 op=0 op_state=1 2024-08-18T16:39:21.298 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.291+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj19]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj19[0] 2024-08-18T16:39:21.298 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.291+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj22 2024-08-18T16:39:21.298 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.291+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj22]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj22 op=0 op_state=1 2024-08-18T16:39:21.298 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.291+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj22]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj22[0] 2024-08-18T16:39:21.298 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.291+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj23 2024-08-18T16:39:21.298 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.291+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj23]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj23 op=0 op_state=1 2024-08-18T16:39:21.298 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.291+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj23]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj23[0] 2024-08-18T16:39:21.298 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.291+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj29 2024-08-18T16:39:21.298 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.291+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj29]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj29 op=0 op_state=1 2024-08-18T16:39:21.299 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.291+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj29]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj29[0] 2024-08-18T16:39:21.299 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.291+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj31 2024-08-18T16:39:21.299 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.291+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj31]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj31 op=0 op_state=1 2024-08-18T16:39:21.299 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.291+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj31]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj31[0] 2024-08-18T16:39:21.299 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.292+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj33 2024-08-18T16:39:21.299 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.292+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj33]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj33 op=0 op_state=1 2024-08-18T16:39:21.299 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.292+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj33]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj33[0] 2024-08-18T16:39:21.299 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.292+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj34 2024-08-18T16:39:21.299 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.292+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj34]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj34 op=0 op_state=1 2024-08-18T16:39:21.299 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.292+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj34]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj34[0] 2024-08-18T16:39:21.299 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.292+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj37 2024-08-18T16:39:21.299 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.292+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj37]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj37 op=0 op_state=1 2024-08-18T16:39:21.299 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.292+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj37]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj37[0] 2024-08-18T16:39:21.300 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.292+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj38 2024-08-18T16:39:21.300 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.292+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj38]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj38 op=0 op_state=1 2024-08-18T16:39:21.300 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.292+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj38]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj38[0] 2024-08-18T16:39:21.300 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.292+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj44 2024-08-18T16:39:21.300 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.292+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj44]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj44 op=0 op_state=1 2024-08-18T16:39:21.300 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.292+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj44]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj44[0] 2024-08-18T16:39:21.300 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.292+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj46 2024-08-18T16:39:21.300 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.292+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj46]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj46 op=0 op_state=1 2024-08-18T16:39:21.300 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.292+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj46]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj46[0] 2024-08-18T16:39:21.300 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.293+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj116]: success 2024-08-18T16:39:21.300 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.293+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj11]: success 2024-08-18T16:39:21.300 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.293+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj51 2024-08-18T16:39:21.301 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.293+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj51]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj51 op=0 op_state=1 2024-08-18T16:39:21.301 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.293+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj51]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj51[0] 2024-08-18T16:39:21.301 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.293+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj55 2024-08-18T16:39:21.301 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.293+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj55]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj55 op=0 op_state=1 2024-08-18T16:39:21.301 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.293+0000 7f818ca10880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj55]: : injecting data sync error on key=obj55 2024-08-18T16:39:21.301 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.293+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj55]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:39:21.301 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.293+0000 7f818ca10880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj55]: ERROR: failed to sync object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj55 2024-08-18T16:39:21.301 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.293+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: a sync operation returned error 2024-08-18T16:39:21.301 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.293+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj57 2024-08-18T16:39:21.301 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.293+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj57]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj57 op=0 op_state=1 2024-08-18T16:39:21.301 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.293+0000 7f818ca10880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj57]: : injecting data sync error on key=obj57 2024-08-18T16:39:21.301 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.293+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj57]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:39:21.301 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.293+0000 7f818ca10880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj57]: ERROR: failed to sync object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj57 2024-08-18T16:39:21.307 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.294+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: a sync operation returned error 2024-08-18T16:39:21.307 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.294+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj58 2024-08-18T16:39:21.307 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.294+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj58]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj58 op=0 op_state=1 2024-08-18T16:39:21.307 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.294+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj58]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj58[0] 2024-08-18T16:39:21.307 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.294+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj115]: success 2024-08-18T16:39:21.307 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.294+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj62 2024-08-18T16:39:21.307 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.294+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj62]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj62 op=0 op_state=1 2024-08-18T16:39:21.307 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.294+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj62]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj62[0] 2024-08-18T16:39:21.308 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.294+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj19]: success 2024-08-18T16:39:21.308 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.294+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj145]: success 2024-08-18T16:39:21.308 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.294+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj22]: success 2024-08-18T16:39:21.308 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.294+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj68 2024-08-18T16:39:21.308 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.294+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj68]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj68 op=0 op_state=1 2024-08-18T16:39:21.308 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.294+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj68]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj68[0] 2024-08-18T16:39:21.308 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.294+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj69 2024-08-18T16:39:21.308 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.294+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj69]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj69 op=0 op_state=1 2024-08-18T16:39:21.308 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.294+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj69]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj69[0] 2024-08-18T16:39:21.308 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.295+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj74 2024-08-18T16:39:21.308 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.295+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj74]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj74 op=0 op_state=1 2024-08-18T16:39:21.308 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.295+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj74]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj74[0] 2024-08-18T16:39:21.309 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.295+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: a sync operation returned error 2024-08-18T16:39:21.309 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.295+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: a sync operation returned error 2024-08-18T16:39:21.309 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.295+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj76 2024-08-18T16:39:21.309 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.295+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj76]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj76 op=0 op_state=1 2024-08-18T16:39:21.309 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.295+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj76]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj76[0] 2024-08-18T16:39:21.309 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.295+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj23]: success 2024-08-18T16:39:21.309 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.295+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj29]: success 2024-08-18T16:39:21.309 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.295+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj85 2024-08-18T16:39:21.309 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.295+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj85]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj85 op=0 op_state=1 2024-08-18T16:39:21.309 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.295+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj85]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj85[0] 2024-08-18T16:39:21.309 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.295+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj87 2024-08-18T16:39:21.310 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.295+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj87]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj87 op=0 op_state=1 2024-08-18T16:39:21.310 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.295+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj38]: success 2024-08-18T16:39:21.310 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.295+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj87]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj87[0] 2024-08-18T16:39:21.310 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.295+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj44]: success 2024-08-18T16:39:21.310 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.295+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj92 2024-08-18T16:39:21.310 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.295+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj92]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj92 op=0 op_state=1 2024-08-18T16:39:21.310 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.295+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj92]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj92[0] 2024-08-18T16:39:21.310 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.295+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj93 2024-08-18T16:39:21.310 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.295+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj93]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj93 op=0 op_state=1 2024-08-18T16:39:21.310 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.295+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj93]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj93[0] 2024-08-18T16:39:21.310 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.295+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj95 2024-08-18T16:39:21.311 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.296+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj95]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj95 op=0 op_state=1 2024-08-18T16:39:21.311 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.296+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj95]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj95[0] 2024-08-18T16:39:21.311 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.296+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: a sync operation returned error 2024-08-18T16:39:21.311 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.296+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj46]: success 2024-08-18T16:39:21.311 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.296+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: a sync operation returned error 2024-08-18T16:39:21.311 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.296+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj185]: success 2024-08-18T16:39:21.311 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.296+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj34]: success 2024-08-18T16:39:21.311 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.297+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj31]: success 2024-08-18T16:39:21.311 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.298+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj155]: success 2024-08-18T16:39:21.311 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.298+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj33]: success 2024-08-18T16:39:21.311 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.298+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj58]: success 2024-08-18T16:39:21.311 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.298+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj51]: success 2024-08-18T16:39:21.311 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.298+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj68]: success 2024-08-18T16:39:21.312 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.298+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj37]: success 2024-08-18T16:39:21.312 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.298+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj87]: success 2024-08-18T16:39:21.312 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.298+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj62]: success 2024-08-18T16:39:21.312 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.298+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj74]: success 2024-08-18T16:39:21.312 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.298+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj95]: success 2024-08-18T16:39:21.312 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.298+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj76]: success 2024-08-18T16:39:21.312 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.298+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj85]: success 2024-08-18T16:39:21.312 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.298+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj69]: success 2024-08-18T16:39:21.312 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.299+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj92]: success 2024-08-18T16:39:21.312 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.299+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj93]: success 2024-08-18T16:39:21.312 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.299+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: updating marker oid=bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26 marker=obj11 2024-08-18T16:39:21.313 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.300+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: backing out with sync_status=-5 2024-08-18T16:39:21.313 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.300+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: ERROR: full sync failed. error: -5 2024-08-18T16:39:21.313 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.301+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: sync status for source bucket: full. lease is: not taken. stop indications is: 0 2024-08-18T16:39:21.313 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.301+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: waiting on bucket lease 2024-08-18T16:39:21.347 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.346+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: status after acquiring the lock is: full 2024-08-18T16:39:21.359 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.358+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: listing bucket for full sync 2024-08-18T16:39:21.367 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.363+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj11 2024-08-18T16:39:21.367 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.363+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj11]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj11 op=0 op_state=1 2024-08-18T16:39:21.367 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.363+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj11]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj11[0] 2024-08-18T16:39:21.367 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.363+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj111 2024-08-18T16:39:21.367 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.363+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj111]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj111 op=0 op_state=1 2024-08-18T16:39:21.367 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.363+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj111]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj111[0] 2024-08-18T16:39:21.367 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.363+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj115 2024-08-18T16:39:21.367 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.363+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj115]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj115 op=0 op_state=1 2024-08-18T16:39:21.367 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.363+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj115]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj115[0] 2024-08-18T16:39:21.367 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.364+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj116 2024-08-18T16:39:21.367 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.364+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj116]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj116 op=0 op_state=1 2024-08-18T16:39:21.368 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.364+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj116]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj116[0] 2024-08-18T16:39:21.368 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.364+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj124 2024-08-18T16:39:21.368 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.364+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj124]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj124 op=0 op_state=1 2024-08-18T16:39:21.368 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.364+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj124]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj124[0] 2024-08-18T16:39:21.368 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.364+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj145 2024-08-18T16:39:21.368 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.364+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj145]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj145 op=0 op_state=1 2024-08-18T16:39:21.368 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.364+0000 7f818ca10880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj145]: : injecting data sync error on key=obj145 2024-08-18T16:39:21.368 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.364+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj145]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:39:21.368 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.364+0000 7f818ca10880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj145]: ERROR: failed to sync object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj145 2024-08-18T16:39:21.368 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.364+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj148 2024-08-18T16:39:21.368 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.364+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj148]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj148 op=0 op_state=1 2024-08-18T16:39:21.368 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.364+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj148]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj148[0] 2024-08-18T16:39:21.369 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.364+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj155 2024-08-18T16:39:21.369 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.364+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj155]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj155 op=0 op_state=1 2024-08-18T16:39:21.369 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.364+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj155]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj155[0] 2024-08-18T16:39:21.369 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.364+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj17 2024-08-18T16:39:21.369 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.364+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj17]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj17 op=0 op_state=1 2024-08-18T16:39:21.369 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.364+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj17]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj17[0] 2024-08-18T16:39:21.369 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.364+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj185 2024-08-18T16:39:21.369 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.364+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj185]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj185 op=0 op_state=1 2024-08-18T16:39:21.369 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.364+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj185]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj185[0] 2024-08-18T16:39:21.369 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.364+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj19 2024-08-18T16:39:21.369 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.364+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj19]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj19 op=0 op_state=1 2024-08-18T16:39:21.369 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.364+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj19]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj19[0] 2024-08-18T16:39:21.370 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.364+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj22 2024-08-18T16:39:21.370 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.364+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj22]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj22 op=0 op_state=1 2024-08-18T16:39:21.370 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.364+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj22]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj22[0] 2024-08-18T16:39:21.370 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.364+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj23 2024-08-18T16:39:21.370 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.364+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj23]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj23 op=0 op_state=1 2024-08-18T16:39:21.370 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.364+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj23]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj23[0] 2024-08-18T16:39:21.370 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.364+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj29 2024-08-18T16:39:21.370 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.364+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj29]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj29 op=0 op_state=1 2024-08-18T16:39:21.370 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.364+0000 7f818ca10880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj29]: : injecting data sync error on key=obj29 2024-08-18T16:39:21.370 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.364+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj29]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:39:21.370 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.364+0000 7f818ca10880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj29]: ERROR: failed to sync object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj29 2024-08-18T16:39:21.370 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.364+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj31 2024-08-18T16:39:21.371 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.364+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj31]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj31 op=0 op_state=1 2024-08-18T16:39:21.371 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.364+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj31]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj31[0] 2024-08-18T16:39:21.371 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.364+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj33 2024-08-18T16:39:21.371 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.364+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj33]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj33 op=0 op_state=1 2024-08-18T16:39:21.371 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.364+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj33]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj33[0] 2024-08-18T16:39:21.371 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.364+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj34 2024-08-18T16:39:21.371 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.364+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj34]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj34 op=0 op_state=1 2024-08-18T16:39:21.371 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.364+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj34]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj34[0] 2024-08-18T16:39:21.371 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.364+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj37 2024-08-18T16:39:21.371 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.364+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj37]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj37 op=0 op_state=1 2024-08-18T16:39:21.371 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.364+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj37]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj37[0] 2024-08-18T16:39:21.371 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.364+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj38 2024-08-18T16:39:21.371 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.364+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj38]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj38 op=0 op_state=1 2024-08-18T16:39:21.372 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.364+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj38]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj38[0] 2024-08-18T16:39:21.372 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.364+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj44 2024-08-18T16:39:21.372 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.365+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj44]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj44 op=0 op_state=1 2024-08-18T16:39:21.372 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.365+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj44]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj44[0] 2024-08-18T16:39:21.372 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.365+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj46 2024-08-18T16:39:21.372 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.365+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj46]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj46 op=0 op_state=1 2024-08-18T16:39:21.372 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.365+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj46]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj46[0] 2024-08-18T16:39:21.372 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.366+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj115]: success 2024-08-18T16:39:21.372 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.366+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj51 2024-08-18T16:39:21.372 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.366+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj51]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj51 op=0 op_state=1 2024-08-18T16:39:21.372 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.366+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj51]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj51[0] 2024-08-18T16:39:21.377 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.366+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: a sync operation returned error 2024-08-18T16:39:21.378 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.366+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj55 2024-08-18T16:39:21.378 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.366+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj55]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj55 op=0 op_state=1 2024-08-18T16:39:21.378 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.366+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj55]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj55[0] 2024-08-18T16:39:21.378 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.367+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: a sync operation returned error 2024-08-18T16:39:21.378 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.367+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj57 2024-08-18T16:39:21.378 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.367+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj57]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj57 op=0 op_state=1 2024-08-18T16:39:21.378 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.367+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj57]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj57[0] 2024-08-18T16:39:21.378 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.367+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj148]: success 2024-08-18T16:39:21.378 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.367+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj58 2024-08-18T16:39:21.378 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.367+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj58]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj58 op=0 op_state=1 2024-08-18T16:39:21.378 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.367+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj58]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj58[0] 2024-08-18T16:39:21.378 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.368+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj124]: success 2024-08-18T16:39:21.379 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.368+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj62 2024-08-18T16:39:21.379 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.368+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj62]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj62 op=0 op_state=1 2024-08-18T16:39:21.379 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.368+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj62]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj62[0] 2024-08-18T16:39:21.379 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.368+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj116]: success 2024-08-18T16:39:21.379 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.368+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj68 2024-08-18T16:39:21.379 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.368+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj68]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj68 op=0 op_state=1 2024-08-18T16:39:21.379 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.368+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj68]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj68[0] 2024-08-18T16:39:21.379 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.369+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj17]: success 2024-08-18T16:39:21.379 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.369+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj69 2024-08-18T16:39:21.379 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.369+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj69]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj69 op=0 op_state=1 2024-08-18T16:39:21.379 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.369+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj69]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj69[0] 2024-08-18T16:39:21.379 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.369+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj155]: success 2024-08-18T16:39:21.380 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.369+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj74 2024-08-18T16:39:21.380 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.369+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj74]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj74 op=0 op_state=1 2024-08-18T16:39:21.380 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.369+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj74]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj74[0] 2024-08-18T16:39:21.380 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.369+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj22]: success 2024-08-18T16:39:21.380 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.369+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj76 2024-08-18T16:39:21.380 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.369+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj76]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj76 op=0 op_state=1 2024-08-18T16:39:21.380 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.369+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj76]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj76[0] 2024-08-18T16:39:21.380 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.369+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj111]: success 2024-08-18T16:39:21.380 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.369+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj185]: success 2024-08-18T16:39:21.380 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.369+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj85 2024-08-18T16:39:21.380 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.369+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj85]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj85 op=0 op_state=1 2024-08-18T16:39:21.380 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.369+0000 7f818ca10880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj85]: : injecting data sync error on key=obj85 2024-08-18T16:39:21.380 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.369+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj85]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:39:21.381 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.369+0000 7f818ca10880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj85]: ERROR: failed to sync object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj85 2024-08-18T16:39:21.381 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.369+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj87 2024-08-18T16:39:21.381 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.369+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj87]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj87 op=0 op_state=1 2024-08-18T16:39:21.381 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.369+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj87]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj87[0] 2024-08-18T16:39:21.381 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.369+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj23]: success 2024-08-18T16:39:21.381 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.369+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj44]: success 2024-08-18T16:39:21.381 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.369+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj92 2024-08-18T16:39:21.381 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.369+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj92]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj92 op=0 op_state=1 2024-08-18T16:39:21.381 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.369+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj92]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj92[0] 2024-08-18T16:39:21.381 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.369+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj93 2024-08-18T16:39:21.381 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.369+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj93]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj93 op=0 op_state=1 2024-08-18T16:39:21.381 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.369+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj93]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj93[0] 2024-08-18T16:39:21.382 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.369+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj55]: success 2024-08-18T16:39:21.382 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.369+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj95 2024-08-18T16:39:21.382 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.369+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj95]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj95 op=0 op_state=1 2024-08-18T16:39:21.382 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.369+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj95]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj95[0] 2024-08-18T16:39:21.382 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.369+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj19]: success 2024-08-18T16:39:21.382 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.369+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj38]: success 2024-08-18T16:39:21.382 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.369+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj31]: success 2024-08-18T16:39:21.382 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.370+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj33]: success 2024-08-18T16:39:21.382 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.370+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj37]: success 2024-08-18T16:39:21.382 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.370+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj11]: success 2024-08-18T16:39:21.382 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.370+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: updating marker oid=bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26 marker=obj124 2024-08-18T16:39:21.382 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.370+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj34]: success 2024-08-18T16:39:21.383 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.370+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj57]: success 2024-08-18T16:39:21.383 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.371+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: a sync operation returned error 2024-08-18T16:39:21.383 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.371+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj58]: success 2024-08-18T16:39:21.383 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.371+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj46]: success 2024-08-18T16:39:21.383 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.371+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj62]: success 2024-08-18T16:39:21.383 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.372+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj87]: success 2024-08-18T16:39:21.383 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.372+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj68]: success 2024-08-18T16:39:21.383 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.372+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj74]: success 2024-08-18T16:39:21.383 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.372+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj93]: success 2024-08-18T16:39:21.383 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.372+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj51]: success 2024-08-18T16:39:21.383 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.372+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj69]: success 2024-08-18T16:39:21.383 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.372+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj76]: success 2024-08-18T16:39:21.383 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.373+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj95]: success 2024-08-18T16:39:21.384 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.373+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj92]: success 2024-08-18T16:39:21.384 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.373+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: backing out with sync_status=-5 2024-08-18T16:39:21.384 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.373+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: ERROR: full sync failed. error: -5 2024-08-18T16:39:21.384 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.374+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: sync status for source bucket: full. lease is: not taken. stop indications is: 0 2024-08-18T16:39:21.384 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.374+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: waiting on bucket lease 2024-08-18T16:39:21.420 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.419+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: status after acquiring the lock is: full 2024-08-18T16:39:21.433 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.431+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: listing bucket for full sync 2024-08-18T16:39:21.438 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.436+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj11 2024-08-18T16:39:21.438 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.436+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj11]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj11 op=0 op_state=1 2024-08-18T16:39:21.438 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.436+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj11]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj11[0] 2024-08-18T16:39:21.439 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.436+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj111 2024-08-18T16:39:21.439 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.436+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj111]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj111 op=0 op_state=1 2024-08-18T16:39:21.439 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.436+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj111]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj111[0] 2024-08-18T16:39:21.439 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.437+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj115 2024-08-18T16:39:21.439 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.437+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj115]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj115 op=0 op_state=1 2024-08-18T16:39:21.439 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.437+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj115]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj115[0] 2024-08-18T16:39:21.439 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.437+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj116 2024-08-18T16:39:21.439 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.437+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj116]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj116 op=0 op_state=1 2024-08-18T16:39:21.439 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.437+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj116]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj116[0] 2024-08-18T16:39:21.439 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.437+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj124 2024-08-18T16:39:21.439 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.437+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj124]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj124 op=0 op_state=1 2024-08-18T16:39:21.439 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.437+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj124]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj124[0] 2024-08-18T16:39:21.440 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.437+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj145 2024-08-18T16:39:21.440 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.437+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj145]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj145 op=0 op_state=1 2024-08-18T16:39:21.440 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.437+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj145]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj145[0] 2024-08-18T16:39:21.440 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.437+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj148 2024-08-18T16:39:21.440 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.437+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj148]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj148 op=0 op_state=1 2024-08-18T16:39:21.440 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.437+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj148]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj148[0] 2024-08-18T16:39:21.440 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.437+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj155 2024-08-18T16:39:21.440 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.437+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj155]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj155 op=0 op_state=1 2024-08-18T16:39:21.440 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.437+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj155]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj155[0] 2024-08-18T16:39:21.440 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.437+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj17 2024-08-18T16:39:21.440 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.437+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj17]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj17 op=0 op_state=1 2024-08-18T16:39:21.440 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.437+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj17]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj17[0] 2024-08-18T16:39:21.440 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.437+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj185 2024-08-18T16:39:21.441 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.437+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj185]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj185 op=0 op_state=1 2024-08-18T16:39:21.441 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.437+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj185]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj185[0] 2024-08-18T16:39:21.441 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.437+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj19 2024-08-18T16:39:21.441 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.437+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj19]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj19 op=0 op_state=1 2024-08-18T16:39:21.441 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.437+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj19]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj19[0] 2024-08-18T16:39:21.441 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.437+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj22 2024-08-18T16:39:21.441 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.437+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj22]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj22 op=0 op_state=1 2024-08-18T16:39:21.441 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.437+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj22]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj22[0] 2024-08-18T16:39:21.441 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.437+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj23 2024-08-18T16:39:21.441 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.437+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj23]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj23 op=0 op_state=1 2024-08-18T16:39:21.441 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.437+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj23]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj23[0] 2024-08-18T16:39:21.441 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.437+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj29 2024-08-18T16:39:21.442 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.437+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj29]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj29 op=0 op_state=1 2024-08-18T16:39:21.442 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.437+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj29]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj29[0] 2024-08-18T16:39:21.442 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.437+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj31 2024-08-18T16:39:21.442 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.437+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj31]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj31 op=0 op_state=1 2024-08-18T16:39:21.442 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.437+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj31]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj31[0] 2024-08-18T16:39:21.442 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.437+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj33 2024-08-18T16:39:21.442 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.437+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj33]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj33 op=0 op_state=1 2024-08-18T16:39:21.442 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.437+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj33]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj33[0] 2024-08-18T16:39:21.442 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.437+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj34 2024-08-18T16:39:21.442 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.437+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj34]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj34 op=0 op_state=1 2024-08-18T16:39:21.442 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.437+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj34]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj34[0] 2024-08-18T16:39:21.442 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.437+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj37 2024-08-18T16:39:21.443 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.437+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj37]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj37 op=0 op_state=1 2024-08-18T16:39:21.443 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.437+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj37]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj37[0] 2024-08-18T16:39:21.443 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.437+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj38 2024-08-18T16:39:21.443 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.437+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj38]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj38 op=0 op_state=1 2024-08-18T16:39:21.443 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.437+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj38]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj38[0] 2024-08-18T16:39:21.443 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.437+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj44 2024-08-18T16:39:21.443 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.437+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj44]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj44 op=0 op_state=1 2024-08-18T16:39:21.443 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.437+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj44]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj44[0] 2024-08-18T16:39:21.443 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.437+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj46 2024-08-18T16:39:21.443 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.437+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj46]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj46 op=0 op_state=1 2024-08-18T16:39:21.443 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.437+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj46]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj46[0] 2024-08-18T16:39:21.449 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.439+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj124]: success 2024-08-18T16:39:21.449 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.439+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj51 2024-08-18T16:39:21.449 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.439+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj51]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj51 op=0 op_state=1 2024-08-18T16:39:21.449 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.439+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj51]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj51[0] 2024-08-18T16:39:21.449 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.440+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj115]: success 2024-08-18T16:39:21.449 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.440+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj55 2024-08-18T16:39:21.450 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.440+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj55]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj55 op=0 op_state=1 2024-08-18T16:39:21.450 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.440+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj55]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj55[0] 2024-08-18T16:39:21.450 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.440+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj111]: success 2024-08-18T16:39:21.450 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.440+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj57 2024-08-18T16:39:21.450 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.440+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj57]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj57 op=0 op_state=1 2024-08-18T16:39:21.450 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.440+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj57]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj57[0] 2024-08-18T16:39:21.450 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.440+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj23]: success 2024-08-18T16:39:21.451 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.440+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj58 2024-08-18T16:39:21.451 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.440+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj58]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj58 op=0 op_state=1 2024-08-18T16:39:21.451 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.440+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj58]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj58[0] 2024-08-18T16:39:21.451 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.441+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj19]: success 2024-08-18T16:39:21.451 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.441+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj62 2024-08-18T16:39:21.451 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.441+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj62]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj62 op=0 op_state=1 2024-08-18T16:39:21.451 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.441+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj62]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj62[0] 2024-08-18T16:39:21.451 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.441+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj116]: success 2024-08-18T16:39:21.451 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.441+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj17]: success 2024-08-18T16:39:21.451 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.441+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj68 2024-08-18T16:39:21.451 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.441+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj68]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj68 op=0 op_state=1 2024-08-18T16:39:21.451 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.441+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj68]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj68[0] 2024-08-18T16:39:21.452 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.441+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj69 2024-08-18T16:39:21.452 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.441+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj69]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj69 op=0 op_state=1 2024-08-18T16:39:21.452 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.441+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj69]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj69[0] 2024-08-18T16:39:21.452 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.441+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj29]: success 2024-08-18T16:39:21.452 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.441+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj74 2024-08-18T16:39:21.452 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.441+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj74]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj74 op=0 op_state=1 2024-08-18T16:39:21.452 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.441+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj74]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj74[0] 2024-08-18T16:39:21.452 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.441+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj38]: success 2024-08-18T16:39:21.452 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.441+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj76 2024-08-18T16:39:21.452 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.441+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj76]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj76 op=0 op_state=1 2024-08-18T16:39:21.452 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.441+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj76]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj76[0] 2024-08-18T16:39:21.452 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.441+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj148]: success 2024-08-18T16:39:21.453 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.441+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj85 2024-08-18T16:39:21.453 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.441+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj85]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj85 op=0 op_state=1 2024-08-18T16:39:21.453 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.441+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj85]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj85[0] 2024-08-18T16:39:21.453 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.442+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj145]: success 2024-08-18T16:39:21.453 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.442+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj87 2024-08-18T16:39:21.453 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.442+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj87]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj87 op=0 op_state=1 2024-08-18T16:39:21.453 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.442+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj87]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj87[0] 2024-08-18T16:39:21.453 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.442+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj155]: success 2024-08-18T16:39:21.453 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.442+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj92 2024-08-18T16:39:21.454 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.442+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj92]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj92 op=0 op_state=1 2024-08-18T16:39:21.454 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.442+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj92]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj92[0] 2024-08-18T16:39:21.454 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.442+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj11]: success 2024-08-18T16:39:21.454 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.442+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: updating marker oid=bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26 marker=obj17 2024-08-18T16:39:21.454 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.443+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj31]: success 2024-08-18T16:39:21.454 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.443+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj93 2024-08-18T16:39:21.454 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.443+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj93]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj93 op=0 op_state=1 2024-08-18T16:39:21.454 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.443+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj93]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj93[0] 2024-08-18T16:39:21.454 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.443+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj22]: success 2024-08-18T16:39:21.454 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.443+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj95 2024-08-18T16:39:21.454 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.443+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj95]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj95 op=0 op_state=1 2024-08-18T16:39:21.454 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.443+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj95]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj95[0] 2024-08-18T16:39:21.455 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.443+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj37]: success 2024-08-18T16:39:21.455 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.443+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj185]: success 2024-08-18T16:39:21.455 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.443+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj44]: success 2024-08-18T16:39:21.455 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.443+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj46]: success 2024-08-18T16:39:21.455 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.443+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj34]: success 2024-08-18T16:39:21.455 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.443+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj57]: success 2024-08-18T16:39:21.455 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.443+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj51]: success 2024-08-18T16:39:21.455 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.443+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj69]: success 2024-08-18T16:39:21.455 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.444+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj62]: success 2024-08-18T16:39:21.455 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.444+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj33]: success 2024-08-18T16:39:21.455 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.444+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: updating marker oid=bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26 marker=obj51 2024-08-18T16:39:21.455 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.444+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj74]: success 2024-08-18T16:39:21.455 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.444+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj55]: success 2024-08-18T16:39:21.456 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.444+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj93]: success 2024-08-18T16:39:21.456 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.445+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj85]: success 2024-08-18T16:39:21.456 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.445+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj92]: success 2024-08-18T16:39:21.456 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.445+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj87]: success 2024-08-18T16:39:21.456 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.445+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj58]: success 2024-08-18T16:39:21.456 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.446+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj76]: success 2024-08-18T16:39:21.456 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.447+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj95]: success 2024-08-18T16:39:21.456 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.447+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj68]: success 2024-08-18T16:39:21.456 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.447+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: updating marker oid=bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26 marker=obj95 2024-08-18T16:39:21.456 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.448+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: set bucket state=incremental 2024-08-18T16:39:21.464 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.449+0000 7f818ca10880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: bucket status objv={r=_CCIj9RdpiSQDUxsQ_xdl50U:149,w=:0} 2024-08-18T16:39:21.464 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.450+0000 7f818ca10880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: WARNING: requested sync of past generation 0 < 6, returning success 2024-08-18T16:39:21.465 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.452+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:39:21.465 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.452+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:39:21.465 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.452+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:39:21.465 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.452+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:39:21.465 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.453+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:39:21.465 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.453+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:39:21.465 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.453+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:1]: sync status for source bucket shard: 2 2024-08-18T16:39:21.465 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.453+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:4]: sync status for source bucket shard: 2 2024-08-18T16:39:21.465 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.453+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: sync status for source bucket shard: 2 2024-08-18T16:39:21.465 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.453+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:1]:inc_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:1]: listing bilog for incremental sync; position= 2024-08-18T16:39:21.465 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.453+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:4]:inc_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:4]: listing bilog for incremental sync; position= 2024-08-18T16:39:21.465 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.453+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:inc_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: listing bilog for incremental sync; position= 2024-08-18T16:39:21.466 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.453+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:2]: sync status for source bucket shard: 2 2024-08-18T16:39:21.466 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.453+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:2]:inc_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:2]: listing bilog for incremental sync; position= 2024-08-18T16:39:21.466 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.453+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:39:21.466 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.453+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:39:21.466 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.453+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:39:21.466 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.453+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:39:21.466 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.453+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:3]: sync status for source bucket shard: 2 2024-08-18T16:39:21.466 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.453+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:3]:inc_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:3]: listing bilog for incremental sync; position= 2024-08-18T16:39:21.466 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.453+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:39:21.466 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.453+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:6]: sync status for source bucket shard: 2 2024-08-18T16:39:21.466 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.454+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:6]:inc_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:6]: listing bilog for incremental sync; position=00000000002.82.6 2024-08-18T16:39:21.466 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.454+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:39:21.467 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.454+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:5]: sync status for source bucket shard: 2 2024-08-18T16:39:21.467 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.454+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:5]:inc_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:5]: listing bilog for incremental sync; position= 2024-08-18T16:39:21.467 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.454+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:39:21.467 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.454+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:8]: sync status for source bucket shard: 2 2024-08-18T16:39:21.467 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.454+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:8]:inc_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:8]: listing bilog for incremental sync; position= 2024-08-18T16:39:21.467 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.454+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:7]: sync status for source bucket shard: 2 2024-08-18T16:39:21.467 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.454+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:7]:inc_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:7]: listing bilog for incremental sync; position= 2024-08-18T16:39:21.467 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.454+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:11]: sync status for source bucket shard: 2 2024-08-18T16:39:21.467 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.454+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:11]:inc_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:11]: listing bilog for incremental sync; position= 2024-08-18T16:39:21.467 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.454+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:39:21.467 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.454+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:39:21.467 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.454+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:9]: sync status for source bucket shard: 2 2024-08-18T16:39:21.467 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.454+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:10]: sync status for source bucket shard: 2 2024-08-18T16:39:21.468 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.454+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:12]: sync status for source bucket shard: 2 2024-08-18T16:39:21.468 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.454+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:9]:inc_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:9]: listing bilog for incremental sync; position= 2024-08-18T16:39:21.468 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.454+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:10]:inc_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:10]: listing bilog for incremental sync; position= 2024-08-18T16:39:21.468 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.454+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:12]:inc_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:12]: listing bilog for incremental sync; position=00000000002.191.6 2024-08-18T16:39:21.468 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.455+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:39:21.468 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.455+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:39:21.468 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.455+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:13]: sync status for source bucket shard: 2 2024-08-18T16:39:21.468 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.455+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:14]: sync status for source bucket shard: 2 2024-08-18T16:39:21.468 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.455+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:13]:inc_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:13]: listing bilog for incremental sync; position= 2024-08-18T16:39:21.468 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.455+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:14]:inc_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:14]: listing bilog for incremental sync; position= 2024-08-18T16:39:21.468 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.455+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:39:21.468 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.455+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:39:21.468 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.455+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:15]: sync status for source bucket shard: 2 2024-08-18T16:39:21.469 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.455+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:15]:inc_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:15]: listing bilog for incremental sync; position= 2024-08-18T16:39:21.469 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.456+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:18]: sync status for source bucket shard: 2 2024-08-18T16:39:21.469 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.456+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:18]:inc_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:18]: listing bilog for incremental sync; position= 2024-08-18T16:39:21.469 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.456+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:17]: sync status for source bucket shard: 2 2024-08-18T16:39:21.469 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.456+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:17]:inc_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:17]: listing bilog for incremental sync; position=00000000002.107.6 2024-08-18T16:39:21.469 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.456+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:19]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:39:21.469 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.456+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:20]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:39:21.469 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.456+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:16]: sync status for source bucket shard: 2 2024-08-18T16:39:21.469 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.456+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:21]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:39:21.469 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.456+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:16]:inc_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:16]: listing bilog for incremental sync; position= 2024-08-18T16:39:21.469 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.456+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:22]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:39:21.469 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.456+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:19]: sync status for source bucket shard: 2 2024-08-18T16:39:21.469 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.456+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:19]:inc_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:19]: listing bilog for incremental sync; position= 2024-08-18T16:39:21.470 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.457+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:24]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:39:21.470 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.457+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:23]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:39:21.470 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.457+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:20]: sync status for source bucket shard: 2 2024-08-18T16:39:21.470 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.457+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:21]: sync status for source bucket shard: 2 2024-08-18T16:39:21.470 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.457+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:20]:inc_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:20]: listing bilog for incremental sync; position= 2024-08-18T16:39:21.470 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.457+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:21]:inc_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:21]: listing bilog for incremental sync; position= 2024-08-18T16:39:21.470 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.457+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:25]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:39:21.470 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.457+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:26]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:39:21.470 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.457+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:22]: sync status for source bucket shard: 2 2024-08-18T16:39:21.470 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.457+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:22]:inc_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:22]: listing bilog for incremental sync; position= 2024-08-18T16:39:21.470 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.457+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:24]: sync status for source bucket shard: 2 2024-08-18T16:39:21.470 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.457+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:28]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:39:21.471 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.457+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:24]:inc_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:24]: listing bilog for incremental sync; position= 2024-08-18T16:39:21.471 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.457+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:23]: sync status for source bucket shard: 2 2024-08-18T16:39:21.471 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.457+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:23]:inc_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:23]: listing bilog for incremental sync; position= 2024-08-18T16:39:21.471 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.458+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:29]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:39:21.471 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.458+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:25]: sync status for source bucket shard: 2 2024-08-18T16:39:21.471 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.458+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:26]: sync status for source bucket shard: 2 2024-08-18T16:39:21.471 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.458+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:27]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:39:21.471 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.458+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:25]:inc_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:25]: listing bilog for incremental sync; position= 2024-08-18T16:39:21.471 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.458+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:26]:inc_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:26]: listing bilog for incremental sync; position= 2024-08-18T16:39:21.471 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.458+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:30]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:39:21.471 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.458+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:28]: sync status for source bucket shard: 2 2024-08-18T16:39:21.471 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.458+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:28]:inc_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:28]: listing bilog for incremental sync; position= 2024-08-18T16:39:21.472 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.458+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:31]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:39:21.472 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.458+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:29]: sync status for source bucket shard: 2 2024-08-18T16:39:21.472 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.458+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:29]:inc_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:29]: listing bilog for incremental sync; position= 2024-08-18T16:39:21.472 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.458+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:27]: sync status for source bucket shard: 2 2024-08-18T16:39:21.472 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.458+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:27]:inc_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:27]: listing bilog for incremental sync; position= 2024-08-18T16:39:21.472 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.459+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:32]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:39:21.472 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.459+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:30]: sync status for source bucket shard: 2 2024-08-18T16:39:21.472 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.459+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:30]:inc_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:30]: listing bilog for incremental sync; position= 2024-08-18T16:39:21.472 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.459+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:33]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:39:21.472 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.459+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:31]: sync status for source bucket shard: 2 2024-08-18T16:39:21.472 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.459+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:31]:inc_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:31]: listing bilog for incremental sync; position= 2024-08-18T16:39:21.472 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.459+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:34]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:39:21.472 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.459+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:32]: sync status for source bucket shard: 2 2024-08-18T16:39:21.473 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.459+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:32]:inc_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:32]: listing bilog for incremental sync; position=00000000001.116.6 2024-08-18T16:39:21.473 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.459+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:35]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:39:21.473 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.459+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:33]: sync status for source bucket shard: 2 2024-08-18T16:39:21.473 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.459+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:33]:inc_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:33]: listing bilog for incremental sync; position= 2024-08-18T16:39:21.473 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.459+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:36]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:39:21.473 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.459+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:34]: sync status for source bucket shard: 2 2024-08-18T16:39:21.473 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.459+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:34]:inc_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:34]: listing bilog for incremental sync; position= 2024-08-18T16:39:21.473 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.460+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:35]: sync status for source bucket shard: 2 2024-08-18T16:39:21.473 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.460+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:35]:inc_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:35]: listing bilog for incremental sync; position= 2024-08-18T16:39:21.473 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.460+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:36]: sync status for source bucket shard: 2 2024-08-18T16:39:21.473 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:21.460+0000 7f818ca10880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:36]:inc_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:36]: listing bilog for incremental sync; position=00000000001.120.6 2024-08-18T16:39:21.523 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'list', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:39:21.523 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'list', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:39:21.523 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket list --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-78 --max-entries 10000 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:39:21.574 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:39:21.574 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:39:21.608 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T16:39:21.609 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:21.609 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj11", 2024-08-18T16:39:21.609 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:21.609 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:21.609 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:39:21.609 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 14 2024-08-18T16:39:21.609 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.609 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:21.609 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:21.609 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:21.609 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:21.609 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 6, 2024-08-18T16:39:21.610 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:23:42.051163Z", 2024-08-18T16:39:21.610 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:21.610 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:21.610 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:21.610 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:21.610 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:21.610 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:21.610 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:21.610 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:21.610 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.610 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.9889212156647452491", 2024-08-18T16:39:21.610 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:21.611 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:21.611 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:21.611 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.611 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:21.611 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj111", 2024-08-18T16:39:21.611 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:21.611 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:21.611 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:39:21.611 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 25 2024-08-18T16:39:21.611 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.611 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:21.611 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:21.611 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:21.611 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:21.612 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-08-18T16:39:21.612 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:36:05.189316Z", 2024-08-18T16:39:21.612 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:21.612 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:21.612 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:21.612 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:21.612 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:21.612 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:21.612 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:21.612 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:21.612 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.612 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.6119517508141186778", 2024-08-18T16:39:21.612 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:21.613 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:21.613 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:21.613 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.613 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:21.613 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj115", 2024-08-18T16:39:21.613 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:21.613 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:21.613 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:39:21.613 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 3 2024-08-18T16:39:21.613 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.613 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:21.613 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:21.613 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:21.613 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:21.614 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 5, 2024-08-18T16:39:21.614 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:27:23.466808Z", 2024-08-18T16:39:21.614 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:21.614 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:21.614 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:21.614 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:21.614 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:21.614 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:21.614 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:21.614 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:21.614 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.614 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.6399818160083281577", 2024-08-18T16:39:21.614 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:21.615 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:21.615 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:21.615 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.615 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:21.615 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj116", 2024-08-18T16:39:21.615 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:21.615 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:21.615 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:39:21.615 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 13 2024-08-18T16:39:21.615 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.615 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:21.615 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:21.615 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:21.615 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:21.616 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 5, 2024-08-18T16:39:21.616 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:30:57.371585Z", 2024-08-18T16:39:21.616 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:21.616 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:21.616 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:21.616 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:21.616 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:21.616 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:21.616 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:21.616 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:21.616 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.616 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.15128885479062847417", 2024-08-18T16:39:21.616 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:21.616 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:21.617 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:21.617 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.617 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:21.617 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj124", 2024-08-18T16:39:21.617 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:21.617 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:21.617 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:39:21.617 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 7 2024-08-18T16:39:21.617 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.617 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:21.617 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:21.617 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:21.617 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:21.618 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 6, 2024-08-18T16:39:21.618 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:33:27.492061Z", 2024-08-18T16:39:21.618 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:21.618 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:21.618 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:21.618 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:21.618 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:21.618 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:21.618 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:21.618 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:21.618 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.618 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.3899074313730904728", 2024-08-18T16:39:21.618 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:21.619 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:21.619 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:21.619 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.619 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:21.619 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj145", 2024-08-18T16:39:21.619 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:21.619 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:21.619 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:39:21.619 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 21 2024-08-18T16:39:21.619 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.619 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:21.619 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:21.619 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:21.619 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:21.620 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:39:21.620 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:30:57.379450Z", 2024-08-18T16:39:21.620 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:21.620 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:21.620 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:21.620 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:21.620 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:21.620 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:21.620 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:21.620 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:21.620 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.620 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.11999609542217578306", 2024-08-18T16:39:21.620 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:21.621 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:21.621 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:21.621 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.621 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:21.621 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj148", 2024-08-18T16:39:21.621 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:21.621 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:21.621 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:39:21.621 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 13 2024-08-18T16:39:21.621 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.621 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:21.621 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:21.621 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:21.622 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:21.622 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 5, 2024-08-18T16:39:21.622 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:36:05.197180Z", 2024-08-18T16:39:21.622 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:21.622 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:21.622 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:21.622 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:21.622 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:21.622 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:21.622 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:21.622 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:21.622 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.622 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.17613569593319029244", 2024-08-18T16:39:21.623 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:21.623 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:21.623 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:21.623 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.623 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:21.623 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj155", 2024-08-18T16:39:21.623 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:21.623 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:21.623 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:39:21.623 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 15 2024-08-18T16:39:21.623 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.623 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:21.623 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:21.623 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:21.624 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:21.624 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:39:21.624 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:33:27.500403Z", 2024-08-18T16:39:21.624 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:21.624 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:21.624 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:21.624 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:21.624 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:21.624 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:21.624 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:21.624 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:21.624 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.624 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.18209861177904032184", 2024-08-18T16:39:21.625 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:21.625 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:21.625 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:21.625 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.625 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:21.625 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj17", 2024-08-18T16:39:21.625 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:21.625 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:21.625 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:39:21.625 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 6 2024-08-18T16:39:21.625 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.625 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:21.625 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:21.625 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:21.626 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:21.626 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 5, 2024-08-18T16:39:21.626 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:24:10.291690Z", 2024-08-18T16:39:21.626 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:21.626 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:21.626 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:21.626 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:21.626 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:21.626 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:21.626 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:21.626 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:21.626 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.626 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.6759974984332877289", 2024-08-18T16:39:21.627 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:21.627 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:21.627 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:21.627 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.627 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:21.627 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj185", 2024-08-18T16:39:21.627 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:21.627 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:21.627 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:39:21.627 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 13 2024-08-18T16:39:21.627 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.627 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:21.627 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:21.627 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:21.628 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:21.628 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-08-18T16:39:21.628 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:36:05.244081Z", 2024-08-18T16:39:21.628 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:21.628 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:21.628 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:21.628 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:21.628 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:21.628 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:21.628 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:21.628 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:21.628 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.628 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.372041751039859441", 2024-08-18T16:39:21.629 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:21.629 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:21.629 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:21.629 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.629 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:21.629 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj19", 2024-08-18T16:39:21.629 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:21.629 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:21.629 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:39:21.629 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 10 2024-08-18T16:39:21.629 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.629 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:21.629 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:21.629 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:21.630 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:21.630 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-08-18T16:39:21.630 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:25:55.960750Z", 2024-08-18T16:39:21.630 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:21.630 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:21.630 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:21.630 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:21.630 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:21.630 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:21.630 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:21.630 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:21.630 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.630 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.3313087035472660803", 2024-08-18T16:39:21.631 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:21.631 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:21.631 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:21.631 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.631 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:21.631 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj22", 2024-08-18T16:39:21.631 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:21.631 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:21.631 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:39:21.631 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 50 2024-08-18T16:39:21.631 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.631 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:21.631 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:21.632 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:21.632 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:21.632 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 5, 2024-08-18T16:39:21.632 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:23:42.058759Z", 2024-08-18T16:39:21.632 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:21.632 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:21.632 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:21.632 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:21.632 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:21.632 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:21.632 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:21.632 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:21.632 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.632 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.10722087597600129896", 2024-08-18T16:39:21.633 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:21.633 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:21.633 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:21.633 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.633 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:21.633 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj23", 2024-08-18T16:39:21.633 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:21.633 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:21.633 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:39:21.633 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 6 2024-08-18T16:39:21.633 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.633 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:21.633 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:21.633 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:21.634 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:21.634 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:39:21.634 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:27:23.361495Z", 2024-08-18T16:39:21.634 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:21.634 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:21.634 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:21.634 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:21.634 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:21.634 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:21.634 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:21.634 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:21.634 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.634 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.12518117691867460845", 2024-08-18T16:39:21.635 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:21.635 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:21.635 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:21.635 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.635 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:21.635 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj29", 2024-08-18T16:39:21.635 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:21.635 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:21.635 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:39:21.635 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 5 2024-08-18T16:39:21.635 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.635 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:21.635 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:21.635 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:21.636 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:21.636 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:39:21.636 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:30:57.311630Z", 2024-08-18T16:39:21.636 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:21.636 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:21.636 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:21.636 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:21.636 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:21.636 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:21.636 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:21.636 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:21.636 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.636 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.9093700191308604259", 2024-08-18T16:39:21.637 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:21.637 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:21.637 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:21.637 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.637 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:21.637 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj31", 2024-08-18T16:39:21.637 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:21.637 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:21.637 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:39:21.637 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 10 2024-08-18T16:39:21.637 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.637 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:21.637 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:21.637 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:21.638 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:21.638 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:39:21.638 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:33:27.426326Z", 2024-08-18T16:39:21.638 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:21.638 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:21.638 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:21.638 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:21.638 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:21.638 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:21.638 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:21.638 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:21.638 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.638 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.4113607296140759139", 2024-08-18T16:39:21.638 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:21.639 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:21.639 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:21.639 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.639 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:21.639 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj33", 2024-08-18T16:39:21.639 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:21.639 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:21.639 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:39:21.639 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 6 2024-08-18T16:39:21.639 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.639 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:21.639 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:21.639 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:21.640 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:21.640 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 6, 2024-08-18T16:39:21.640 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:23:42.064530Z", 2024-08-18T16:39:21.640 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:21.640 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:21.640 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:21.640 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:21.640 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:21.640 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:21.640 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:21.640 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:21.640 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.640 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.15286180760377802005", 2024-08-18T16:39:21.640 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:21.641 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:21.641 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:21.641 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.641 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:21.641 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj34", 2024-08-18T16:39:21.641 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:21.641 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:21.641 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:39:21.641 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 10 2024-08-18T16:39:21.641 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.641 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:21.641 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:21.641 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:21.642 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:21.642 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:39:21.642 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:24:10.298744Z", 2024-08-18T16:39:21.642 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:21.642 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:21.642 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:21.642 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:21.642 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:21.642 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:21.642 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:21.642 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:21.642 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.642 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.1382908489229446564", 2024-08-18T16:39:21.643 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:21.643 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:21.643 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:21.643 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.643 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:21.643 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj37", 2024-08-18T16:39:21.643 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:21.643 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:21.643 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:39:21.643 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 11 2024-08-18T16:39:21.643 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.643 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:21.643 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:21.643 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:21.644 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:21.644 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:39:21.644 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:36:05.133754Z", 2024-08-18T16:39:21.644 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:21.644 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:21.644 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:21.644 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:21.644 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:21.644 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:21.644 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:21.644 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:21.644 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.644 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.10395008102202495793", 2024-08-18T16:39:21.644 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:21.645 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:21.645 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:21.645 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.645 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:21.645 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj38", 2024-08-18T16:39:21.645 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:21.645 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:21.645 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:39:21.645 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 17 2024-08-18T16:39:21.645 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.645 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:21.645 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:21.645 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:21.646 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:21.646 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-08-18T16:39:21.646 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:25:55.967916Z", 2024-08-18T16:39:21.646 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:21.646 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:21.646 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:21.646 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:21.646 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:21.646 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:21.646 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:21.646 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:21.646 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.646 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.3777469865024363097", 2024-08-18T16:39:21.647 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:21.647 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:21.647 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:21.647 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.647 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:21.647 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj44", 2024-08-18T16:39:21.647 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:21.647 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:21.647 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:39:21.647 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 8 2024-08-18T16:39:21.647 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.647 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:21.647 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:21.647 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:21.648 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:21.648 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 6, 2024-08-18T16:39:21.648 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:23:42.115387Z", 2024-08-18T16:39:21.648 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:21.648 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:21.648 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:21.648 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:21.648 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:21.648 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:21.648 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:21.648 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:21.648 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.648 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.2838957566806869052", 2024-08-18T16:39:21.649 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:21.649 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:21.649 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:21.649 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.649 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:21.649 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj46", 2024-08-18T16:39:21.649 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:21.649 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:21.649 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:39:21.649 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 9 2024-08-18T16:39:21.649 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.649 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:21.649 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:21.650 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:21.650 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:21.650 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:39:21.650 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:27:23.407342Z", 2024-08-18T16:39:21.650 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:21.650 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:21.650 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:21.650 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:21.650 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:21.650 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:21.650 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:21.650 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:21.650 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.651 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.7304181808045275938", 2024-08-18T16:39:21.651 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:21.651 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:21.651 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:21.651 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.651 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:21.651 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj51", 2024-08-18T16:39:21.651 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:21.651 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:21.651 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:39:21.651 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 5 2024-08-18T16:39:21.651 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.651 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:21.651 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:21.652 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:21.652 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:21.652 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 6, 2024-08-18T16:39:21.652 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:24:10.305333Z", 2024-08-18T16:39:21.652 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:21.652 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:21.652 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:21.652 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:21.652 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:21.652 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:21.652 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:21.652 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:21.652 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.652 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.2913498005472717509", 2024-08-18T16:39:21.653 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:21.653 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:21.653 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:21.653 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.653 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:21.653 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj55", 2024-08-18T16:39:21.653 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:21.653 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:21.653 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:39:21.653 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 7 2024-08-18T16:39:21.653 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.653 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:21.653 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:21.654 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:21.654 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:21.654 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 6, 2024-08-18T16:39:21.654 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:23:42.122245Z", 2024-08-18T16:39:21.654 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:21.654 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:21.654 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:21.654 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:21.654 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:21.654 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:21.654 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:21.654 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:21.654 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.655 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.5436641643066532846", 2024-08-18T16:39:21.655 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:21.655 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:21.655 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:21.655 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.655 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:21.655 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj57", 2024-08-18T16:39:21.655 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:21.655 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:21.655 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:39:21.655 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 29 2024-08-18T16:39:21.655 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.655 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:21.655 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:21.656 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:21.656 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:21.656 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:39:21.656 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:25:55.973669Z", 2024-08-18T16:39:21.656 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:21.656 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:21.656 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:21.656 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:21.656 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:21.656 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:21.656 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:21.656 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:21.656 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.656 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.9901401109371881948", 2024-08-18T16:39:21.657 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:21.657 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:21.657 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:21.657 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.657 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:21.657 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj58", 2024-08-18T16:39:21.657 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:21.657 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:21.657 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:39:21.657 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 7 2024-08-18T16:39:21.657 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.657 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:21.657 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:21.657 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:21.658 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:21.658 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 6, 2024-08-18T16:39:21.658 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:30:57.319413Z", 2024-08-18T16:39:21.658 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:21.658 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:21.658 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:21.658 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:21.658 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:21.658 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:21.658 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:21.658 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:21.659 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.659 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.9507689502494592331", 2024-08-18T16:39:21.659 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:21.659 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:21.659 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:21.659 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.659 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:21.659 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj62", 2024-08-18T16:39:21.659 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:21.659 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:21.659 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:39:21.659 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 51 2024-08-18T16:39:21.659 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.660 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:21.660 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:21.660 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:21.660 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:21.660 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 6, 2024-08-18T16:39:21.660 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:33:27.434791Z", 2024-08-18T16:39:21.660 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:21.660 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:21.660 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:21.660 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:21.660 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:21.660 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:21.660 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:21.660 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:21.661 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.661 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.16197264008125151659", 2024-08-18T16:39:21.661 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:21.661 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:21.661 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:21.661 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.661 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:21.661 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj68", 2024-08-18T16:39:21.661 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:21.661 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:21.661 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:39:21.661 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 12 2024-08-18T16:39:21.661 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.662 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:21.662 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:21.662 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:21.662 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:21.662 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 6, 2024-08-18T16:39:21.662 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:24:10.351074Z", 2024-08-18T16:39:21.662 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:21.662 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:21.662 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:21.662 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:21.662 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:21.662 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:21.662 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:21.663 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:21.663 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.663 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.12739846555303172313", 2024-08-18T16:39:21.663 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:21.663 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:21.663 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:21.663 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.663 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:21.663 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj69", 2024-08-18T16:39:21.663 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:21.663 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:21.663 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:39:21.663 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 12 2024-08-18T16:39:21.664 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.664 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:21.664 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:21.664 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:21.664 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:21.664 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:39:21.664 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:27:23.414590Z", 2024-08-18T16:39:21.664 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:21.664 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:21.664 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:21.664 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:21.664 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:21.664 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:21.664 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:21.665 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:21.665 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.665 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.6317092130447753347", 2024-08-18T16:39:21.665 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:21.665 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:21.665 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:21.665 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.665 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:21.665 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj74", 2024-08-18T16:39:21.665 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:21.665 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:21.665 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:39:21.665 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 8 2024-08-18T16:39:21.665 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.666 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:21.666 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:21.666 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:21.666 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:21.666 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 5, 2024-08-18T16:39:21.666 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:36:05.141727Z", 2024-08-18T16:39:21.666 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:21.666 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:21.666 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:21.666 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:21.666 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:21.666 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:21.666 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:21.667 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:21.667 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.667 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.7265404272103059924", 2024-08-18T16:39:21.667 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:21.667 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:21.667 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:21.667 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.667 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:21.667 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj76", 2024-08-18T16:39:21.667 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:21.667 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:21.667 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:39:21.667 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 12 2024-08-18T16:39:21.667 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.668 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:21.668 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:21.668 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:21.668 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:21.668 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:39:21.668 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:25:56.019500Z", 2024-08-18T16:39:21.668 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:21.668 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:21.668 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:21.668 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:21.668 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:21.668 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:21.668 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:21.669 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:21.669 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.669 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.9103800521599819169", 2024-08-18T16:39:21.669 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:21.669 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:21.669 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:21.669 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.669 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:21.669 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj85", 2024-08-18T16:39:21.669 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:21.669 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:21.669 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:39:21.670 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 12 2024-08-18T16:39:21.671 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.671 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:21.671 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:21.671 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:21.671 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:21.671 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-08-18T16:39:21.671 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:24:10.358501Z", 2024-08-18T16:39:21.671 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:21.671 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:21.671 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:21.671 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:21.671 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:21.672 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:21.672 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:21.672 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:21.672 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.672 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.14645986973619251717", 2024-08-18T16:39:21.672 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:21.672 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:21.672 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:21.672 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.672 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:21.672 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj87", 2024-08-18T16:39:21.672 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:21.672 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:21.673 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:39:21.673 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 5 2024-08-18T16:39:21.673 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.673 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:21.673 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:21.673 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:21.673 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:21.673 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-08-18T16:39:21.673 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:30:57.327145Z", 2024-08-18T16:39:21.673 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:21.673 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:21.673 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:21.674 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:21.674 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:21.674 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:21.674 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:21.674 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:21.674 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.674 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.15363169285135208126", 2024-08-18T16:39:21.674 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:21.674 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:21.674 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:21.674 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.674 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:21.675 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj92", 2024-08-18T16:39:21.675 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:21.675 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:21.675 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:39:21.675 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 20 2024-08-18T16:39:21.675 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.675 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:21.675 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:21.675 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:21.675 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:21.675 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-08-18T16:39:21.675 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:27:23.459526Z", 2024-08-18T16:39:21.675 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:21.676 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:21.676 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:21.676 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:21.676 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:21.676 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:21.676 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:21.676 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:21.676 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.676 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.15204855023445239626", 2024-08-18T16:39:21.676 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:21.676 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:21.676 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:21.676 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.677 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:21.677 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj93", 2024-08-18T16:39:21.677 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:21.677 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:21.677 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:39:21.677 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 11 2024-08-18T16:39:21.677 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.677 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:21.677 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:21.677 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:21.677 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:21.677 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 5, 2024-08-18T16:39:21.677 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:33:27.484033Z", 2024-08-18T16:39:21.677 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:21.678 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:21.678 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:21.678 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:21.678 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:21.678 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:21.678 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:21.678 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:21.678 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.678 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.915037665819258209", 2024-08-18T16:39:21.678 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:21.678 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:21.678 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:21.678 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.679 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:21.679 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj95", 2024-08-18T16:39:21.679 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:21.679 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:21.679 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:39:21.679 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 20 2024-08-18T16:39:21.679 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.679 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:21.679 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:21.679 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:21.679 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:21.679 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:39:21.679 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:25:56.027064Z", 2024-08-18T16:39:21.680 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:21.680 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:21.680 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:21.680 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:21.680 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:21.680 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:21.680 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:21.680 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:21.680 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:21.680 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.5376182089991821317", 2024-08-18T16:39:21.680 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:21.680 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:21.680 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:21.681 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:39:21.681 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T16:39:21.681 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'list', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--max-entries', '10000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:39:21.681 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'list', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--max-entries', '10000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:39:21.682 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket list --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-78 --max-entries 10000 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:39:21.732 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:39:21.732 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:39:21.766 INFO:teuthology.orchestra.run.smithi193.stdout:[ 2024-08-18T16:39:21.766 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:39:21.766 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj11", 2024-08-18T16:39:21.766 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:39:21.766 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:39:21.766 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:39:21.766 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 16 2024-08-18T16:39:21.766 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.766 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:39:21.767 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:39:21.767 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:39:21.767 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:39:21.767 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 16, 2024-08-18T16:39:21.767 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:23:42.051163Z", 2024-08-18T16:39:21.767 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:21.767 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:39:21.767 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:21.767 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:21.767 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:21.767 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:39:21.767 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:39:21.767 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:39:21.768 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.768 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_EmMqzxQw7107oc5eooZkOe92CnNj5Of", 2024-08-18T16:39:21.768 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:39:21.768 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:39:21.768 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:39:21.768 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.768 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:39:21.768 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj111", 2024-08-18T16:39:21.768 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:39:21.768 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:39:21.768 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:39:21.768 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 29 2024-08-18T16:39:21.768 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.769 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:39:21.769 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:39:21.769 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:39:21.769 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:39:21.769 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 16, 2024-08-18T16:39:21.769 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:36:05.189316Z", 2024-08-18T16:39:21.769 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:21.769 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:39:21.769 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:21.769 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:21.769 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:21.769 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:39:21.769 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:39:21.769 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:39:21.770 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.770 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_aYujydtKN5oO9t5eyzcNKrZCQXdSLwt", 2024-08-18T16:39:21.770 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:39:21.770 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:39:21.770 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:39:21.770 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.770 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:39:21.770 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj115", 2024-08-18T16:39:21.770 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:39:21.770 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:39:21.770 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:39:21.770 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 5 2024-08-18T16:39:21.770 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.771 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:39:21.771 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:39:21.771 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:39:21.771 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:39:21.771 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 6, 2024-08-18T16:39:21.771 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:27:23.466808Z", 2024-08-18T16:39:21.771 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:21.771 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:39:21.771 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:21.771 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:21.771 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:21.771 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:39:21.772 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:39:21.772 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:39:21.772 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.772 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_wqM1lVnP6nZZfMjEqrLtpZxeRw-Grpp", 2024-08-18T16:39:21.772 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:39:21.772 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:39:21.772 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:39:21.772 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.772 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:39:21.772 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj116", 2024-08-18T16:39:21.772 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:39:21.772 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:39:21.773 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:39:21.773 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 11 2024-08-18T16:39:21.773 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.773 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:39:21.773 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:39:21.773 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:39:21.773 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:39:21.773 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 6, 2024-08-18T16:39:21.773 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:30:57.371585Z", 2024-08-18T16:39:21.773 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:21.773 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:39:21.773 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:21.773 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:21.774 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:21.774 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:39:21.774 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:39:21.774 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:39:21.774 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.774 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_0ssO5QX0AUabgBxDLtUS-GOtUpPtxpB", 2024-08-18T16:39:21.774 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:39:21.774 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:39:21.774 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:39:21.774 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.774 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:39:21.774 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj124", 2024-08-18T16:39:21.774 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:39:21.774 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:39:21.775 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:39:21.775 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 15 2024-08-18T16:39:21.775 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.775 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:39:21.775 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:39:21.775 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:39:21.775 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:39:21.775 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 16, 2024-08-18T16:39:21.775 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:33:27.492061Z", 2024-08-18T16:39:21.775 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:21.776 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:39:21.776 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:21.776 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:21.776 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:21.776 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:39:21.776 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:39:21.776 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:39:21.776 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.776 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_zFn9EsgV3fOk06gG8yScAfaM82VqJJT", 2024-08-18T16:39:21.776 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:39:21.776 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:39:21.776 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:39:21.776 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.777 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:39:21.777 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj145", 2024-08-18T16:39:21.777 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:39:21.777 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:39:21.777 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:39:21.777 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 20 2024-08-18T16:39:21.777 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.777 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:39:21.777 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:39:21.777 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:39:21.777 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:39:21.777 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 6, 2024-08-18T16:39:21.777 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:30:57.379450Z", 2024-08-18T16:39:21.777 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:21.778 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:39:21.778 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:21.778 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:21.778 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:21.778 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:39:21.778 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:39:21.778 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:39:21.778 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.778 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_pLJm2YC838fuNwFo1bIz7bzeP7W2KGg", 2024-08-18T16:39:21.778 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:39:21.778 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:39:21.778 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:39:21.778 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.778 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:39:21.779 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj148", 2024-08-18T16:39:21.779 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:39:21.779 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:39:21.779 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:39:21.779 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 11 2024-08-18T16:39:21.779 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.779 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:39:21.779 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:39:21.779 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:39:21.779 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:39:21.779 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 6, 2024-08-18T16:39:21.779 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:36:05.197180Z", 2024-08-18T16:39:21.779 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:21.780 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:39:21.780 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:21.780 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:21.780 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:21.780 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:39:21.780 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:39:21.780 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:39:21.780 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.780 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_8D6gN01khs-zSxppBhG8Byi8_qMp8cv", 2024-08-18T16:39:21.780 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:39:21.780 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:39:21.780 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:39:21.780 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.780 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:39:21.781 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj155", 2024-08-18T16:39:21.781 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:39:21.781 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:39:21.781 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:39:21.781 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 11 2024-08-18T16:39:21.781 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.781 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:39:21.781 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:39:21.781 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:39:21.781 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:39:21.781 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 5, 2024-08-18T16:39:21.781 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:33:27.500403Z", 2024-08-18T16:39:21.781 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:21.781 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:39:21.782 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:21.782 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:21.782 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:21.782 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:39:21.782 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:39:21.782 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:39:21.782 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.782 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_sJZBD9RQEtw9_TZSHriEdzwomoTizMJ", 2024-08-18T16:39:21.782 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:39:21.782 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:39:21.782 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:39:21.782 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.782 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:39:21.782 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj17", 2024-08-18T16:39:21.783 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:39:21.783 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:39:21.783 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:39:21.783 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 9 2024-08-18T16:39:21.783 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.783 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:39:21.783 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:39:21.783 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:39:21.783 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:39:21.783 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 16, 2024-08-18T16:39:21.783 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:24:10.291690Z", 2024-08-18T16:39:21.783 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:21.783 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:39:21.784 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:21.784 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:21.784 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:21.784 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:39:21.784 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:39:21.784 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:39:21.784 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.784 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_wA3tQwbXV4CUYw8FR9sg_bClkYK9i3W", 2024-08-18T16:39:21.784 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:39:21.784 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:39:21.784 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:39:21.784 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.784 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:39:21.785 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj185", 2024-08-18T16:39:21.785 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:39:21.785 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:39:21.785 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:39:21.785 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 15 2024-08-18T16:39:21.785 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.785 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:39:21.785 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:39:21.785 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:39:21.785 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:39:21.785 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 3, 2024-08-18T16:39:21.785 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:36:05.244081Z", 2024-08-18T16:39:21.785 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:21.786 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:39:21.786 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:21.786 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:21.786 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:21.786 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:39:21.786 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:39:21.786 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:39:21.786 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.786 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_cmepO8rj-2Nqh43swntDP0WBsxH5B6b", 2024-08-18T16:39:21.786 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:39:21.786 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:39:21.786 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:39:21.786 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.787 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:39:21.787 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj19", 2024-08-18T16:39:21.787 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:39:21.787 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:39:21.787 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:39:21.787 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 10 2024-08-18T16:39:21.787 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.787 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:39:21.787 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:39:21.787 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:39:21.787 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:39:21.787 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 16, 2024-08-18T16:39:21.787 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:25:55.960750Z", 2024-08-18T16:39:21.787 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:21.788 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:39:21.788 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:21.788 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:21.788 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:21.788 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:39:21.788 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:39:21.788 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:39:21.788 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.788 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_-MRyz69iv0GzNoEJ_HpNUJwmLdnVtDM", 2024-08-18T16:39:21.788 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:39:21.788 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:39:21.788 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:39:21.788 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.789 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:39:21.789 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj22", 2024-08-18T16:39:21.789 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:39:21.789 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:39:21.789 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:39:21.789 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 53 2024-08-18T16:39:21.789 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.789 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:39:21.789 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:39:21.789 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:39:21.789 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:39:21.789 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 6, 2024-08-18T16:39:21.789 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:23:42.058759Z", 2024-08-18T16:39:21.789 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:21.790 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:39:21.790 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:21.790 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:21.790 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:21.790 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:39:21.790 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:39:21.790 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:39:21.790 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.790 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_mezGpEZjp3JckgqVbmyhzJY2WFf9Ps2", 2024-08-18T16:39:21.790 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:39:21.790 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:39:21.790 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:39:21.790 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.791 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:39:21.791 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj23", 2024-08-18T16:39:21.791 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:39:21.791 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:39:21.791 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:39:21.791 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 10 2024-08-18T16:39:21.791 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.791 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:39:21.791 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:39:21.791 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:39:21.791 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:39:21.791 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 5, 2024-08-18T16:39:21.791 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:27:23.361495Z", 2024-08-18T16:39:21.791 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:21.792 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:39:21.792 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:21.792 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:21.792 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:21.792 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:39:21.792 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:39:21.792 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:39:21.792 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.792 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_g8ag_cwe986AlttOTpiUCCsaf2IwnQd", 2024-08-18T16:39:21.792 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:39:21.792 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:39:21.792 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:39:21.792 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.792 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:39:21.793 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj29", 2024-08-18T16:39:21.793 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:39:21.793 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:39:21.793 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:39:21.793 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 7 2024-08-18T16:39:21.793 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.793 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:39:21.793 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:39:21.793 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:39:21.793 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:39:21.793 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 16, 2024-08-18T16:39:21.793 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:30:57.311630Z", 2024-08-18T16:39:21.793 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:21.794 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:39:21.794 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:21.794 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:21.794 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:21.794 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:39:21.794 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:39:21.794 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:39:21.794 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.794 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_VT6j72Ke0mZIQaChpiPLHxPNZ41ClOP", 2024-08-18T16:39:21.794 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:39:21.794 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:39:21.794 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:39:21.794 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.794 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:39:21.795 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj31", 2024-08-18T16:39:21.795 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:39:21.795 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:39:21.795 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:39:21.795 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 10 2024-08-18T16:39:21.795 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.795 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:39:21.795 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:39:21.795 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:39:21.795 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:39:21.795 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 16, 2024-08-18T16:39:21.795 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:33:27.426326Z", 2024-08-18T16:39:21.795 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:21.796 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:39:21.796 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:21.796 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:21.796 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:21.796 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:39:21.796 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:39:21.796 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:39:21.796 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.796 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_qgPcNvhwL_5MWzhs9PxVynZygXcxnA-", 2024-08-18T16:39:21.796 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:39:21.796 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:39:21.796 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:39:21.796 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.796 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:39:21.797 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj33", 2024-08-18T16:39:21.797 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:39:21.797 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:39:21.797 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:39:21.797 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 8 2024-08-18T16:39:21.797 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.797 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:39:21.797 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:39:21.797 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:39:21.797 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:39:21.797 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 6, 2024-08-18T16:39:21.797 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:23:42.064530Z", 2024-08-18T16:39:21.797 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:21.797 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:39:21.798 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:21.798 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:21.798 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:21.798 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:39:21.798 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:39:21.798 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:39:21.798 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.798 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_wAPDhao0ej3vAVQvmUI0GYC7DwFGrkP", 2024-08-18T16:39:21.798 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:39:21.798 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:39:21.798 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:39:21.798 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.798 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:39:21.799 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj34", 2024-08-18T16:39:21.799 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:39:21.799 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:39:21.799 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:39:21.799 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 18 2024-08-18T16:39:21.799 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.799 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:39:21.799 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:39:21.799 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:39:21.799 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:39:21.799 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 5, 2024-08-18T16:39:21.799 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:24:10.298744Z", 2024-08-18T16:39:21.799 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:21.799 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:39:21.800 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:21.800 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:21.800 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:21.800 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:39:21.800 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:39:21.800 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:39:21.800 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.800 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_ovuxOhLx7SBJf6KKeRLqVYHTTgs-DTL", 2024-08-18T16:39:21.800 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:39:21.800 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:39:21.800 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:39:21.800 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.800 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:39:21.801 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj37", 2024-08-18T16:39:21.801 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:39:21.801 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:39:21.801 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:39:21.801 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 17 2024-08-18T16:39:21.801 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.801 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:39:21.801 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:39:21.801 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:39:21.801 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:39:21.801 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 6, 2024-08-18T16:39:21.801 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:36:05.133754Z", 2024-08-18T16:39:21.801 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:21.801 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:39:21.802 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:21.802 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:21.802 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:21.802 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:39:21.802 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:39:21.802 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:39:21.802 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.802 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_9RObiQvh5MYh_CiFc5_zrv4S0GHcNY7", 2024-08-18T16:39:21.802 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:39:21.802 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:39:21.802 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:39:21.802 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.802 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:39:21.803 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj38", 2024-08-18T16:39:21.803 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:39:21.803 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:39:21.803 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:39:21.803 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 10 2024-08-18T16:39:21.803 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.803 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:39:21.803 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:39:21.803 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:39:21.803 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:39:21.803 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 6, 2024-08-18T16:39:21.803 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:25:55.967916Z", 2024-08-18T16:39:21.803 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:21.803 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:39:21.804 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:21.804 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:21.804 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:21.804 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:39:21.804 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:39:21.804 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:39:21.804 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.804 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_Qb7r0pY7NsPkHLN18WhiIvlY4s00MMi", 2024-08-18T16:39:21.804 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:39:21.804 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:39:21.804 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:39:21.804 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.804 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:39:21.804 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj44", 2024-08-18T16:39:21.805 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:39:21.805 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:39:21.805 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:39:21.805 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 10 2024-08-18T16:39:21.805 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.805 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:39:21.805 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:39:21.805 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:39:21.805 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:39:21.805 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 3, 2024-08-18T16:39:21.805 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:23:42.115387Z", 2024-08-18T16:39:21.805 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:21.805 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:39:21.805 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:21.806 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:21.806 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:21.806 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:39:21.806 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:39:21.806 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:39:21.806 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.806 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_ZM2yhnRSOxOGtQPxXPGlZyzLFfYIbM0", 2024-08-18T16:39:21.806 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:39:21.806 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:39:21.806 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:39:21.806 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.806 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:39:21.806 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj46", 2024-08-18T16:39:21.807 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:39:21.807 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:39:21.807 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:39:21.807 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 11 2024-08-18T16:39:21.807 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.807 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:39:21.807 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:39:21.807 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:39:21.807 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:39:21.807 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 16, 2024-08-18T16:39:21.807 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:27:23.407342Z", 2024-08-18T16:39:21.807 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:21.807 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:39:21.807 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:21.808 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:21.808 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:21.808 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:39:21.808 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:39:21.808 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:39:21.808 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.810 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_1LmMzNGwgiZ7vDNqP64djoFyUqK__U7", 2024-08-18T16:39:21.810 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:39:21.810 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:39:21.810 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:39:21.810 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.810 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:39:21.810 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj51", 2024-08-18T16:39:21.810 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:39:21.810 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:39:21.810 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:39:21.811 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 7 2024-08-18T16:39:21.811 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.811 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:39:21.811 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:39:21.811 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:39:21.811 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:39:21.811 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 5, 2024-08-18T16:39:21.811 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:24:10.305333Z", 2024-08-18T16:39:21.811 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:21.811 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:39:21.811 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:21.811 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:21.811 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:21.812 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:39:21.812 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:39:21.812 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:39:21.812 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.812 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_MBKVcMW-jTvkFcLqFpNbF4rkMtO5s75", 2024-08-18T16:39:21.812 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:39:21.812 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:39:21.812 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:39:21.812 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.812 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:39:21.812 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj55", 2024-08-18T16:39:21.812 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:39:21.812 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:39:21.812 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:39:21.813 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 9 2024-08-18T16:39:21.813 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.813 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:39:21.813 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:39:21.813 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:39:21.813 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:39:21.813 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 3, 2024-08-18T16:39:21.813 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:23:42.122245Z", 2024-08-18T16:39:21.813 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:21.813 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:39:21.813 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:21.813 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:21.813 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:21.813 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:39:21.814 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:39:21.814 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:39:21.814 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.814 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_YUS4H00tIicekyP_e3K3GUwD2PLt7Od", 2024-08-18T16:39:21.814 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:39:21.814 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:39:21.814 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:39:21.814 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.814 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:39:21.814 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj57", 2024-08-18T16:39:21.814 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:39:21.814 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:39:21.814 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:39:21.815 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 13 2024-08-18T16:39:21.815 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.815 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:39:21.815 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:39:21.815 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:39:21.815 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:39:21.815 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 16, 2024-08-18T16:39:21.815 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:25:55.973669Z", 2024-08-18T16:39:21.815 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:21.815 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:39:21.815 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:21.815 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:21.815 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:21.815 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:39:21.816 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:39:21.816 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:39:21.816 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.816 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_72Byd2n2R79npPeY3aWb2uKX6EMu1IR", 2024-08-18T16:39:21.816 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:39:21.816 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:39:21.816 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:39:21.816 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.816 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:39:21.816 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj58", 2024-08-18T16:39:21.816 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:39:21.816 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:39:21.816 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:39:21.817 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 11 2024-08-18T16:39:21.817 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.817 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:39:21.817 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:39:21.817 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:39:21.817 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:39:21.817 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 16, 2024-08-18T16:39:21.817 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:30:57.319413Z", 2024-08-18T16:39:21.817 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:21.817 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:39:21.817 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:21.817 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:21.817 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:21.817 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:39:21.818 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:39:21.818 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:39:21.818 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.818 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_WL9YWyhRbwAYLmg1Y3Ri3EYJaKbbX5H", 2024-08-18T16:39:21.818 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:39:21.818 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:39:21.818 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:39:21.818 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.818 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:39:21.818 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj62", 2024-08-18T16:39:21.818 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:39:21.818 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:39:21.818 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:39:21.819 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 52 2024-08-18T16:39:21.819 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.819 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:39:21.819 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:39:21.819 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:39:21.819 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:39:21.819 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 3, 2024-08-18T16:39:21.819 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:33:27.434791Z", 2024-08-18T16:39:21.819 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:21.819 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:39:21.819 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:21.819 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:21.819 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:21.819 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:39:21.820 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:39:21.820 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:39:21.820 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.820 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_bc8kXm9vKqfP5d3zYRyCS8Z3uRtZhu3", 2024-08-18T16:39:21.820 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:39:21.820 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:39:21.820 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:39:21.820 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.820 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:39:21.820 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj68", 2024-08-18T16:39:21.820 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:39:21.820 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:39:21.820 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:39:21.820 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 16 2024-08-18T16:39:21.821 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.821 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:39:21.821 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:39:21.821 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:39:21.821 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:39:21.821 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 16, 2024-08-18T16:39:21.821 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:24:10.351074Z", 2024-08-18T16:39:21.821 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:21.821 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:39:21.821 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:21.821 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:21.821 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:21.821 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:39:21.821 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:39:21.822 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:39:21.822 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.822 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_kfPQ3dhoQJB27owtJl2jBD5cMsxAJqr", 2024-08-18T16:39:21.822 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:39:21.822 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:39:21.822 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:39:21.822 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.822 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:39:21.822 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj69", 2024-08-18T16:39:21.822 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:39:21.822 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:39:21.822 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:39:21.822 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 14 2024-08-18T16:39:21.823 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.823 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:39:21.823 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:39:21.823 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:39:21.823 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:39:21.823 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 16, 2024-08-18T16:39:21.823 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:27:23.414590Z", 2024-08-18T16:39:21.823 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:21.823 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:39:21.823 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:21.823 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:21.823 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:21.823 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:39:21.823 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:39:21.824 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:39:21.824 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.824 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_TajS0OQu4haG51zKbuoP7dBXR4dk3M-", 2024-08-18T16:39:21.824 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:39:21.824 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:39:21.824 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:39:21.824 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.824 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:39:21.824 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj74", 2024-08-18T16:39:21.824 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:39:21.824 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:39:21.824 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:39:21.824 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 10 2024-08-18T16:39:21.824 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.825 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:39:21.825 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:39:21.825 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:39:21.825 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:39:21.825 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 16, 2024-08-18T16:39:21.825 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:36:05.141727Z", 2024-08-18T16:39:21.825 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:21.825 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:39:21.825 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:21.825 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:21.825 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:21.825 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:39:21.825 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:39:21.825 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:39:21.826 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.826 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_gKrkL3--grsoXHlKuzCVdh2rR0_lDWu", 2024-08-18T16:39:21.826 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:39:21.826 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:39:21.826 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:39:21.826 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.826 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:39:21.826 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj76", 2024-08-18T16:39:21.826 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:39:21.826 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:39:21.826 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:39:21.826 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 12 2024-08-18T16:39:21.826 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.826 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:39:21.827 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:39:21.827 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:39:21.827 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:39:21.827 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 5, 2024-08-18T16:39:21.827 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:25:56.019500Z", 2024-08-18T16:39:21.827 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:21.827 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:39:21.827 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:21.827 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:21.827 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:21.827 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:39:21.827 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:39:21.827 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:39:21.827 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.828 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_9C3V-3_dtPt9hZi3ayRllorABGMaAq7", 2024-08-18T16:39:21.828 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:39:21.828 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:39:21.828 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:39:21.828 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.828 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:39:21.828 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj85", 2024-08-18T16:39:21.828 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:39:21.828 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:39:21.828 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:39:21.828 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 12 2024-08-18T16:39:21.828 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.828 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:39:21.828 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:39:21.829 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:39:21.829 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:39:21.829 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 16, 2024-08-18T16:39:21.829 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:24:10.358501Z", 2024-08-18T16:39:21.829 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:21.829 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:39:21.829 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:21.829 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:21.829 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:21.829 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:39:21.829 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:39:21.829 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:39:21.829 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.829 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_K2AJn7DUzvZIFPvNMtiNIdNf-1PSpu3", 2024-08-18T16:39:21.830 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:39:21.830 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:39:21.830 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:39:21.830 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.830 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:39:21.830 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj87", 2024-08-18T16:39:21.830 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:39:21.830 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:39:21.830 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:39:21.830 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 7 2024-08-18T16:39:21.830 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.830 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:39:21.830 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:39:21.831 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:39:21.832 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:39:21.832 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 16, 2024-08-18T16:39:21.832 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:30:57.327145Z", 2024-08-18T16:39:21.832 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:21.832 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:39:21.832 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:21.832 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:21.832 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:21.832 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:39:21.832 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:39:21.832 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:39:21.832 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.833 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_RcbiK4WZ1gRPBYChi8AVBHPRHgE_bMO", 2024-08-18T16:39:21.833 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:39:21.833 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:39:21.833 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:39:21.833 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.833 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:39:21.833 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj92", 2024-08-18T16:39:21.833 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:39:21.833 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:39:21.833 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:39:21.833 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 21 2024-08-18T16:39:21.833 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.833 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:39:21.833 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:39:21.834 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:39:21.834 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:39:21.834 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 6, 2024-08-18T16:39:21.834 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:27:23.459526Z", 2024-08-18T16:39:21.834 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:21.834 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:39:21.834 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:21.834 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:21.834 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:21.834 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:39:21.834 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:39:21.834 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:39:21.834 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.835 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_53RdBR82NZHpZDOaBHJIVjwgin0QYba", 2024-08-18T16:39:21.835 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:39:21.835 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:39:21.835 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:39:21.835 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.835 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:39:21.835 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj93", 2024-08-18T16:39:21.835 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:39:21.835 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:39:21.835 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:39:21.835 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 11 2024-08-18T16:39:21.835 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.835 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:39:21.835 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:39:21.836 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:39:21.836 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:39:21.836 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 6, 2024-08-18T16:39:21.836 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:33:27.484033Z", 2024-08-18T16:39:21.836 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:21.836 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:39:21.836 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:21.836 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:21.836 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:21.836 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:39:21.836 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:39:21.836 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:39:21.836 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.836 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_6lbtSRYgccV6EQF_puu9YNdpZcz4pSt", 2024-08-18T16:39:21.837 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:39:21.837 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:39:21.837 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:39:21.837 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.837 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:39:21.837 INFO:teuthology.orchestra.run.smithi193.stdout: "name": "obj95", 2024-08-18T16:39:21.837 INFO:teuthology.orchestra.run.smithi193.stdout: "instance": "", 2024-08-18T16:39:21.837 INFO:teuthology.orchestra.run.smithi193.stdout: "ver": { 2024-08-18T16:39:21.837 INFO:teuthology.orchestra.run.smithi193.stdout: "pool": 9, 2024-08-18T16:39:21.837 INFO:teuthology.orchestra.run.smithi193.stdout: "epoch": 14 2024-08-18T16:39:21.837 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.837 INFO:teuthology.orchestra.run.smithi193.stdout: "locator": "", 2024-08-18T16:39:21.837 INFO:teuthology.orchestra.run.smithi193.stdout: "exists": true, 2024-08-18T16:39:21.837 INFO:teuthology.orchestra.run.smithi193.stdout: "meta": { 2024-08-18T16:39:21.837 INFO:teuthology.orchestra.run.smithi193.stdout: "category": 1, 2024-08-18T16:39:21.838 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 6, 2024-08-18T16:39:21.838 INFO:teuthology.orchestra.run.smithi193.stdout: "mtime": "2024-08-18T16:25:56.027064Z", 2024-08-18T16:39:21.838 INFO:teuthology.orchestra.run.smithi193.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:21.838 INFO:teuthology.orchestra.run.smithi193.stdout: "storage_class": "", 2024-08-18T16:39:21.838 INFO:teuthology.orchestra.run.smithi193.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:21.838 INFO:teuthology.orchestra.run.smithi193.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:21.838 INFO:teuthology.orchestra.run.smithi193.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:21.838 INFO:teuthology.orchestra.run.smithi193.stdout: "accounted_size": 3, 2024-08-18T16:39:21.838 INFO:teuthology.orchestra.run.smithi193.stdout: "user_data": "", 2024-08-18T16:39:21.838 INFO:teuthology.orchestra.run.smithi193.stdout: "appendable": false 2024-08-18T16:39:21.838 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:39:21.838 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "_qOROfbqvBDurmdnhE3Zr2FAXznsJELX", 2024-08-18T16:39:21.838 INFO:teuthology.orchestra.run.smithi193.stdout: "flags": 0, 2024-08-18T16:39:21.839 INFO:teuthology.orchestra.run.smithi193.stdout: "pending_map": [], 2024-08-18T16:39:21.839 INFO:teuthology.orchestra.run.smithi193.stdout: "versioned_epoch": 0 2024-08-18T16:39:21.839 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:39:21.839 INFO:teuthology.orchestra.run.smithi193.stdout:] 2024-08-18T16:39:21.839 INFO:tasks.rgw.c2.client.0:Restarting daemon 2024-08-18T16:39:21.840 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2024-08-18T16:39:21.842 INFO:tasks.rgw.c2.client.0:Started 2024-08-18T16:39:21.842 DEBUG:teuthology.orchestra.run.smithi193:> curl http://smithi193.front.sepia.ceph.com:8000 2024-08-18T16:39:21.861 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-08-18T16:39:21.862 INFO:teuthology.orchestra.run.smithi193.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-08-18T16:39:21.862 INFO:teuthology.orchestra.run.smithi193.stderr: Dload Upload Total Spent Left Speed 2024-08-18T16:39:21.862 INFO:teuthology.orchestra.run.smithi193.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2024-08-18T16:39:21.862 INFO:teuthology.orchestra.run.smithi193.stderr:curl: (7) Failed to connect to smithi193.front.sepia.ceph.com port 8000: Connection refused 2024-08-18T16:39:22.863 DEBUG:teuthology.orchestra.run.smithi193:> curl http://smithi193.front.sepia.ceph.com:8000 2024-08-18T16:39:22.883 INFO:teuthology.orchestra.run.smithi193.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-08-18T16:39:22.883 INFO:teuthology.orchestra.run.smithi193.stderr: Dload Upload Total Spent Left Speed 2024-08-18T16:39:22.883 INFO:teuthology.orchestra.run.smithi193.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-08-18T16:39:22.884 INFO:teuthology.orchestra.run.smithi193.stdout:anonymous 2024-08-18T16:39:22.884 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2024-08-18T16:39:22.884 INFO:teuthology.orchestra.run:waiting for 300 2024-08-18T16:39:22.886 INFO:tasks.rgw.c1.client.1.smithi134.stdout:2024-08-18T16:39:22.885+0000 7f377c5ca640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 83654) UID: 0 2024-08-18T16:39:22.886 INFO:tasks.rgw.c1.client.1.smithi134.stdout:2024-08-18T16:39:22.885+0000 7f378262f880 -1 shutting down 2024-08-18T16:39:28.590 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2024-08-18T16:39:28.986 INFO:tasks.rgw.c1.client.1:Stopped 2024-08-18T16:39:28.986 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'list', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:39:28.987 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'list', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:39:28.987 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket list --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-78 --max-entries 10000 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:39:29.045 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:39:29.045 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:39:29.080 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T16:39:29.080 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:29.080 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj11", 2024-08-18T16:39:29.080 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:29.080 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:29.080 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:39:29.080 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 15 2024-08-18T16:39:29.081 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.081 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:29.081 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:29.081 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:29.081 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:29.081 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 6, 2024-08-18T16:39:29.081 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:23:42.051163Z", 2024-08-18T16:39:29.081 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:29.081 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:29.081 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:29.081 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:29.081 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:29.081 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:29.082 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:29.082 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:29.082 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.082 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_mfvtRGZAiphlov7-Cd7nafulBDDeasl", 2024-08-18T16:39:29.082 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:29.082 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:29.082 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:29.082 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.082 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:29.082 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj111", 2024-08-18T16:39:29.082 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:29.082 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:29.082 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:39:29.083 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 24 2024-08-18T16:39:29.083 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.083 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:29.083 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:29.083 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:29.083 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:29.083 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-08-18T16:39:29.083 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:36:05.189316Z", 2024-08-18T16:39:29.083 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:29.083 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:29.083 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:29.083 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:29.083 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:29.084 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:29.084 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:29.084 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:29.084 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.084 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_AeNWlhmsJgt-cUFxmTcKCTf_TzuPHV5", 2024-08-18T16:39:29.084 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:29.084 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:29.084 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:29.084 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.084 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:29.084 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj115", 2024-08-18T16:39:29.084 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:29.084 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:29.084 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:39:29.085 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 3 2024-08-18T16:39:29.085 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.085 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:29.085 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:29.085 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:29.085 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:29.085 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:39:29.085 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:27:23.466808Z", 2024-08-18T16:39:29.085 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:29.086 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:29.086 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:29.086 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:29.086 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:29.086 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:29.086 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:29.086 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:29.086 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.086 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_P096CHTiwy5JL-neucK2B2W5DRAx9Yb", 2024-08-18T16:39:29.086 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:29.086 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:29.086 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:29.087 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.087 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:29.087 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj116", 2024-08-18T16:39:29.087 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:29.087 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:29.087 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:39:29.087 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 15 2024-08-18T16:39:29.087 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.087 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:29.087 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:29.087 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:29.087 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:29.087 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 5, 2024-08-18T16:39:29.088 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:30:57.371585Z", 2024-08-18T16:39:29.088 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:29.088 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:29.088 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:29.088 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:29.088 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:29.088 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:29.088 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:29.088 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:29.088 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.088 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_lXyXZR-emEDTbe1rWOzF2dNlO-x31He", 2024-08-18T16:39:29.088 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:29.088 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:29.088 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:29.089 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.089 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:29.089 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj124", 2024-08-18T16:39:29.089 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:29.089 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:29.089 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:39:29.089 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 7 2024-08-18T16:39:29.089 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.089 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:29.089 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:29.089 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:29.089 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:29.089 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 6, 2024-08-18T16:39:29.089 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:33:27.492061Z", 2024-08-18T16:39:29.090 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:29.090 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:29.090 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:29.090 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:29.090 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:29.090 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:29.090 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:29.090 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:29.090 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.090 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_sOyCu8nF5IjXKnR22WXFeuZr5wEj1_L", 2024-08-18T16:39:29.090 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:29.090 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:29.091 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:29.091 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.091 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:29.091 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj145", 2024-08-18T16:39:29.091 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:29.091 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:29.091 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:39:29.091 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 19 2024-08-18T16:39:29.091 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.091 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:29.091 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:29.091 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:29.091 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:29.091 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 5, 2024-08-18T16:39:29.092 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:30:57.379450Z", 2024-08-18T16:39:29.092 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:29.092 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:29.092 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:29.092 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:29.092 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:29.092 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:29.092 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:29.092 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:29.092 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.092 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_kpcl7Mj9VTx3qguakfcfiivHhZJMSxh", 2024-08-18T16:39:29.092 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:29.092 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:29.092 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:29.093 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.093 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:29.093 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj148", 2024-08-18T16:39:29.093 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:29.093 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:29.093 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:39:29.093 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 6 2024-08-18T16:39:29.093 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.093 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:29.093 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:29.093 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:29.093 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:29.093 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 6, 2024-08-18T16:39:29.093 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:36:05.197180Z", 2024-08-18T16:39:29.094 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:29.094 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:29.094 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:29.094 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:29.094 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:29.094 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:29.094 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:29.094 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:29.094 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.094 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_52YjiIHgYlE_xF1_kxbYvQrT1AG5LD3", 2024-08-18T16:39:29.094 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:29.094 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:29.094 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:29.095 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.095 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:29.095 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj155", 2024-08-18T16:39:29.095 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:29.095 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:29.095 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:39:29.095 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 7 2024-08-18T16:39:29.095 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.095 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:29.095 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:29.095 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:29.095 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:29.096 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 5, 2024-08-18T16:39:29.096 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:33:27.500403Z", 2024-08-18T16:39:29.096 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:29.096 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:29.096 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:29.096 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:29.096 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:29.096 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:29.096 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:29.096 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:29.097 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.097 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_2nasRcLhTrCqjl8JCUO9rEeHyvr1XUW", 2024-08-18T16:39:29.097 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:29.097 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:29.097 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:29.097 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.097 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:29.097 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj17", 2024-08-18T16:39:29.097 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:29.097 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:29.097 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:39:29.097 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 6 2024-08-18T16:39:29.097 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.098 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:29.098 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:29.098 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:29.098 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:29.098 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:39:29.098 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:24:10.291690Z", 2024-08-18T16:39:29.098 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:29.098 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:29.098 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:29.098 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:29.098 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:29.098 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:29.098 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:29.099 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:29.099 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.099 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_sq7QBdWqoP7LerpGRFBkFA7ifVbro3Q", 2024-08-18T16:39:29.099 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:29.099 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:29.099 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:29.099 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.099 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:29.099 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj185", 2024-08-18T16:39:29.099 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:29.099 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:29.099 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:39:29.099 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 11 2024-08-18T16:39:29.099 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.100 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:29.100 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:29.100 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:29.100 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:29.100 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-08-18T16:39:29.100 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:36:05.244081Z", 2024-08-18T16:39:29.100 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:29.100 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:29.100 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:29.100 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:29.100 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:29.100 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:29.100 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:29.100 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:29.101 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.101 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_eyRJVgoss2YAXCFsDtANEsSbpET5g2S", 2024-08-18T16:39:29.101 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:29.101 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:29.101 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:29.101 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.101 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:29.101 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj19", 2024-08-18T16:39:29.101 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:29.101 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:29.101 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:39:29.101 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 6 2024-08-18T16:39:29.101 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.101 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:29.101 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:29.102 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:29.102 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:29.102 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 6, 2024-08-18T16:39:29.102 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:25:55.960750Z", 2024-08-18T16:39:29.102 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:29.102 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:29.102 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:29.102 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:29.102 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:29.102 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:29.102 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:29.102 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:29.103 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.103 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_ucioMzd1Dq29GZy1je814fryV7BcCl0", 2024-08-18T16:39:29.103 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:29.103 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:29.103 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:29.103 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.103 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:29.103 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj22", 2024-08-18T16:39:29.103 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:29.103 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:29.103 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:39:29.103 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 48 2024-08-18T16:39:29.103 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.103 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:29.104 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:29.104 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:29.104 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:29.104 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 5, 2024-08-18T16:39:29.104 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:23:42.058759Z", 2024-08-18T16:39:29.104 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:29.104 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:29.104 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:29.104 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:29.104 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:29.104 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:29.104 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:29.104 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:29.105 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.105 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_kiQWZ74u3J2uAGhF6gzGSPzPn_QI0Ie", 2024-08-18T16:39:29.105 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:29.105 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:29.105 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:29.105 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.105 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:29.105 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj23", 2024-08-18T16:39:29.105 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:29.105 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:29.105 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:39:29.105 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 6 2024-08-18T16:39:29.105 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.106 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:29.106 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:29.106 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:29.106 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:29.106 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-08-18T16:39:29.106 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:27:23.361495Z", 2024-08-18T16:39:29.106 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:29.106 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:29.106 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:29.106 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:29.106 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:29.106 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:29.106 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:29.107 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:29.107 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.107 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_0WNuo70it-orx9P2AuYofMjA7A64Jze", 2024-08-18T16:39:29.107 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:29.107 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:29.107 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:29.107 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.107 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:29.107 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj29", 2024-08-18T16:39:29.107 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:29.107 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:29.107 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:39:29.107 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 5 2024-08-18T16:39:29.107 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.108 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:29.108 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:29.108 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:29.108 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:29.108 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 6, 2024-08-18T16:39:29.108 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:30:57.311630Z", 2024-08-18T16:39:29.108 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:29.108 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:29.108 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:29.108 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:29.108 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:29.108 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:29.108 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:29.109 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:29.109 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.109 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_ot8aloMjUEAJ2q0p0iHcj-e2xYrBW96", 2024-08-18T16:39:29.109 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:29.109 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:29.109 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:29.110 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.110 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:29.110 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj31", 2024-08-18T16:39:29.110 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:29.110 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:29.110 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:39:29.111 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 8 2024-08-18T16:39:29.111 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.111 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:29.111 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:29.111 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:29.111 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:29.112 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 5, 2024-08-18T16:39:29.112 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:33:27.426326Z", 2024-08-18T16:39:29.112 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:29.112 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:29.112 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:29.112 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:29.112 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:29.112 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:29.112 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:29.112 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:29.112 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.112 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_HgJOI46pk6xOFYf2ie2Dmagdue74x4Y", 2024-08-18T16:39:29.113 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:29.113 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:29.113 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:29.113 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.113 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:29.113 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj33", 2024-08-18T16:39:29.113 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:29.114 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:29.114 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:39:29.114 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 8 2024-08-18T16:39:29.114 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.114 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:29.114 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:29.115 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:29.115 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:29.115 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-08-18T16:39:29.115 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:23:42.064530Z", 2024-08-18T16:39:29.115 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:29.115 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:29.116 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:29.116 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:29.116 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:29.116 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:29.116 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:29.116 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:29.117 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.117 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_czJyfXNgPLj-mIy6bXFosoJ5fRhuB-9", 2024-08-18T16:39:29.117 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:29.117 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:29.117 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:29.117 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.118 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:29.118 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj34", 2024-08-18T16:39:29.118 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:29.118 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:29.118 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:39:29.118 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 13 2024-08-18T16:39:29.119 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.119 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:29.119 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:29.119 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:29.120 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:29.120 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 5, 2024-08-18T16:39:29.120 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:24:10.298744Z", 2024-08-18T16:39:29.120 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:29.121 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:29.121 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:29.121 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:29.121 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:29.121 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:29.121 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:29.122 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:29.122 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.122 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_9GaiprXpiTIIsu_1xNQ9mHlB2bPMC9r", 2024-08-18T16:39:29.122 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:29.122 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:29.123 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:29.123 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.123 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:29.123 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj37", 2024-08-18T16:39:29.123 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:29.123 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:29.124 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:39:29.124 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 14 2024-08-18T16:39:29.124 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.124 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:29.124 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:29.124 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:29.125 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:29.125 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 5, 2024-08-18T16:39:29.125 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:36:05.133754Z", 2024-08-18T16:39:29.125 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:29.125 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:29.125 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:29.126 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:29.126 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:29.126 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:29.126 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:29.126 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:29.127 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.127 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_LXCylTuS0VLesc9vN34hj5zHZZWHbGf", 2024-08-18T16:39:29.127 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:29.127 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:29.127 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:29.128 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.128 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:29.128 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj38", 2024-08-18T16:39:29.128 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:29.129 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:29.129 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:39:29.129 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 8 2024-08-18T16:39:29.129 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.129 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:29.129 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:29.130 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:29.130 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:29.130 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 6, 2024-08-18T16:39:29.130 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:25:55.967916Z", 2024-08-18T16:39:29.131 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:29.131 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:29.131 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:29.131 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:29.131 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:29.131 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:29.131 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:29.131 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:29.132 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.132 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_8r6sCKyamzLVExatDv4CVXyDQBTHAOk", 2024-08-18T16:39:29.132 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:29.132 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:29.132 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:29.132 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.132 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:29.132 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj44", 2024-08-18T16:39:29.132 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:29.132 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:29.132 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:39:29.132 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 8 2024-08-18T16:39:29.133 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.133 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:29.133 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:29.133 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:29.133 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:29.133 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:39:29.133 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:23:42.115387Z", 2024-08-18T16:39:29.133 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:29.133 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:29.133 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:29.133 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:29.133 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:29.134 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:29.134 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:29.134 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:29.134 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.134 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_XxLMp-t4O_6eWKZcSkM9ESbR_rNWQcD", 2024-08-18T16:39:29.134 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:29.134 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:29.134 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:29.134 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.134 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:29.134 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj46", 2024-08-18T16:39:29.134 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:29.135 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:29.135 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:39:29.135 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 9 2024-08-18T16:39:29.135 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.135 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:29.135 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:29.135 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:29.135 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:29.135 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 5, 2024-08-18T16:39:29.135 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:27:23.407342Z", 2024-08-18T16:39:29.135 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:29.135 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:29.135 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:29.135 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:29.136 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:29.136 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:29.136 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:29.136 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:29.136 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.136 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_I4OxKQVfeqjaC7ZuJNeyRyGpOCnCbik", 2024-08-18T16:39:29.136 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:29.136 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:29.136 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:29.136 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.136 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:29.136 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj51", 2024-08-18T16:39:29.136 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:29.136 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:29.137 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:39:29.137 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 7 2024-08-18T16:39:29.137 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.137 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:29.137 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:29.137 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:29.137 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:29.137 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:39:29.137 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:24:10.305333Z", 2024-08-18T16:39:29.137 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:29.137 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:29.137 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:29.138 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:29.138 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:29.138 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:29.138 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:29.138 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:29.138 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.138 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_KqplNeOm2nw6ANcGyDFiW9ZZxGrw0nz", 2024-08-18T16:39:29.138 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:29.138 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:29.138 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:29.138 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.139 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:29.139 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj55", 2024-08-18T16:39:29.139 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:29.139 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:29.139 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:39:29.139 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 5 2024-08-18T16:39:29.139 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.139 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:29.139 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:29.139 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:29.139 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:29.139 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-08-18T16:39:29.139 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:23:42.122245Z", 2024-08-18T16:39:29.140 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:29.140 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:29.140 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:29.140 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:29.140 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:29.140 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:29.140 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:29.140 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:29.140 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.140 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_gdN2-ypFxxwhePEp2dlmg_7dCNkVyi6", 2024-08-18T16:39:29.140 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:29.140 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:29.140 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:29.141 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.141 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:29.141 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj57", 2024-08-18T16:39:29.141 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:29.141 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:29.141 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:39:29.141 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 11 2024-08-18T16:39:29.142 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.142 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:29.142 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:29.142 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:29.142 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:29.143 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:39:29.143 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:25:55.973669Z", 2024-08-18T16:39:29.143 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:29.143 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:29.143 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:29.144 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:29.144 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:29.144 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:29.144 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:29.144 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:29.144 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.144 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_20Gt2Zb-ot8FtyRMor6FoT1EOMICM3U", 2024-08-18T16:39:29.144 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:29.144 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:29.144 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:29.144 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.144 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:29.145 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj58", 2024-08-18T16:39:29.145 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:29.145 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:29.145 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:39:29.145 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 7 2024-08-18T16:39:29.145 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.145 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:29.145 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:29.145 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:29.145 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:29.145 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 6, 2024-08-18T16:39:29.145 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:30:57.319413Z", 2024-08-18T16:39:29.145 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:29.146 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:29.146 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:29.146 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:29.146 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:29.146 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:29.146 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:29.146 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:29.146 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.147 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_qCdEJ6_AjoXT0jG2GtNWWUIXT0MGIEc", 2024-08-18T16:39:29.147 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:29.147 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:29.147 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:29.147 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.147 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:29.147 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj62", 2024-08-18T16:39:29.147 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:29.147 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:29.147 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:39:29.148 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 49 2024-08-18T16:39:29.148 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.148 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:29.148 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:29.148 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:29.148 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:29.148 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 5, 2024-08-18T16:39:29.148 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:33:27.434791Z", 2024-08-18T16:39:29.148 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:29.148 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:29.148 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:29.148 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:29.149 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:29.149 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:29.149 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:29.149 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:29.149 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.149 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_0WWZzkncSJhWgm7CnY1-cDVBpwnK79Z", 2024-08-18T16:39:29.149 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:29.149 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:29.149 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:29.149 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.149 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:29.149 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj68", 2024-08-18T16:39:29.149 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:29.150 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:29.150 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:39:29.150 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 10 2024-08-18T16:39:29.150 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.150 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:29.150 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:29.150 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:29.150 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:29.150 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 5, 2024-08-18T16:39:29.150 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:24:10.351074Z", 2024-08-18T16:39:29.150 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:29.150 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:29.150 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:29.151 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:29.151 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:29.151 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:29.151 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:29.151 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:29.151 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.151 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_5HwFIXcg-Q469RQTbqE57smwihmmCX9", 2024-08-18T16:39:29.151 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:29.151 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:29.151 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:29.151 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.151 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:29.151 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj69", 2024-08-18T16:39:29.152 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:29.152 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:29.152 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:39:29.152 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 14 2024-08-18T16:39:29.152 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.152 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:29.152 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:29.152 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:29.152 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:29.152 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:39:29.152 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:27:23.414590Z", 2024-08-18T16:39:29.152 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:29.153 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:29.153 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:29.153 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:29.153 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:29.153 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:29.153 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:29.153 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:29.153 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.153 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_cZU9Vmd-uAdA4ycGRhVkr42Ngh7qnVh", 2024-08-18T16:39:29.153 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:29.153 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:29.154 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:29.154 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.154 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:29.154 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj74", 2024-08-18T16:39:29.154 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:29.154 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:29.154 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:39:29.154 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 6 2024-08-18T16:39:29.154 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.154 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:29.154 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:29.155 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:29.155 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:29.155 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-08-18T16:39:29.155 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:36:05.141727Z", 2024-08-18T16:39:29.155 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:29.155 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:29.155 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:29.155 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:29.155 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:29.155 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:29.155 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:29.155 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:29.155 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.156 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_FwXhQOOI65L3y4nQoyZ5xoaiUML2XV6", 2024-08-18T16:39:29.156 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:29.156 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:29.156 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:29.156 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.156 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:29.156 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj76", 2024-08-18T16:39:29.156 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:29.156 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:29.156 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:39:29.156 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 5 2024-08-18T16:39:29.156 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.157 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:29.157 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:29.157 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:29.157 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:29.157 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-08-18T16:39:29.157 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:25:56.019500Z", 2024-08-18T16:39:29.157 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:29.157 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:29.157 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:29.158 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:29.158 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:29.158 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:29.158 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:29.158 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:29.159 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.159 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_TT7OlK4G_fuXoSsSEiyvEYVgNCYfmOl", 2024-08-18T16:39:29.159 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:29.159 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:29.159 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:29.159 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.159 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:29.159 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj85", 2024-08-18T16:39:29.159 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:29.159 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:29.159 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:39:29.159 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 10 2024-08-18T16:39:29.159 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.160 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:29.160 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:29.160 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:29.160 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:29.160 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 5, 2024-08-18T16:39:29.160 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:24:10.358501Z", 2024-08-18T16:39:29.160 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:29.160 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:29.160 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:29.161 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:29.161 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:29.161 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:29.161 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:29.161 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:29.161 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.161 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_A3yQYwlQ1XR07MYr5WUZeercJGI-Pep", 2024-08-18T16:39:29.161 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:29.161 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:29.161 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:29.161 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.161 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:29.162 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj87", 2024-08-18T16:39:29.162 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:29.162 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:29.162 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:39:29.162 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 5 2024-08-18T16:39:29.162 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.162 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:29.162 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:29.163 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:29.163 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:29.164 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 6, 2024-08-18T16:39:29.164 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:30:57.327145Z", 2024-08-18T16:39:29.164 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:29.164 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:29.164 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:29.164 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:29.164 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:29.164 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:29.164 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:29.164 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:29.164 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.165 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_dpc7AV6P5bG3gVZo0Zmw7ZFJ4pFtZiI", 2024-08-18T16:39:29.165 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:29.165 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:29.165 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:29.165 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.165 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:29.165 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj92", 2024-08-18T16:39:29.165 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:29.166 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:29.166 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:39:29.166 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 18 2024-08-18T16:39:29.166 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.167 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:29.167 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:29.167 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:29.167 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:29.168 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-08-18T16:39:29.168 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:27:23.459526Z", 2024-08-18T16:39:29.168 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:29.168 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:29.168 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:29.168 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:29.168 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:29.168 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:29.168 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:29.168 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:29.168 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.168 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_VOjs2WF872PYA5L-kWMv1xCH1DKb88X", 2024-08-18T16:39:29.168 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:29.169 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:29.169 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:29.169 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.169 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:29.169 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj93", 2024-08-18T16:39:29.169 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:29.169 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:29.169 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:39:29.170 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 7 2024-08-18T16:39:29.170 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.170 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:29.170 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:29.170 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:29.170 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:29.170 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:39:29.170 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:33:27.484033Z", 2024-08-18T16:39:29.171 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:29.171 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:29.171 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:29.171 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:29.171 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:29.171 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:29.171 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:29.171 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:29.171 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.172 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_zc4jjce0iYmfKNVqUVYi5N-Prj8AUv8", 2024-08-18T16:39:29.172 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:29.172 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:29.172 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:29.172 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.172 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:29.172 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj95", 2024-08-18T16:39:29.172 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:29.172 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:29.172 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:39:29.173 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 16 2024-08-18T16:39:29.173 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.173 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:29.173 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:29.173 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:29.173 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:29.173 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 6, 2024-08-18T16:39:29.173 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:25:56.027064Z", 2024-08-18T16:39:29.173 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:29.173 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:29.173 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:29.173 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:29.173 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:29.173 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:29.174 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:29.174 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:29.174 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:29.174 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_MQWp_D9z4BcVpaAwTr8dkeTEqf_r1lL", 2024-08-18T16:39:29.174 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:29.174 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:29.174 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:29.174 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:39:29.174 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T16:39:29.175 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:29.175 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:29.175 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-78 --object obj11 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:39:29.226 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:39:29.226 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:39:29.317 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj111', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:29.317 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj111', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:29.317 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-78 --object obj111 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:39:29.368 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:39:29.368 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:39:29.458 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj115', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:29.458 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj115', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:29.458 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-78 --object obj115 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:39:29.508 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:39:29.508 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:39:29.607 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj116', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:29.607 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj116', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:29.607 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-78 --object obj116 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:39:29.656 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:39:29.656 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:39:29.748 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj124', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:29.748 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj124', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:29.749 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-78 --object obj124 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:39:29.797 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:39:29.798 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:39:29.888 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj145', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:29.888 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj145', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:29.888 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-78 --object obj145 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:39:29.937 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:39:29.937 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:39:30.026 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj148', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:30.027 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj148', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:30.027 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-78 --object obj148 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:39:30.076 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:39:30.076 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:39:30.165 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj155', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:30.165 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj155', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:30.166 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-78 --object obj155 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:39:30.214 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:39:30.214 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:39:30.304 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:30.304 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:30.304 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-78 --object obj17 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:39:30.352 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:39:30.353 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:39:30.444 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj185', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:30.444 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj185', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:30.444 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-78 --object obj185 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:39:30.493 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:39:30.493 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:39:30.584 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:30.584 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:30.584 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-78 --object obj19 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:39:30.634 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:39:30.634 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:39:30.723 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:30.723 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:30.723 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-78 --object obj22 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:39:30.772 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:39:30.772 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:39:30.865 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:30.865 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:30.865 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-78 --object obj23 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:39:30.914 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:39:30.914 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:39:31.003 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj29', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:31.003 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj29', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:31.003 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-78 --object obj29 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:39:31.054 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:39:31.054 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:39:31.143 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj31', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:31.143 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj31', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:31.143 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-78 --object obj31 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:39:31.193 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:39:31.193 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:39:31.283 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj33', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:31.283 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj33', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:31.283 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-78 --object obj33 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:39:31.332 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:39:31.333 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:39:31.421 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj34', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:31.421 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj34', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:31.421 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-78 --object obj34 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:39:31.470 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:39:31.470 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:39:31.562 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj37', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:31.562 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj37', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:31.562 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-78 --object obj37 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:39:31.611 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:39:31.611 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:39:31.701 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj38', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:31.701 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj38', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:31.702 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-78 --object obj38 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:39:31.750 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:39:31.750 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:39:31.845 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj44', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:31.845 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj44', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:31.846 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-78 --object obj44 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:39:31.896 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:39:31.896 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:39:31.987 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj46', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:31.987 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj46', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:31.987 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-78 --object obj46 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:39:32.037 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:39:32.037 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:39:32.126 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj51', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:32.126 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj51', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:32.126 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-78 --object obj51 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:39:32.176 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:39:32.176 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:39:32.267 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:32.267 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:32.268 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-78 --object obj55 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:39:32.316 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:39:32.317 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:39:32.406 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj57', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:32.406 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj57', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:32.407 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-78 --object obj57 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:39:32.455 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:39:32.455 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:39:32.544 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj58', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:32.544 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj58', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:32.544 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-78 --object obj58 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:39:32.593 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:39:32.593 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:39:32.685 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:32.685 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:32.685 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-78 --object obj62 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:39:32.734 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:39:32.734 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:39:32.822 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:32.822 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:32.822 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-78 --object obj68 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:39:32.871 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:39:32.871 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:39:32.962 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:32.963 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:32.963 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-78 --object obj69 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:39:33.011 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:39:33.011 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:39:33.099 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:33.100 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:33.100 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-78 --object obj74 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:39:33.149 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:39:33.149 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:39:33.244 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:33.244 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:33.244 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-78 --object obj76 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:39:33.294 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:39:33.294 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:39:33.385 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj85', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:33.385 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj85', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:33.386 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-78 --object obj85 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:39:33.434 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:39:33.434 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:39:33.525 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj87', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:33.525 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj87', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:33.526 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-78 --object obj87 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:39:33.575 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:39:33.575 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:39:33.666 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj92', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:33.666 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj92', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:33.666 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-78 --object obj92 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:39:33.715 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:39:33.715 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:39:33.803 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj93', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:33.803 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj93', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:33.804 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-78 --object obj93 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:39:33.853 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:39:33.853 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:39:33.944 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj95', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:33.945 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--object', 'obj95', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:33.945 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-78 --object obj95 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:39:33.993 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:39:33.993 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:39:34.082 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'trim', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:34.082 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'trim', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:34.082 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog trim --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-78 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:39:34.129 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:39:34.129 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:39:34.182 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'init', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:34.182 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'init', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:34.182 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync init --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-78 --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:39:34.194 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2024-08-18T16:39:34.234 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:39:34.234 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:39:34.280 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'run', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:34.281 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'run', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:39:34.281 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync run --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-78 --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:39:34.331 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:39:34.331 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:39:34.403 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.402+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: sync status for source bucket: init. lease is: not taken. stop indications is: 0 2024-08-18T16:39:34.403 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.402+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: waiting on bucket lease 2024-08-18T16:39:34.408 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.408+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: status after acquiring the lock is: init 2024-08-18T16:39:34.420 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.419+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: listing bucket for full sync 2024-08-18T16:39:34.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.465+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj11 2024-08-18T16:39:34.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.465+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj11]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj11 op=0 op_state=1 2024-08-18T16:39:34.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.465+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj11]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj11[0] 2024-08-18T16:39:34.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.465+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj111 2024-08-18T16:39:34.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.465+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj111]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj111 op=0 op_state=1 2024-08-18T16:39:34.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.465+0000 7f7a82c11880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj111]: : injecting data sync error on key=obj111 2024-08-18T16:39:34.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.465+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj111]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:39:34.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.465+0000 7f7a82c11880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj111]: ERROR: failed to sync object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj111 2024-08-18T16:39:34.473 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.465+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj115 2024-08-18T16:39:34.473 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.465+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj115]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj115 op=0 op_state=1 2024-08-18T16:39:34.473 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.465+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj115]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj115[0] 2024-08-18T16:39:34.473 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.465+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj116 2024-08-18T16:39:34.473 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.465+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj116]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj116 op=0 op_state=1 2024-08-18T16:39:34.473 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.465+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj116]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj116[0] 2024-08-18T16:39:34.473 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.465+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj124 2024-08-18T16:39:34.473 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.465+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj124]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj124 op=0 op_state=1 2024-08-18T16:39:34.473 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.465+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj124]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj124[0] 2024-08-18T16:39:34.473 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.465+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj145 2024-08-18T16:39:34.473 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.465+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj145]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj145 op=0 op_state=1 2024-08-18T16:39:34.473 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.465+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj145]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj145[0] 2024-08-18T16:39:34.474 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.465+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj148 2024-08-18T16:39:34.474 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.465+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj148]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj148 op=0 op_state=1 2024-08-18T16:39:34.474 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.465+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj148]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj148[0] 2024-08-18T16:39:34.474 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.465+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj155 2024-08-18T16:39:34.474 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.465+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj155]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj155 op=0 op_state=1 2024-08-18T16:39:34.474 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.465+0000 7f7a82c11880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj155]: : injecting data sync error on key=obj155 2024-08-18T16:39:34.474 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.465+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj155]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:39:34.474 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.465+0000 7f7a82c11880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj155]: ERROR: failed to sync object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj155 2024-08-18T16:39:34.474 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.465+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj17 2024-08-18T16:39:34.474 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.465+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj17]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj17 op=0 op_state=1 2024-08-18T16:39:34.474 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.465+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj17]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj17[0] 2024-08-18T16:39:34.475 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.465+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj185 2024-08-18T16:39:34.475 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.465+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj185]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj185 op=0 op_state=1 2024-08-18T16:39:34.475 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.465+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj185]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj185[0] 2024-08-18T16:39:34.475 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.465+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj19 2024-08-18T16:39:34.475 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.465+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj19]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj19 op=0 op_state=1 2024-08-18T16:39:34.475 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.465+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj19]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj19[0] 2024-08-18T16:39:34.475 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.465+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj22 2024-08-18T16:39:34.475 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.465+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj22]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj22 op=0 op_state=1 2024-08-18T16:39:34.475 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.465+0000 7f7a82c11880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj22]: : injecting data sync error on key=obj22 2024-08-18T16:39:34.475 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.466+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj22]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:39:34.475 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.466+0000 7f7a82c11880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj22]: ERROR: failed to sync object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj22 2024-08-18T16:39:34.475 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.466+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj23 2024-08-18T16:39:34.476 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.466+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj23]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj23 op=0 op_state=1 2024-08-18T16:39:34.476 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.466+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj23]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj23[0] 2024-08-18T16:39:34.476 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.466+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj29 2024-08-18T16:39:34.476 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.466+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj29]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj29 op=0 op_state=1 2024-08-18T16:39:34.476 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.466+0000 7f7a82c11880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj29]: : injecting data sync error on key=obj29 2024-08-18T16:39:34.476 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.466+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj29]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:39:34.476 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.466+0000 7f7a82c11880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj29]: ERROR: failed to sync object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj29 2024-08-18T16:39:34.476 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.466+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj31 2024-08-18T16:39:34.476 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.466+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj31]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj31 op=0 op_state=1 2024-08-18T16:39:34.476 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.466+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj31]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj31[0] 2024-08-18T16:39:34.476 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.466+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj33 2024-08-18T16:39:34.476 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.466+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj33]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj33 op=0 op_state=1 2024-08-18T16:39:34.476 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.466+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj33]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj33[0] 2024-08-18T16:39:34.477 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.466+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj34 2024-08-18T16:39:34.477 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.466+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj34]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj34 op=0 op_state=1 2024-08-18T16:39:34.477 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.466+0000 7f7a82c11880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj34]: : injecting data sync error on key=obj34 2024-08-18T16:39:34.477 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.466+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj34]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:39:34.477 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.466+0000 7f7a82c11880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj34]: ERROR: failed to sync object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj34 2024-08-18T16:39:34.477 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.466+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj37 2024-08-18T16:39:34.477 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.466+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj37]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj37 op=0 op_state=1 2024-08-18T16:39:34.477 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.466+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj37]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj37[0] 2024-08-18T16:39:34.477 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.466+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj38 2024-08-18T16:39:34.477 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.466+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj38]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj38 op=0 op_state=1 2024-08-18T16:39:34.477 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.466+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj38]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj38[0] 2024-08-18T16:39:34.478 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.466+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj44 2024-08-18T16:39:34.478 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.466+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj44]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj44 op=0 op_state=1 2024-08-18T16:39:34.478 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.466+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj44]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj44[0] 2024-08-18T16:39:34.478 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.466+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj46 2024-08-18T16:39:34.478 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.466+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj46]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj46 op=0 op_state=1 2024-08-18T16:39:34.478 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.466+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj46]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj46[0] 2024-08-18T16:39:34.478 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.468+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: a sync operation returned error 2024-08-18T16:39:34.478 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.468+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj51 2024-08-18T16:39:34.478 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.468+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj51]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj51 op=0 op_state=1 2024-08-18T16:39:34.478 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.468+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj51]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj51[0] 2024-08-18T16:39:34.478 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.468+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: a sync operation returned error 2024-08-18T16:39:34.478 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.468+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj55 2024-08-18T16:39:34.478 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.468+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj55]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj55 op=0 op_state=1 2024-08-18T16:39:34.479 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.468+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj55]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj55[0] 2024-08-18T16:39:34.479 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.468+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: a sync operation returned error 2024-08-18T16:39:34.479 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.468+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj57 2024-08-18T16:39:34.479 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.468+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj57]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj57 op=0 op_state=1 2024-08-18T16:39:34.479 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.468+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj57]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj57[0] 2024-08-18T16:39:34.479 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.468+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: a sync operation returned error 2024-08-18T16:39:34.479 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.468+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj58 2024-08-18T16:39:34.479 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.468+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj58]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj58 op=0 op_state=1 2024-08-18T16:39:34.479 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.468+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj58]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj58[0] 2024-08-18T16:39:34.479 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.470+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: a sync operation returned error 2024-08-18T16:39:34.479 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.470+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj62 2024-08-18T16:39:34.479 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.470+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj62]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj62 op=0 op_state=1 2024-08-18T16:39:34.480 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.470+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj62]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj62[0] 2024-08-18T16:39:34.495 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.494+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj23]: success 2024-08-18T16:39:34.495 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.494+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj68 2024-08-18T16:39:34.495 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.494+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj68]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj68 op=0 op_state=1 2024-08-18T16:39:34.495 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.494+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj68]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj68[0] 2024-08-18T16:39:34.496 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.494+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj11]: success 2024-08-18T16:39:34.496 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.494+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj69 2024-08-18T16:39:34.496 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.494+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj69]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj69 op=0 op_state=1 2024-08-18T16:39:34.496 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.494+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj69]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj69[0] 2024-08-18T16:39:34.496 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.494+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj58]: success 2024-08-18T16:39:34.496 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.494+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj74 2024-08-18T16:39:34.496 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.494+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj74]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj74 op=0 op_state=1 2024-08-18T16:39:34.496 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.494+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj74]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj74[0] 2024-08-18T16:39:34.498 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.495+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj115]: success 2024-08-18T16:39:34.498 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.495+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj33]: success 2024-08-18T16:39:34.498 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.495+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj76 2024-08-18T16:39:34.498 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.495+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj76]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj76 op=0 op_state=1 2024-08-18T16:39:34.498 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.495+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj76]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj76[0] 2024-08-18T16:39:34.498 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.495+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj85 2024-08-18T16:39:34.498 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.495+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj85]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj85 op=0 op_state=1 2024-08-18T16:39:34.498 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.495+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj37]: success 2024-08-18T16:39:34.498 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.495+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj51]: success 2024-08-18T16:39:34.499 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.495+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj85]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj85[0] 2024-08-18T16:39:34.499 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.495+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj38]: success 2024-08-18T16:39:34.499 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.495+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj19]: success 2024-08-18T16:39:34.499 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.495+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj55]: success 2024-08-18T16:39:34.499 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.495+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj87 2024-08-18T16:39:34.499 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.495+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj87]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj87 op=0 op_state=1 2024-08-18T16:39:34.499 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.495+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj87]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj87[0] 2024-08-18T16:39:34.499 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.495+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj92 2024-08-18T16:39:34.499 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.495+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj92]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj92 op=0 op_state=1 2024-08-18T16:39:34.499 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.495+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj92]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj92[0] 2024-08-18T16:39:34.499 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.495+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj93 2024-08-18T16:39:34.499 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.495+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj93]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj93 op=0 op_state=1 2024-08-18T16:39:34.500 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.495+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj93]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj93[0] 2024-08-18T16:39:34.500 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.495+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj95 2024-08-18T16:39:34.500 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.496+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj95]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj95 op=0 op_state=1 2024-08-18T16:39:34.500 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.496+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj95]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj95[0] 2024-08-18T16:39:34.500 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.496+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj124]: success 2024-08-18T16:39:34.500 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.496+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj57]: success 2024-08-18T16:39:34.501 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.498+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj31]: success 2024-08-18T16:39:34.501 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.498+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj185]: success 2024-08-18T16:39:34.501 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.498+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj145]: success 2024-08-18T16:39:34.501 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.499+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj17]: success 2024-08-18T16:39:34.501 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.499+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj116]: success 2024-08-18T16:39:34.501 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.499+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj46]: success 2024-08-18T16:39:34.501 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.499+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj44]: success 2024-08-18T16:39:34.502 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.499+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj148]: success 2024-08-18T16:39:34.525 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.523+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj62]: success 2024-08-18T16:39:34.544 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.543+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj95]: success 2024-08-18T16:39:34.544 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.544+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj85]: success 2024-08-18T16:39:34.545 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.544+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj69]: success 2024-08-18T16:39:34.546 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.545+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj68]: success 2024-08-18T16:39:34.546 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.545+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj87]: success 2024-08-18T16:39:34.547 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.546+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj74]: success 2024-08-18T16:39:34.547 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.546+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj76]: success 2024-08-18T16:39:34.547 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.546+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj93]: success 2024-08-18T16:39:34.548 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.546+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj92]: success 2024-08-18T16:39:34.548 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.546+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: updating marker oid=bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26 marker=obj11 2024-08-18T16:39:34.549 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.548+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: backing out with sync_status=-5 2024-08-18T16:39:34.549 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.548+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: ERROR: full sync failed. error: -5 2024-08-18T16:39:34.550 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.549+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: sync status for source bucket: full. lease is: not taken. stop indications is: 0 2024-08-18T16:39:34.550 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.549+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: waiting on bucket lease 2024-08-18T16:39:34.554 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.553+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: status after acquiring the lock is: full 2024-08-18T16:39:34.566 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.565+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: listing bucket for full sync 2024-08-18T16:39:34.614 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.611+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj11 2024-08-18T16:39:34.614 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.611+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj11]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj11 op=0 op_state=1 2024-08-18T16:39:34.614 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.611+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj11]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj11[0] 2024-08-18T16:39:34.614 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.611+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj111 2024-08-18T16:39:34.614 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.611+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj111]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj111 op=0 op_state=1 2024-08-18T16:39:34.614 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.611+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj111]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj111[0] 2024-08-18T16:39:34.615 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.611+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj115 2024-08-18T16:39:34.615 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.611+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj115]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj115 op=0 op_state=1 2024-08-18T16:39:34.615 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.611+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj115]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj115[0] 2024-08-18T16:39:34.615 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.611+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj116 2024-08-18T16:39:34.615 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.611+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj116]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj116 op=0 op_state=1 2024-08-18T16:39:34.615 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.611+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj116]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj116[0] 2024-08-18T16:39:34.615 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.611+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj124 2024-08-18T16:39:34.615 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.611+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj124]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj124 op=0 op_state=1 2024-08-18T16:39:34.615 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.611+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj124]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj124[0] 2024-08-18T16:39:34.615 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.611+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj145 2024-08-18T16:39:34.615 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.611+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj145]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj145 op=0 op_state=1 2024-08-18T16:39:34.616 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.611+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj145]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj145[0] 2024-08-18T16:39:34.616 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.611+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj148 2024-08-18T16:39:34.616 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.611+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj148]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj148 op=0 op_state=1 2024-08-18T16:39:34.616 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.611+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj148]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj148[0] 2024-08-18T16:39:34.616 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.611+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj155 2024-08-18T16:39:34.616 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.611+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj155]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj155 op=0 op_state=1 2024-08-18T16:39:34.616 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.611+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj155]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj155[0] 2024-08-18T16:39:34.616 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.611+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj17 2024-08-18T16:39:34.617 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.611+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj17]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj17 op=0 op_state=1 2024-08-18T16:39:34.617 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.611+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj17]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj17[0] 2024-08-18T16:39:34.617 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.611+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj185 2024-08-18T16:39:34.617 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.611+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj185]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj185 op=0 op_state=1 2024-08-18T16:39:34.617 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.611+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj185]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj185[0] 2024-08-18T16:39:34.617 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.611+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj19 2024-08-18T16:39:34.617 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.611+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj19]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj19 op=0 op_state=1 2024-08-18T16:39:34.617 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.611+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj19]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj19[0] 2024-08-18T16:39:34.617 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.611+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj22 2024-08-18T16:39:34.617 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.611+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj22]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj22 op=0 op_state=1 2024-08-18T16:39:34.617 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.611+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj22]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj22[0] 2024-08-18T16:39:34.617 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.612+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj23 2024-08-18T16:39:34.618 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.612+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj23]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj23 op=0 op_state=1 2024-08-18T16:39:34.618 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.612+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj23]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj23[0] 2024-08-18T16:39:34.618 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.612+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj29 2024-08-18T16:39:34.618 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.612+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj29]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj29 op=0 op_state=1 2024-08-18T16:39:34.618 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.612+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj29]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj29[0] 2024-08-18T16:39:34.618 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.612+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj31 2024-08-18T16:39:34.618 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.612+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj31]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj31 op=0 op_state=1 2024-08-18T16:39:34.618 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.612+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj31]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj31[0] 2024-08-18T16:39:34.618 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.612+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj33 2024-08-18T16:39:34.618 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.612+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj33]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj33 op=0 op_state=1 2024-08-18T16:39:34.619 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.612+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj33]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj33[0] 2024-08-18T16:39:34.619 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.612+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj34 2024-08-18T16:39:34.619 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.612+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj34]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj34 op=0 op_state=1 2024-08-18T16:39:34.619 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.612+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj34]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj34[0] 2024-08-18T16:39:34.619 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.612+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj37 2024-08-18T16:39:34.619 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.612+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj37]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj37 op=0 op_state=1 2024-08-18T16:39:34.619 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.612+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj37]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj37[0] 2024-08-18T16:39:34.619 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.612+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj38 2024-08-18T16:39:34.619 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.612+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj38]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj38 op=0 op_state=1 2024-08-18T16:39:34.619 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.612+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj38]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj38[0] 2024-08-18T16:39:34.619 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.612+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj44 2024-08-18T16:39:34.620 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.612+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj44]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj44 op=0 op_state=1 2024-08-18T16:39:34.620 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.612+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj44]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj44[0] 2024-08-18T16:39:34.620 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.612+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj46 2024-08-18T16:39:34.620 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.612+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj46]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj46 op=0 op_state=1 2024-08-18T16:39:34.620 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.612+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj46]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj46[0] 2024-08-18T16:39:34.624 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.614+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj11]: success 2024-08-18T16:39:34.625 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.614+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj51 2024-08-18T16:39:34.625 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.614+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj51]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj51 op=0 op_state=1 2024-08-18T16:39:34.625 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.614+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj51]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj51[0] 2024-08-18T16:39:34.625 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.615+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj145]: success 2024-08-18T16:39:34.625 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.615+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj55 2024-08-18T16:39:34.625 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.615+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj55]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj55 op=0 op_state=1 2024-08-18T16:39:34.625 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.615+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj55]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj55[0] 2024-08-18T16:39:34.625 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.615+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj115]: success 2024-08-18T16:39:34.625 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.615+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj57 2024-08-18T16:39:34.625 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.615+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj57]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj57 op=0 op_state=1 2024-08-18T16:39:34.626 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.615+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj57]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj57[0] 2024-08-18T16:39:34.626 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.616+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj124]: success 2024-08-18T16:39:34.626 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.616+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj58 2024-08-18T16:39:34.626 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.616+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj58]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj58 op=0 op_state=1 2024-08-18T16:39:34.626 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.616+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj58]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj58[0] 2024-08-18T16:39:34.626 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.616+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj148]: success 2024-08-18T16:39:34.626 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.616+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj62 2024-08-18T16:39:34.626 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.616+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj62]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj62 op=0 op_state=1 2024-08-18T16:39:34.626 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.616+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj62]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj62[0] 2024-08-18T16:39:34.626 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.616+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj17]: success 2024-08-18T16:39:34.626 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.616+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj68 2024-08-18T16:39:34.627 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.617+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj68]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj68 op=0 op_state=1 2024-08-18T16:39:34.627 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.617+0000 7f7a82c11880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj68]: : injecting data sync error on key=obj68 2024-08-18T16:39:34.627 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.617+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj68]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:39:34.627 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.617+0000 7f7a82c11880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj68]: ERROR: failed to sync object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj68 2024-08-18T16:39:34.627 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.617+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj19]: success 2024-08-18T16:39:34.627 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.617+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj69 2024-08-18T16:39:34.627 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.617+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj69]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj69 op=0 op_state=1 2024-08-18T16:39:34.627 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.617+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj69]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj69[0] 2024-08-18T16:39:34.627 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.617+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj46]: success 2024-08-18T16:39:34.627 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.617+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj74 2024-08-18T16:39:34.627 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.617+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj74]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj74 op=0 op_state=1 2024-08-18T16:39:34.627 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.617+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj74]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj74[0] 2024-08-18T16:39:34.628 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.617+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj23]: success 2024-08-18T16:39:34.628 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.617+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj76 2024-08-18T16:39:34.628 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.618+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj76]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj76 op=0 op_state=1 2024-08-18T16:39:34.628 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.618+0000 7f7a82c11880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj76]: : injecting data sync error on key=obj76 2024-08-18T16:39:34.628 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.618+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj76]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:39:34.628 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.618+0000 7f7a82c11880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj76]: ERROR: failed to sync object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj76 2024-08-18T16:39:34.628 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.618+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj116]: success 2024-08-18T16:39:34.628 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.618+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj85 2024-08-18T16:39:34.628 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.618+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj85]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj85 op=0 op_state=1 2024-08-18T16:39:34.628 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.618+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj85]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj85[0] 2024-08-18T16:39:34.628 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.619+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj44]: success 2024-08-18T16:39:34.628 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.619+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj87 2024-08-18T16:39:34.628 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.619+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj87]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj87 op=0 op_state=1 2024-08-18T16:39:34.629 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.619+0000 7f7a82c11880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj87]: : injecting data sync error on key=obj87 2024-08-18T16:39:34.629 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.619+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj87]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:39:34.629 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.619+0000 7f7a82c11880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj87]: ERROR: failed to sync object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj87 2024-08-18T16:39:34.629 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.620+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj31]: success 2024-08-18T16:39:34.629 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.620+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj92 2024-08-18T16:39:34.629 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.620+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj92]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj92 op=0 op_state=1 2024-08-18T16:39:34.629 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.620+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj92]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj92[0] 2024-08-18T16:39:34.629 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.620+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj37]: success 2024-08-18T16:39:34.629 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.620+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj93 2024-08-18T16:39:34.629 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.620+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj93]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj93 op=0 op_state=1 2024-08-18T16:39:34.629 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.621+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj93]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj93[0] 2024-08-18T16:39:34.629 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.622+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj38]: success 2024-08-18T16:39:34.630 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.622+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj51]: success 2024-08-18T16:39:34.630 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.622+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj95 2024-08-18T16:39:34.630 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.622+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj95]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj95 op=0 op_state=1 2024-08-18T16:39:34.630 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.622+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj95]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj95[0] 2024-08-18T16:39:34.630 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.622+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj33]: success 2024-08-18T16:39:34.630 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.622+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj57]: success 2024-08-18T16:39:34.630 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.622+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: a sync operation returned error 2024-08-18T16:39:34.630 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.622+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj62]: success 2024-08-18T16:39:34.630 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.622+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj55]: success 2024-08-18T16:39:34.630 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.622+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj74]: success 2024-08-18T16:39:34.630 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.623+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj185]: success 2024-08-18T16:39:34.630 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.623+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj58]: success 2024-08-18T16:39:34.631 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.623+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj85]: success 2024-08-18T16:39:34.631 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.623+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj69]: success 2024-08-18T16:39:34.631 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.623+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: a sync operation returned error 2024-08-18T16:39:34.631 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.623+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj92]: success 2024-08-18T16:39:34.631 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.624+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: a sync operation returned error 2024-08-18T16:39:34.631 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.624+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj93]: success 2024-08-18T16:39:34.632 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.624+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj111]: success 2024-08-18T16:39:34.632 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.624+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: updating marker oid=bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26 marker=obj148 2024-08-18T16:39:34.632 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.625+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj95]: success 2024-08-18T16:39:34.632 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.625+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj155]: success 2024-08-18T16:39:34.632 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.625+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj22]: success 2024-08-18T16:39:34.632 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.626+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj34]: success 2024-08-18T16:39:34.664 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.663+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj29]: success 2024-08-18T16:39:34.664 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.663+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: updating marker oid=bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26 marker=obj62 2024-08-18T16:39:34.665 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.664+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: backing out with sync_status=-5 2024-08-18T16:39:34.665 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.664+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: ERROR: full sync failed. error: -5 2024-08-18T16:39:34.666 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.665+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: sync status for source bucket: full. lease is: not taken. stop indications is: 0 2024-08-18T16:39:34.666 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.665+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: waiting on bucket lease 2024-08-18T16:39:34.671 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.671+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: status after acquiring the lock is: full 2024-08-18T16:39:34.683 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.682+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: listing bucket for full sync 2024-08-18T16:39:34.730 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.727+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj11 2024-08-18T16:39:34.731 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.727+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj11]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj11 op=0 op_state=1 2024-08-18T16:39:34.731 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.727+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj11]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj11[0] 2024-08-18T16:39:34.731 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.727+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj111 2024-08-18T16:39:34.731 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.727+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj111]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj111 op=0 op_state=1 2024-08-18T16:39:34.731 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.727+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj111]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj111[0] 2024-08-18T16:39:34.731 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.727+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj115 2024-08-18T16:39:34.731 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.727+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj115]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj115 op=0 op_state=1 2024-08-18T16:39:34.732 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.727+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj115]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj115[0] 2024-08-18T16:39:34.732 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.727+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj116 2024-08-18T16:39:34.732 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.727+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj116]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj116 op=0 op_state=1 2024-08-18T16:39:34.732 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.727+0000 7f7a82c11880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj116]: : injecting data sync error on key=obj116 2024-08-18T16:39:34.732 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.727+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj116]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:39:34.733 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.727+0000 7f7a82c11880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj116]: ERROR: failed to sync object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj116 2024-08-18T16:39:34.733 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.727+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj124 2024-08-18T16:39:34.733 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.727+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj124]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj124 op=0 op_state=1 2024-08-18T16:39:34.734 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.727+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj124]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj124[0] 2024-08-18T16:39:34.734 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.727+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj145 2024-08-18T16:39:34.734 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.727+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj145]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj145 op=0 op_state=1 2024-08-18T16:39:34.734 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.727+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj145]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj145[0] 2024-08-18T16:39:34.734 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.727+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj148 2024-08-18T16:39:34.734 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.727+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj148]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj148 op=0 op_state=1 2024-08-18T16:39:34.735 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.727+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj148]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj148[0] 2024-08-18T16:39:34.735 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.727+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj155 2024-08-18T16:39:34.735 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.727+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj155]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj155 op=0 op_state=1 2024-08-18T16:39:34.735 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.727+0000 7f7a82c11880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj155]: : injecting data sync error on key=obj155 2024-08-18T16:39:34.735 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.727+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj155]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:39:34.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.727+0000 7f7a82c11880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj155]: ERROR: failed to sync object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj155 2024-08-18T16:39:34.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.727+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj17 2024-08-18T16:39:34.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.727+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj17]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj17 op=0 op_state=1 2024-08-18T16:39:34.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.727+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj17]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj17[0] 2024-08-18T16:39:34.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.727+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj185 2024-08-18T16:39:34.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.727+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj185]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj185 op=0 op_state=1 2024-08-18T16:39:34.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.727+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj185]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj185[0] 2024-08-18T16:39:34.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.727+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj19 2024-08-18T16:39:34.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.727+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj19]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj19 op=0 op_state=1 2024-08-18T16:39:34.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.727+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj19]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj19[0] 2024-08-18T16:39:34.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.727+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj22 2024-08-18T16:39:34.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.727+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj22]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj22 op=0 op_state=1 2024-08-18T16:39:34.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.727+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj22]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj22[0] 2024-08-18T16:39:34.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.727+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj23 2024-08-18T16:39:34.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.727+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj23]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj23 op=0 op_state=1 2024-08-18T16:39:34.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.727+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj23]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj23[0] 2024-08-18T16:39:34.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.727+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj29 2024-08-18T16:39:34.739 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.728+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj29]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj29 op=0 op_state=1 2024-08-18T16:39:34.739 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.728+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj29]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj29[0] 2024-08-18T16:39:34.739 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.728+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj31 2024-08-18T16:39:34.739 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.728+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj31]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj31 op=0 op_state=1 2024-08-18T16:39:34.739 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.728+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj31]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj31[0] 2024-08-18T16:39:34.740 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.728+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj33 2024-08-18T16:39:34.740 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.728+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj33]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj33 op=0 op_state=1 2024-08-18T16:39:34.740 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.728+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj33]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj33[0] 2024-08-18T16:39:34.740 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.728+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj34 2024-08-18T16:39:34.740 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.728+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj34]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj34 op=0 op_state=1 2024-08-18T16:39:34.740 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.728+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj34]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj34[0] 2024-08-18T16:39:34.741 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.728+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj37 2024-08-18T16:39:34.741 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.728+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj37]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj37 op=0 op_state=1 2024-08-18T16:39:34.741 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.728+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj37]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj37[0] 2024-08-18T16:39:34.741 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.728+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj38 2024-08-18T16:39:34.741 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.728+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj38]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj38 op=0 op_state=1 2024-08-18T16:39:34.742 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.728+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj38]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj38[0] 2024-08-18T16:39:34.742 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.728+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj44 2024-08-18T16:39:34.742 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.728+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj44]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj44 op=0 op_state=1 2024-08-18T16:39:34.742 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.728+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj44]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj44[0] 2024-08-18T16:39:34.742 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.728+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj46 2024-08-18T16:39:34.743 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.728+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj46]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj46 op=0 op_state=1 2024-08-18T16:39:34.743 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.728+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj46]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj46[0] 2024-08-18T16:39:34.753 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.731+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj115]: success 2024-08-18T16:39:34.753 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.731+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj51 2024-08-18T16:39:34.754 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.731+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj51]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj51 op=0 op_state=1 2024-08-18T16:39:34.754 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.731+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj51]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj51[0] 2024-08-18T16:39:34.754 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.732+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj11]: success 2024-08-18T16:39:34.754 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.732+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj55 2024-08-18T16:39:34.754 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.732+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj55]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj55 op=0 op_state=1 2024-08-18T16:39:34.754 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.732+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj55]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj55[0] 2024-08-18T16:39:34.755 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.732+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj17]: success 2024-08-18T16:39:34.755 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.732+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: a sync operation returned error 2024-08-18T16:39:34.755 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.732+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj57 2024-08-18T16:39:34.755 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.732+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj57]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj57 op=0 op_state=1 2024-08-18T16:39:34.755 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.732+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj57]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj57[0] 2024-08-18T16:39:34.756 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.732+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj58 2024-08-18T16:39:34.756 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.732+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj58]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj58 op=0 op_state=1 2024-08-18T16:39:34.756 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.732+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj58]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj58[0] 2024-08-18T16:39:34.756 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.732+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj185]: success 2024-08-18T16:39:34.756 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.732+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj145]: success 2024-08-18T16:39:34.757 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.732+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: a sync operation returned error 2024-08-18T16:39:34.757 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.732+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj124]: success 2024-08-18T16:39:34.757 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.732+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj44]: success 2024-08-18T16:39:34.757 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.732+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj62 2024-08-18T16:39:34.757 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.732+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj62]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj62 op=0 op_state=1 2024-08-18T16:39:34.758 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.732+0000 7f7a82c11880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj62]: : injecting data sync error on key=obj62 2024-08-18T16:39:34.758 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.732+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj62]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:39:34.758 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.732+0000 7f7a82c11880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj62]: ERROR: failed to sync object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj62 2024-08-18T16:39:34.758 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.732+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj68 2024-08-18T16:39:34.758 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.732+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj68]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj68 op=0 op_state=1 2024-08-18T16:39:34.759 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.732+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj68]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj68[0] 2024-08-18T16:39:34.759 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.732+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj69 2024-08-18T16:39:34.759 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.732+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj69]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj69 op=0 op_state=1 2024-08-18T16:39:34.759 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.732+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj69]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj69[0] 2024-08-18T16:39:34.759 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.732+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj74 2024-08-18T16:39:34.760 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.732+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj74]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj74 op=0 op_state=1 2024-08-18T16:39:34.760 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.732+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj74]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj74[0] 2024-08-18T16:39:34.760 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.733+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj76 2024-08-18T16:39:34.760 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.733+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj76]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj76 op=0 op_state=1 2024-08-18T16:39:34.760 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.733+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj76]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj76[0] 2024-08-18T16:39:34.760 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.733+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj148]: success 2024-08-18T16:39:34.761 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.733+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj85 2024-08-18T16:39:34.761 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.733+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj85]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj85 op=0 op_state=1 2024-08-18T16:39:34.761 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.733+0000 7f7a82c11880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj85]: : injecting data sync error on key=obj85 2024-08-18T16:39:34.761 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.733+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj85]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:39:34.761 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.733+0000 7f7a82c11880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj85]: ERROR: failed to sync object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj85 2024-08-18T16:39:34.761 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.734+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj22]: success 2024-08-18T16:39:34.761 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.734+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj38]: success 2024-08-18T16:39:34.762 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.734+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj33]: success 2024-08-18T16:39:34.762 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.734+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj29]: success 2024-08-18T16:39:34.762 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.734+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj87 2024-08-18T16:39:34.762 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.734+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj87]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj87 op=0 op_state=1 2024-08-18T16:39:34.762 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.734+0000 7f7a82c11880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj87]: : injecting data sync error on key=obj87 2024-08-18T16:39:34.762 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.734+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj87]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:39:34.762 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.734+0000 7f7a82c11880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj87]: ERROR: failed to sync object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj87 2024-08-18T16:39:34.762 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.734+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj92 2024-08-18T16:39:34.762 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.734+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj92]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj92 op=0 op_state=1 2024-08-18T16:39:34.762 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.734+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj92]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj92[0] 2024-08-18T16:39:34.762 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.734+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj93 2024-08-18T16:39:34.762 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.734+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj93]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj93 op=0 op_state=1 2024-08-18T16:39:34.763 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.734+0000 7f7a82c11880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj93]: : injecting data sync error on key=obj93 2024-08-18T16:39:34.763 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.734+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj93]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:39:34.763 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.734+0000 7f7a82c11880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj93]: ERROR: failed to sync object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj93 2024-08-18T16:39:34.763 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.734+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj95 2024-08-18T16:39:34.763 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.734+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj95]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj95 op=0 op_state=1 2024-08-18T16:39:34.763 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.734+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj95]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj95[0] 2024-08-18T16:39:34.763 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.737+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj111]: success 2024-08-18T16:39:34.763 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.737+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: updating marker oid=bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26 marker=obj115 2024-08-18T16:39:34.763 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.737+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj19]: success 2024-08-18T16:39:34.763 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.737+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj31]: success 2024-08-18T16:39:34.763 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.737+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj51]: success 2024-08-18T16:39:34.763 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.737+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj46]: success 2024-08-18T16:39:34.763 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.737+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: a sync operation returned error 2024-08-18T16:39:34.764 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.737+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: a sync operation returned error 2024-08-18T16:39:34.764 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.737+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: a sync operation returned error 2024-08-18T16:39:34.764 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.738+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj23]: success 2024-08-18T16:39:34.764 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.738+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj37]: success 2024-08-18T16:39:34.764 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.738+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj34]: success 2024-08-18T16:39:34.764 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.738+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj68]: success 2024-08-18T16:39:34.764 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.738+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj57]: success 2024-08-18T16:39:34.764 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.738+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj76]: success 2024-08-18T16:39:34.764 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.738+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj55]: success 2024-08-18T16:39:34.764 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.738+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj95]: success 2024-08-18T16:39:34.764 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.738+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj92]: success 2024-08-18T16:39:34.764 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.738+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj58]: success 2024-08-18T16:39:34.765 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.738+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj69]: success 2024-08-18T16:39:34.765 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.739+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj74]: success 2024-08-18T16:39:34.765 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.739+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: a sync operation returned error 2024-08-18T16:39:34.765 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.739+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: backing out with sync_status=-5 2024-08-18T16:39:34.765 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.739+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: ERROR: full sync failed. error: -5 2024-08-18T16:39:34.765 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.740+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: sync status for source bucket: full. lease is: not taken. stop indications is: 0 2024-08-18T16:39:34.765 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.740+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: waiting on bucket lease 2024-08-18T16:39:34.785 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.784+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: status after acquiring the lock is: full 2024-08-18T16:39:34.796 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.796+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: listing bucket for full sync 2024-08-18T16:39:34.844 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.841+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj11 2024-08-18T16:39:34.845 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.841+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj11]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj11 op=0 op_state=1 2024-08-18T16:39:34.845 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.841+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj11]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj11[0] 2024-08-18T16:39:34.845 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.841+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj111 2024-08-18T16:39:34.845 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.841+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj111]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj111 op=0 op_state=1 2024-08-18T16:39:34.845 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.841+0000 7f7a82c11880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj111]: : injecting data sync error on key=obj111 2024-08-18T16:39:34.845 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.841+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj111]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:39:34.845 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.841+0000 7f7a82c11880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj111]: ERROR: failed to sync object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj111 2024-08-18T16:39:34.845 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.841+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj115 2024-08-18T16:39:34.845 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.841+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj115]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj115 op=0 op_state=1 2024-08-18T16:39:34.846 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.841+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj115]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj115[0] 2024-08-18T16:39:34.846 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.841+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj116 2024-08-18T16:39:34.846 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.841+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj116]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj116 op=0 op_state=1 2024-08-18T16:39:34.846 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.841+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj116]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj116[0] 2024-08-18T16:39:34.846 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.841+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj124 2024-08-18T16:39:34.846 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.841+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj124]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj124 op=0 op_state=1 2024-08-18T16:39:34.846 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.841+0000 7f7a82c11880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj124]: : injecting data sync error on key=obj124 2024-08-18T16:39:34.846 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.841+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj124]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:39:34.846 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.841+0000 7f7a82c11880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj124]: ERROR: failed to sync object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj124 2024-08-18T16:39:34.846 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.841+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj145 2024-08-18T16:39:34.847 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.841+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj145]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj145 op=0 op_state=1 2024-08-18T16:39:34.847 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.841+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj145]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj145[0] 2024-08-18T16:39:34.847 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.842+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj148 2024-08-18T16:39:34.847 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.842+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj148]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj148 op=0 op_state=1 2024-08-18T16:39:34.847 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.842+0000 7f7a82c11880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj148]: : injecting data sync error on key=obj148 2024-08-18T16:39:34.847 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.842+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj148]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:39:34.847 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.842+0000 7f7a82c11880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj148]: ERROR: failed to sync object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj148 2024-08-18T16:39:34.847 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.842+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj155 2024-08-18T16:39:34.847 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.842+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj155]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj155 op=0 op_state=1 2024-08-18T16:39:34.847 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.842+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj155]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj155[0] 2024-08-18T16:39:34.847 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.842+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj17 2024-08-18T16:39:34.847 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.842+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj17]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj17 op=0 op_state=1 2024-08-18T16:39:34.848 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.842+0000 7f7a82c11880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj17]: : injecting data sync error on key=obj17 2024-08-18T16:39:34.848 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.842+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj17]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:39:34.848 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.842+0000 7f7a82c11880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj17]: ERROR: failed to sync object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj17 2024-08-18T16:39:34.848 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.842+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj185 2024-08-18T16:39:34.848 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.842+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj185]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj185 op=0 op_state=1 2024-08-18T16:39:34.848 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.842+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj185]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj185[0] 2024-08-18T16:39:34.848 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.842+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj19 2024-08-18T16:39:34.848 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.842+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj19]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj19 op=0 op_state=1 2024-08-18T16:39:34.848 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.842+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj19]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj19[0] 2024-08-18T16:39:34.848 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.842+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj22 2024-08-18T16:39:34.848 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.842+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj22]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj22 op=0 op_state=1 2024-08-18T16:39:34.848 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.842+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj22]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj22[0] 2024-08-18T16:39:34.849 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.843+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj23 2024-08-18T16:39:34.849 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.843+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj23]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj23 op=0 op_state=1 2024-08-18T16:39:34.849 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.843+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj23]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj23[0] 2024-08-18T16:39:34.849 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.843+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj29 2024-08-18T16:39:34.849 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.843+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj29]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj29 op=0 op_state=1 2024-08-18T16:39:34.849 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.843+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj29]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj29[0] 2024-08-18T16:39:34.849 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.843+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj31 2024-08-18T16:39:34.849 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.843+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj31]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj31 op=0 op_state=1 2024-08-18T16:39:34.849 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.843+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj31]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj31[0] 2024-08-18T16:39:34.849 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.843+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj33 2024-08-18T16:39:34.849 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.843+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj33]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj33 op=0 op_state=1 2024-08-18T16:39:34.849 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.843+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj33]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj33[0] 2024-08-18T16:39:34.850 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.843+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj34 2024-08-18T16:39:34.850 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.843+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj34]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj34 op=0 op_state=1 2024-08-18T16:39:34.850 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.843+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj34]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj34[0] 2024-08-18T16:39:34.850 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.843+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj37 2024-08-18T16:39:34.850 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.843+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj37]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj37 op=0 op_state=1 2024-08-18T16:39:34.850 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.843+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj37]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj37[0] 2024-08-18T16:39:34.850 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.843+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj38 2024-08-18T16:39:34.850 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.843+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj38]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj38 op=0 op_state=1 2024-08-18T16:39:34.850 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.843+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj38]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj38[0] 2024-08-18T16:39:34.850 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.843+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj44 2024-08-18T16:39:34.850 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.843+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj44]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj44 op=0 op_state=1 2024-08-18T16:39:34.850 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.843+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj44]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj44[0] 2024-08-18T16:39:34.851 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.843+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj46 2024-08-18T16:39:34.851 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.843+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj46]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj46 op=0 op_state=1 2024-08-18T16:39:34.851 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.843+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj46]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj46[0] 2024-08-18T16:39:34.853 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.846+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj185]: success 2024-08-18T16:39:34.853 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.846+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj116]: success 2024-08-18T16:39:34.853 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.846+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj115]: success 2024-08-18T16:39:34.853 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.846+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: a sync operation returned error 2024-08-18T16:39:34.853 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.846+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: a sync operation returned error 2024-08-18T16:39:34.853 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.846+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj51 2024-08-18T16:39:34.854 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.846+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj51]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj51 op=0 op_state=1 2024-08-18T16:39:34.854 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.846+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj51]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj51[0] 2024-08-18T16:39:34.854 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.846+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj55 2024-08-18T16:39:34.854 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.846+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj55]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj55 op=0 op_state=1 2024-08-18T16:39:34.854 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.846+0000 7f7a82c11880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj55]: : injecting data sync error on key=obj55 2024-08-18T16:39:34.854 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.846+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj55]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:39:34.854 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.846+0000 7f7a82c11880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj55]: ERROR: failed to sync object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj55 2024-08-18T16:39:34.854 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.846+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj57 2024-08-18T16:39:34.855 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.846+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj57]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj57 op=0 op_state=1 2024-08-18T16:39:34.855 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.846+0000 7f7a82c11880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj57]: : injecting data sync error on key=obj57 2024-08-18T16:39:34.855 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.846+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj57]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:39:34.855 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.846+0000 7f7a82c11880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj57]: ERROR: failed to sync object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj57 2024-08-18T16:39:34.855 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.846+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj58 2024-08-18T16:39:34.855 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.846+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj58]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj58 op=0 op_state=1 2024-08-18T16:39:34.855 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.846+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj58]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj58[0] 2024-08-18T16:39:34.855 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.846+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj62 2024-08-18T16:39:34.855 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.846+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj62]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj62 op=0 op_state=1 2024-08-18T16:39:34.855 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.846+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj62]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj62[0] 2024-08-18T16:39:34.856 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.846+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj11]: success 2024-08-18T16:39:34.856 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.846+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj68 2024-08-18T16:39:34.856 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.847+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj68]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj68 op=0 op_state=1 2024-08-18T16:39:34.856 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.847+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj68]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj68[0] 2024-08-18T16:39:34.856 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.847+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj145]: success 2024-08-18T16:39:34.856 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.847+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj22]: success 2024-08-18T16:39:34.856 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.847+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj69 2024-08-18T16:39:34.856 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.847+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj69]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj69 op=0 op_state=1 2024-08-18T16:39:34.856 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.847+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj69]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj69[0] 2024-08-18T16:39:34.856 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.847+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj74 2024-08-18T16:39:34.856 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.847+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj74]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj74 op=0 op_state=1 2024-08-18T16:39:34.856 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.847+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj74]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj74[0] 2024-08-18T16:39:34.857 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.848+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj155]: success 2024-08-18T16:39:34.857 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.848+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj46]: success 2024-08-18T16:39:34.857 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.848+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj76 2024-08-18T16:39:34.857 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.848+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj76]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj76 op=0 op_state=1 2024-08-18T16:39:34.857 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.848+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj76]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj76[0] 2024-08-18T16:39:34.857 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.848+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj85 2024-08-18T16:39:34.857 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.848+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj85]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj85 op=0 op_state=1 2024-08-18T16:39:34.857 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.848+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj85]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj85[0] 2024-08-18T16:39:34.857 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.848+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj44]: success 2024-08-18T16:39:34.857 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.849+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj23]: success 2024-08-18T16:39:34.857 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.849+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj87 2024-08-18T16:39:34.858 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.850+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj87]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj87 op=0 op_state=1 2024-08-18T16:39:34.858 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.850+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj87]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj87[0] 2024-08-18T16:39:34.858 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.850+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj92 2024-08-18T16:39:34.858 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.850+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj92]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj92 op=0 op_state=1 2024-08-18T16:39:34.858 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.850+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj38]: success 2024-08-18T16:39:34.858 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.850+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj92]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj92[0] 2024-08-18T16:39:34.858 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.851+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: a sync operation returned error 2024-08-18T16:39:34.858 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.851+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: a sync operation returned error 2024-08-18T16:39:34.858 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.851+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj33]: success 2024-08-18T16:39:34.858 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.851+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj34]: success 2024-08-18T16:39:34.858 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.851+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj31]: success 2024-08-18T16:39:34.859 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.851+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj37]: success 2024-08-18T16:39:34.859 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.851+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj29]: success 2024-08-18T16:39:34.859 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.851+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj93 2024-08-18T16:39:34.859 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.851+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj93]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj93 op=0 op_state=1 2024-08-18T16:39:34.859 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.851+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj51]: success 2024-08-18T16:39:34.859 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.851+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj93]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj93[0] 2024-08-18T16:39:34.859 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.851+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj62]: success 2024-08-18T16:39:34.859 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.851+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj95 2024-08-18T16:39:34.859 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.851+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj95]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj95 op=0 op_state=1 2024-08-18T16:39:34.859 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.851+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj95]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj95[0] 2024-08-18T16:39:34.859 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.851+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: a sync operation returned error 2024-08-18T16:39:34.859 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.851+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: a sync operation returned error 2024-08-18T16:39:34.860 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.851+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj19]: success 2024-08-18T16:39:34.860 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.851+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj58]: success 2024-08-18T16:39:34.860 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.851+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj69]: success 2024-08-18T16:39:34.860 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.851+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj85]: success 2024-08-18T16:39:34.860 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.852+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj74]: success 2024-08-18T16:39:34.860 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.852+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj68]: success 2024-08-18T16:39:34.860 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.852+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj76]: success 2024-08-18T16:39:34.861 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.853+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj92]: success 2024-08-18T16:39:34.861 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.853+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj87]: success 2024-08-18T16:39:34.861 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.853+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj93]: success 2024-08-18T16:39:34.861 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.853+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj95]: success 2024-08-18T16:39:34.861 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.853+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: updating marker oid=bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26 marker=obj11 2024-08-18T16:39:34.861 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.854+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: backing out with sync_status=-5 2024-08-18T16:39:34.861 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.854+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: ERROR: full sync failed. error: -5 2024-08-18T16:39:34.862 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.856+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: sync status for source bucket: full. lease is: not taken. stop indications is: 0 2024-08-18T16:39:34.862 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.856+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: waiting on bucket lease 2024-08-18T16:39:34.901 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.900+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: status after acquiring the lock is: full 2024-08-18T16:39:34.913 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.912+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: listing bucket for full sync 2024-08-18T16:39:34.961 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.958+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj11 2024-08-18T16:39:34.961 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.958+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj11]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj11 op=0 op_state=1 2024-08-18T16:39:34.961 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.958+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj11]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj11[0] 2024-08-18T16:39:34.961 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.958+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj111 2024-08-18T16:39:34.961 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.958+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj111]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj111 op=0 op_state=1 2024-08-18T16:39:34.961 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.958+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj111]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj111[0] 2024-08-18T16:39:34.962 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.958+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj115 2024-08-18T16:39:34.962 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.958+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj115]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj115 op=0 op_state=1 2024-08-18T16:39:34.962 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.958+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj115]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj115[0] 2024-08-18T16:39:34.962 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.958+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj116 2024-08-18T16:39:34.962 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.958+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj116]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj116 op=0 op_state=1 2024-08-18T16:39:34.963 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.958+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj116]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj116[0] 2024-08-18T16:39:34.963 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.958+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj124 2024-08-18T16:39:34.963 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.958+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj124]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj124 op=0 op_state=1 2024-08-18T16:39:34.963 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.958+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj124]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj124[0] 2024-08-18T16:39:34.963 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.958+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj145 2024-08-18T16:39:34.964 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.958+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj145]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj145 op=0 op_state=1 2024-08-18T16:39:34.964 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.958+0000 7f7a82c11880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj145]: : injecting data sync error on key=obj145 2024-08-18T16:39:34.964 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.958+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj145]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:39:34.964 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.958+0000 7f7a82c11880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj145]: ERROR: failed to sync object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj145 2024-08-18T16:39:34.964 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.958+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj148 2024-08-18T16:39:34.964 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.958+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj148]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj148 op=0 op_state=1 2024-08-18T16:39:34.965 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.958+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj148]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj148[0] 2024-08-18T16:39:34.965 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.958+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj155 2024-08-18T16:39:34.965 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.958+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj155]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj155 op=0 op_state=1 2024-08-18T16:39:34.965 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.958+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj155]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj155[0] 2024-08-18T16:39:34.965 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.958+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj17 2024-08-18T16:39:34.966 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.959+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj17]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj17 op=0 op_state=1 2024-08-18T16:39:34.966 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.959+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj17]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj17[0] 2024-08-18T16:39:34.966 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.959+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj185 2024-08-18T16:39:34.966 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.959+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj185]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj185 op=0 op_state=1 2024-08-18T16:39:34.966 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.959+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj185]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj185[0] 2024-08-18T16:39:34.966 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.959+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj19 2024-08-18T16:39:34.967 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.959+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj19]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj19 op=0 op_state=1 2024-08-18T16:39:34.967 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.959+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj19]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj19[0] 2024-08-18T16:39:34.967 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.959+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj22 2024-08-18T16:39:34.967 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.959+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj22]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj22 op=0 op_state=1 2024-08-18T16:39:34.967 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.959+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj22]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj22[0] 2024-08-18T16:39:34.968 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.959+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj23 2024-08-18T16:39:34.968 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.959+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj23]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj23 op=0 op_state=1 2024-08-18T16:39:34.968 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.959+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj23]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj23[0] 2024-08-18T16:39:34.968 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.959+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj29 2024-08-18T16:39:34.969 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.959+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj29]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj29 op=0 op_state=1 2024-08-18T16:39:34.969 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.959+0000 7f7a82c11880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj29]: : injecting data sync error on key=obj29 2024-08-18T16:39:34.969 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.959+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj29]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:39:34.969 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.959+0000 7f7a82c11880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj29]: ERROR: failed to sync object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj29 2024-08-18T16:39:34.969 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.959+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj31 2024-08-18T16:39:34.969 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.959+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj31]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj31 op=0 op_state=1 2024-08-18T16:39:34.970 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.959+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj31]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj31[0] 2024-08-18T16:39:34.970 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.959+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj33 2024-08-18T16:39:34.970 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.959+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj33]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj33 op=0 op_state=1 2024-08-18T16:39:34.970 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.959+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj33]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj33[0] 2024-08-18T16:39:34.970 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.959+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj34 2024-08-18T16:39:34.971 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.959+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj34]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj34 op=0 op_state=1 2024-08-18T16:39:34.971 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.959+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj34]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj34[0] 2024-08-18T16:39:34.971 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.959+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj37 2024-08-18T16:39:34.971 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.959+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj37]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj37 op=0 op_state=1 2024-08-18T16:39:34.971 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.959+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj37]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj37[0] 2024-08-18T16:39:34.971 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.959+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj38 2024-08-18T16:39:34.972 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.959+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj38]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj38 op=0 op_state=1 2024-08-18T16:39:34.972 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.959+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj38]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj38[0] 2024-08-18T16:39:34.972 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.959+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj44 2024-08-18T16:39:34.972 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.959+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj44]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj44 op=0 op_state=1 2024-08-18T16:39:34.972 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.959+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj44]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj44[0] 2024-08-18T16:39:34.973 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.959+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj46 2024-08-18T16:39:34.973 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.959+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj46]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj46 op=0 op_state=1 2024-08-18T16:39:34.973 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.959+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj46]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj46[0] 2024-08-18T16:39:34.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.962+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj155]: success 2024-08-18T16:39:34.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.962+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj115]: success 2024-08-18T16:39:34.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.962+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj124]: success 2024-08-18T16:39:34.978 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.962+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj111]: success 2024-08-18T16:39:34.978 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.962+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj51 2024-08-18T16:39:34.978 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.962+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj51]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj51 op=0 op_state=1 2024-08-18T16:39:34.979 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.962+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj51]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj51[0] 2024-08-18T16:39:34.979 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.962+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj55 2024-08-18T16:39:34.979 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.962+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj55]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj55 op=0 op_state=1 2024-08-18T16:39:34.979 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.962+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj55]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj55[0] 2024-08-18T16:39:34.980 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.962+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj57 2024-08-18T16:39:34.980 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.962+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj57]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj57 op=0 op_state=1 2024-08-18T16:39:34.980 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.962+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj57]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj57[0] 2024-08-18T16:39:34.980 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.962+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj58 2024-08-18T16:39:34.980 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.962+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj58]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj58 op=0 op_state=1 2024-08-18T16:39:34.980 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.962+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj58]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj58[0] 2024-08-18T16:39:34.981 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.963+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj11]: success 2024-08-18T16:39:34.981 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.963+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj62 2024-08-18T16:39:34.981 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.963+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj62]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj62 op=0 op_state=1 2024-08-18T16:39:34.981 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.963+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj62]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj62[0] 2024-08-18T16:39:34.981 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.963+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj31]: success 2024-08-18T16:39:34.981 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.963+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj68 2024-08-18T16:39:34.981 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.963+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj68]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj68 op=0 op_state=1 2024-08-18T16:39:34.982 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.963+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj19]: success 2024-08-18T16:39:34.982 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.963+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj68]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj68[0] 2024-08-18T16:39:34.982 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.963+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj69 2024-08-18T16:39:34.982 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.963+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj69]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj69 op=0 op_state=1 2024-08-18T16:39:34.982 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.963+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj69]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj69[0] 2024-08-18T16:39:34.983 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.963+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj34]: success 2024-08-18T16:39:34.983 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.963+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj74 2024-08-18T16:39:34.983 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.963+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj74]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj74 op=0 op_state=1 2024-08-18T16:39:34.983 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.963+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj74]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj74[0] 2024-08-18T16:39:34.983 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.963+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj37]: success 2024-08-18T16:39:34.983 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.963+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj76 2024-08-18T16:39:34.983 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.963+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj76]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj76 op=0 op_state=1 2024-08-18T16:39:34.984 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.963+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj76]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj76[0] 2024-08-18T16:39:34.984 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.963+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj44]: success 2024-08-18T16:39:34.984 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.963+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj85 2024-08-18T16:39:34.984 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.963+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj85]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj85 op=0 op_state=1 2024-08-18T16:39:34.984 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.963+0000 7f7a82c11880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj85]: : injecting data sync error on key=obj85 2024-08-18T16:39:34.984 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.963+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj85]: failed, retcode=-5 ((5) Input/output error) 2024-08-18T16:39:34.985 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.963+0000 7f7a82c11880 0 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj85]: ERROR: failed to sync object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj85 2024-08-18T16:39:34.985 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.964+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj17]: success 2024-08-18T16:39:34.985 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.964+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj87 2024-08-18T16:39:34.985 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.964+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj87]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj87 op=0 op_state=1 2024-08-18T16:39:34.985 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.964+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj87]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj87[0] 2024-08-18T16:39:34.986 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.964+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj46]: success 2024-08-18T16:39:34.986 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.964+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj92 2024-08-18T16:39:34.986 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.964+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj92]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj92 op=0 op_state=1 2024-08-18T16:39:34.986 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.964+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj92]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj92[0] 2024-08-18T16:39:34.986 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.966+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj38]: success 2024-08-18T16:39:34.986 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.966+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj22]: success 2024-08-18T16:39:34.987 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.966+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj23]: success 2024-08-18T16:39:34.987 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.966+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj93 2024-08-18T16:39:34.987 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.966+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj93]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj93 op=0 op_state=1 2024-08-18T16:39:34.987 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.966+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj93]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj93[0] 2024-08-18T16:39:34.987 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.966+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj95 2024-08-18T16:39:34.988 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.966+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj95]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj95 op=0 op_state=1 2024-08-18T16:39:34.988 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.966+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj95]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj95[0] 2024-08-18T16:39:34.988 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.966+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj58]: success 2024-08-18T16:39:34.988 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.966+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj74]: success 2024-08-18T16:39:34.988 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.967+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: a sync operation returned error 2024-08-18T16:39:34.988 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.968+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: a sync operation returned error 2024-08-18T16:39:34.988 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.968+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: a sync operation returned error 2024-08-18T16:39:34.988 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.968+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj51]: success 2024-08-18T16:39:34.988 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.968+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj68]: success 2024-08-18T16:39:34.989 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.968+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj62]: success 2024-08-18T16:39:34.989 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.968+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj185]: success 2024-08-18T16:39:34.989 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.968+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj33]: success 2024-08-18T16:39:34.989 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.968+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj69]: success 2024-08-18T16:39:34.989 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.968+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj92]: success 2024-08-18T16:39:34.989 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.968+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj87]: success 2024-08-18T16:39:34.989 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.969+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj76]: success 2024-08-18T16:39:34.989 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.969+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj148]: success 2024-08-18T16:39:34.990 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.970+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj116]: success 2024-08-18T16:39:34.990 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.970+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: updating marker oid=bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26 marker=obj124 2024-08-18T16:39:34.990 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.970+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj55]: success 2024-08-18T16:39:34.990 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.970+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj95]: success 2024-08-18T16:39:34.990 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.970+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj57]: success 2024-08-18T16:39:34.990 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.970+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj93]: success 2024-08-18T16:39:34.990 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.971+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: backing out with sync_status=-5 2024-08-18T16:39:34.991 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.971+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: ERROR: full sync failed. error: -5 2024-08-18T16:39:34.991 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.972+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: sync status for source bucket: full. lease is: not taken. stop indications is: 0 2024-08-18T16:39:34.991 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:34.972+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: waiting on bucket lease 2024-08-18T16:39:35.018 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.017+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: status after acquiring the lock is: full 2024-08-18T16:39:35.030 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.029+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: listing bucket for full sync 2024-08-18T16:39:35.081 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.075+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj11 2024-08-18T16:39:35.081 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.075+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj11]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj11 op=0 op_state=1 2024-08-18T16:39:35.082 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.075+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj11]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj11[0] 2024-08-18T16:39:35.082 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.075+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj111 2024-08-18T16:39:35.082 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.075+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj111]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj111 op=0 op_state=1 2024-08-18T16:39:35.082 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.075+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj111]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj111[0] 2024-08-18T16:39:35.082 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.075+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj115 2024-08-18T16:39:35.083 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.075+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj115]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj115 op=0 op_state=1 2024-08-18T16:39:35.083 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.075+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj115]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj115[0] 2024-08-18T16:39:35.083 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.075+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj116 2024-08-18T16:39:35.083 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.075+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj116]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj116 op=0 op_state=1 2024-08-18T16:39:35.084 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.075+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj116]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj116[0] 2024-08-18T16:39:35.084 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.075+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj124 2024-08-18T16:39:35.084 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.075+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj124]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj124 op=0 op_state=1 2024-08-18T16:39:35.084 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.075+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj124]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj124[0] 2024-08-18T16:39:35.084 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.075+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj145 2024-08-18T16:39:35.084 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.075+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj145]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj145 op=0 op_state=1 2024-08-18T16:39:35.084 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.075+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj145]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj145[0] 2024-08-18T16:39:35.084 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.075+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj148 2024-08-18T16:39:35.084 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.075+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj148]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj148 op=0 op_state=1 2024-08-18T16:39:35.084 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.075+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj148]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj148[0] 2024-08-18T16:39:35.084 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.075+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj155 2024-08-18T16:39:35.085 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.075+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj155]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj155 op=0 op_state=1 2024-08-18T16:39:35.085 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.075+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj155]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj155[0] 2024-08-18T16:39:35.085 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.076+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj17 2024-08-18T16:39:35.085 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.076+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj17]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj17 op=0 op_state=1 2024-08-18T16:39:35.085 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.076+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj17]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj17[0] 2024-08-18T16:39:35.085 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.076+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj185 2024-08-18T16:39:35.085 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.076+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj185]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj185 op=0 op_state=1 2024-08-18T16:39:35.085 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.076+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj185]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj185[0] 2024-08-18T16:39:35.085 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.076+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj19 2024-08-18T16:39:35.085 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.076+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj19]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj19 op=0 op_state=1 2024-08-18T16:39:35.085 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.076+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj19]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj19[0] 2024-08-18T16:39:35.085 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.076+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj22 2024-08-18T16:39:35.085 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.076+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj22]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj22 op=0 op_state=1 2024-08-18T16:39:35.086 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.076+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj22]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj22[0] 2024-08-18T16:39:35.086 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.076+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj23 2024-08-18T16:39:35.086 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.076+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj23]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj23 op=0 op_state=1 2024-08-18T16:39:35.086 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.076+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj23]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj23[0] 2024-08-18T16:39:35.086 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.076+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj29 2024-08-18T16:39:35.086 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.076+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj29]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj29 op=0 op_state=1 2024-08-18T16:39:35.086 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.076+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj29]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj29[0] 2024-08-18T16:39:35.086 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.076+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj31 2024-08-18T16:39:35.086 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.076+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj31]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj31 op=0 op_state=1 2024-08-18T16:39:35.086 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.076+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj31]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj31[0] 2024-08-18T16:39:35.086 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.076+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj33 2024-08-18T16:39:35.087 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.076+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj33]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj33 op=0 op_state=1 2024-08-18T16:39:35.087 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.076+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj33]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj33[0] 2024-08-18T16:39:35.087 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.076+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj34 2024-08-18T16:39:35.087 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.076+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj34]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj34 op=0 op_state=1 2024-08-18T16:39:35.087 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.076+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj34]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj34[0] 2024-08-18T16:39:35.088 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.076+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj37 2024-08-18T16:39:35.088 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.076+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj37]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj37 op=0 op_state=1 2024-08-18T16:39:35.088 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.076+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj37]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj37[0] 2024-08-18T16:39:35.088 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.076+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj38 2024-08-18T16:39:35.088 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.076+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj38]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj38 op=0 op_state=1 2024-08-18T16:39:35.088 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.076+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj38]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj38[0] 2024-08-18T16:39:35.088 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.076+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj44 2024-08-18T16:39:35.089 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.076+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj44]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj44 op=0 op_state=1 2024-08-18T16:39:35.089 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.076+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj44]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj44[0] 2024-08-18T16:39:35.089 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.076+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj46 2024-08-18T16:39:35.089 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.076+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj46]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj46 op=0 op_state=1 2024-08-18T16:39:35.089 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.076+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj46]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj46[0] 2024-08-18T16:39:35.089 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.080+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj145]: success 2024-08-18T16:39:35.089 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.080+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj51 2024-08-18T16:39:35.089 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.080+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj51]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj51 op=0 op_state=1 2024-08-18T16:39:35.089 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.080+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj51]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj51[0] 2024-08-18T16:39:35.089 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.080+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj155]: success 2024-08-18T16:39:35.089 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.080+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj115]: success 2024-08-18T16:39:35.089 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.080+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj148]: success 2024-08-18T16:39:35.090 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.080+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj116]: success 2024-08-18T16:39:35.090 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.080+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj124]: success 2024-08-18T16:39:35.090 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.080+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj111]: success 2024-08-18T16:39:35.090 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.080+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj11]: success 2024-08-18T16:39:35.090 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.080+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj55 2024-08-18T16:39:35.090 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.080+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj55]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj55 op=0 op_state=1 2024-08-18T16:39:35.090 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.080+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj55]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj55[0] 2024-08-18T16:39:35.090 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.080+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj57 2024-08-18T16:39:35.090 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.080+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj57]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj57 op=0 op_state=1 2024-08-18T16:39:35.090 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.080+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj57]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj57[0] 2024-08-18T16:39:35.090 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.080+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj58 2024-08-18T16:39:35.090 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.080+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj58]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj58 op=0 op_state=1 2024-08-18T16:39:35.091 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.080+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj58]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj58[0] 2024-08-18T16:39:35.091 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.080+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj62 2024-08-18T16:39:35.091 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.080+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj62]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj62 op=0 op_state=1 2024-08-18T16:39:35.091 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.080+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj62]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj62[0] 2024-08-18T16:39:35.091 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.080+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj68 2024-08-18T16:39:35.091 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.081+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj68]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj68 op=0 op_state=1 2024-08-18T16:39:35.091 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.081+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj68]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj68[0] 2024-08-18T16:39:35.091 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.081+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj69 2024-08-18T16:39:35.092 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.081+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj69]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj69 op=0 op_state=1 2024-08-18T16:39:35.092 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.081+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj69]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj69[0] 2024-08-18T16:39:35.096 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.081+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj74 2024-08-18T16:39:35.096 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.081+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj74]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj74 op=0 op_state=1 2024-08-18T16:39:35.096 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.081+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj74]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj74[0] 2024-08-18T16:39:35.096 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.081+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj34]: success 2024-08-18T16:39:35.096 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.081+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj19]: success 2024-08-18T16:39:35.096 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.081+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj76 2024-08-18T16:39:35.096 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.081+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj76]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj76 op=0 op_state=1 2024-08-18T16:39:35.096 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.081+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj76]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj76[0] 2024-08-18T16:39:35.096 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.081+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj85 2024-08-18T16:39:35.096 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.081+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj85]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj85 op=0 op_state=1 2024-08-18T16:39:35.097 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.081+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj85]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj85[0] 2024-08-18T16:39:35.097 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.081+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj23]: success 2024-08-18T16:39:35.097 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.081+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj87 2024-08-18T16:39:35.097 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.081+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj87]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj87 op=0 op_state=1 2024-08-18T16:39:35.097 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.081+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj87]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj87[0] 2024-08-18T16:39:35.097 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.081+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj31]: success 2024-08-18T16:39:35.097 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.081+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj92 2024-08-18T16:39:35.097 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.081+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj92]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj92 op=0 op_state=1 2024-08-18T16:39:35.097 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.081+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj92]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj92[0] 2024-08-18T16:39:35.097 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.082+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj37]: success 2024-08-18T16:39:35.097 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.082+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj93 2024-08-18T16:39:35.097 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.082+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj93]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj93 op=0 op_state=1 2024-08-18T16:39:35.098 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.082+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj93]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj93[0] 2024-08-18T16:39:35.098 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.083+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj17]: success 2024-08-18T16:39:35.098 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.083+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: updating marker oid=bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26 marker=obj17 2024-08-18T16:39:35.098 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.083+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj29]: success 2024-08-18T16:39:35.098 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.083+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: [full sync] syncing object: kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0/obj95 2024-08-18T16:39:35.098 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.083+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj95]: bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b=bucket sync single entry (source_zone=96bf8502-2f39-4d53-8196-3378c28da3b4) b= log_entry=obj95 op=0 op_state=1 2024-08-18T16:39:35.098 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.083+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj95]: bucket sync: sync obj: 96bf8502-2f39-4d53-8196-3378c28da3b4/:kibgmb-78[96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26])/obj95[0] 2024-08-18T16:39:35.098 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.083+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj44]: success 2024-08-18T16:39:35.098 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.083+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj46]: success 2024-08-18T16:39:35.098 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.084+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj185]: success 2024-08-18T16:39:35.098 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.084+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj33]: success 2024-08-18T16:39:35.098 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.084+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj22]: success 2024-08-18T16:39:35.099 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.084+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj38]: success 2024-08-18T16:39:35.099 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.086+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj55]: success 2024-08-18T16:39:35.099 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.086+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj69]: success 2024-08-18T16:39:35.099 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.086+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj58]: success 2024-08-18T16:39:35.099 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.086+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj85]: success 2024-08-18T16:39:35.099 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.086+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj68]: success 2024-08-18T16:39:35.099 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.086+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj74]: success 2024-08-18T16:39:35.099 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.086+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj76]: success 2024-08-18T16:39:35.099 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.086+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj87]: success 2024-08-18T16:39:35.099 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.086+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj57]: success 2024-08-18T16:39:35.099 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.086+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj92]: success 2024-08-18T16:39:35.099 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.087+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj93]: success 2024-08-18T16:39:35.099 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.087+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj62]: success 2024-08-18T16:39:35.100 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.087+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj51]: success 2024-08-18T16:39:35.100 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.087+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: updating marker oid=bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26 marker=obj93 2024-08-18T16:39:35.100 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.087+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:entry[obj95]: success 2024-08-18T16:39:35.100 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.087+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: updating marker oid=bucket.full-sync-status.96bf8502-2f39-4d53-8196-3378c28da3b4:kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26 marker=obj95 2024-08-18T16:39:35.100 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.090+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: set bucket state=incremental 2024-08-18T16:39:35.100 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.091+0000 7f7a82c11880 5 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:full_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: bucket status objv={r=_jIycT6MSetLnuGq5SYmpham:150,w=:0} 2024-08-18T16:39:35.100 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.092+0000 7f7a82c11880 10 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: WARNING: requested sync of past generation 0 < 6, returning success 2024-08-18T16:39:35.100 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.094+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:39:35.100 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.094+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:39:35.100 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.094+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:39:35.100 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.094+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:39:35.101 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.094+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: sync status for source bucket shard: 2 2024-08-18T16:39:35.101 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.094+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]:inc_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:0]: listing bilog for incremental sync; position= 2024-08-18T16:39:35.101 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.094+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:1]: sync status for source bucket shard: 2 2024-08-18T16:39:35.101 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.094+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:1]:inc_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:1]: listing bilog for incremental sync; position= 2024-08-18T16:39:35.101 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.095+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:3]: sync status for source bucket shard: 2 2024-08-18T16:39:35.101 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.095+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:3]:inc_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:3]: listing bilog for incremental sync; position= 2024-08-18T16:39:35.101 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.095+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:39:35.101 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.095+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:5]: sync status for source bucket shard: 2 2024-08-18T16:39:35.102 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.095+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:5]:inc_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:5]: listing bilog for incremental sync; position= 2024-08-18T16:39:35.102 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.095+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:39:35.102 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.095+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:2]: sync status for source bucket shard: 2 2024-08-18T16:39:35.102 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.095+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:2]:inc_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:2]: listing bilog for incremental sync; position= 2024-08-18T16:39:35.108 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.096+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:39:35.108 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.096+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:4]: sync status for source bucket shard: 2 2024-08-18T16:39:35.108 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.096+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:4]:inc_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:4]: listing bilog for incremental sync; position= 2024-08-18T16:39:35.108 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.096+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:39:35.109 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.096+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:39:35.109 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.096+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:6]: sync status for source bucket shard: 2 2024-08-18T16:39:35.109 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.096+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:6]:inc_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:6]: listing bilog for incremental sync; position=00000000002.82.6 2024-08-18T16:39:35.109 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.097+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:39:35.109 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.097+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:39:35.109 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.097+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:39:35.109 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.097+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:39:35.109 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.097+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:7]: sync status for source bucket shard: 2 2024-08-18T16:39:35.109 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.097+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:7]:inc_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:7]: listing bilog for incremental sync; position= 2024-08-18T16:39:35.109 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.097+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:39:35.109 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.097+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:8]: sync status for source bucket shard: 2 2024-08-18T16:39:35.109 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.097+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:39:35.110 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.097+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:8]:inc_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:8]: listing bilog for incremental sync; position= 2024-08-18T16:39:35.110 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.097+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:39:35.110 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.097+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:39:35.110 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.098+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:9]: sync status for source bucket shard: 2 2024-08-18T16:39:35.110 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.098+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:9]:inc_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:9]: listing bilog for incremental sync; position= 2024-08-18T16:39:35.110 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.098+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:10]: sync status for source bucket shard: 2 2024-08-18T16:39:35.110 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.098+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:10]:inc_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:10]: listing bilog for incremental sync; position= 2024-08-18T16:39:35.110 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.098+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:39:35.110 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.098+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:14]: sync status for source bucket shard: 2 2024-08-18T16:39:35.110 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.098+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:15]: sync status for source bucket shard: 2 2024-08-18T16:39:35.110 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.098+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:11]: sync status for source bucket shard: 2 2024-08-18T16:39:35.110 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.098+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:14]:inc_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:14]: listing bilog for incremental sync; position= 2024-08-18T16:39:35.111 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.098+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:15]:inc_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:15]: listing bilog for incremental sync; position= 2024-08-18T16:39:35.111 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.098+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:11]:inc_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:11]: listing bilog for incremental sync; position= 2024-08-18T16:39:35.111 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.098+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:12]: sync status for source bucket shard: 2 2024-08-18T16:39:35.111 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.098+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:13]: sync status for source bucket shard: 2 2024-08-18T16:39:35.111 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.098+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:12]:inc_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:12]: listing bilog for incremental sync; position=00000000002.191.6 2024-08-18T16:39:35.111 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.098+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:13]:inc_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:13]: listing bilog for incremental sync; position= 2024-08-18T16:39:35.111 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.099+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:19]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:39:35.111 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.099+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:39:35.111 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.099+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:16]: sync status for source bucket shard: 2 2024-08-18T16:39:35.111 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.099+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:16]:inc_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:16]: listing bilog for incremental sync; position= 2024-08-18T16:39:35.111 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.099+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:20]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:39:35.111 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.099+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:22]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:39:35.112 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.099+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:17]: sync status for source bucket shard: 2 2024-08-18T16:39:35.112 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.099+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:17]:inc_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:17]: listing bilog for incremental sync; position=00000000002.107.6 2024-08-18T16:39:35.112 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.100+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:21]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:39:35.112 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.100+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:20]: sync status for source bucket shard: 2 2024-08-18T16:39:35.112 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.100+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:20]:inc_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:20]: listing bilog for incremental sync; position= 2024-08-18T16:39:35.112 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.100+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:23]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:39:35.112 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.100+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:24]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:39:35.112 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.100+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:27]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:39:35.112 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.100+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:26]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:39:35.112 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.100+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:25]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:39:35.112 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.100+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:19]: sync status for source bucket shard: 2 2024-08-18T16:39:35.112 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.100+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:18]: sync status for source bucket shard: 2 2024-08-18T16:39:35.112 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.100+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:19]:inc_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:19]: listing bilog for incremental sync; position= 2024-08-18T16:39:35.113 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.100+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:18]:inc_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:18]: listing bilog for incremental sync; position= 2024-08-18T16:39:35.113 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.100+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:28]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:39:35.113 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.101+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:29]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:39:35.113 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.101+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:22]: sync status for source bucket shard: 2 2024-08-18T16:39:35.113 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.101+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:22]:inc_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:22]: listing bilog for incremental sync; position= 2024-08-18T16:39:35.113 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.101+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:30]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:39:35.113 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.101+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:23]: sync status for source bucket shard: 2 2024-08-18T16:39:35.113 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.101+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:23]:inc_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:23]: listing bilog for incremental sync; position= 2024-08-18T16:39:35.113 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.101+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:31]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:39:35.113 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.102+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:32]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:39:35.114 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.102+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:34]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:39:35.114 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.102+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:21]: sync status for source bucket shard: 2 2024-08-18T16:39:35.114 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.102+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:27]: sync status for source bucket shard: 2 2024-08-18T16:39:35.114 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.102+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:24]: sync status for source bucket shard: 2 2024-08-18T16:39:35.114 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.102+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:21]:inc_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:21]: listing bilog for incremental sync; position= 2024-08-18T16:39:35.114 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.102+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:27]:inc_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:27]: listing bilog for incremental sync; position= 2024-08-18T16:39:35.114 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.102+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:24]:inc_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:24]: listing bilog for incremental sync; position= 2024-08-18T16:39:35.114 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.102+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:25]: sync status for source bucket shard: 2 2024-08-18T16:39:35.115 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.102+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:29]: sync status for source bucket shard: 2 2024-08-18T16:39:35.115 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.102+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:30]: sync status for source bucket shard: 2 2024-08-18T16:39:35.115 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.102+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:25]:inc_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:25]: listing bilog for incremental sync; position= 2024-08-18T16:39:35.115 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.102+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:29]:inc_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:29]: listing bilog for incremental sync; position= 2024-08-18T16:39:35.115 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.102+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:30]:inc_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:30]: listing bilog for incremental sync; position= 2024-08-18T16:39:35.115 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.102+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:32]: sync status for source bucket shard: 2 2024-08-18T16:39:35.115 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.103+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:32]:inc_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:32]: listing bilog for incremental sync; position=00000000001.116.6 2024-08-18T16:39:35.115 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.103+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:31]: sync status for source bucket shard: 2 2024-08-18T16:39:35.116 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.103+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:31]:inc_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:31]: listing bilog for incremental sync; position= 2024-08-18T16:39:35.116 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.103+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:34]: sync status for source bucket shard: 2 2024-08-18T16:39:35.116 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.103+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:34]:inc_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:34]: listing bilog for incremental sync; position= 2024-08-18T16:39:35.116 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.103+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:33]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:39:35.116 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.103+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:35]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:39:35.116 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.103+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:26]: sync status for source bucket shard: 2 2024-08-18T16:39:35.116 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.104+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:26]:inc_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:26]: listing bilog for incremental sync; position= 2024-08-18T16:39:35.116 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.104+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:36]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-08-18T16:39:35.116 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.104+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:28]: sync status for source bucket shard: 2 2024-08-18T16:39:35.116 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.104+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:28]:inc_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:28]: listing bilog for incremental sync; position= 2024-08-18T16:39:35.116 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.104+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:35]: sync status for source bucket shard: 2 2024-08-18T16:39:35.116 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.104+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:35]:inc_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:35]: listing bilog for incremental sync; position= 2024-08-18T16:39:35.116 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.104+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:36]: sync status for source bucket shard: 2 2024-08-18T16:39:35.117 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.104+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:36]:inc_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:36]: listing bilog for incremental sync; position=00000000001.120.6 2024-08-18T16:39:35.117 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.105+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:33]: sync status for source bucket shard: 2 2024-08-18T16:39:35.117 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:35.105+0000 7f7a82c11880 20 RGW-SYNC:bucket[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26<-kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:33]:inc_sync[kibgmb-78:96bf8502-2f39-4d53-8196-3378c28da3b4.6026.26:33]: listing bilog for incremental sync; position= 2024-08-18T16:39:35.171 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'list', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:39:35.171 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'list', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:39:35.171 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket list --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-78 --max-entries 10000 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:39:35.224 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:39:35.224 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:39:35.259 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T16:39:35.259 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:35.259 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj11", 2024-08-18T16:39:35.259 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:35.259 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:35.260 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:39:35.260 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 14 2024-08-18T16:39:35.260 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.260 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:35.260 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:35.260 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:35.260 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:35.260 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 6, 2024-08-18T16:39:35.260 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:23:42.051163Z", 2024-08-18T16:39:35.260 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:35.260 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:35.260 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:35.260 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:35.260 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:35.261 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:35.261 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:35.261 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:35.261 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.261 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.9889212156647452491", 2024-08-18T16:39:35.261 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:35.261 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:35.261 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:35.261 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.261 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:35.262 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj111", 2024-08-18T16:39:35.262 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:35.262 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:35.262 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:39:35.262 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 25 2024-08-18T16:39:35.262 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.262 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:35.262 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:35.262 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:35.262 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:35.262 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-08-18T16:39:35.262 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:36:05.189316Z", 2024-08-18T16:39:35.263 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:35.263 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:35.263 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:35.263 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:35.263 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:35.263 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:35.263 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:35.263 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:35.263 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.264 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.6119517508141186778", 2024-08-18T16:39:35.264 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:35.264 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:35.264 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:35.265 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.265 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:35.265 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj115", 2024-08-18T16:39:35.265 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:35.265 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:35.265 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:39:35.265 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 3 2024-08-18T16:39:35.265 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.265 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:35.266 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:35.266 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:35.266 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:35.266 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 5, 2024-08-18T16:39:35.266 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:27:23.466808Z", 2024-08-18T16:39:35.266 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:35.266 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:35.266 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:35.266 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:35.266 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:35.266 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:35.266 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:35.267 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:35.267 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.267 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.6399818160083281577", 2024-08-18T16:39:35.267 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:35.267 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:35.267 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:35.267 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.267 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:35.267 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj116", 2024-08-18T16:39:35.267 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:35.267 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:35.267 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:39:35.268 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 13 2024-08-18T16:39:35.268 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.268 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:35.268 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:35.268 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:35.268 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:35.268 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 5, 2024-08-18T16:39:35.268 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:30:57.371585Z", 2024-08-18T16:39:35.268 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:35.268 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:35.269 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:35.269 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:35.269 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:35.269 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:35.269 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:35.270 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:35.270 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.270 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.15128885479062847417", 2024-08-18T16:39:35.270 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:35.270 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:35.270 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:35.271 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.271 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:35.271 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj124", 2024-08-18T16:39:35.271 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:35.271 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:35.271 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:39:35.271 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 7 2024-08-18T16:39:35.272 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.272 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:35.272 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:35.272 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:35.272 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:35.272 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 6, 2024-08-18T16:39:35.272 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:33:27.492061Z", 2024-08-18T16:39:35.272 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:35.273 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:35.273 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:35.273 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:35.273 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:35.273 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:35.273 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:35.273 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:35.273 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.273 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.3899074313730904728", 2024-08-18T16:39:35.273 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:35.273 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:35.273 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:35.274 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.274 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:35.274 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj145", 2024-08-18T16:39:35.274 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:35.274 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:35.274 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:39:35.275 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 21 2024-08-18T16:39:35.275 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.275 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:35.275 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:35.275 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:35.275 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:35.275 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:39:35.275 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:30:57.379450Z", 2024-08-18T16:39:35.275 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:35.275 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:35.275 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:35.276 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:35.276 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:35.276 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:35.276 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:35.276 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:35.277 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.277 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.11999609542217578306", 2024-08-18T16:39:35.277 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:35.277 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:35.277 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:35.277 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.277 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:35.277 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj148", 2024-08-18T16:39:35.277 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:35.277 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:35.277 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:39:35.277 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 13 2024-08-18T16:39:35.277 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.278 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:35.278 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:35.278 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:35.278 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:35.278 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 5, 2024-08-18T16:39:35.278 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:36:05.197180Z", 2024-08-18T16:39:35.278 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:35.278 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:35.278 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:35.278 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:35.278 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:35.279 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:35.279 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:35.279 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:35.279 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.279 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.17613569593319029244", 2024-08-18T16:39:35.279 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:35.279 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:35.279 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:35.279 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.279 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:35.280 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj155", 2024-08-18T16:39:35.280 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:35.280 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:35.280 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:39:35.280 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 15 2024-08-18T16:39:35.280 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.280 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:35.280 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:35.280 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:35.280 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:35.280 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:39:35.280 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:33:27.500403Z", 2024-08-18T16:39:35.280 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:35.280 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:35.281 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:35.281 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:35.281 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:35.281 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:35.281 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:35.281 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:35.282 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.282 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.18209861177904032184", 2024-08-18T16:39:35.282 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:35.282 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:35.282 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:35.282 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.282 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:35.282 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj17", 2024-08-18T16:39:35.282 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:35.282 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:35.282 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:39:35.282 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 6 2024-08-18T16:39:35.283 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.283 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:35.283 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:35.283 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:35.283 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:35.283 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 5, 2024-08-18T16:39:35.283 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:24:10.291690Z", 2024-08-18T16:39:35.283 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:35.283 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:35.283 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:35.283 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:35.283 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:35.283 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:35.283 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:35.284 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:35.284 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.284 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.6759974984332877289", 2024-08-18T16:39:35.284 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:35.284 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:35.284 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:35.284 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.284 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:35.284 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj185", 2024-08-18T16:39:35.284 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:35.284 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:35.284 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:39:35.284 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 13 2024-08-18T16:39:35.284 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.285 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:35.285 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:35.285 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:35.285 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:35.285 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-08-18T16:39:35.285 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:36:05.244081Z", 2024-08-18T16:39:35.285 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:35.285 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:35.285 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:35.285 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:35.285 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:35.285 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:35.285 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:35.285 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:35.286 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.286 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.372041751039859441", 2024-08-18T16:39:35.286 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:35.286 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:35.286 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:35.286 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.286 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:35.286 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj19", 2024-08-18T16:39:35.286 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:35.286 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:35.286 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:39:35.286 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 10 2024-08-18T16:39:35.286 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.287 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:35.287 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:35.287 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:35.287 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:35.287 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-08-18T16:39:35.287 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:25:55.960750Z", 2024-08-18T16:39:35.287 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:35.287 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:35.287 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:35.287 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:35.287 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:35.287 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:35.287 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:35.288 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:35.288 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.288 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.3313087035472660803", 2024-08-18T16:39:35.288 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:35.288 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:35.288 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:35.288 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.288 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:35.288 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj22", 2024-08-18T16:39:35.288 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:35.288 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:35.288 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:39:35.288 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 50 2024-08-18T16:39:35.288 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.289 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:35.289 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:35.289 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:35.289 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:35.289 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 5, 2024-08-18T16:39:35.289 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:23:42.058759Z", 2024-08-18T16:39:35.289 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:35.289 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:35.289 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:35.289 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:35.289 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:35.289 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:35.289 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:35.290 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:35.290 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.290 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.10722087597600129896", 2024-08-18T16:39:35.290 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:35.290 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:35.290 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:35.290 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.290 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:35.290 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj23", 2024-08-18T16:39:35.290 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:35.290 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:35.290 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:39:35.290 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 6 2024-08-18T16:39:35.291 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.291 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:35.291 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:35.291 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:35.291 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:35.291 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:39:35.291 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:27:23.361495Z", 2024-08-18T16:39:35.291 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:35.291 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:35.291 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:35.291 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:35.291 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:35.291 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:35.292 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:35.292 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:35.292 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.292 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.12518117691867460845", 2024-08-18T16:39:35.292 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:35.292 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:35.292 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:35.292 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.292 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:35.292 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj29", 2024-08-18T16:39:35.292 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:35.292 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:35.292 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:39:35.293 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 5 2024-08-18T16:39:35.293 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.293 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:35.293 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:35.293 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:35.293 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:35.293 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:39:35.293 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:30:57.311630Z", 2024-08-18T16:39:35.293 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:35.293 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:35.293 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:35.293 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:35.293 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:35.293 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:35.294 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:35.294 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:35.294 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.294 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.9093700191308604259", 2024-08-18T16:39:35.294 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:35.294 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:35.294 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:35.294 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.294 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:35.294 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj31", 2024-08-18T16:39:35.294 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:35.295 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:35.295 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:39:35.295 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 10 2024-08-18T16:39:35.295 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.295 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:35.295 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:35.295 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:35.295 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:35.295 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:39:35.295 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:33:27.426326Z", 2024-08-18T16:39:35.295 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:35.295 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:35.296 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:35.296 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:35.296 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:35.296 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:35.296 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:35.296 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:35.296 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.296 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.4113607296140759139", 2024-08-18T16:39:35.296 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:35.296 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:35.296 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:35.296 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.296 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:35.296 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj33", 2024-08-18T16:39:35.297 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:35.297 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:35.297 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:39:35.297 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 6 2024-08-18T16:39:35.297 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.297 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:35.297 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:35.297 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:35.297 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:35.297 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 6, 2024-08-18T16:39:35.297 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:23:42.064530Z", 2024-08-18T16:39:35.297 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:35.297 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:35.297 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:35.298 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:35.298 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:35.298 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:35.298 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:35.298 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:35.298 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.298 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.15286180760377802005", 2024-08-18T16:39:35.298 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:35.298 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:35.298 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:35.298 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.298 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:35.298 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj34", 2024-08-18T16:39:35.299 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:35.299 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:35.299 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:39:35.299 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 10 2024-08-18T16:39:35.299 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.299 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:35.299 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:35.299 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:35.299 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:35.299 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:39:35.299 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:24:10.298744Z", 2024-08-18T16:39:35.299 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:35.299 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:35.300 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:35.300 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:35.300 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:35.300 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:35.300 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:35.300 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:35.300 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.300 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.1382908489229446564", 2024-08-18T16:39:35.300 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:35.300 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:35.301 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:35.301 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.301 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:35.301 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj37", 2024-08-18T16:39:35.301 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:35.301 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:35.301 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:39:35.302 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 11 2024-08-18T16:39:35.302 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.302 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:35.302 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:35.302 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:35.303 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:35.303 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:39:35.303 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:36:05.133754Z", 2024-08-18T16:39:35.303 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:35.303 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:35.303 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:35.303 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:35.304 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:35.304 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:35.304 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:35.304 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:35.304 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.304 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.10395008102202495793", 2024-08-18T16:39:35.305 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:35.305 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:35.305 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:35.305 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.305 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:35.306 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj38", 2024-08-18T16:39:35.306 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:35.306 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:35.306 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:39:35.306 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 17 2024-08-18T16:39:35.306 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.307 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:35.307 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:35.307 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:35.307 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:35.307 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-08-18T16:39:35.308 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:25:55.967916Z", 2024-08-18T16:39:35.308 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:35.308 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:35.308 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:35.308 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:35.309 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:35.309 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:35.309 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:35.309 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:35.309 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.309 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.3777469865024363097", 2024-08-18T16:39:35.310 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:35.310 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:35.310 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:35.310 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.310 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:35.310 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj44", 2024-08-18T16:39:35.311 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:35.311 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:35.311 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:39:35.311 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 8 2024-08-18T16:39:35.311 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.311 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:35.312 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:35.312 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:35.312 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:35.312 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 6, 2024-08-18T16:39:35.312 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:23:42.115387Z", 2024-08-18T16:39:35.312 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:35.313 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:35.313 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:35.313 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:35.313 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:35.313 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:35.314 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:35.314 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:35.314 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.314 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.2838957566806869052", 2024-08-18T16:39:35.314 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:35.315 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:35.315 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:35.315 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.315 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:35.315 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj46", 2024-08-18T16:39:35.316 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:35.316 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:35.316 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:39:35.316 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 9 2024-08-18T16:39:35.316 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.317 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:35.317 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:35.317 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:35.317 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:35.317 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:39:35.317 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:27:23.407342Z", 2024-08-18T16:39:35.317 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:35.318 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:35.318 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:35.318 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:35.318 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:35.318 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:35.318 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:35.318 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:35.319 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.319 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.7304181808045275938", 2024-08-18T16:39:35.319 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:35.319 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:35.320 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:35.320 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.320 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:35.320 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj51", 2024-08-18T16:39:35.320 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:35.320 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:35.320 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:39:35.320 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 5 2024-08-18T16:39:35.320 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.320 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:35.321 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:35.321 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:35.321 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:35.321 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 6, 2024-08-18T16:39:35.321 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:24:10.305333Z", 2024-08-18T16:39:35.321 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:35.321 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:35.321 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:35.322 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:35.322 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:35.322 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:35.322 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:35.322 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:35.322 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.322 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.2913498005472717509", 2024-08-18T16:39:35.322 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:35.322 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:35.322 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:35.322 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.323 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:35.323 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj55", 2024-08-18T16:39:35.323 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:35.323 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:35.323 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:39:35.323 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 7 2024-08-18T16:39:35.323 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.323 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:35.323 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:35.323 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:35.323 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:35.323 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 6, 2024-08-18T16:39:35.323 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:23:42.122245Z", 2024-08-18T16:39:35.324 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:35.324 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:35.324 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:35.324 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:35.324 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:35.324 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:35.324 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:35.324 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:35.324 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.324 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.5436641643066532846", 2024-08-18T16:39:35.324 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:35.324 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:35.324 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:35.324 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.325 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:35.325 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj57", 2024-08-18T16:39:35.325 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:35.325 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:35.325 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:39:35.325 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 29 2024-08-18T16:39:35.325 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.325 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:35.325 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:35.325 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:35.325 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:35.326 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:39:35.326 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:25:55.973669Z", 2024-08-18T16:39:35.326 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:35.326 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:35.326 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:35.326 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:35.326 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:35.326 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:35.326 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:35.326 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:35.326 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.326 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.9901401109371881948", 2024-08-18T16:39:35.327 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:35.327 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:35.327 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:35.327 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.327 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:35.327 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj58", 2024-08-18T16:39:35.327 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:35.327 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:35.327 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:39:35.327 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 7 2024-08-18T16:39:35.327 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.327 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:35.327 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:35.327 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:35.328 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:35.328 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 6, 2024-08-18T16:39:35.328 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:30:57.319413Z", 2024-08-18T16:39:35.328 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:35.328 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:35.328 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:35.328 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:35.328 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:35.328 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:35.328 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:35.328 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:35.328 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.328 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.9507689502494592331", 2024-08-18T16:39:35.328 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:35.329 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:35.329 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:35.329 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.329 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:35.329 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj62", 2024-08-18T16:39:35.329 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:35.329 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:35.329 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:39:35.329 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 51 2024-08-18T16:39:35.329 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.329 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:35.329 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:35.329 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:35.330 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:35.330 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 6, 2024-08-18T16:39:35.330 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:33:27.434791Z", 2024-08-18T16:39:35.330 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:35.330 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:35.330 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:35.330 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:35.330 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:35.330 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:35.330 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:35.330 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:35.330 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.331 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.16197264008125151659", 2024-08-18T16:39:35.331 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:35.331 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:35.331 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:35.331 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.331 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:35.331 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj68", 2024-08-18T16:39:35.331 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:35.331 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:35.331 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:39:35.331 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 12 2024-08-18T16:39:35.331 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.331 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:35.332 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:35.332 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:35.332 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:35.332 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 6, 2024-08-18T16:39:35.332 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:24:10.351074Z", 2024-08-18T16:39:35.332 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:35.332 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:35.332 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:35.332 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:35.332 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:35.332 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:35.332 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:35.332 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:35.332 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.333 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.12739846555303172313", 2024-08-18T16:39:35.333 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:35.333 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:35.333 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:35.333 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.333 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:35.333 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj69", 2024-08-18T16:39:35.333 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:35.333 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:35.333 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:39:35.333 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 12 2024-08-18T16:39:35.333 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.333 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:35.333 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:35.334 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:35.334 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:35.334 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:39:35.334 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:27:23.414590Z", 2024-08-18T16:39:35.334 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:35.334 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:35.334 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:35.334 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:35.334 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:35.334 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:35.334 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:35.334 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:35.335 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.335 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.6317092130447753347", 2024-08-18T16:39:35.335 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:35.335 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:35.335 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:35.335 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.335 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:35.335 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj74", 2024-08-18T16:39:35.335 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:35.335 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:35.335 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:39:35.335 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 8 2024-08-18T16:39:35.335 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.336 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:35.336 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:35.336 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:35.336 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:35.336 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 5, 2024-08-18T16:39:35.336 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:36:05.141727Z", 2024-08-18T16:39:35.336 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:35.336 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:35.336 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:35.336 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:35.336 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:35.336 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:35.336 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:35.336 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:35.337 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.337 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.7265404272103059924", 2024-08-18T16:39:35.337 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:35.337 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:35.337 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:35.337 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.337 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:35.337 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj76", 2024-08-18T16:39:35.337 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:35.337 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:35.337 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:39:35.337 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 12 2024-08-18T16:39:35.337 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.338 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:35.338 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:35.338 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:35.338 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:35.338 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:39:35.338 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:25:56.019500Z", 2024-08-18T16:39:35.338 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:35.338 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:35.338 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:35.338 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:35.338 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:35.338 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:35.338 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:35.339 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:35.339 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.339 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.9103800521599819169", 2024-08-18T16:39:35.339 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:35.339 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:35.339 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:35.339 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.339 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:35.339 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj85", 2024-08-18T16:39:35.339 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:35.339 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:35.339 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:39:35.340 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 12 2024-08-18T16:39:35.340 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.341 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:35.341 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:35.341 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:35.341 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:35.341 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-08-18T16:39:35.341 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:24:10.358501Z", 2024-08-18T16:39:35.341 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:35.341 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:35.341 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:35.341 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:35.341 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:35.341 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:35.342 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:35.342 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:35.342 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.342 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.14645986973619251717", 2024-08-18T16:39:35.342 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:35.342 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:35.342 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:35.342 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.342 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:35.342 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj87", 2024-08-18T16:39:35.342 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:35.342 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:35.343 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:39:35.343 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 5 2024-08-18T16:39:35.343 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.343 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:35.343 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:35.343 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:35.343 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:35.343 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-08-18T16:39:35.343 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:30:57.327145Z", 2024-08-18T16:39:35.343 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:35.343 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:35.343 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:35.343 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:35.344 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:35.344 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:35.344 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:35.344 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:35.344 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.344 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.15363169285135208126", 2024-08-18T16:39:35.344 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:35.344 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:35.344 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:35.345 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.345 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:35.345 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj92", 2024-08-18T16:39:35.345 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:35.345 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:35.345 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:39:35.345 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 20 2024-08-18T16:39:35.345 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.345 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:35.345 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:35.345 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:35.345 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:35.345 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-08-18T16:39:35.346 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:27:23.459526Z", 2024-08-18T16:39:35.346 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:35.346 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:35.346 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:35.346 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:35.346 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:35.346 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:35.346 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:35.346 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:35.346 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.346 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.15204855023445239626", 2024-08-18T16:39:35.346 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:35.347 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:35.347 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:35.347 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.347 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:35.347 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj93", 2024-08-18T16:39:35.347 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:35.347 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:35.347 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:39:35.347 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 11 2024-08-18T16:39:35.347 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.347 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:35.347 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:35.347 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:35.347 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:35.348 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 5, 2024-08-18T16:39:35.349 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:33:27.484033Z", 2024-08-18T16:39:35.349 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:35.349 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:35.349 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:35.349 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:35.349 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:35.349 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:35.350 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:35.350 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:35.350 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.350 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.915037665819258209", 2024-08-18T16:39:35.350 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:35.350 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:35.350 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:35.350 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.350 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:35.350 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj95", 2024-08-18T16:39:35.351 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:35.351 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:35.351 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-08-18T16:39:35.351 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 20 2024-08-18T16:39:35.351 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.351 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:35.351 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:35.351 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:35.351 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:35.351 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:39:35.351 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:25:56.027064Z", 2024-08-18T16:39:35.351 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:35.351 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:35.351 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:35.352 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:35.352 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:35.352 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:35.352 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:35.352 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:35.352 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.352 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "96bf8502-2f39-4d53-8196-3378c28da3b4.6024.5376182089991821317", 2024-08-18T16:39:35.352 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:35.352 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:35.352 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:35.352 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:39:35.352 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T16:39:35.353 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'list', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:39:35.353 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'list', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-78', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:39:35.354 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket list --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-78 --max-entries 10000 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:39:35.404 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:39:35.404 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:39:35.436 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T16:39:35.437 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:35.437 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj11", 2024-08-18T16:39:35.437 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:35.437 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:35.437 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:39:35.437 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 17 2024-08-18T16:39:35.437 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.437 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:35.437 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:35.437 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:35.437 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:35.437 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:39:35.437 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:23:42.051163Z", 2024-08-18T16:39:35.438 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:35.438 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:35.438 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:35.438 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:35.438 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:35.438 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:35.438 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:35.438 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:35.438 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.438 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_x2Mr2NTeNQkkN0EuA8-G8PGF0lHCtKp", 2024-08-18T16:39:35.438 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:35.438 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:35.439 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:35.439 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.439 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:35.439 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj111", 2024-08-18T16:39:35.439 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:35.439 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:35.439 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:39:35.439 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 26 2024-08-18T16:39:35.439 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.439 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:35.439 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:35.439 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:35.439 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:35.440 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-08-18T16:39:35.440 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:36:05.189316Z", 2024-08-18T16:39:35.440 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:35.440 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:35.440 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:35.440 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:35.440 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:35.440 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:35.440 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:35.440 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:35.440 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.440 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_O9muq_jGccDNhkJuPeZxo6ORIfJqiqN", 2024-08-18T16:39:35.440 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:35.441 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:35.441 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:35.441 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.441 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:35.441 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj115", 2024-08-18T16:39:35.441 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:35.441 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:35.441 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:39:35.441 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 5 2024-08-18T16:39:35.441 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.441 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:35.441 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:35.441 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:35.442 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:35.442 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:39:35.442 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:27:23.466808Z", 2024-08-18T16:39:35.442 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:35.442 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:35.442 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:35.442 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:35.442 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:35.442 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:35.442 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:35.442 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:35.442 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.443 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_YL00HEXSSHuKrK8si5v8ipub6HJVeQR", 2024-08-18T16:39:35.443 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:35.443 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:35.443 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:35.443 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.443 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:35.443 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj116", 2024-08-18T16:39:35.443 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:35.443 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:35.443 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:39:35.443 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 17 2024-08-18T16:39:35.443 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.443 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:35.444 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:35.444 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:35.444 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:35.444 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 5, 2024-08-18T16:39:35.444 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:30:57.371585Z", 2024-08-18T16:39:35.444 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:35.444 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:35.444 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:35.444 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:35.444 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:35.444 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:35.444 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:35.444 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:35.444 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.445 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_UnP4N2STaesyf3-ICKYokBWeBvt_QoH", 2024-08-18T16:39:35.445 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:35.445 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:35.445 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:35.445 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.445 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:35.445 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj124", 2024-08-18T16:39:35.445 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:35.445 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:35.445 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:39:35.445 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 9 2024-08-18T16:39:35.445 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.445 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:35.446 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:35.446 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:35.446 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:35.446 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:39:35.446 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:33:27.492061Z", 2024-08-18T16:39:35.446 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:35.446 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:35.446 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:35.446 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:35.446 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:35.446 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:35.446 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:35.447 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:35.447 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.447 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_mZnzzVw4o6UZycWOsEku1_L62k_zZgj", 2024-08-18T16:39:35.447 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:35.447 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:35.447 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:35.447 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.447 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:35.447 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj145", 2024-08-18T16:39:35.447 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:35.447 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:35.447 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:39:35.447 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 22 2024-08-18T16:39:35.448 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.448 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:35.448 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:35.448 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:35.448 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:35.448 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:39:35.448 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:30:57.379450Z", 2024-08-18T16:39:35.448 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:35.448 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:35.448 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:35.448 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:35.448 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:35.448 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:35.448 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:35.449 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:35.449 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.449 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_xWJWvRCAdzKqnjAUgpOr87cdLss9hLw", 2024-08-18T16:39:35.449 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:35.449 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:35.449 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:35.449 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.449 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:35.449 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj148", 2024-08-18T16:39:35.449 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:35.449 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:35.449 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:39:35.450 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 9 2024-08-18T16:39:35.450 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.450 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:35.450 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:35.450 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:35.450 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:35.450 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-08-18T16:39:35.450 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:36:05.197180Z", 2024-08-18T16:39:35.450 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:35.450 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:35.450 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:35.450 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:35.451 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:35.451 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:35.451 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:35.451 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:35.451 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.451 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_OObPTr98zXzbAnYrLemu_vNjptzkiIk", 2024-08-18T16:39:35.451 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:35.451 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:35.451 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:35.451 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.451 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:35.451 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj155", 2024-08-18T16:39:35.451 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:35.451 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:35.452 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:39:35.452 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 9 2024-08-18T16:39:35.452 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.452 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:35.452 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:35.452 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:35.452 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:35.452 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 6, 2024-08-18T16:39:35.452 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:33:27.500403Z", 2024-08-18T16:39:35.452 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:35.452 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:35.452 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:35.452 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:35.453 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:35.453 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:35.453 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:35.453 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:35.453 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.453 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_LeAgq9dL3i83w14ce_2BYfP_UndKGXG", 2024-08-18T16:39:35.453 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:35.453 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:35.453 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:35.453 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.453 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:35.453 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj17", 2024-08-18T16:39:35.453 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:35.454 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:35.454 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:39:35.454 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 8 2024-08-18T16:39:35.454 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.454 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:35.454 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:35.454 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:35.454 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:35.454 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 5, 2024-08-18T16:39:35.454 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:24:10.291690Z", 2024-08-18T16:39:35.454 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:35.454 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:35.455 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:35.455 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:35.455 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:35.455 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:35.455 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:35.455 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:35.455 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.455 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_dx8gveiYKak6FG8kqxSbWKAxLVQnSbC", 2024-08-18T16:39:35.455 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:35.455 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:35.455 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:35.455 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.455 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:35.455 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj185", 2024-08-18T16:39:35.456 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:35.456 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:35.456 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:39:35.456 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 14 2024-08-18T16:39:35.456 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.456 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:35.456 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:35.456 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:35.456 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:35.456 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:39:35.456 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:36:05.244081Z", 2024-08-18T16:39:35.456 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:35.456 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:35.456 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:35.457 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:35.457 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:35.457 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:35.457 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:35.457 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:35.457 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.457 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_2UKN1VD79rhkZHdpTe4SjR8kBuFmTAN", 2024-08-18T16:39:35.457 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:35.457 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:35.457 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:35.457 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.457 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:35.457 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj19", 2024-08-18T16:39:35.458 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:35.458 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:35.458 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:39:35.458 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 10 2024-08-18T16:39:35.458 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.458 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:35.458 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:35.458 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:35.458 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:35.458 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 6, 2024-08-18T16:39:35.458 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:25:55.960750Z", 2024-08-18T16:39:35.458 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:35.458 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:35.459 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:35.459 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:35.459 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:35.459 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:35.459 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:35.459 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:35.459 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.459 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_heZy6lg-fKus8znILZC5i1ucpngg3PF", 2024-08-18T16:39:35.459 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:35.459 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:35.459 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:35.459 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.459 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:35.460 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj22", 2024-08-18T16:39:35.460 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:35.460 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:35.460 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:39:35.460 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 53 2024-08-18T16:39:35.460 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.460 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:35.460 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:35.460 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:35.460 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:35.460 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-08-18T16:39:35.460 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:23:42.058759Z", 2024-08-18T16:39:35.460 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:35.460 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:35.461 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:35.461 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:35.461 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:35.461 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:35.461 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:35.461 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:35.461 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.461 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_EqvQ3AfCcscWxPWXtqoevIDk7PCdS-R", 2024-08-18T16:39:35.461 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:35.461 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:35.461 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:35.461 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.461 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:35.461 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj23", 2024-08-18T16:39:35.462 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:35.462 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:35.462 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:39:35.462 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 10 2024-08-18T16:39:35.462 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.462 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:35.462 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:35.462 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:35.462 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:35.462 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 5, 2024-08-18T16:39:35.462 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:27:23.361495Z", 2024-08-18T16:39:35.462 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:35.463 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:35.463 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:35.463 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:35.463 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:35.463 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:35.463 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:35.463 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:35.463 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.463 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_x9kkT1jQAdpTi1vb6hdynmzcv6LrNZW", 2024-08-18T16:39:35.463 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:35.463 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:35.463 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:35.463 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.464 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:35.464 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj29", 2024-08-18T16:39:35.464 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:35.464 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:35.464 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:39:35.464 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 7 2024-08-18T16:39:35.464 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.464 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:35.464 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:35.464 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:35.464 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:35.464 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:39:35.464 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:30:57.311630Z", 2024-08-18T16:39:35.464 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:35.465 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:35.465 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:35.465 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:35.465 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:35.465 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:35.465 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:35.465 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:35.465 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.465 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_8d5bobqaXxA0mh2qOSqE4OqdAUCjGD_", 2024-08-18T16:39:35.465 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:35.465 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:35.465 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:35.465 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.466 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:35.466 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj31", 2024-08-18T16:39:35.466 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:35.466 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:35.466 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:39:35.466 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 10 2024-08-18T16:39:35.466 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.466 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:35.466 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:35.466 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:35.466 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:35.466 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:39:35.466 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:33:27.426326Z", 2024-08-18T16:39:35.467 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:35.467 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:35.467 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:35.467 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:35.467 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:35.467 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:35.467 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:35.467 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:35.467 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.467 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_wj19CssEsxi-60uhV9nNAdjyt_6sqKe", 2024-08-18T16:39:35.467 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:35.467 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:35.468 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:35.468 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.468 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:35.468 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj33", 2024-08-18T16:39:35.468 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:35.468 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:35.468 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:39:35.468 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 10 2024-08-18T16:39:35.468 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.468 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:35.468 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:35.468 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:35.468 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:35.469 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:39:35.469 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:23:42.064530Z", 2024-08-18T16:39:35.469 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:35.469 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:35.469 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:35.469 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:35.469 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:35.469 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:35.469 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:35.469 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:35.469 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.469 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_6iwLafOWWt4UfRg1D9Dmh33JnKD9UdM", 2024-08-18T16:39:35.469 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:35.470 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:35.470 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:35.470 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.470 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:35.470 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj34", 2024-08-18T16:39:35.470 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:35.470 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:35.470 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:39:35.470 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 18 2024-08-18T16:39:35.470 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.470 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:35.470 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:35.470 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:35.471 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:35.471 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 5, 2024-08-18T16:39:35.471 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:24:10.298744Z", 2024-08-18T16:39:35.471 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:35.471 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:35.471 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:35.471 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:35.471 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:35.471 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:35.471 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:35.471 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:35.471 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.471 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_0pYfvl5abtFLCeqE8p42XjQZEDDPC2G", 2024-08-18T16:39:35.471 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:35.472 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:35.472 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:35.472 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.472 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:35.472 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj37", 2024-08-18T16:39:35.472 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:35.472 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:35.472 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:39:35.472 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 17 2024-08-18T16:39:35.472 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.472 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:35.472 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:35.472 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:35.473 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:35.473 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 6, 2024-08-18T16:39:35.473 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:36:05.133754Z", 2024-08-18T16:39:35.473 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:35.473 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:35.473 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:35.473 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:35.473 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:35.473 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:35.473 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:35.473 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:35.473 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.474 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_1aGYR3nZPLH6wNTfwUcUoz7yhV91ib4", 2024-08-18T16:39:35.474 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:35.474 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:35.474 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:35.474 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.474 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:35.474 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj38", 2024-08-18T16:39:35.474 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:35.474 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:35.474 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:39:35.474 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 10 2024-08-18T16:39:35.474 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.475 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:35.475 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:35.475 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:35.475 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:35.475 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:39:35.475 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:25:55.967916Z", 2024-08-18T16:39:35.475 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:35.475 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:35.475 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:35.475 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:35.475 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:35.475 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:35.475 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:35.476 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:35.476 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.476 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_4PkW0WQJSQtbX7WC45RUUisG-bO_o7L", 2024-08-18T16:39:35.476 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:35.476 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:35.476 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:35.476 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.476 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:35.476 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj44", 2024-08-18T16:39:35.476 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:35.476 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:35.476 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:39:35.476 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 12 2024-08-18T16:39:35.477 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.477 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:35.477 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:35.477 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:35.477 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:35.477 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 6, 2024-08-18T16:39:35.477 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:23:42.115387Z", 2024-08-18T16:39:35.477 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:35.477 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:35.477 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:35.477 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:35.477 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:35.477 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:35.478 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:35.478 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:35.478 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.478 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_UKItzdy-Z71tKIN4C0LYTvr0T-YVk1Z", 2024-08-18T16:39:35.478 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:35.478 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:35.478 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:35.478 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.478 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:35.478 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj46", 2024-08-18T16:39:35.478 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:35.478 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:35.478 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:39:35.479 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 13 2024-08-18T16:39:35.479 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.479 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:35.479 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:35.479 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:35.479 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:35.479 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:39:35.479 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:27:23.407342Z", 2024-08-18T16:39:35.479 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:35.479 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:35.479 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:35.479 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:35.479 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:35.480 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:35.480 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:35.480 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:35.480 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.480 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_ySKis0NG89cLSwqop9uwLYxWFgoTByb", 2024-08-18T16:39:35.480 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:35.480 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:35.480 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:35.480 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.480 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:35.480 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj51", 2024-08-18T16:39:35.480 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:35.481 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:35.481 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:39:35.481 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 9 2024-08-18T16:39:35.481 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.481 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:35.481 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:35.481 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:35.481 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:35.481 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 5, 2024-08-18T16:39:35.481 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:24:10.305333Z", 2024-08-18T16:39:35.481 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:35.481 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:35.481 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:35.482 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:35.482 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:35.482 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:35.482 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:35.482 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:35.482 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.482 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_ojyphEJx_vyQrmLgV622MJXU7cuLsXQ", 2024-08-18T16:39:35.482 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:35.482 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:35.482 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:35.482 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.482 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:35.483 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj55", 2024-08-18T16:39:35.483 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:35.483 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:35.483 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:39:35.483 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 9 2024-08-18T16:39:35.483 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.483 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:35.483 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:35.483 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:35.483 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:35.483 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:39:35.483 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:23:42.122245Z", 2024-08-18T16:39:35.483 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:35.483 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:35.484 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:35.484 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:35.484 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:35.484 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:35.484 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:35.484 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:35.484 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.484 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_yWSpxcyb5DjsBr-aN0WbEeN02W1DbF5", 2024-08-18T16:39:35.484 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:35.484 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:35.484 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:35.484 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.484 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:35.485 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj57", 2024-08-18T16:39:35.485 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:35.485 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:35.485 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:39:35.485 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 13 2024-08-18T16:39:35.485 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.485 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:35.485 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:35.485 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:35.485 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:35.485 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:39:35.485 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:25:55.973669Z", 2024-08-18T16:39:35.486 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:35.486 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:35.486 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:35.486 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:35.486 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:35.486 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:35.486 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:35.486 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:35.486 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.486 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_qndG3j9N62OA0ucepOeghx-gkOIHQNV", 2024-08-18T16:39:35.486 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:35.486 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:35.486 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:35.487 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.487 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:35.487 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj58", 2024-08-18T16:39:35.487 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:35.487 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:35.487 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:39:35.487 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 11 2024-08-18T16:39:35.487 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.487 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:35.487 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:35.487 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:35.487 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:35.487 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:39:35.488 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:30:57.319413Z", 2024-08-18T16:39:35.488 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:35.488 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:35.488 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:35.488 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:35.488 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:35.488 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:35.488 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:35.488 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:35.488 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.488 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_pFMqKXQ6TTwLe17IlBHmOBNHFzZTAnQ", 2024-08-18T16:39:35.488 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:35.488 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:35.489 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:35.489 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.489 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:35.489 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj62", 2024-08-18T16:39:35.489 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:35.489 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:35.489 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:39:35.489 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 52 2024-08-18T16:39:35.489 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.489 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:35.489 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:35.489 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:35.489 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:35.490 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:39:35.490 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:33:27.434791Z", 2024-08-18T16:39:35.490 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:35.490 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:35.490 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:35.490 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:35.490 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:35.490 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:35.490 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:35.490 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:35.490 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.490 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "__1C5tMvURqXNpxNMb51KW7P8rxwYXBp", 2024-08-18T16:39:35.490 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:35.491 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:35.491 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:35.491 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.491 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:35.491 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj68", 2024-08-18T16:39:35.491 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:35.491 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:35.491 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:39:35.491 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 15 2024-08-18T16:39:35.491 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.491 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:35.491 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:35.491 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:35.492 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:35.492 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:39:35.492 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:24:10.351074Z", 2024-08-18T16:39:35.492 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:35.492 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:35.492 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:35.492 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:35.492 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:35.492 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:35.492 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:35.492 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:35.492 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.492 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_AMZ_ksPhtFId2xClE_xoxGTiA1Tm8m3", 2024-08-18T16:39:35.492 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:35.493 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:35.493 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:35.493 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.493 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:35.493 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj69", 2024-08-18T16:39:35.493 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:35.493 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:35.493 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:39:35.493 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 16 2024-08-18T16:39:35.493 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.493 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:35.493 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:35.493 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:35.493 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:35.494 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 5, 2024-08-18T16:39:35.494 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:27:23.414590Z", 2024-08-18T16:39:35.494 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:35.494 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:35.494 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:35.494 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:35.494 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:35.494 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:35.494 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:35.494 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:35.494 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.494 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_Hl_YD1L0giLBby8BAzA88Z-8EOkBrzD", 2024-08-18T16:39:35.495 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:35.495 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:35.495 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:35.495 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.495 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:35.495 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj74", 2024-08-18T16:39:35.495 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:35.495 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:35.495 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:39:35.495 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 10 2024-08-18T16:39:35.495 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.495 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:35.496 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:35.496 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:35.496 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:35.496 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 6, 2024-08-18T16:39:35.496 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:36:05.141727Z", 2024-08-18T16:39:35.496 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:35.496 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:35.496 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:35.496 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:35.496 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:35.496 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:35.496 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:35.496 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:35.497 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.497 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_KVNiJk098GGdZSvAKENE7iV-gai926S", 2024-08-18T16:39:35.497 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:35.497 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:35.497 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:35.497 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.497 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:35.497 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj76", 2024-08-18T16:39:35.497 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:35.497 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:35.497 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:39:35.497 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 10 2024-08-18T16:39:35.497 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.497 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:35.498 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:35.498 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:35.498 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:35.498 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-08-18T16:39:35.498 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:25:56.019500Z", 2024-08-18T16:39:35.498 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:35.498 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:35.498 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:35.498 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:35.498 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:35.498 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:35.498 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:35.499 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:35.499 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.499 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_7XkmWZiTtNuUsebp1yPD8fgN-7D59gY", 2024-08-18T16:39:35.499 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:35.499 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:35.499 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:35.499 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.499 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:35.499 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj85", 2024-08-18T16:39:35.499 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:35.499 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:35.499 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:39:35.499 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 12 2024-08-18T16:39:35.499 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.500 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:35.500 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:35.500 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:35.500 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:35.500 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-08-18T16:39:35.500 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:24:10.358501Z", 2024-08-18T16:39:35.500 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:35.500 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:35.500 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:35.500 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:35.500 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:35.500 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:35.500 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:35.501 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:35.501 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.501 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_WbFsCvzVo6G6mB6iGqgByek1iODxtqn", 2024-08-18T16:39:35.501 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:35.501 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:35.501 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:35.501 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.501 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:35.501 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj87", 2024-08-18T16:39:35.501 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:35.501 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:35.501 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:39:35.501 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 7 2024-08-18T16:39:35.502 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.502 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:35.503 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:35.503 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:35.503 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:35.503 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:39:35.503 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:30:57.327145Z", 2024-08-18T16:39:35.503 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:35.503 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:35.503 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:35.503 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:35.504 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:35.504 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:35.504 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:35.504 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:35.504 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.504 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_Fao0bCXGtmwz4RYFyh22lvAZDgXM75F", 2024-08-18T16:39:35.504 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:35.504 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:35.504 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:35.504 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.504 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:35.504 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj92", 2024-08-18T16:39:35.504 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:35.505 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:35.505 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:39:35.505 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 23 2024-08-18T16:39:35.505 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.505 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:35.505 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:35.505 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:35.505 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:35.505 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-08-18T16:39:35.505 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:27:23.459526Z", 2024-08-18T16:39:35.505 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:35.505 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:35.505 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:35.506 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:35.506 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:35.506 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:35.506 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:35.506 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:35.506 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.506 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_kYO8weugkcsTt_PlhchiTyULGWmqgY_", 2024-08-18T16:39:35.506 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:35.506 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:35.506 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:35.506 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.506 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:35.507 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj93", 2024-08-18T16:39:35.507 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:35.507 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:35.507 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:39:35.507 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 11 2024-08-18T16:39:35.507 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.507 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:35.507 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:35.507 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:35.507 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:35.507 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-08-18T16:39:35.507 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:33:27.484033Z", 2024-08-18T16:39:35.507 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:35.507 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:35.508 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:35.508 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:35.508 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:35.508 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:35.508 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:35.508 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:35.508 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.508 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_BpIAv5pVqfCe600_dnUNfpxk0aBW2nQ", 2024-08-18T16:39:35.508 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:35.508 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:35.508 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:35.508 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.508 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:39:35.509 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "obj95", 2024-08-18T16:39:35.509 INFO:teuthology.orchestra.run.smithi134.stdout: "instance": "", 2024-08-18T16:39:35.509 INFO:teuthology.orchestra.run.smithi134.stdout: "ver": { 2024-08-18T16:39:35.509 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 14, 2024-08-18T16:39:35.509 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 18 2024-08-18T16:39:35.509 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.509 INFO:teuthology.orchestra.run.smithi134.stdout: "locator": "", 2024-08-18T16:39:35.509 INFO:teuthology.orchestra.run.smithi134.stdout: "exists": true, 2024-08-18T16:39:35.509 INFO:teuthology.orchestra.run.smithi134.stdout: "meta": { 2024-08-18T16:39:35.509 INFO:teuthology.orchestra.run.smithi134.stdout: "category": 1, 2024-08-18T16:39:35.509 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 16, 2024-08-18T16:39:35.509 INFO:teuthology.orchestra.run.smithi134.stdout: "mtime": "2024-08-18T16:25:56.027064Z", 2024-08-18T16:39:35.509 INFO:teuthology.orchestra.run.smithi134.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-08-18T16:39:35.510 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_class": "", 2024-08-18T16:39:35.510 INFO:teuthology.orchestra.run.smithi134.stdout: "owner": "rgw-multisite-test-user", 2024-08-18T16:39:35.510 INFO:teuthology.orchestra.run.smithi134.stdout: "owner_display_name": "Multisite Test User", 2024-08-18T16:39:35.510 INFO:teuthology.orchestra.run.smithi134.stdout: "content_type": "application/octet-stream", 2024-08-18T16:39:35.510 INFO:teuthology.orchestra.run.smithi134.stdout: "accounted_size": 3, 2024-08-18T16:39:35.510 INFO:teuthology.orchestra.run.smithi134.stdout: "user_data": "", 2024-08-18T16:39:35.510 INFO:teuthology.orchestra.run.smithi134.stdout: "appendable": false 2024-08-18T16:39:35.510 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:39:35.510 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "_Cg-y52YWOxdNoItO4SvV2VGTd4LuE-x", 2024-08-18T16:39:35.510 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 0, 2024-08-18T16:39:35.510 INFO:teuthology.orchestra.run.smithi134.stdout: "pending_map": [], 2024-08-18T16:39:35.510 INFO:teuthology.orchestra.run.smithi134.stdout: "versioned_epoch": 0 2024-08-18T16:39:35.510 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:39:35.511 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T16:39:35.511 INFO:tasks.rgw.c1.client.1:Restarting daemon 2024-08-18T16:39:35.511 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2024-08-18T16:39:35.513 INFO:tasks.rgw.c1.client.1:Started 2024-08-18T16:39:35.514 DEBUG:teuthology.orchestra.run.smithi134:> curl http://smithi134.front.sepia.ceph.com:8001 2024-08-18T16:39:35.532 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-08-18T16:39:35.533 INFO:teuthology.orchestra.run.smithi134.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-08-18T16:39:35.533 INFO:teuthology.orchestra.run.smithi134.stderr: Dload Upload Total Spent Left Speed 2024-08-18T16:39:35.533 INFO:teuthology.orchestra.run.smithi134.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2024-08-18T16:39:35.533 INFO:teuthology.orchestra.run.smithi134.stderr:curl: (7) Failed to connect to smithi134.front.sepia.ceph.com port 8001: Connection refused 2024-08-18T16:39:36.534 DEBUG:teuthology.orchestra.run.smithi134:> curl http://smithi134.front.sepia.ceph.com:8001 2024-08-18T16:39:36.554 INFO:teuthology.orchestra.run.smithi134.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-08-18T16:39:36.554 INFO:teuthology.orchestra.run.smithi134.stderr: Dload Upload Total Spent Left Speed 2024-08-18T16:39:36.554 INFO:teuthology.orchestra.run.smithi134.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-08-18T16:39:36.555 INFO:teuthology.orchestra.run.smithi134.stdout:anonymous 2024-08-18T16:39:36.555 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:39:36.555 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:39:36.556 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-78 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T16:39:36.617 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:39:36.617 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:39:36.660 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:36.659+0000 7fbec773c880 1 bucket sync caught up with source: 2024-08-18T16:39:36.660 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , 00000000002.82.6, , , , , , 00000000002.191.6, , , , , 00000000002.107.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.120.6] 2024-08-18T16:39:36.660 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , 00000000002.82.6, , , , , , 00000000002.191.6, , , , , 00000000002.107.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.120.6] 2024-08-18T16:39:36.660 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:39:36.659+0000 7fbec773c880 0 bucket checkpoint complete 2024-08-18T16:39:36.675 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:39:36.675 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:39:36.675 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-78 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:39:36.743 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:39:36.743 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:39:36.782 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:36.781+0000 7f0257523880 1 bucket sync caught up with source: 2024-08-18T16:39:36.782 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , 00000000002.82.6, , , , , , 00000000002.191.6, , , , , 00000000002.107.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.120.6] 2024-08-18T16:39:36.782 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , , 00000000002.82.6, , , , , , 00000000002.191.6, , , , , 00000000002.107.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.120.6] 2024-08-18T16:39:36.782 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:36.781+0000 7f0257523880 0 bucket checkpoint complete 2024-08-18T16:39:36.796 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:39:36.796 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:39:36.796 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-78 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:39:36.850 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:39:36.850 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:39:36.886 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:36.885+0000 7fb81ff42880 1 bucket sync caught up with source: 2024-08-18T16:39:36.886 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000009.393.6, 00000000006.266.6, 00000000006.382.6, 00000000015.250.6, 00000000012.219.6, 00000000003.414.6, 00000000003.268.6, 00000000012.303.6, 00000000021.392.6, 00000000018.303.6] 2024-08-18T16:39:36.886 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000009.393.6, 00000000006.266.6, 00000000006.382.6, 00000000015.250.6, 00000000012.219.6, 00000000003.414.6, 00000000003.268.6, 00000000012.303.6, 00000000021.392.6, 00000000018.303.6] 2024-08-18T16:39:36.886 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:36.885+0000 7fb81ff42880 0 bucket checkpoint complete 2024-08-18T16:39:36.900 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:39:36.900 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:39:36.901 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-78 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:39:36.954 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:39:36.954 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:39:36.989 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:36.988+0000 7fbf80830880 1 waiting for incremental sync to catch up: 2024-08-18T16:39:36.989 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000009.393.6, 00000000006.266.6, 00000000002.370.6, 00000000015.250.6, 00000000012.219.6, 00000000003.414.6, 00000000001.244.6, 00000000012.303.6, 00000000017.383.6, 00000000006.277.6] 2024-08-18T16:39:36.989 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000009.393.6, 00000000006.266.6, 00000000006.382.6, 00000000015.250.6, 00000000012.219.6, 00000000003.414.6, 00000000003.268.6, 00000000012.303.6, 00000000021.392.6, 00000000018.303.6] 2024-08-18T16:39:41.990 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:41.989+0000 7fbf80830880 1 waiting for incremental sync to catch up: 2024-08-18T16:39:41.991 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000009.393.6, 00000000006.266.6, 00000000002.370.6, 00000000015.250.6, 00000000012.219.6, 00000000003.414.6, 00000000001.244.6, 00000000012.303.6, 00000000017.383.6, 00000000006.277.6] 2024-08-18T16:39:41.991 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000009.393.6, 00000000006.266.6, 00000000006.382.6, 00000000015.250.6, 00000000012.219.6, 00000000003.414.6, 00000000003.268.6, 00000000012.303.6, 00000000021.392.6, 00000000018.303.6] 2024-08-18T16:39:46.992 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:46.991+0000 7fbf80830880 1 waiting for incremental sync to catch up: 2024-08-18T16:39:46.992 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000009.393.6, 00000000006.266.6, 00000000002.370.6, 00000000015.250.6, 00000000012.219.6, 00000000003.414.6, 00000000001.244.6, 00000000012.303.6, 00000000017.383.6, 00000000006.277.6] 2024-08-18T16:39:46.992 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000009.393.6, 00000000006.266.6, 00000000006.382.6, 00000000015.250.6, 00000000012.219.6, 00000000003.414.6, 00000000003.268.6, 00000000012.303.6, 00000000021.392.6, 00000000018.303.6] 2024-08-18T16:39:51.993 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:51.992+0000 7fbf80830880 1 waiting for incremental sync to catch up: 2024-08-18T16:39:51.993 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000009.393.6, 00000000006.266.6, 00000000002.370.6, 00000000015.250.6, 00000000012.219.6, 00000000003.414.6, 00000000001.244.6, 00000000012.303.6, 00000000017.383.6, 00000000006.277.6] 2024-08-18T16:39:51.993 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000009.393.6, 00000000006.266.6, 00000000006.382.6, 00000000015.250.6, 00000000012.219.6, 00000000003.414.6, 00000000003.268.6, 00000000012.303.6, 00000000021.392.6, 00000000018.303.6] 2024-08-18T16:39:56.994 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:39:56.993+0000 7fbf80830880 1 waiting for incremental sync to catch up: 2024-08-18T16:39:56.994 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000009.393.6, 00000000006.266.6, 00000000002.370.6, 00000000015.250.6, 00000000012.219.6, 00000000003.414.6, 00000000001.244.6, 00000000012.303.6, 00000000017.383.6, 00000000006.277.6] 2024-08-18T16:39:56.995 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000009.393.6, 00000000006.266.6, 00000000006.382.6, 00000000015.250.6, 00000000012.219.6, 00000000003.414.6, 00000000003.268.6, 00000000012.303.6, 00000000021.392.6, 00000000018.303.6] 2024-08-18T16:40:01.995 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:40:01.994+0000 7fbf80830880 1 waiting for incremental sync to catch up: 2024-08-18T16:40:01.996 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000009.393.6, 00000000006.266.6, 00000000002.370.6, 00000000015.250.6, 00000000012.219.6, 00000000003.414.6, 00000000001.244.6, 00000000012.303.6, 00000000017.383.6, 00000000006.277.6] 2024-08-18T16:40:01.996 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000009.393.6, 00000000006.266.6, 00000000006.382.6, 00000000015.250.6, 00000000012.219.6, 00000000003.414.6, 00000000003.268.6, 00000000012.303.6, 00000000021.392.6, 00000000018.303.6] 2024-08-18T16:40:06.997 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:40:06.996+0000 7fbf80830880 1 waiting for incremental sync to catch up: 2024-08-18T16:40:06.997 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000009.393.6, 00000000006.266.6, 00000000002.370.6, 00000000015.250.6, 00000000012.219.6, 00000000003.414.6, 00000000001.244.6, 00000000012.303.6, 00000000017.383.6, 00000000006.277.6] 2024-08-18T16:40:06.997 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000009.393.6, 00000000006.266.6, 00000000006.382.6, 00000000015.250.6, 00000000012.219.6, 00000000003.414.6, 00000000003.268.6, 00000000012.303.6, 00000000021.392.6, 00000000018.303.6] 2024-08-18T16:40:11.998 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:40:11.997+0000 7fbf80830880 1 waiting for incremental sync to catch up: 2024-08-18T16:40:11.998 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000009.393.6, 00000000006.266.6, 00000000002.370.6, 00000000015.250.6, 00000000012.219.6, 00000000003.414.6, 00000000001.244.6, 00000000012.303.6, 00000000017.383.6, 00000000006.277.6] 2024-08-18T16:40:11.998 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000009.393.6, 00000000006.266.6, 00000000006.382.6, 00000000015.250.6, 00000000012.219.6, 00000000003.414.6, 00000000003.268.6, 00000000012.303.6, 00000000021.392.6, 00000000018.303.6] 2024-08-18T16:40:16.999 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:40:16.998+0000 7fbf80830880 1 waiting for incremental sync to catch up: 2024-08-18T16:40:16.999 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000009.393.6, 00000000006.266.6, 00000000002.370.6, 00000000015.250.6, 00000000012.219.6, 00000000003.414.6, 00000000001.244.6, 00000000012.303.6, 00000000017.383.6, 00000000006.277.6] 2024-08-18T16:40:16.999 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000009.393.6, 00000000006.266.6, 00000000006.382.6, 00000000015.250.6, 00000000012.219.6, 00000000003.414.6, 00000000003.268.6, 00000000012.303.6, 00000000021.392.6, 00000000018.303.6] 2024-08-18T16:40:22.000 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:40:21.999+0000 7fbf80830880 1 waiting for incremental sync to catch up: 2024-08-18T16:40:22.001 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000009.393.6, 00000000006.266.6, 00000000002.370.6, 00000000015.250.6, 00000000012.219.6, 00000000003.414.6, 00000000001.244.6, 00000000012.303.6, 00000000017.383.6, 00000000006.277.6] 2024-08-18T16:40:22.001 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000009.393.6, 00000000006.266.6, 00000000006.382.6, 00000000015.250.6, 00000000012.219.6, 00000000003.414.6, 00000000003.268.6, 00000000012.303.6, 00000000021.392.6, 00000000018.303.6] 2024-08-18T16:40:27.002 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:40:27.001+0000 7fbf80830880 1 waiting for incremental sync to catch up: 2024-08-18T16:40:27.002 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000009.393.6, 00000000006.266.6, 00000000002.370.6, 00000000015.250.6, 00000000012.219.6, 00000000003.414.6, 00000000001.244.6, 00000000012.303.6, 00000000017.383.6, 00000000006.277.6] 2024-08-18T16:40:27.002 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000009.393.6, 00000000006.266.6, 00000000006.382.6, 00000000015.250.6, 00000000012.219.6, 00000000003.414.6, 00000000003.268.6, 00000000012.303.6, 00000000021.392.6, 00000000018.303.6] 2024-08-18T16:40:32.003 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:40:32.002+0000 7fbf80830880 1 waiting for incremental sync to catch up: 2024-08-18T16:40:32.003 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000009.393.6, 00000000006.266.6, 00000000002.370.6, 00000000015.250.6, 00000000012.219.6, 00000000003.414.6, 00000000001.244.6, 00000000012.303.6, 00000000017.383.6, 00000000006.277.6] 2024-08-18T16:40:32.003 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000009.393.6, 00000000006.266.6, 00000000006.382.6, 00000000015.250.6, 00000000012.219.6, 00000000003.414.6, 00000000003.268.6, 00000000012.303.6, 00000000021.392.6, 00000000018.303.6] 2024-08-18T16:40:37.004 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:40:37.003+0000 7fbf80830880 1 waiting for incremental sync to catch up: 2024-08-18T16:40:37.004 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000009.393.6, 00000000006.266.6, 00000000002.370.6, 00000000015.250.6, 00000000012.219.6, 00000000003.414.6, 00000000001.244.6, 00000000012.303.6, 00000000017.383.6, 00000000006.277.6] 2024-08-18T16:40:37.004 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000009.393.6, 00000000006.266.6, 00000000006.382.6, 00000000015.250.6, 00000000012.219.6, 00000000003.414.6, 00000000003.268.6, 00000000012.303.6, 00000000021.392.6, 00000000018.303.6] 2024-08-18T16:40:42.006 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:40:42.005+0000 7fbf80830880 1 waiting for incremental sync to catch up: 2024-08-18T16:40:42.006 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000009.393.6, 00000000006.266.6, 00000000002.370.6, 00000000015.250.6, 00000000012.219.6, 00000000003.414.6, 00000000001.244.6, 00000000012.303.6, 00000000017.383.6, 00000000006.277.6] 2024-08-18T16:40:42.006 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000009.393.6, 00000000006.266.6, 00000000006.382.6, 00000000015.250.6, 00000000012.219.6, 00000000003.414.6, 00000000003.268.6, 00000000012.303.6, 00000000021.392.6, 00000000018.303.6] 2024-08-18T16:40:47.007 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:40:47.006+0000 7fbf80830880 1 waiting for incremental sync to catch up: 2024-08-18T16:40:47.007 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000009.393.6, 00000000006.266.6, 00000000002.370.6, 00000000015.250.6, 00000000012.219.6, 00000000003.414.6, 00000000001.244.6, 00000000012.303.6, 00000000017.383.6, 00000000006.277.6] 2024-08-18T16:40:47.007 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000009.393.6, 00000000006.266.6, 00000000006.382.6, 00000000015.250.6, 00000000012.219.6, 00000000003.414.6, 00000000003.268.6, 00000000012.303.6, 00000000021.392.6, 00000000018.303.6] 2024-08-18T16:40:52.008 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:40:52.006+0000 7fbf80830880 1 waiting for incremental sync to catch up: 2024-08-18T16:40:52.008 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000009.393.6, 00000000006.266.6, 00000000002.370.6, 00000000015.250.6, 00000000012.219.6, 00000000003.414.6, 00000000001.244.6, 00000000012.303.6, 00000000017.383.6, 00000000006.277.6] 2024-08-18T16:40:52.008 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000009.393.6, 00000000006.266.6, 00000000006.382.6, 00000000015.250.6, 00000000012.219.6, 00000000003.414.6, 00000000003.268.6, 00000000012.303.6, 00000000021.392.6, 00000000018.303.6] 2024-08-18T16:40:57.009 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:40:57.008+0000 7fbf80830880 1 waiting for incremental sync to catch up: 2024-08-18T16:40:57.009 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000009.393.6, 00000000006.266.6, 00000000002.370.6, 00000000015.250.6, 00000000012.219.6, 00000000003.414.6, 00000000001.244.6, 00000000012.303.6, 00000000019.386.6, 00000000006.277.6] 2024-08-18T16:40:57.010 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000009.393.6, 00000000006.266.6, 00000000006.382.6, 00000000015.250.6, 00000000012.219.6, 00000000003.414.6, 00000000003.268.6, 00000000012.303.6, 00000000021.392.6, 00000000018.303.6] 2024-08-18T16:41:02.010 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:41:02.009+0000 7fbf80830880 1 waiting for incremental sync to catch up: 2024-08-18T16:41:02.011 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000009.393.6, 00000000006.266.6, 00000000002.370.6, 00000000015.250.6, 00000000012.219.6, 00000000003.414.6, 00000000001.244.6, 00000000012.303.6, 00000000019.386.6, 00000000006.277.6] 2024-08-18T16:41:02.011 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000009.393.6, 00000000006.266.6, 00000000006.382.6, 00000000015.250.6, 00000000012.219.6, 00000000003.414.6, 00000000003.268.6, 00000000012.303.6, 00000000021.392.6, 00000000018.303.6] 2024-08-18T16:41:07.012 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:41:07.011+0000 7fbf80830880 1 waiting for incremental sync to catch up: 2024-08-18T16:41:07.012 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000009.393.6, 00000000006.266.6, 00000000002.370.6, 00000000015.250.6, 00000000012.219.6, 00000000003.414.6, 00000000001.244.6, 00000000012.303.6, 00000000019.386.6, 00000000006.277.6] 2024-08-18T16:41:07.012 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000009.393.6, 00000000006.266.6, 00000000006.382.6, 00000000015.250.6, 00000000012.219.6, 00000000003.414.6, 00000000003.268.6, 00000000012.303.6, 00000000021.392.6, 00000000018.303.6] 2024-08-18T16:41:12.013 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:41:12.012+0000 7fbf80830880 1 waiting for incremental sync to catch up: 2024-08-18T16:41:12.013 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000009.393.6, 00000000006.266.6, 00000000002.370.6, 00000000015.250.6, 00000000012.219.6, 00000000003.414.6, 00000000001.244.6, 00000000012.303.6, 00000000019.386.6, 00000000006.277.6] 2024-08-18T16:41:12.013 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000009.393.6, 00000000006.266.6, 00000000006.382.6, 00000000015.250.6, 00000000012.219.6, 00000000003.414.6, 00000000003.268.6, 00000000012.303.6, 00000000021.392.6, 00000000018.303.6] 2024-08-18T16:41:17.014 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:41:17.013+0000 7fbf80830880 1 waiting for incremental sync to catch up: 2024-08-18T16:41:17.015 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000009.393.6, 00000000006.266.6, 00000000002.370.6, 00000000015.250.6, 00000000012.219.6, 00000000003.414.6, 00000000001.244.6, 00000000012.303.6, 00000000019.386.6, 00000000006.277.6] 2024-08-18T16:41:17.015 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000009.393.6, 00000000006.266.6, 00000000006.382.6, 00000000015.250.6, 00000000012.219.6, 00000000003.414.6, 00000000003.268.6, 00000000012.303.6, 00000000021.392.6, 00000000018.303.6] 2024-08-18T16:41:22.015 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:41:22.014+0000 7fbf80830880 1 waiting for incremental sync to catch up: 2024-08-18T16:41:22.015 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000009.393.6, 00000000006.266.6, 00000000002.370.6, 00000000015.250.6, 00000000012.219.6, 00000000003.414.6, 00000000001.244.6, 00000000012.303.6, 00000000019.386.6, 00000000006.277.6] 2024-08-18T16:41:22.015 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000009.393.6, 00000000006.266.6, 00000000006.382.6, 00000000015.250.6, 00000000012.219.6, 00000000003.414.6, 00000000003.268.6, 00000000012.303.6, 00000000021.392.6, 00000000018.303.6] 2024-08-18T16:41:27.017 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:41:27.015+0000 7fbf80830880 1 waiting for incremental sync to catch up: 2024-08-18T16:41:27.017 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000009.393.6, 00000000006.266.6, 00000000002.370.6, 00000000015.250.6, 00000000012.219.6, 00000000003.414.6, 00000000001.244.6, 00000000012.303.6, 00000000019.386.6, 00000000006.277.6] 2024-08-18T16:41:27.017 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000009.393.6, 00000000006.266.6, 00000000006.382.6, 00000000015.250.6, 00000000012.219.6, 00000000003.414.6, 00000000003.268.6, 00000000012.303.6, 00000000021.392.6, 00000000018.303.6] 2024-08-18T16:41:32.017 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:41:32.016+0000 7fbf80830880 1 waiting for incremental sync to catch up: 2024-08-18T16:41:32.018 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000009.393.6, 00000000006.266.6, 00000000002.370.6, 00000000015.250.6, 00000000012.219.6, 00000000003.414.6, 00000000001.244.6, 00000000012.303.6, 00000000019.386.6, 00000000006.277.6] 2024-08-18T16:41:32.018 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000009.393.6, 00000000006.266.6, 00000000006.382.6, 00000000015.250.6, 00000000012.219.6, 00000000003.414.6, 00000000003.268.6, 00000000012.303.6, 00000000021.392.6, 00000000018.303.6] 2024-08-18T16:41:37.017 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:41:37.016+0000 7fbf80830880 1 waiting for incremental sync to catch up: 2024-08-18T16:41:37.054 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000009.393.6, 00000000006.266.6, 00000000006.382.6, 00000000015.250.6, 00000000012.219.6, 00000000003.414.6, 00000000001.244.6, 00000000012.303.6, 00000000019.386.6, 00000000015.298.6] 2024-08-18T16:41:37.054 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000009.393.6, 00000000006.266.6, 00000000006.382.6, 00000000015.250.6, 00000000012.219.6, 00000000003.414.6, 00000000003.268.6, 00000000012.303.6, 00000000021.392.6, 00000000018.303.6] 2024-08-18T16:41:42.019 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:41:42.018+0000 7fbf80830880 1 waiting for incremental sync to catch up: 2024-08-18T16:41:42.019 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000009.393.6, 00000000006.266.6, 00000000006.382.6, 00000000015.250.6, 00000000012.219.6, 00000000003.414.6, 00000000001.244.6, 00000000012.303.6, 00000000019.386.6, 00000000015.298.6] 2024-08-18T16:41:42.019 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000009.393.6, 00000000006.266.6, 00000000006.382.6, 00000000015.250.6, 00000000012.219.6, 00000000003.414.6, 00000000003.268.6, 00000000012.303.6, 00000000021.392.6, 00000000018.303.6] 2024-08-18T16:41:47.020 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:41:47.019+0000 7fbf80830880 1 waiting for incremental sync to catch up: 2024-08-18T16:41:47.021 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000009.393.6, 00000000006.266.6, 00000000006.382.6, 00000000015.250.6, 00000000012.219.6, 00000000003.414.6, 00000000001.244.6, 00000000012.303.6, 00000000019.386.6, 00000000015.298.6] 2024-08-18T16:41:47.021 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000009.393.6, 00000000006.266.6, 00000000006.382.6, 00000000015.250.6, 00000000012.219.6, 00000000003.414.6, 00000000003.268.6, 00000000012.303.6, 00000000021.392.6, 00000000018.303.6] 2024-08-18T16:41:52.022 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:41:52.020+0000 7fbf80830880 1 waiting for incremental sync to catch up: 2024-08-18T16:41:52.022 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000009.393.6, 00000000006.266.6, 00000000006.382.6, 00000000015.250.6, 00000000012.219.6, 00000000003.414.6, 00000000001.244.6, 00000000012.303.6, 00000000019.386.6, 00000000015.298.6] 2024-08-18T16:41:52.022 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000009.393.6, 00000000006.266.6, 00000000006.382.6, 00000000015.250.6, 00000000012.219.6, 00000000003.414.6, 00000000003.268.6, 00000000012.303.6, 00000000021.392.6, 00000000018.303.6] 2024-08-18T16:41:57.023 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:41:57.022+0000 7fbf80830880 1 waiting for incremental sync to catch up: 2024-08-18T16:41:57.023 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000009.393.6, 00000000006.266.6, 00000000006.382.6, 00000000015.250.6, 00000000012.219.6, 00000000003.414.6, 00000000001.244.6, 00000000012.303.6, 00000000019.386.6, 00000000015.298.6] 2024-08-18T16:41:57.023 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000009.393.6, 00000000006.266.6, 00000000006.382.6, 00000000015.250.6, 00000000012.219.6, 00000000003.414.6, 00000000003.268.6, 00000000012.303.6, 00000000021.392.6, 00000000018.303.6] 2024-08-18T16:42:02.024 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:42:02.023+0000 7fbf80830880 1 waiting for incremental sync to catch up: 2024-08-18T16:42:02.024 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000009.393.6, 00000000006.266.6, 00000000006.382.6, 00000000015.250.6, 00000000012.219.6, 00000000003.414.6, 00000000001.244.6, 00000000012.303.6, 00000000019.386.6, 00000000015.298.6] 2024-08-18T16:42:02.025 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000009.393.6, 00000000006.266.6, 00000000006.382.6, 00000000015.250.6, 00000000012.219.6, 00000000003.414.6, 00000000003.268.6, 00000000012.303.6, 00000000021.392.6, 00000000018.303.6] 2024-08-18T16:42:07.025 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:42:07.024+0000 7fbf80830880 1 waiting for incremental sync to catch up: 2024-08-18T16:42:07.026 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000009.393.6, 00000000006.266.6, 00000000006.382.6, 00000000015.250.6, 00000000012.219.6, 00000000003.414.6, 00000000001.244.6, 00000000012.303.6, 00000000019.386.6, 00000000015.298.6] 2024-08-18T16:42:07.026 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000009.393.6, 00000000006.266.6, 00000000006.382.6, 00000000015.250.6, 00000000012.219.6, 00000000003.414.6, 00000000003.268.6, 00000000012.303.6, 00000000021.392.6, 00000000018.303.6] 2024-08-18T16:42:12.026 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:42:12.025+0000 7fbf80830880 1 waiting for incremental sync to catch up: 2024-08-18T16:42:12.027 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000009.393.6, 00000000006.266.6, 00000000006.382.6, 00000000015.250.6, 00000000012.219.6, 00000000003.414.6, 00000000001.244.6, 00000000012.303.6, 00000000019.386.6, 00000000015.298.6] 2024-08-18T16:42:12.027 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000009.393.6, 00000000006.266.6, 00000000006.382.6, 00000000015.250.6, 00000000012.219.6, 00000000003.414.6, 00000000003.268.6, 00000000012.303.6, 00000000021.392.6, 00000000018.303.6] 2024-08-18T16:42:17.028 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:42:17.027+0000 7fbf80830880 1 waiting for incremental sync to catch up: 2024-08-18T16:42:17.028 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000009.393.6, 00000000006.266.6, 00000000006.382.6, 00000000015.250.6, 00000000012.219.6, 00000000003.414.6, 00000000001.244.6, 00000000012.303.6, 00000000021.392.6, 00000000015.298.6] 2024-08-18T16:42:17.028 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000009.393.6, 00000000006.266.6, 00000000006.382.6, 00000000015.250.6, 00000000012.219.6, 00000000003.414.6, 00000000003.268.6, 00000000012.303.6, 00000000021.392.6, 00000000018.303.6] 2024-08-18T16:42:22.029 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:42:22.028+0000 7fbf80830880 1 waiting for incremental sync to catch up: 2024-08-18T16:42:22.029 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000009.393.6, 00000000006.266.6, 00000000006.382.6, 00000000015.250.6, 00000000012.219.6, 00000000003.414.6, 00000000001.244.6, 00000000012.303.6, 00000000021.392.6, 00000000015.298.6] 2024-08-18T16:42:22.029 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000009.393.6, 00000000006.266.6, 00000000006.382.6, 00000000015.250.6, 00000000012.219.6, 00000000003.414.6, 00000000003.268.6, 00000000012.303.6, 00000000021.392.6, 00000000018.303.6] 2024-08-18T16:42:27.030 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:42:27.029+0000 7fbf80830880 1 waiting for incremental sync to catch up: 2024-08-18T16:42:27.031 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000009.393.6, 00000000006.266.6, 00000000006.382.6, 00000000015.250.6, 00000000012.219.6, 00000000003.414.6, 00000000001.244.6, 00000000012.303.6, 00000000021.392.6, 00000000015.298.6] 2024-08-18T16:42:27.031 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000009.393.6, 00000000006.266.6, 00000000006.382.6, 00000000015.250.6, 00000000012.219.6, 00000000003.414.6, 00000000003.268.6, 00000000012.303.6, 00000000021.392.6, 00000000018.303.6] 2024-08-18T16:42:32.031 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:42:32.030+0000 7fbf80830880 1 waiting for incremental sync to catch up: 2024-08-18T16:42:32.031 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000009.393.6, 00000000006.266.6, 00000000006.382.6, 00000000015.250.6, 00000000012.219.6, 00000000003.414.6, 00000000001.244.6, 00000000012.303.6, 00000000021.392.6, 00000000015.298.6] 2024-08-18T16:42:32.032 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000009.393.6, 00000000006.266.6, 00000000006.382.6, 00000000015.250.6, 00000000012.219.6, 00000000003.414.6, 00000000003.268.6, 00000000012.303.6, 00000000021.392.6, 00000000018.303.6] 2024-08-18T16:42:37.032 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:42:37.031+0000 7fbf80830880 1 bucket sync caught up with source: 2024-08-18T16:42:37.033 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000009.393.6, 00000000006.266.6, 00000000006.382.6, 00000000015.250.6, 00000000012.219.6, 00000000003.414.6, 00000000003.268.6, 00000000012.303.6, 00000000021.392.6, 00000000018.303.6] 2024-08-18T16:42:37.033 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000009.393.6, 00000000006.266.6, 00000000006.382.6, 00000000015.250.6, 00000000012.219.6, 00000000003.414.6, 00000000003.268.6, 00000000012.303.6, 00000000021.392.6, 00000000018.303.6] 2024-08-18T16:42:37.033 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:42:37.031+0000 7fbf80830880 0 bucket checkpoint complete 2024-08-18T16:42:37.048 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:42:37.048 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:42:37.048 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-78 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:42:37.103 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:42:37.103 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:42:37.138 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:42:37.137+0000 7f556d010880 1 bucket sync caught up with source: 2024-08-18T16:42:37.138 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, 00000000009.275.6, 00000000006.526.6, 00000000006.270.6, 00000000015.314.6, 00000000012.172.6, 00000000003.540.6, 00000000003.528.6, 00000000012.213.6, 00000000021.274.6, 00000000018.185.6] 2024-08-18T16:42:37.138 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, 00000000009.275.6, 00000000006.526.6, 00000000006.270.6, 00000000015.314.6, 00000000012.172.6, 00000000003.540.6, 00000000003.528.6, 00000000012.213.6, 00000000021.274.6, 00000000018.185.6] 2024-08-18T16:42:37.138 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:42:37.137+0000 7f556d010880 0 bucket checkpoint complete 2024-08-18T16:42:37.152 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:42:37.152 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-78', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:42:37.152 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-78 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T16:42:37.214 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:42:37.214 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:42:37.254 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:42:37.254+0000 7f1fac320880 1 bucket sync caught up with source: 2024-08-18T16:42:37.254 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, 00000000009.275.6, 00000000006.526.6, 00000000006.270.6, 00000000015.314.6, 00000000012.172.6, 00000000003.540.6, 00000000003.528.6, 00000000012.213.6, 00000000021.274.6, 00000000018.185.6] 2024-08-18T16:42:37.255 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, 00000000009.275.6, 00000000006.526.6, 00000000006.270.6, 00000000015.314.6, 00000000012.172.6, 00000000003.540.6, 00000000003.528.6, 00000000012.213.6, 00000000021.274.6, 00000000018.185.6] 2024-08-18T16:42:37.255 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:42:37.254+0000 7f1fac320880 0 bucket checkpoint complete 2024-08-18T16:42:37.270 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-78 zones={test-zone2, test-zone1} 2024-08-18T16:42:37.282 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:42:37.287 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-08-18T16:42:37.287 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2024-08-18T16:42:37.287 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2024-08-18T16:42:37.287 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2024-08-18T16:42:37.287 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2024-08-18T16:42:37.287 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2024-08-18T16:42:37.287 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2024-08-18T16:42:37.287 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2024-08-18T16:42:37.287 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-08-18T16:42:37.287 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2024-08-18T16:42:37.288 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-08-18T16:42:37.288 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-08-18T16:42:37.288 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2024-08-18T16:42:37.288 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2024-08-18T16:42:37.288 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2024-08-18T16:42:37.288 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-08-18T16:42:37.288 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-08-18T16:42:37.288 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2024-08-18T16:42:37.288 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-08-18T16:42:37.288 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-08-18T16:42:37.288 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2024-08-18T16:42:37.288 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-08-18T16:42:37.288 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-08-18T16:42:37.288 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-08-18T16:42:37.288 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2024-08-18T16:42:37.289 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2024-08-18T16:42:37.289 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-08-18T16:42:37.289 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2024-08-18T16:42:37.289 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2024-08-18T16:42:37.289 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-08-18T16:42:37.289 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-08-18T16:42:37.289 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2024-08-18T16:42:37.289 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2024-08-18T16:42:37.289 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2024-08-18T16:42:37.289 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-08-18T16:42:37.289 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:42:37.295 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-08-18T16:42:37.296 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2024-08-18T16:42:37.296 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2024-08-18T16:42:37.296 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2024-08-18T16:42:37.296 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2024-08-18T16:42:37.296 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2024-08-18T16:42:37.296 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2024-08-18T16:42:37.296 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2024-08-18T16:42:37.296 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-08-18T16:42:37.296 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2024-08-18T16:42:37.296 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-08-18T16:42:37.296 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-08-18T16:42:37.296 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2024-08-18T16:42:37.296 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2024-08-18T16:42:37.296 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2024-08-18T16:42:37.297 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-08-18T16:42:37.297 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-08-18T16:42:37.297 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2024-08-18T16:42:37.297 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-08-18T16:42:37.297 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-08-18T16:42:37.297 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2024-08-18T16:42:37.297 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-08-18T16:42:37.297 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-08-18T16:42:37.297 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-08-18T16:42:37.297 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2024-08-18T16:42:37.297 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2024-08-18T16:42:37.297 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-08-18T16:42:37.297 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2024-08-18T16:42:37.297 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2024-08-18T16:42:37.297 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-08-18T16:42:37.297 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-08-18T16:42:37.298 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2024-08-18T16:42:37.298 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2024-08-18T16:42:37.298 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2024-08-18T16:42:37.298 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-08-18T16:42:37.306 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T16:42:37.313 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T16:42:37.372 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T16:42:37.381 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2024-08-18T16:42:37.430 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2024-08-18T16:42:37.441 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2024-08-18T16:42:37.494 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-08-18T16:42:37.501 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-08-18T16:42:37.513 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-08-18T16:42:37.566 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-08-18T16:42:37.573 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-08-18T16:42:37.674 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-08-18T16:42:37.683 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2024-08-18T16:42:37.690 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2024-08-18T16:42:37.744 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2024-08-18T16:42:37.795 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-08-18T16:42:37.801 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-08-18T16:42:37.858 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-08-18T16:42:37.868 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2024-08-18T16:42:37.918 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2024-08-18T16:42:37.930 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2024-08-18T16:42:37.982 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2024-08-18T16:42:37.989 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2024-08-18T16:42:38.001 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2024-08-18T16:42:38.055 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-08-18T16:42:38.061 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-08-18T16:42:38.118 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-08-18T16:42:38.127 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2024-08-18T16:42:38.181 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2024-08-18T16:42:38.194 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2024-08-18T16:42:38.246 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-08-18T16:42:38.294 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-08-18T16:42:38.307 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-08-18T16:42:38.358 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T16:42:38.365 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T16:42:38.422 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T16:42:38.472 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-08-18T16:42:38.479 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-08-18T16:42:38.534 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-08-18T16:42:38.584 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-08-18T16:42:38.591 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-08-18T16:42:38.645 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-08-18T16:42:38.699 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2024-08-18T16:42:38.751 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2024-08-18T16:42:38.806 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2024-08-18T16:42:38.815 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T16:42:38.867 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T16:42:38.880 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T16:42:38.932 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-08-18T16:42:38.939 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-08-18T16:42:38.996 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-08-18T16:42:39.005 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2024-08-18T16:42:39.054 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2024-08-18T16:42:39.066 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2024-08-18T16:42:39.122 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-08-18T16:42:39.129 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-08-18T16:42:39.186 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-08-18T16:42:39.195 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T16:42:39.246 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T16:42:39.261 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T16:42:39.314 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-08-18T16:42:39.322 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-08-18T16:42:39.334 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-08-18T16:42:39.390 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-08-18T16:42:39.397 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-08-18T16:42:39.410 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-08-18T16:42:39.464 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T16:42:39.514 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T16:42:39.526 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T16:42:39.578 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-08-18T16:42:39.585 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-08-18T16:42:39.637 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-08-18T16:42:39.645 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-08-18T16:42:39.652 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-08-18T16:42:39.705 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-08-18T16:42:39.714 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2024-08-18T16:42:39.764 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2024-08-18T16:42:39.818 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2024-08-18T16:42:39.827 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-08-18T16:42:39.834 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-08-18T16:42:39.888 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-08-18T16:42:39.939 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-08-18T16:42:39.946 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-08-18T16:42:39.958 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-08-18T16:42:40.014 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2024-08-18T16:42:40.021 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2024-08-18T16:42:40.073 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2024-08-18T16:42:40.127 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-08-18T16:42:40.134 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-08-18T16:42:40.146 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-08-18T16:42:40.197 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-08-18T16:42:40.203 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-08-18T16:42:40.260 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-08-18T16:42:40.310 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-08-18T16:42:40.318 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-08-18T16:42:40.330 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-08-18T16:42:40.382 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-08-18T16:42:40.388 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-08-18T16:42:40.442 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-08-18T16:42:40.451 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2024-08-18T16:42:40.499 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2024-08-18T16:42:40.511 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2024-08-18T16:42:40.561 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-08-18T16:42:40.568 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-08-18T16:42:40.622 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-08-18T16:42:40.633 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-78 zones={test-zone2, test-zone1} 2024-08-18T16:42:40.633 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-78 zones={test-zone3, test-zone1} 2024-08-18T16:42:40.641 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:42:40.645 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-08-18T16:42:40.645 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2024-08-18T16:42:40.645 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2024-08-18T16:42:40.645 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2024-08-18T16:42:40.645 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2024-08-18T16:42:40.645 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2024-08-18T16:42:40.645 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2024-08-18T16:42:40.645 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2024-08-18T16:42:40.646 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-08-18T16:42:40.646 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2024-08-18T16:42:40.646 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-08-18T16:42:40.646 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-08-18T16:42:40.646 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2024-08-18T16:42:40.646 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2024-08-18T16:42:40.646 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2024-08-18T16:42:40.646 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-08-18T16:42:40.646 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-08-18T16:42:40.646 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2024-08-18T16:42:40.646 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-08-18T16:42:40.647 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-08-18T16:42:40.647 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2024-08-18T16:42:40.647 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-08-18T16:42:40.647 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-08-18T16:42:40.647 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-08-18T16:42:40.647 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2024-08-18T16:42:40.647 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2024-08-18T16:42:40.647 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-08-18T16:42:40.647 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2024-08-18T16:42:40.647 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2024-08-18T16:42:40.647 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-08-18T16:42:40.647 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-08-18T16:42:40.647 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2024-08-18T16:42:40.647 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2024-08-18T16:42:40.647 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2024-08-18T16:42:40.648 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-08-18T16:42:40.648 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:42:40.653 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-08-18T16:42:40.653 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2024-08-18T16:42:40.653 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2024-08-18T16:42:40.653 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2024-08-18T16:42:40.653 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2024-08-18T16:42:40.653 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2024-08-18T16:42:40.653 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2024-08-18T16:42:40.653 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2024-08-18T16:42:40.653 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-08-18T16:42:40.653 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2024-08-18T16:42:40.653 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-08-18T16:42:40.654 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-08-18T16:42:40.654 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2024-08-18T16:42:40.654 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2024-08-18T16:42:40.654 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2024-08-18T16:42:40.654 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-08-18T16:42:40.654 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-08-18T16:42:40.654 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2024-08-18T16:42:40.654 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-08-18T16:42:40.654 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-08-18T16:42:40.654 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2024-08-18T16:42:40.654 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-08-18T16:42:40.654 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-08-18T16:42:40.654 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-08-18T16:42:40.654 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2024-08-18T16:42:40.655 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2024-08-18T16:42:40.655 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-08-18T16:42:40.655 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2024-08-18T16:42:40.655 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2024-08-18T16:42:40.655 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-08-18T16:42:40.655 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-08-18T16:42:40.655 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2024-08-18T16:42:40.655 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2024-08-18T16:42:40.655 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2024-08-18T16:42:40.655 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-08-18T16:42:40.664 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T16:42:40.714 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T16:42:40.727 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T16:42:40.777 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2024-08-18T16:42:40.783 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2024-08-18T16:42:40.837 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2024-08-18T16:42:40.845 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-08-18T16:42:40.852 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-08-18T16:42:40.904 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-08-18T16:42:40.913 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-08-18T16:42:40.920 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-08-18T16:42:40.973 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-08-18T16:42:41.022 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2024-08-18T16:42:41.028 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2024-08-18T16:42:41.040 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2024-08-18T16:42:41.090 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-08-18T16:42:41.096 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-08-18T16:42:41.149 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-08-18T16:42:41.158 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2024-08-18T16:42:41.207 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2024-08-18T16:42:41.219 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2024-08-18T16:42:41.268 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2024-08-18T16:42:41.275 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2024-08-18T16:42:41.331 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2024-08-18T16:42:41.380 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-08-18T16:42:41.387 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-08-18T16:42:41.440 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-08-18T16:42:41.491 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2024-08-18T16:42:41.501 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2024-08-18T16:42:41.553 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2024-08-18T16:42:41.603 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-08-18T16:42:41.650 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-08-18T16:42:41.661 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-08-18T16:42:41.714 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T16:42:41.720 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T16:42:41.777 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T16:42:41.785 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-08-18T16:42:41.792 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-08-18T16:42:41.845 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-08-18T16:42:41.895 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-08-18T16:42:41.901 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-08-18T16:42:41.916 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-08-18T16:42:41.966 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2024-08-18T16:42:41.972 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2024-08-18T16:42:42.026 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2024-08-18T16:42:42.034 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T16:42:42.088 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T16:42:42.099 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T16:42:42.153 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-08-18T16:42:42.159 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-08-18T16:42:42.211 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-08-18T16:42:42.263 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2024-08-18T16:42:42.270 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2024-08-18T16:42:42.281 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2024-08-18T16:42:42.334 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-08-18T16:42:42.341 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-08-18T16:42:42.392 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-08-18T16:42:42.401 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T16:42:42.407 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T16:42:42.461 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T16:42:42.470 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-08-18T16:42:42.476 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-08-18T16:42:42.532 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-08-18T16:42:42.585 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-08-18T16:42:42.591 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-08-18T16:42:42.644 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-08-18T16:42:42.695 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T16:42:42.701 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T16:42:42.713 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T16:42:42.767 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-08-18T16:42:42.818 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-08-18T16:42:42.829 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-08-18T16:42:42.882 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-08-18T16:42:42.889 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-08-18T16:42:42.945 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-08-18T16:42:42.955 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2024-08-18T16:42:43.004 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2024-08-18T16:42:43.058 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2024-08-18T16:42:43.067 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-08-18T16:42:43.074 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-08-18T16:42:43.132 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-08-18T16:42:43.183 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-08-18T16:42:43.189 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-08-18T16:42:43.201 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-08-18T16:42:43.254 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2024-08-18T16:42:43.260 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2024-08-18T16:42:43.273 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2024-08-18T16:42:43.326 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-08-18T16:42:43.332 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-08-18T16:42:43.385 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-08-18T16:42:43.434 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-08-18T16:42:43.441 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-08-18T16:42:43.452 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-08-18T16:42:43.502 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-08-18T16:42:43.508 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-08-18T16:42:43.562 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-08-18T16:42:43.572 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-08-18T16:42:43.578 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-08-18T16:42:43.631 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-08-18T16:42:43.680 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2024-08-18T16:42:43.687 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2024-08-18T16:42:43.739 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2024-08-18T16:42:43.795 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-08-18T16:42:43.801 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-08-18T16:42:43.813 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-08-18T16:42:43.950 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-78 zones={test-zone3, test-zone1} 2024-08-18T16:42:43.950 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-78 zones={test-zone3, test-zone2} 2024-08-18T16:42:43.955 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:42:43.970 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-08-18T16:42:43.970 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2024-08-18T16:42:43.970 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2024-08-18T16:42:43.970 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2024-08-18T16:42:43.971 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2024-08-18T16:42:43.971 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2024-08-18T16:42:43.971 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2024-08-18T16:42:43.971 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2024-08-18T16:42:43.971 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-08-18T16:42:43.971 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2024-08-18T16:42:43.971 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-08-18T16:42:43.971 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-08-18T16:42:43.971 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2024-08-18T16:42:43.971 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2024-08-18T16:42:43.972 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2024-08-18T16:42:43.972 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-08-18T16:42:43.972 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-08-18T16:42:43.972 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2024-08-18T16:42:43.972 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-08-18T16:42:43.972 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-08-18T16:42:43.972 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2024-08-18T16:42:43.972 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-08-18T16:42:43.972 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-08-18T16:42:43.972 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-08-18T16:42:43.972 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2024-08-18T16:42:43.972 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2024-08-18T16:42:43.972 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-08-18T16:42:43.972 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2024-08-18T16:42:43.973 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2024-08-18T16:42:43.973 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-08-18T16:42:43.973 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-08-18T16:42:43.973 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2024-08-18T16:42:43.973 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2024-08-18T16:42:43.973 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2024-08-18T16:42:43.973 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-08-18T16:42:43.973 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:42:43.981 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-08-18T16:42:43.981 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2024-08-18T16:42:43.981 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2024-08-18T16:42:43.981 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2024-08-18T16:42:43.981 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2024-08-18T16:42:43.981 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2024-08-18T16:42:43.981 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2024-08-18T16:42:43.981 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2024-08-18T16:42:43.981 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-08-18T16:42:43.982 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2024-08-18T16:42:43.982 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-08-18T16:42:43.982 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-08-18T16:42:43.982 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2024-08-18T16:42:43.982 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2024-08-18T16:42:43.982 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2024-08-18T16:42:43.982 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-08-18T16:42:43.982 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-08-18T16:42:43.982 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2024-08-18T16:42:43.982 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-08-18T16:42:43.982 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-08-18T16:42:43.982 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2024-08-18T16:42:43.982 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-08-18T16:42:43.982 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-08-18T16:42:43.983 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-08-18T16:42:43.983 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2024-08-18T16:42:43.983 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2024-08-18T16:42:43.983 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-08-18T16:42:43.983 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2024-08-18T16:42:43.983 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2024-08-18T16:42:43.983 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-08-18T16:42:43.983 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-08-18T16:42:43.983 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2024-08-18T16:42:43.983 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2024-08-18T16:42:43.983 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2024-08-18T16:42:43.983 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-08-18T16:42:43.995 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T16:42:44.086 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T16:42:44.104 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T16:42:44.159 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2024-08-18T16:42:44.209 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2024-08-18T16:42:44.265 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2024-08-18T16:42:44.317 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-08-18T16:42:44.324 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-08-18T16:42:44.377 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-08-18T16:42:44.430 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-08-18T16:42:44.437 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-08-18T16:42:44.489 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-08-18T16:42:44.541 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2024-08-18T16:42:44.548 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2024-08-18T16:42:44.600 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2024-08-18T16:42:44.649 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-08-18T16:42:44.656 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-08-18T16:42:44.708 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-08-18T16:42:44.716 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2024-08-18T16:42:44.766 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2024-08-18T16:42:44.778 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2024-08-18T16:42:44.786 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2024-08-18T16:42:44.836 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2024-08-18T16:42:44.890 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2024-08-18T16:42:44.899 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-08-18T16:42:44.906 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-08-18T16:42:44.959 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-08-18T16:42:45.008 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2024-08-18T16:42:45.017 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2024-08-18T16:42:45.072 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2024-08-18T16:42:45.125 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-08-18T16:42:45.132 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-08-18T16:42:45.184 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-08-18T16:42:45.233 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T16:42:45.240 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T16:42:45.293 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T16:42:45.342 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-08-18T16:42:45.349 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-08-18T16:42:45.400 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-08-18T16:42:45.451 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-08-18T16:42:45.458 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-08-18T16:42:45.470 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-08-18T16:42:45.519 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2024-08-18T16:42:45.527 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2024-08-18T16:42:45.582 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2024-08-18T16:42:45.631 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T16:42:45.639 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T16:42:45.691 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T16:42:45.743 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-08-18T16:42:45.750 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-08-18T16:42:45.803 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-08-18T16:42:45.852 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2024-08-18T16:42:45.858 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2024-08-18T16:42:45.911 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2024-08-18T16:42:45.959 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-08-18T16:42:45.966 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-08-18T16:42:46.018 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-08-18T16:42:46.027 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T16:42:46.078 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T16:42:46.089 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T16:42:46.141 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-08-18T16:42:46.148 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-08-18T16:42:46.201 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-08-18T16:42:46.253 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-08-18T16:42:46.261 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-08-18T16:42:46.312 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-08-18T16:42:46.361 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T16:42:46.368 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T16:42:46.421 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T16:42:46.474 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-08-18T16:42:46.481 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-08-18T16:42:46.536 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-08-18T16:42:46.545 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-08-18T16:42:46.552 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-08-18T16:42:46.604 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-08-18T16:42:46.653 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2024-08-18T16:42:46.660 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2024-08-18T16:42:46.712 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2024-08-18T16:42:46.720 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-08-18T16:42:46.770 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-08-18T16:42:46.783 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-08-18T16:42:46.835 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-08-18T16:42:46.842 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-08-18T16:42:46.895 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-08-18T16:42:46.944 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2024-08-18T16:42:46.952 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2024-08-18T16:42:47.005 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2024-08-18T16:42:47.053 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-08-18T16:42:47.060 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-08-18T16:42:47.113 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-08-18T16:42:47.165 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-08-18T16:42:47.172 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-08-18T16:42:47.224 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-08-18T16:42:47.274 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-08-18T16:42:47.282 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-08-18T16:42:47.336 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-08-18T16:42:47.344 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-08-18T16:42:47.350 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-08-18T16:42:47.403 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-08-18T16:42:47.455 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2024-08-18T16:42:47.463 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2024-08-18T16:42:47.515 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2024-08-18T16:42:47.568 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-08-18T16:42:47.574 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-08-18T16:42:47.626 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-08-18T16:42:47.634 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-78 zones={test-zone3, test-zone2} 2024-08-18T16:42:47.635 INFO:tasks.rgw_multisite_tests:Create several generations of objects, trash them, then run bucket sync init ... ok 2024-08-18T16:42:47.635 INFO:rgw_multi.tests:create role zone=test-zone1 name=roles-kibgmb-1 2024-08-18T16:42:47.647 INFO:rgw_multi.tests:create role zone=test-zone2 name=roles-kibgmb-2 2024-08-18T16:42:47.665 INFO:rgw_multi.tests:create role zone=test-zone3 name=roles-kibgmb-3 2024-08-18T16:42:47.681 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:42:47.681 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:42:47.681 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:42:47.730 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:42:47.731 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:42:47.761 INFO:teuthology.orchestra.run.smithi134.stderr:No --period given, using current period=01050e30-179b-43be-bdbb-3bb23cf37ae8 2024-08-18T16:42:47.777 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T16:42:47.777 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:42:47.777 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T16:42:47.777 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T16:42:47.777 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:42:47.777 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:42:47.777 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T16:42:47.778 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T16:42:47.778 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:42:47.778 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:42:47.778 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T16:42:47.778 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:42:47.672461Z" 2024-08-18T16:42:47.778 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:42:47.778 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:42:47.778 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723999367.643367_1116.1", 2024-08-18T16:42:47.778 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:42:47.643367Z" 2024-08-18T16:42:47.778 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:42:47.778 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T16:42:47.779 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-08-18T16:42:47.780 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:42:47.780 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:42:47.780 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:42:47.833 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:42:47.834 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:42:47.866 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:42:47.866+0000 7f47e1136880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:42:47.868 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:42:47.867+0000 7f47e1136880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:42:47.882 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T16:42:47.882 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T16:42:47.883 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T16:42:47.883 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T16:42:47.883 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T16:42:47.883 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T16:42:47.883 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T16:42:47.883 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:42:47.883 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T16:42:47.883 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:42:47.883 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T16:42:47.883 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:42:47.883 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:42:47.883 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T16:42:47.884 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:42:47.884 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T16:42:47.884 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:42:47.884 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T16:42:47.884 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T16:42:47.884 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:42:47.884 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:42:47.884 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:42:47.884 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T16:42:47.884 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:42:47.884 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:42:47.885 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T16:42:47.885 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:42:47.885 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:42:47.885 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:42:47.885 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T16:42:47.885 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T16:42:47.885 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:42:47.885 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:42:47.885 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:42:47.885 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T16:42:47.885 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:42:47.885 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:42:47.885 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T16:42:47.886 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:42:47.886 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:42:47.886 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:42:47.886 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T16:42:47.886 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T16:42:47.886 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:42:47.886 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:42:47.886 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:42:47.886 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T16:42:47.886 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:42:47.886 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:42:47.886 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.643367_1116.1", 2024-08-18T16:42:47.886 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:42:47.887 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:42:47.887 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:42:47.887 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.643367Z", 2024-08-18T16:42:47.887 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T16:42:47.887 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:42:47.887 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:42:47.887 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T16:42:47.887 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:42:47.887 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T16:42:47.887 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T16:42:47.887 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T16:42:47.887 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:42:47.887 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T16:42:47Z" 2024-08-18T16:42:47.887 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T16:42:47.888 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-08-18T16:42:47.889 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-08-18T16:42:47.889 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:42:47.889 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:42:47.889 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:42:47.936 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:42:47.936 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:42:47.965 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:42:47.964+0000 7f41ae034880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:42:47.966 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:42:47.965+0000 7f41ae034880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:42:47.979 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T16:42:47.979 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T16:42:47.979 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T16:42:47.979 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T16:42:47.979 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T16:42:47.979 INFO:teuthology.orchestra.run.smithi134.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T16:42:47.979 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T16:42:47.979 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:42:47.980 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T16:42:47.980 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:42:47.980 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T16:42:47.980 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:42:47.980 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T16:42:47.980 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T16:42:47.980 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:42:47.980 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 1, 2024-08-18T16:42:47.980 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:42:47.980 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T16:42:47.980 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T16:42:47.980 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:42:47.980 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:42:47.980 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:42:47.980 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T16:42:47.981 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:42:47.981 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T16:42:47.981 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T16:42:47.981 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:42:47.981 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:42:47.981 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:42:47.981 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T16:42:47.981 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T16:42:47.981 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:42:47.981 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:42:47.981 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:42:47.981 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T16:42:47.981 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:42:47.982 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T16:42:47.982 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T16:42:47.982 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:42:47.982 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:42:47.982 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:42:47.982 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T16:42:47.982 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T16:42:47.982 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:42:47.982 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:42:47.982 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:42:47.982 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T16:42:47.982 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:42:47.982 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T16:42:47.982 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723998965.073459_1065.1", 2024-08-18T16:42:47.983 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:42:47.983 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:42:47.983 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:42:47.983 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:36:05.073459Z", 2024-08-18T16:42:47.983 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T16:42:47.983 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:42:47.983 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:42:47.983 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T16:42:47.983 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:42:47.983 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T16:42:47.983 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 1, 2024-08-18T16:42:47.983 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 1 2024-08-18T16:42:47.983 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:42:47.983 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T16:42:47Z" 2024-08-18T16:42:47.983 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T16:42:47.984 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1723999367.643367_1116.1 target=1_1723998965.073459_1065.1 2024-08-18T16:42:52.985 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:42:52.985 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:42:52.985 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:42:53.041 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:42:53.041 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:42:53.075 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:42:53.074+0000 7fd5abb24880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:42:53.076 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:42:53.075+0000 7fd5abb24880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:42:53.090 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T16:42:53.091 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T16:42:53.091 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T16:42:53.091 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T16:42:53.091 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T16:42:53.091 INFO:teuthology.orchestra.run.smithi134.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T16:42:53.092 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T16:42:53.092 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:42:53.092 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T16:42:53.092 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:42:53.092 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T16:42:53.092 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:42:53.092 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T16:42:53.092 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T16:42:53.092 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:42:53.093 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 1, 2024-08-18T16:42:53.093 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:42:53.093 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T16:42:53.093 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T16:42:53.093 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:42:53.093 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:42:53.093 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:42:53.093 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T16:42:53.093 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:42:53.094 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T16:42:53.094 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T16:42:53.094 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:42:53.094 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:42:53.094 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:42:53.094 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T16:42:53.095 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T16:42:53.095 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:42:53.095 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:42:53.095 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:42:53.095 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T16:42:53.095 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:42:53.095 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T16:42:53.095 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T16:42:53.095 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:42:53.095 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:42:53.095 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:42:53.095 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T16:42:53.095 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T16:42:53.095 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:42:53.095 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:42:53.096 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:42:53.096 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T16:42:53.096 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:42:53.096 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T16:42:53.096 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723999367.643367_1116.1", 2024-08-18T16:42:53.096 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:42:53.096 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:42:53.096 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:42:53.096 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:42:47.643367Z", 2024-08-18T16:42:53.096 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T16:42:53.096 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:42:53.096 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:42:53.096 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T16:42:53.097 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:42:53.097 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T16:42:53.097 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 1, 2024-08-18T16:42:53.097 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 1 2024-08-18T16:42:53.097 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:42:53.097 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T16:42:53Z" 2024-08-18T16:42:53.097 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T16:42:53.098 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-08-18T16:42:53.098 INFO:tasks.rgw_multi.zone_rados:comparing role=roles-kibgmb-1 zones={test-zone2, test-zone1} 2024-08-18T16:42:53.103 DEBUG:tasks.rgw_multi.zone_rados:comparing role name=roles-kibgmb-1 2024-08-18T16:42:53.104 INFO:tasks.rgw_multi.zone_rados:success, role identical: role=roles-kibgmb-1 zones={test-zone2, test-zone1} 2024-08-18T16:42:53.104 INFO:tasks.rgw_multi.zone_rados:comparing role=roles-kibgmb-1 zones={test-zone3, test-zone1} 2024-08-18T16:42:53.108 DEBUG:tasks.rgw_multi.zone_rados:comparing role name=roles-kibgmb-1 2024-08-18T16:42:53.108 INFO:tasks.rgw_multi.zone_rados:success, role identical: role=roles-kibgmb-1 zones={test-zone3, test-zone1} 2024-08-18T16:42:53.108 INFO:tasks.rgw_multi.zone_rados:comparing role=roles-kibgmb-2 zones={test-zone1, test-zone2} 2024-08-18T16:42:53.112 DEBUG:tasks.rgw_multi.zone_rados:comparing role name=roles-kibgmb-2 2024-08-18T16:42:53.112 INFO:tasks.rgw_multi.zone_rados:success, role identical: role=roles-kibgmb-2 zones={test-zone1, test-zone2} 2024-08-18T16:42:53.112 INFO:tasks.rgw_multi.zone_rados:comparing role=roles-kibgmb-2 zones={test-zone3, test-zone2} 2024-08-18T16:42:53.116 DEBUG:tasks.rgw_multi.zone_rados:comparing role name=roles-kibgmb-2 2024-08-18T16:42:53.116 INFO:tasks.rgw_multi.zone_rados:success, role identical: role=roles-kibgmb-2 zones={test-zone3, test-zone2} 2024-08-18T16:42:53.116 INFO:tasks.rgw_multi.zone_rados:comparing role=roles-kibgmb-3 zones={test-zone1, test-zone3} 2024-08-18T16:42:53.121 DEBUG:tasks.rgw_multi.zone_rados:comparing role name=roles-kibgmb-3 2024-08-18T16:42:53.121 INFO:tasks.rgw_multi.zone_rados:success, role identical: role=roles-kibgmb-3 zones={test-zone1, test-zone3} 2024-08-18T16:42:53.121 INFO:tasks.rgw_multi.zone_rados:comparing role=roles-kibgmb-3 zones={test-zone2, test-zone3} 2024-08-18T16:42:53.125 DEBUG:tasks.rgw_multi.zone_rados:comparing role name=roles-kibgmb-3 2024-08-18T16:42:53.125 INFO:tasks.rgw_multi.zone_rados:success, role identical: role=roles-kibgmb-3 zones={test-zone2, test-zone3} 2024-08-18T16:42:53.125 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_role_sync ... ok 2024-08-18T16:42:53.141 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:42:53.141 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:42:53.141 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:42:53.188 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:42:53.188 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:42:53.223 INFO:teuthology.orchestra.run.smithi134.stderr:No --period given, using current period=01050e30-179b-43be-bdbb-3bb23cf37ae8 2024-08-18T16:42:53.237 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T16:42:53.237 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:42:53.237 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T16:42:53.237 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T16:42:53.237 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:42:53.237 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:42:53.238 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T16:42:53.238 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T16:42:53.238 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:42:53.238 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:42:53.238 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T16:42:53.238 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:42:47.672461Z" 2024-08-18T16:42:53.238 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:42:53.238 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:42:53.238 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723999373.139848_1121.1", 2024-08-18T16:42:53.238 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:42:53.139848Z" 2024-08-18T16:42:53.239 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:42:53.239 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T16:42:53.240 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-08-18T16:42:53.240 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:42:53.240 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:42:53.240 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:42:53.296 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:42:53.296 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:42:53.332 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:42:53.331+0000 7f6d82c3f880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:42:53.333 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:42:53.332+0000 7f6d82c3f880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:42:53.348 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T16:42:53.348 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T16:42:53.348 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T16:42:53.348 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T16:42:53.348 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T16:42:53.348 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T16:42:53.348 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T16:42:53.348 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:42:53.348 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T16:42:53.348 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:42:53.348 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T16:42:53.348 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:42:53.349 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:42:53.349 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T16:42:53.349 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:42:53.349 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T16:42:53.349 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:42:53.349 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T16:42:53.349 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T16:42:53.349 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:42:53.349 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:42:53.349 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:42:53.350 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T16:42:53.350 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:42:53.350 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:42:53.350 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T16:42:53.350 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:42:53.350 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:42:53.350 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:42:53.350 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T16:42:53.350 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T16:42:53.350 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:42:53.350 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:42:53.350 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:42:53.351 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T16:42:53.351 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:42:53.351 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:42:53.351 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T16:42:53.351 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:42:53.351 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:42:53.351 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:42:53.351 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T16:42:53.351 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T16:42:53.351 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:42:53.351 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:42:53.351 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:42:53.351 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T16:42:53.352 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:42:53.352 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:42:53.352 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999373.133509_1118.1", 2024-08-18T16:42:53.352 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:42:53.352 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:42:53.352 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:42:53.352 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:53.133509Z", 2024-08-18T16:42:53.352 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T16:42:53.352 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:42:53.352 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:42:53.352 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T16:42:53.352 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:42:53.352 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T16:42:53.352 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T16:42:53.352 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T16:42:53.353 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:42:53.353 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T16:42:53Z" 2024-08-18T16:42:53.353 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T16:42:53.354 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1723999373.139848_1121.1 target=1_1723999373.133509_1118.1 2024-08-18T16:42:58.354 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:42:58.355 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:42:58.355 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:42:58.410 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:42:58.410 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:42:58.443 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:42:58.442+0000 7fb3fd12d880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:42:58.444 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:42:58.443+0000 7fb3fd12d880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:42:58.459 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T16:42:58.459 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T16:42:58.459 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T16:42:58.459 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T16:42:58.459 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T16:42:58.459 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T16:42:58.459 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T16:42:58.460 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:42:58.460 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T16:42:58.460 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:42:58.460 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T16:42:58.460 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:42:58.460 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:42:58.460 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T16:42:58.460 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:42:58.460 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T16:42:58.460 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:42:58.460 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T16:42:58.460 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T16:42:58.461 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:42:58.461 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:42:58.461 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:42:58.461 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T16:42:58.461 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:42:58.461 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:42:58.461 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T16:42:58.461 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:42:58.461 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:42:58.461 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:42:58.461 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T16:42:58.461 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T16:42:58.461 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:42:58.461 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:42:58.462 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:42:58.462 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T16:42:58.462 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:42:58.462 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:42:58.462 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T16:42:58.462 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:42:58.462 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:42:58.462 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:42:58.462 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T16:42:58.462 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T16:42:58.462 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:42:58.462 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:42:58.463 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:42:58.463 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T16:42:58.463 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:42:58.463 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:42:58.463 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999373.139848_1121.1", 2024-08-18T16:42:58.463 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:42:58.463 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:42:58.463 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:42:58.463 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:53.139848Z", 2024-08-18T16:42:58.463 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T16:42:58.463 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:42:58.463 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:42:58.463 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T16:42:58.465 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:42:58.465 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T16:42:58.466 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T16:42:58.466 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T16:42:58.466 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:42:58.466 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T16:42:58Z" 2024-08-18T16:42:58.466 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T16:42:58.470 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-08-18T16:42:58.473 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-08-18T16:42:58.473 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:42:58.473 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:42:58.473 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:42:58.528 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:42:58.528 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:42:58.561 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:42:58.560+0000 7ff91b329880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:42:58.562 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:42:58.561+0000 7ff91b329880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:42:58.576 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T16:42:58.576 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T16:42:58.576 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T16:42:58.576 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T16:42:58.576 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T16:42:58.576 INFO:teuthology.orchestra.run.smithi134.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T16:42:58.577 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T16:42:58.577 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:42:58.577 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T16:42:58.577 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:42:58.577 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T16:42:58.577 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:42:58.577 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T16:42:58.577 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T16:42:58.577 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:42:58.577 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 1, 2024-08-18T16:42:58.577 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:42:58.577 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T16:42:58.578 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T16:42:58.578 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:42:58.578 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:42:58.578 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:42:58.578 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T16:42:58.578 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:42:58.578 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T16:42:58.578 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T16:42:58.578 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:42:58.578 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:42:58.578 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:42:58.578 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T16:42:58.578 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T16:42:58.579 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:42:58.579 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:42:58.579 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:42:58.579 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T16:42:58.579 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:42:58.579 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T16:42:58.579 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T16:42:58.579 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:42:58.579 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:42:58.579 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:42:58.579 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T16:42:58.579 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T16:42:58.579 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:42:58.580 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:42:58.580 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:42:58.580 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T16:42:58.580 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:42:58.580 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T16:42:58.580 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723999373.139848_1121.1", 2024-08-18T16:42:58.580 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:42:58.580 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:42:58.580 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:42:58.580 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:42:53.139848Z", 2024-08-18T16:42:58.580 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T16:42:58.580 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:42:58.580 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:42:58.580 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T16:42:58.581 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:42:58.581 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T16:42:58.581 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 1, 2024-08-18T16:42:58.581 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 1 2024-08-18T16:42:58.581 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:42:58.581 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T16:42:58Z" 2024-08-18T16:42:58.581 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T16:42:58.582 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-08-18T16:42:58.582 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2024-08-18T16:42:58.582 INFO:teuthology.orchestra.run:waiting for 300 2024-08-18T16:42:58.584 INFO:tasks.rgw.c2.client.0.smithi193.stdout:2024-08-18T16:42:58.582+0000 7ffbe0e6b640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 80518) UID: 0 2024-08-18T16:42:58.584 INFO:tasks.rgw.c2.client.0.smithi193.stdout:2024-08-18T16:42:58.582+0000 7ffbe6e2f880 -1 shutting down 2024-08-18T16:43:02.943 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2024-08-18T16:43:04.684 INFO:tasks.rgw.c2.client.0:Stopped 2024-08-18T16:43:04.807 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:43:04.807 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'init', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:43:04.807 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync init --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-08-18T16:43:04.863 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:43:04.863 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:43:04.899 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:43:04.898+0000 7f80a0232880 5 RGW-SYNC:data: waiting on data sync status lease 2024-08-18T16:43:04.900 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:43:04.900+0000 7f80a0232880 5 RGW-SYNC:data: acquired data sync status lease 2024-08-18T16:43:04.924 INFO:tasks.rgw.c2.client.0:Restarting daemon 2024-08-18T16:43:04.924 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2024-08-18T16:43:04.927 INFO:tasks.rgw.c2.client.0:Started 2024-08-18T16:43:04.927 DEBUG:teuthology.orchestra.run.smithi193:> curl http://smithi193.front.sepia.ceph.com:8000 2024-08-18T16:43:04.946 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-08-18T16:43:04.947 INFO:teuthology.orchestra.run.smithi193.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-08-18T16:43:04.947 INFO:teuthology.orchestra.run.smithi193.stderr: Dload Upload Total Spent Left Speed 2024-08-18T16:43:04.947 INFO:teuthology.orchestra.run.smithi193.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2024-08-18T16:43:04.947 INFO:teuthology.orchestra.run.smithi193.stderr:curl: (7) Failed to connect to smithi193.front.sepia.ceph.com port 8000: Connection refused 2024-08-18T16:43:05.948 DEBUG:teuthology.orchestra.run.smithi193:> curl http://smithi193.front.sepia.ceph.com:8000 2024-08-18T16:43:05.968 INFO:teuthology.orchestra.run.smithi193.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-08-18T16:43:05.968 INFO:teuthology.orchestra.run.smithi193.stderr: Dload Upload Total Spent Left Speed 2024-08-18T16:43:05.969 INFO:teuthology.orchestra.run.smithi193.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-08-18T16:43:05.969 INFO:teuthology.orchestra.run.smithi193.stdout:anonymous 2024-08-18T16:43:05.970 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-79', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:43:05.970 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-79', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:43:05.970 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-79 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T16:43:06.063 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:43:06.064 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:43:06.101 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:43:06.100+0000 7fa8c8926880 1 waiting to reach incremental sync.. 2024-08-18T16:43:08.648 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2024-08-18T16:43:11.102 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:43:11.101+0000 7fa8c8926880 1 waiting to reach incremental sync.. 2024-08-18T16:43:16.103 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:43:16.102+0000 7fa8c8926880 1 waiting to reach incremental sync.. 2024-08-18T16:43:21.103 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:43:21.102+0000 7fa8c8926880 1 waiting to reach incremental sync.. 2024-08-18T16:43:26.104 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:43:26.103+0000 7fa8c8926880 1 waiting to reach incremental sync.. 2024-08-18T16:43:31.104 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:43:31.103+0000 7fa8c8926880 1 waiting to reach incremental sync.. 2024-08-18T16:43:36.105 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:43:36.104+0000 7fa8c8926880 1 waiting to reach incremental sync.. 2024-08-18T16:43:41.105 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:43:41.104+0000 7fa8c8926880 1 waiting to reach incremental sync.. 2024-08-18T16:43:46.106 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:43:46.105+0000 7fa8c8926880 1 waiting to reach incremental sync.. 2024-08-18T16:43:51.106 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:43:51.105+0000 7fa8c8926880 1 waiting to reach incremental sync.. 2024-08-18T16:43:56.106 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:43:56.106+0000 7fa8c8926880 1 waiting to reach incremental sync.. 2024-08-18T16:44:01.107 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:44:01.106+0000 7fa8c8926880 1 waiting to reach incremental sync.. 2024-08-18T16:44:06.107 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:44:06.107+0000 7fa8c8926880 1 waiting to reach incremental sync.. 2024-08-18T16:44:11.108 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:44:11.107+0000 7fa8c8926880 1 waiting to reach incremental sync.. 2024-08-18T16:44:16.109 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:44:16.108+0000 7fa8c8926880 1 waiting to reach incremental sync.. 2024-08-18T16:44:21.109 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:44:21.108+0000 7fa8c8926880 1 waiting to reach incremental sync.. 2024-08-18T16:44:26.110 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:44:26.109+0000 7fa8c8926880 1 waiting to reach incremental sync.. 2024-08-18T16:44:31.110 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:44:31.109+0000 7fa8c8926880 1 waiting to reach incremental sync.. 2024-08-18T16:44:36.111 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:44:36.110+0000 7fa8c8926880 1 waiting to reach incremental sync.. 2024-08-18T16:44:41.112 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:44:41.111+0000 7fa8c8926880 1 waiting to reach incremental sync.. 2024-08-18T16:44:46.112 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:44:46.111+0000 7fa8c8926880 1 waiting to reach incremental sync.. 2024-08-18T16:44:51.113 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:44:51.112+0000 7fa8c8926880 1 waiting to reach incremental sync.. 2024-08-18T16:44:56.113 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:44:56.113+0000 7fa8c8926880 1 waiting to reach incremental sync.. 2024-08-18T16:45:01.114 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:45:01.113+0000 7fa8c8926880 1 waiting to reach incremental sync.. 2024-08-18T16:45:06.115 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:45:06.113+0000 7fa8c8926880 1 waiting to reach incremental sync.. 2024-08-18T16:45:11.116 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:45:11.114+0000 7fa8c8926880 1 waiting to reach incremental sync.. 2024-08-18T16:45:16.116 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:45:16.115+0000 7fa8c8926880 1 waiting to reach incremental sync.. 2024-08-18T16:45:21.117 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:45:21.116+0000 7fa8c8926880 1 waiting to reach incremental sync.. 2024-08-18T16:45:26.117 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:45:26.116+0000 7fa8c8926880 1 waiting to reach incremental sync.. 2024-08-18T16:45:31.118 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:45:31.117+0000 7fa8c8926880 1 waiting to reach incremental sync.. 2024-08-18T16:45:36.119 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:45:36.118+0000 7fa8c8926880 1 bucket sync caught up with source: 2024-08-18T16:45:36.119 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , 00000000001.119.6, 00000000002.205.6, 00000000002.105.6] 2024-08-18T16:45:36.119 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , , , 00000000001.119.6, 00000000002.205.6, 00000000002.105.6] 2024-08-18T16:45:36.119 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:45:36.118+0000 7fa8c8926880 0 bucket checkpoint complete 2024-08-18T16:45:36.140 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-79', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:45:36.141 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-79', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:45:36.141 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-79 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:45:36.202 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:45:36.202 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:45:36.240 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:45:36.239+0000 7f6b92d42880 1 bucket sync caught up with source: 2024-08-18T16:45:36.241 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , , , 00000000001.119.6, 00000000002.205.6, 00000000002.105.6] 2024-08-18T16:45:36.241 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , , , , 00000000001.119.6, 00000000002.205.6, 00000000002.105.6] 2024-08-18T16:45:36.241 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:45:36.239+0000 7f6b92d42880 0 bucket checkpoint complete 2024-08-18T16:45:36.255 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-79', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:45:36.255 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-79', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:45:36.256 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-79 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:45:36.309 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:45:36.309 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:45:36.347 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:45:36.346+0000 7f631a72c880 1 bucket sync caught up with source: 2024-08-18T16:45:36.347 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , , , 00000000001.248.6, 00000000002.503.6, 00000000002.309.6] 2024-08-18T16:45:36.347 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , , , , 00000000001.248.6, 00000000002.503.6, 00000000002.309.6] 2024-08-18T16:45:36.348 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:45:36.346+0000 7f631a72c880 0 bucket checkpoint complete 2024-08-18T16:45:36.361 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-79', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:45:36.362 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-79', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:45:36.362 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-79 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:45:36.416 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:45:36.416 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:45:36.451 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:45:36.450+0000 7f6d1c142880 1 bucket sync caught up with source: 2024-08-18T16:45:36.451 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , , , 00000000001.248.6, 00000000002.503.6, 00000000002.309.6] 2024-08-18T16:45:36.451 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , , , , 00000000001.248.6, 00000000002.503.6, 00000000002.309.6] 2024-08-18T16:45:36.451 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:45:36.450+0000 7f6d1c142880 0 bucket checkpoint complete 2024-08-18T16:45:36.466 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-79', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:45:36.466 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-79', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:45:36.466 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-79 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:45:36.519 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:45:36.519 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:45:36.554 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:45:36.553+0000 7f7e65932880 1 bucket sync caught up with source: 2024-08-18T16:45:36.554 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , , , 00000000001.123.6, 00000000002.268.6, 00000000002.203.6] 2024-08-18T16:45:36.554 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , , , , 00000000001.123.6, 00000000002.268.6, 00000000002.203.6] 2024-08-18T16:45:36.554 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:45:36.553+0000 7f7e65932880 0 bucket checkpoint complete 2024-08-18T16:45:36.568 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-79', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:45:36.568 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-79', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:45:36.568 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-79 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T16:45:36.630 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:45:36.631 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:45:36.670 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:45:36.669+0000 7fee36327880 1 bucket sync caught up with source: 2024-08-18T16:45:36.670 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , 00000000001.123.6, 00000000002.268.6, 00000000002.203.6] 2024-08-18T16:45:36.670 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , , , 00000000001.123.6, 00000000002.268.6, 00000000002.203.6] 2024-08-18T16:45:36.670 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:45:36.669+0000 7fee36327880 0 bucket checkpoint complete 2024-08-18T16:45:36.686 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-79 zones={test-zone2, test-zone1} 2024-08-18T16:45:36.696 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:45:36.699 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-08-18T16:45:36.699 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-08-18T16:45:36.700 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-08-18T16:45:36.700 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-08-18T16:45:36.700 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-08-18T16:45:36.700 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:45:36.702 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-08-18T16:45:36.702 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-08-18T16:45:36.702 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-08-18T16:45:36.702 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-08-18T16:45:36.702 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-08-18T16:45:36.707 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T16:45:36.714 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T16:45:36.725 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T16:45:36.777 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T16:45:36.784 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T16:45:36.837 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T16:45:36.889 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T16:45:36.896 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T16:45:36.948 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T16:45:36.956 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T16:45:36.963 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T16:45:36.974 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T16:45:37.028 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T16:45:37.035 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T16:45:37.093 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T16:45:37.145 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-79 zones={test-zone2, test-zone1} 2024-08-18T16:45:37.146 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-79 zones={test-zone3, test-zone1} 2024-08-18T16:45:37.152 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:45:37.154 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-08-18T16:45:37.154 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-08-18T16:45:37.154 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-08-18T16:45:37.154 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-08-18T16:45:37.154 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-08-18T16:45:37.155 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:45:37.158 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-08-18T16:45:37.158 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-08-18T16:45:37.158 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-08-18T16:45:37.158 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-08-18T16:45:37.158 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-08-18T16:45:37.163 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T16:45:37.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T16:45:37.225 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T16:45:37.233 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T16:45:37.240 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T16:45:37.297 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T16:45:37.353 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T16:45:37.360 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T16:45:37.414 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T16:45:37.422 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T16:45:37.475 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T16:45:37.487 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T16:45:37.535 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T16:45:37.541 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T16:45:37.594 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T16:45:37.604 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-79 zones={test-zone3, test-zone1} 2024-08-18T16:45:37.604 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-79 zones={test-zone3, test-zone2} 2024-08-18T16:45:37.608 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:45:37.611 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-08-18T16:45:37.611 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-08-18T16:45:37.611 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-08-18T16:45:37.611 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-08-18T16:45:37.611 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-08-18T16:45:37.611 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:45:37.614 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-08-18T16:45:37.614 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-08-18T16:45:37.614 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-08-18T16:45:37.614 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-08-18T16:45:37.614 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-08-18T16:45:37.619 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T16:45:37.672 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T16:45:37.684 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T16:45:37.693 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T16:45:37.700 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T16:45:37.798 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T16:45:37.848 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T16:45:37.898 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T16:45:37.938 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T16:45:37.987 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T16:45:37.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T16:45:38.046 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T16:45:38.096 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T16:45:38.103 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T16:45:38.159 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T16:45:38.211 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-79 zones={test-zone3, test-zone2} 2024-08-18T16:45:38.212 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:45:38.212 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:45:38.212 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:45:38.259 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:45:38.260 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:45:38.309 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T16:45:38.309 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:45:38.310 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000031632", 2024-08-18T16:45:38.310 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:43:13.880609Z" 2024-08-18T16:45:38.310 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:45:38.310 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:45:38.310 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000024109", 2024-08-18T16:45:38.310 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:36:05.109760Z" 2024-08-18T16:45:38.310 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:45:38.310 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:45:38.310 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000037925", 2024-08-18T16:45:38.310 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:43:04.758926Z" 2024-08-18T16:45:38.310 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:45:38.311 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:45:38.311 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000034672", 2024-08-18T16:45:38.311 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:43:13.882480Z" 2024-08-18T16:45:38.311 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:45:38.311 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T16:45:38.312 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-08-18T16:45:38.312 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:45:38.312 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:45:38.313 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:45:38.369 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:45:38.369 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:45:38.420 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T16:45:38.420 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T16:45:38.420 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T16:45:38.420 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T16:45:38.420 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T16:45:38.420 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 1766851417195514391 2024-08-18T16:45:38.420 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:45:38.421 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T16:45:38.421 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:45:38.421 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T16:45:38.421 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:45:38.421 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T16:45:38.421 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000031632", 2024-08-18T16:45:38.421 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:45:38.421 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:45:38.421 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:45:38.421 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:43:13.880609Z" 2024-08-18T16:45:38.421 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:45:38.422 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:45:38.422 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:45:38.422 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T16:45:38.422 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:45:38.422 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T16:45:38.422 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000024109", 2024-08-18T16:45:38.422 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:45:38.422 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:45:38.422 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:45:38.422 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:36:05.109760Z" 2024-08-18T16:45:38.423 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:45:38.423 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:45:38.423 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:45:38.423 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T16:45:38.423 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:45:38.423 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T16:45:38.423 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037925", 2024-08-18T16:45:38.423 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:45:38.423 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 70, 2024-08-18T16:45:38.423 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:45:38.423 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:43:04.758926Z" 2024-08-18T16:45:38.423 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:45:38.424 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:45:38.424 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:45:38.424 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T16:45:38.424 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:45:38.424 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T16:45:38.424 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000034672", 2024-08-18T16:45:38.424 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:45:38.424 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T16:45:38.424 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:45:38.424 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:43:13.882480Z" 2024-08-18T16:45:38.424 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:45:38.424 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:45:38.424 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T16:45:38.424 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:45:38.425 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T16:45:38.425 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 79, 2024-08-18T16:45:38.425 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 79 2024-08-18T16:45:38.425 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:45:38.425 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T16:45:38Z" 2024-08-18T16:45:38.425 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T16:45:38.426 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-08-18T16:45:38.426 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:45:38.426 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:45:38.426 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:45:38.473 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:45:38.473 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:45:38.523 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T16:45:38.523 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:45:38.523 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000031632", 2024-08-18T16:45:38.523 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:43:13.880609Z" 2024-08-18T16:45:38.523 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:45:38.523 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:45:38.523 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000024109", 2024-08-18T16:45:38.524 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:36:05.109760Z" 2024-08-18T16:45:38.524 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:45:38.524 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:45:38.524 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000037925", 2024-08-18T16:45:38.524 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:43:04.758926Z" 2024-08-18T16:45:38.524 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:45:38.524 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:45:38.524 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000034672", 2024-08-18T16:45:38.524 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:43:13.882480Z" 2024-08-18T16:45:38.524 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:45:38.524 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T16:45:38.525 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2024-08-18T16:45:38.525 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:45:38.525 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:45:38.526 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:45:38.574 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:45:38.574 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:45:38.617 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T16:45:38.618 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T16:45:38.618 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T16:45:38.618 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T16:45:38.618 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T16:45:38.618 INFO:teuthology.orchestra.run.smithi134.stdout: "instance_id": 6908254841848037526 2024-08-18T16:45:38.618 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:45:38.618 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T16:45:38.618 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:45:38.619 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T16:45:38.619 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:45:38.619 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T16:45:38.619 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000031632", 2024-08-18T16:45:38.619 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:45:38.619 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:45:38.619 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:45:38.619 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:43:13.880609Z" 2024-08-18T16:45:38.619 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:45:38.619 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:45:38.619 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:45:38.619 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T16:45:38.619 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:45:38.620 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T16:45:38.620 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000024109", 2024-08-18T16:45:38.620 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:45:38.620 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:45:38.620 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:45:38.620 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:36:05.109760Z" 2024-08-18T16:45:38.620 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:45:38.620 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:45:38.620 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:45:38.621 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T16:45:38.621 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:45:38.621 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T16:45:38.621 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000037925", 2024-08-18T16:45:38.621 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:45:38.621 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:45:38.621 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:45:38.622 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:43:04.758926Z" 2024-08-18T16:45:38.622 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:45:38.622 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:45:38.622 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:45:38.622 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T16:45:38.622 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:45:38.622 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T16:45:38.622 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000034672", 2024-08-18T16:45:38.622 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:45:38.622 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:45:38.622 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:45:38.622 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:43:13.882480Z" 2024-08-18T16:45:38.622 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:45:38.623 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:45:38.623 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T16:45:38.623 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:45:38.623 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T16:45:38.623 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 0, 2024-08-18T16:45:38.623 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 0 2024-08-18T16:45:38.623 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:45:38.623 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T16:45:38Z" 2024-08-18T16:45:38.623 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T16:45:38.624 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2024-08-18T16:45:38.624 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:45:38.624 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:45:38.625 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:45:38.674 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:45:38.674 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:45:38.725 INFO:teuthology.orchestra.run.smithi193.stdout:[ 2024-08-18T16:45:38.726 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:45:38.726 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000014929", 2024-08-18T16:45:38.726 INFO:teuthology.orchestra.run.smithi193.stdout: "last_update": "2024-08-18T16:44:55.032157Z" 2024-08-18T16:45:38.726 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:45:38.726 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:45:38.726 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000010376", 2024-08-18T16:45:38.726 INFO:teuthology.orchestra.run.smithi193.stdout: "last_update": "2024-08-18T16:39:21.505176Z" 2024-08-18T16:45:38.726 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:45:38.726 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:45:38.726 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000020381", 2024-08-18T16:45:38.727 INFO:teuthology.orchestra.run.smithi193.stdout: "last_update": "2024-08-18T16:44:35.206148Z" 2024-08-18T16:45:38.727 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:45:38.727 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:45:38.727 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000020101", 2024-08-18T16:45:38.727 INFO:teuthology.orchestra.run.smithi193.stdout: "last_update": "2024-08-18T16:44:55.033950Z" 2024-08-18T16:45:38.727 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:45:38.727 INFO:teuthology.orchestra.run.smithi193.stdout:] 2024-08-18T16:45:38.728 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2024-08-18T16:45:38.728 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:45:38.728 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:45:38.728 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:45:38.777 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:45:38.777 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:45:38.822 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T16:45:38.822 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T16:45:38.822 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T16:45:38.822 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T16:45:38.822 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T16:45:38.822 INFO:teuthology.orchestra.run.smithi134.stdout: "instance_id": 15510818030922206205 2024-08-18T16:45:38.822 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:45:38.822 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T16:45:38.823 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:45:38.823 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T16:45:38.823 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:45:38.823 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T16:45:38.823 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000014929", 2024-08-18T16:45:38.823 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:45:38.823 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:45:38.823 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:45:38.823 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:44:55.032157Z" 2024-08-18T16:45:38.823 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:45:38.823 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:45:38.824 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:45:38.824 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T16:45:38.824 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:45:38.824 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T16:45:38.824 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000010376", 2024-08-18T16:45:38.824 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:45:38.824 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:45:38.824 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:45:38.824 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:39:21.505176Z" 2024-08-18T16:45:38.824 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:45:38.824 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:45:38.824 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:45:38.824 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T16:45:38.825 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:45:38.825 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T16:45:38.825 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000020381", 2024-08-18T16:45:38.825 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:45:38.825 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:45:38.825 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:45:38.825 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:44:35.206148Z" 2024-08-18T16:45:38.825 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:45:38.825 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:45:38.825 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:45:38.825 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T16:45:38.825 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:45:38.826 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T16:45:38.826 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000020101", 2024-08-18T16:45:38.826 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:45:38.826 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:45:38.826 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:45:38.826 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:44:55.033950Z" 2024-08-18T16:45:38.826 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:45:38.826 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:45:38.826 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T16:45:38.826 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:45:38.826 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T16:45:38.826 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 0, 2024-08-18T16:45:38.827 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 0 2024-08-18T16:45:38.827 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:45:38.827 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T16:45:38Z" 2024-08-18T16:45:38.827 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T16:45:38.828 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2024-08-18T16:45:38.828 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:45:38.828 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:45:38.828 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:45:38.876 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:45:38.876 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:45:38.926 INFO:teuthology.orchestra.run.smithi193.stdout:[ 2024-08-18T16:45:38.926 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:45:38.926 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000014929", 2024-08-18T16:45:38.927 INFO:teuthology.orchestra.run.smithi193.stdout: "last_update": "2024-08-18T16:44:55.032157Z" 2024-08-18T16:45:38.927 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:45:38.927 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:45:38.927 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000010376", 2024-08-18T16:45:38.927 INFO:teuthology.orchestra.run.smithi193.stdout: "last_update": "2024-08-18T16:39:21.505176Z" 2024-08-18T16:45:38.927 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:45:38.927 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:45:38.927 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000020381", 2024-08-18T16:45:38.927 INFO:teuthology.orchestra.run.smithi193.stdout: "last_update": "2024-08-18T16:44:35.206148Z" 2024-08-18T16:45:38.927 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:45:38.927 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:45:38.927 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000020101", 2024-08-18T16:45:38.928 INFO:teuthology.orchestra.run.smithi193.stdout: "last_update": "2024-08-18T16:44:55.033950Z" 2024-08-18T16:45:38.928 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:45:38.928 INFO:teuthology.orchestra.run.smithi193.stdout:] 2024-08-18T16:45:38.929 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2024-08-18T16:45:38.929 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:45:38.929 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:45:38.929 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:45:38.979 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:45:38.979 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:45:39.024 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T16:45:39.024 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T16:45:39.024 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T16:45:39.024 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T16:45:39.025 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T16:45:39.025 INFO:teuthology.orchestra.run.smithi134.stdout: "instance_id": 9282142518545740118 2024-08-18T16:45:39.025 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:45:39.025 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T16:45:39.025 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:45:39.025 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T16:45:39.025 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:45:39.025 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T16:45:39.025 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000014929", 2024-08-18T16:45:39.025 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:45:39.025 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:45:39.026 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:45:39.026 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:44:55.032157Z" 2024-08-18T16:45:39.026 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:45:39.026 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:45:39.026 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:45:39.026 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T16:45:39.026 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:45:39.026 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T16:45:39.026 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000010376", 2024-08-18T16:45:39.026 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:45:39.027 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:45:39.027 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:45:39.027 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:39:21.505176Z" 2024-08-18T16:45:39.027 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:45:39.027 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:45:39.027 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:45:39.027 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T16:45:39.027 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:45:39.027 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T16:45:39.027 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000020381", 2024-08-18T16:45:39.027 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:45:39.028 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:45:39.028 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:45:39.028 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:44:35.206148Z" 2024-08-18T16:45:39.028 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:45:39.028 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:45:39.028 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:45:39.028 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T16:45:39.028 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:45:39.028 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T16:45:39.028 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000020101", 2024-08-18T16:45:39.028 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:45:39.028 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:45:39.029 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:45:39.029 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:44:55.033950Z" 2024-08-18T16:45:39.029 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:45:39.029 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:45:39.029 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T16:45:39.029 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:45:39.029 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T16:45:39.029 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 0, 2024-08-18T16:45:39.029 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 0 2024-08-18T16:45:39.029 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:45:39.029 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T16:45:39Z" 2024-08-18T16:45:39.030 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T16:45:39.031 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2024-08-18T16:45:39.031 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:45:39.031 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:45:39.031 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:45:39.077 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:45:39.077 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:45:39.127 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T16:45:39.127 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:45:39.127 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000014648", 2024-08-18T16:45:39.127 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:43:37.616463Z" 2024-08-18T16:45:39.127 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:45:39.128 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:45:39.128 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000010376", 2024-08-18T16:45:39.128 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:39:35.151572Z" 2024-08-18T16:45:39.128 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:45:39.128 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:45:39.128 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000020240", 2024-08-18T16:45:39.128 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:43:16.856638Z" 2024-08-18T16:45:39.128 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:45:39.128 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:45:39.128 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000021087", 2024-08-18T16:45:39.128 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:43:37.618174Z" 2024-08-18T16:45:39.129 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:45:39.129 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T16:45:39.130 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2024-08-18T16:45:39.130 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:45:39.130 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:45:39.130 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:45:39.179 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:45:39.179 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:45:39.224 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T16:45:39.224 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T16:45:39.225 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T16:45:39.225 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T16:45:39.225 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T16:45:39.225 INFO:teuthology.orchestra.run.smithi134.stdout: "instance_id": 16929744281786545893 2024-08-18T16:45:39.225 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:45:39.225 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T16:45:39.225 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:45:39.225 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T16:45:39.225 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:45:39.225 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T16:45:39.225 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000014648", 2024-08-18T16:45:39.225 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:45:39.225 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:45:39.226 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:45:39.226 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:43:37.616463Z" 2024-08-18T16:45:39.226 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:45:39.226 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:45:39.226 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:45:39.226 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T16:45:39.226 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:45:39.226 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T16:45:39.226 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000010376", 2024-08-18T16:45:39.226 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:45:39.227 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:45:39.227 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:45:39.227 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:39:35.151572Z" 2024-08-18T16:45:39.227 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:45:39.227 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:45:39.227 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:45:39.227 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T16:45:39.227 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:45:39.227 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T16:45:39.227 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000020240", 2024-08-18T16:45:39.228 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:45:39.228 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:45:39.228 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:45:39.228 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:43:16.856638Z" 2024-08-18T16:45:39.228 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:45:39.228 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:45:39.228 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:45:39.228 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T16:45:39.228 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:45:39.228 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T16:45:39.229 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000021087", 2024-08-18T16:45:39.229 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:45:39.229 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:45:39.229 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:45:39.229 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:43:37.618174Z" 2024-08-18T16:45:39.229 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:45:39.229 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:45:39.229 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T16:45:39.229 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:45:39.229 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T16:45:39.229 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 0, 2024-08-18T16:45:39.229 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 0 2024-08-18T16:45:39.230 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:45:39.230 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T16:45:39Z" 2024-08-18T16:45:39.230 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T16:45:39.231 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2024-08-18T16:45:39.232 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:45:39.232 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:45:39.232 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:45:39.279 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:45:39.280 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:45:39.329 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T16:45:39.329 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:45:39.329 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000014648", 2024-08-18T16:45:39.330 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:43:37.616463Z" 2024-08-18T16:45:39.330 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:45:39.330 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:45:39.330 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000010376", 2024-08-18T16:45:39.330 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:39:35.151572Z" 2024-08-18T16:45:39.330 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:45:39.330 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:45:39.330 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000020240", 2024-08-18T16:45:39.330 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:43:16.856638Z" 2024-08-18T16:45:39.330 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:45:39.330 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:45:39.331 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000021087", 2024-08-18T16:45:39.331 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:43:37.618174Z" 2024-08-18T16:45:39.331 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:45:39.331 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T16:45:39.332 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2024-08-18T16:45:39.332 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:45:39.332 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:45:39.332 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:45:39.383 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:45:39.383 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:45:39.428 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T16:45:39.429 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T16:45:39.429 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T16:45:39.429 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T16:45:39.429 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T16:45:39.429 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17055221296585213178 2024-08-18T16:45:39.429 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:45:39.429 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T16:45:39.429 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:45:39.429 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T16:45:39.429 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:45:39.430 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T16:45:39.430 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000014648", 2024-08-18T16:45:39.430 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:45:39.430 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:45:39.430 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:45:39.430 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:43:37.616463Z" 2024-08-18T16:45:39.430 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:45:39.430 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:45:39.430 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:45:39.430 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T16:45:39.430 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:45:39.431 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T16:45:39.431 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000010376", 2024-08-18T16:45:39.431 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:45:39.431 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:45:39.431 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:45:39.431 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:39:35.151572Z" 2024-08-18T16:45:39.431 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:45:39.431 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:45:39.431 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:45:39.431 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T16:45:39.431 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:45:39.431 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T16:45:39.431 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000020240", 2024-08-18T16:45:39.432 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:45:39.432 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:45:39.432 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:45:39.432 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:43:16.856638Z" 2024-08-18T16:45:39.432 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:45:39.432 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:45:39.432 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:45:39.432 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T16:45:39.432 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:45:39.432 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T16:45:39.432 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000021087", 2024-08-18T16:45:39.432 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:45:39.432 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:45:39.433 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:45:39.433 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:43:37.618174Z" 2024-08-18T16:45:39.433 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:45:39.433 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:45:39.433 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T16:45:39.433 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:45:39.433 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T16:45:39.433 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 0, 2024-08-18T16:45:39.433 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 0 2024-08-18T16:45:39.433 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:45:39.433 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T16:45:39Z" 2024-08-18T16:45:39.434 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T16:45:39.435 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2024-08-18T16:45:39.435 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_full_sync_after_data_sync_init ... ok 2024-08-18T16:45:39.452 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:45:39.452 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:45:39.452 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:45:39.498 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:45:39.498 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:45:39.529 INFO:teuthology.orchestra.run.smithi134.stderr:No --period given, using current period=01050e30-179b-43be-bdbb-3bb23cf37ae8 2024-08-18T16:45:39.542 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T16:45:39.542 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:45:39.542 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T16:45:39.543 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T16:45:39.543 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:45:39.543 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:45:39.543 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T16:45:39.543 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T16:45:39.543 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:45:39.543 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:45:39.543 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T16:45:39.543 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:42:47.672461Z" 2024-08-18T16:45:39.543 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:45:39.543 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:45:39.543 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723999539.450359_1148.1", 2024-08-18T16:45:39.544 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:45:39.450359Z" 2024-08-18T16:45:39.544 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:45:39.544 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T16:45:39.545 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-08-18T16:45:39.545 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:45:39.545 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:45:39.545 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:45:39.593 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:45:39.593 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:45:39.624 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:45:39.624+0000 7f3e19c1f880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:45:39.625 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:45:39.624+0000 7f3e19c1f880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:45:39.638 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T16:45:39.638 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T16:45:39.638 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T16:45:39.638 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T16:45:39.639 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T16:45:39.639 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T16:45:39.639 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T16:45:39.639 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:45:39.639 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T16:45:39.639 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:45:39.639 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T16:45:39.639 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:45:39.639 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:45:39.639 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T16:45:39.639 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:45:39.639 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T16:45:39.639 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:45:39.640 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T16:45:39.640 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T16:45:39.640 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:45:39.640 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:45:39.640 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:45:39.640 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T16:45:39.640 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:45:39.640 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:45:39.640 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T16:45:39.641 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:45:39.641 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:45:39.641 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:45:39.641 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T16:45:39.641 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T16:45:39.641 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:45:39.641 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:45:39.641 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:45:39.641 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T16:45:39.641 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:45:39.641 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:45:39.641 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T16:45:39.641 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:45:39.642 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:45:39.642 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:45:39.642 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T16:45:39.642 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T16:45:39.642 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:45:39.642 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:45:39.642 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:45:39.642 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T16:45:39.642 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:45:39.642 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:45:39.642 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999539.444493_1145.1", 2024-08-18T16:45:39.642 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:45:39.642 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:45:39.643 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:45:39.643 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:45:39.444493Z", 2024-08-18T16:45:39.643 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T16:45:39.643 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:45:39.643 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:45:39.643 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T16:45:39.643 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:45:39.643 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T16:45:39.643 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T16:45:39.643 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T16:45:39.643 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:45:39.643 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T16:45:39Z" 2024-08-18T16:45:39.644 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T16:45:39.644 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1723999539.450359_1148.1 target=1_1723999539.444493_1145.1 2024-08-18T16:45:44.645 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:45:44.646 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:45:44.646 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:45:44.704 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:45:44.704 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:45:44.738 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:45:44.737+0000 7f679c410880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:45:44.739 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:45:44.738+0000 7f679c410880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:45:44.754 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T16:45:44.754 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T16:45:44.754 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T16:45:44.754 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T16:45:44.754 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T16:45:44.754 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T16:45:44.754 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T16:45:44.754 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:45:44.754 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T16:45:44.754 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:45:44.755 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T16:45:44.755 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:45:44.755 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:45:44.755 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T16:45:44.755 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:45:44.755 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T16:45:44.755 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:45:44.755 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T16:45:44.755 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T16:45:44.755 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:45:44.755 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:45:44.755 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:45:44.755 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T16:45:44.755 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:45:44.756 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:45:44.756 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T16:45:44.756 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:45:44.756 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:45:44.756 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:45:44.756 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T16:45:44.756 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T16:45:44.756 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:45:44.756 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:45:44.756 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:45:44.756 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T16:45:44.756 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:45:44.756 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:45:44.757 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T16:45:44.757 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:45:44.757 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:45:44.757 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:45:44.757 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T16:45:44.757 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T16:45:44.757 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:45:44.757 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:45:44.757 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:45:44.757 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T16:45:44.757 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:45:44.757 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:45:44.757 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999539.450359_1148.1", 2024-08-18T16:45:44.757 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:45:44.758 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:45:44.758 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:45:44.758 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:45:39.450359Z", 2024-08-18T16:45:44.758 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T16:45:44.758 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:45:44.758 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:45:44.758 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T16:45:44.758 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:45:44.758 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T16:45:44.758 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T16:45:44.758 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T16:45:44.758 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:45:44.759 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T16:45:44Z" 2024-08-18T16:45:44.759 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T16:45:44.760 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-08-18T16:45:44.760 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-08-18T16:45:44.760 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:45:44.760 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:45:44.760 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:45:44.815 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:45:44.815 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:45:44.849 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:45:44.848+0000 7f276ec62880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:45:44.849 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:45:44.849+0000 7f276ec62880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:45:44.864 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T16:45:44.864 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T16:45:44.864 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T16:45:44.864 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T16:45:44.864 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T16:45:44.864 INFO:teuthology.orchestra.run.smithi134.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T16:45:44.864 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T16:45:44.864 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:45:44.864 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T16:45:44.865 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:45:44.865 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T16:45:44.865 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:45:44.865 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T16:45:44.865 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T16:45:44.865 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:45:44.865 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 1, 2024-08-18T16:45:44.865 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:45:44.865 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T16:45:44.865 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T16:45:44.865 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:45:44.865 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:45:44.866 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:45:44.866 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T16:45:44.866 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:45:44.866 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T16:45:44.866 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T16:45:44.866 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:45:44.866 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:45:44.866 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:45:44.866 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T16:45:44.866 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T16:45:44.866 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:45:44.867 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:45:44.867 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:45:44.867 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T16:45:44.867 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:45:44.867 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T16:45:44.867 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T16:45:44.867 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:45:44.867 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:45:44.867 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:45:44.868 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T16:45:44.868 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T16:45:44.868 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:45:44.868 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:45:44.868 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:45:44.868 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T16:45:44.868 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:45:44.868 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T16:45:44.868 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723999539.450359_1148.1", 2024-08-18T16:45:44.868 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:45:44.868 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:45:44.868 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:45:44.868 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:45:39.450359Z", 2024-08-18T16:45:44.869 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T16:45:44.869 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:45:44.869 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:45:44.869 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T16:45:44.869 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:45:44.869 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T16:45:44.869 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 1, 2024-08-18T16:45:44.869 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 1 2024-08-18T16:45:44.869 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:45:44.869 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T16:45:44Z" 2024-08-18T16:45:44.869 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T16:45:44.870 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-08-18T16:45:44.871 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2024-08-18T16:45:44.871 INFO:teuthology.orchestra.run:waiting for 300 2024-08-18T16:45:44.871 INFO:tasks.rgw.c2.client.0.smithi193.stdout:2024-08-18T16:45:44.871+0000 7f7edd979640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 81835) UID: 0 2024-08-18T16:45:44.872 INFO:tasks.rgw.c2.client.0.smithi193.stdout:2024-08-18T16:45:44.871+0000 7f7ee3a2f880 -1 shutting down 2024-08-18T16:45:48.314 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2024-08-18T16:45:50.972 INFO:tasks.rgw.c2.client.0:Stopped 2024-08-18T16:45:51.087 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-80', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:45:51.087 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-80', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:45:51.087 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-80 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:45:51.144 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:45:51.144 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:45:51.191 INFO:teuthology.orchestra.run.smithi134.stdout:tenant: 2024-08-18T16:45:51.192 INFO:teuthology.orchestra.run.smithi134.stdout:bucket name: kibgmb-80 2024-08-18T16:45:51.195 INFO:teuthology.orchestra.run.smithi134.stdout:total entries: 5 2024-08-18T16:45:51.347 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-80', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:45:51.347 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-80', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:45:51.347 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-80 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:45:51.401 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:45:51.401 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:45:51.445 INFO:teuthology.orchestra.run.smithi134.stdout:tenant: 2024-08-18T16:45:51.445 INFO:teuthology.orchestra.run.smithi134.stdout:bucket name: kibgmb-80 2024-08-18T16:45:51.452 INFO:teuthology.orchestra.run.smithi134.stdout:total entries: 10 2024-08-18T16:45:51.569 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-80', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:45:51.570 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-80', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:45:51.570 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-80 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:45:51.620 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:45:51.620 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:45:51.666 INFO:teuthology.orchestra.run.smithi134.stdout:tenant: 2024-08-18T16:45:51.666 INFO:teuthology.orchestra.run.smithi134.stdout:bucket name: kibgmb-80 2024-08-18T16:45:51.672 INFO:teuthology.orchestra.run.smithi134.stdout:total entries: 15 2024-08-18T16:45:51.842 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-80', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:45:51.843 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-80', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:45:51.843 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-80 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:45:51.893 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:45:51.893 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:45:51.939 INFO:teuthology.orchestra.run.smithi134.stdout:tenant: 2024-08-18T16:45:51.940 INFO:teuthology.orchestra.run.smithi134.stdout:bucket name: kibgmb-80 2024-08-18T16:45:51.947 INFO:teuthology.orchestra.run.smithi134.stdout:total entries: 20 2024-08-18T16:45:52.119 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-80', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:45:52.119 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-80', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:45:52.120 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-80 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:45:52.169 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:45:52.170 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:45:52.217 INFO:teuthology.orchestra.run.smithi134.stdout:tenant: 2024-08-18T16:45:52.217 INFO:teuthology.orchestra.run.smithi134.stdout:bucket name: kibgmb-80 2024-08-18T16:45:52.227 INFO:teuthology.orchestra.run.smithi134.stdout:total entries: 25 2024-08-18T16:45:52.412 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-80', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:45:52.412 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-80', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:45:52.412 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-80 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:45:52.461 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:45:52.462 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:45:52.511 INFO:teuthology.orchestra.run.smithi134.stdout:tenant: 2024-08-18T16:45:52.512 INFO:teuthology.orchestra.run.smithi134.stdout:bucket name: kibgmb-80 2024-08-18T16:45:52.522 INFO:teuthology.orchestra.run.smithi134.stdout:total entries: 30 2024-08-18T16:45:52.711 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:45:52.711 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'init', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:45:52.711 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync init --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-08-18T16:45:52.767 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:45:52.767 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:45:52.804 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:45:52.803+0000 7f2207722880 5 RGW-SYNC:data: waiting on data sync status lease 2024-08-18T16:45:52.805 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:45:52.804+0000 7f2207722880 5 RGW-SYNC:data: acquired data sync status lease 2024-08-18T16:45:52.831 INFO:tasks.rgw.c2.client.0:Restarting daemon 2024-08-18T16:45:52.831 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2024-08-18T16:45:52.833 INFO:tasks.rgw.c2.client.0:Started 2024-08-18T16:45:52.833 DEBUG:teuthology.orchestra.run.smithi193:> curl http://smithi193.front.sepia.ceph.com:8000 2024-08-18T16:45:52.852 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-08-18T16:45:52.853 INFO:teuthology.orchestra.run.smithi193.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-08-18T16:45:52.853 INFO:teuthology.orchestra.run.smithi193.stderr: Dload Upload Total Spent Left Speed 2024-08-18T16:45:52.853 INFO:teuthology.orchestra.run.smithi193.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2024-08-18T16:45:52.853 INFO:teuthology.orchestra.run.smithi193.stderr:curl: (7) Failed to connect to smithi193.front.sepia.ceph.com port 8000: Connection refused 2024-08-18T16:45:53.854 DEBUG:teuthology.orchestra.run.smithi193:> curl http://smithi193.front.sepia.ceph.com:8000 2024-08-18T16:45:53.873 INFO:teuthology.orchestra.run.smithi193.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-08-18T16:45:53.873 INFO:teuthology.orchestra.run.smithi193.stderr: Dload Upload Total Spent Left Speed 2024-08-18T16:45:53.873 INFO:teuthology.orchestra.run.smithi193.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-08-18T16:45:53.874 INFO:teuthology.orchestra.run.smithi193.stdout:anonymous 2024-08-18T16:45:53.874 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-80', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:45:53.874 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-80', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:45:53.875 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-80 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T16:45:53.968 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:45:53.969 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:45:54.007 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:45:54.005+0000 7ff92a924880 1 waiting to reach incremental sync.. 2024-08-18T16:45:54.020 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2024-08-18T16:45:59.007 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:45:59.006+0000 7ff92a924880 1 waiting to reach incremental sync.. 2024-08-18T16:46:04.007 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:46:04.006+0000 7ff92a924880 1 waiting to reach incremental sync.. 2024-08-18T16:46:09.008 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:46:09.006+0000 7ff92a924880 1 waiting to reach incremental sync.. 2024-08-18T16:46:14.009 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:46:14.008+0000 7ff92a924880 1 waiting to reach incremental sync.. 2024-08-18T16:46:19.009 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:46:19.008+0000 7ff92a924880 1 waiting to reach incremental sync.. 2024-08-18T16:46:24.009 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:46:24.008+0000 7ff92a924880 1 waiting to reach incremental sync.. 2024-08-18T16:46:29.010 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:46:29.009+0000 7ff92a924880 1 waiting to reach incremental sync.. 2024-08-18T16:46:34.010 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:46:34.009+0000 7ff92a924880 1 waiting to reach incremental sync.. 2024-08-18T16:46:39.011 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:46:39.010+0000 7ff92a924880 1 waiting to reach incremental sync.. 2024-08-18T16:46:44.012 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:46:44.011+0000 7ff92a924880 1 waiting to reach incremental sync.. 2024-08-18T16:46:49.012 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:46:49.011+0000 7ff92a924880 1 waiting to reach incremental sync.. 2024-08-18T16:46:54.013 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:46:54.012+0000 7ff92a924880 1 waiting to reach incremental sync.. 2024-08-18T16:46:59.014 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:46:59.012+0000 7ff92a924880 1 waiting to reach incremental sync.. 2024-08-18T16:47:04.014 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:47:04.013+0000 7ff92a924880 1 waiting to reach incremental sync.. 2024-08-18T16:47:09.014 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:47:09.013+0000 7ff92a924880 1 waiting to reach incremental sync.. 2024-08-18T16:47:14.015 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:47:14.013+0000 7ff92a924880 1 waiting to reach incremental sync.. 2024-08-18T16:47:19.015 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:47:19.014+0000 7ff92a924880 1 waiting to reach incremental sync.. 2024-08-18T16:47:24.017 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:47:24.016+0000 7ff92a924880 1 waiting to reach incremental sync.. 2024-08-18T16:47:29.017 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:47:29.016+0000 7ff92a924880 1 waiting to reach incremental sync.. 2024-08-18T16:47:34.018 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:47:34.017+0000 7ff92a924880 1 waiting to reach incremental sync.. 2024-08-18T16:47:39.019 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:47:39.018+0000 7ff92a924880 1 waiting to reach incremental sync.. 2024-08-18T16:47:44.020 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:47:44.018+0000 7ff92a924880 1 waiting to reach incremental sync.. 2024-08-18T16:47:49.020 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:47:49.019+0000 7ff92a924880 1 waiting to reach incremental sync.. 2024-08-18T16:47:54.020 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:47:54.020+0000 7ff92a924880 1 waiting to reach incremental sync.. 2024-08-18T16:47:59.024 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:47:59.023+0000 7ff92a924880 1 bucket sync caught up with source: 2024-08-18T16:47:59.024 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.161.6, , , , , 00000000002.151.6, , , , , , , , , , , , , , , 00000000001.136.6, , , , 00000000001.144.6] 2024-08-18T16:47:59.024 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , 00000000002.134.6, , , , , , 00000000002.161.6, , , , , 00000000002.151.6, , , , , , , , , , , , , , , 00000000001.136.6, , , , 00000000001.144.6] 2024-08-18T16:47:59.024 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:47:59.023+0000 7ff92a924880 0 bucket checkpoint complete 2024-08-18T16:47:59.042 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-80', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:47:59.042 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-80', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:47:59.043 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-80 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:47:59.104 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:47:59.105 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:47:59.146 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:47:59.145+0000 7f9465686880 1 bucket sync caught up with source: 2024-08-18T16:47:59.146 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.161.6, , , , , 00000000002.151.6, , , , , , , , , , , , , , , 00000000001.136.6, , , , 00000000001.144.6] 2024-08-18T16:47:59.147 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , , 00000000002.134.6, , , , , , 00000000002.161.6, , , , , 00000000002.151.6, , , , , , , , , , , , , , , 00000000001.136.6, , , , 00000000001.144.6] 2024-08-18T16:47:59.147 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:47:59.145+0000 7f9465686880 0 bucket checkpoint complete 2024-08-18T16:47:59.161 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-80', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:47:59.161 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-80', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:47:59.162 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-80 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:47:59.216 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:47:59.217 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:47:59.252 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:47:59.251+0000 7f4fa8930880 1 waiting to reach incremental sync.. 2024-08-18T16:48:04.252 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:48:04.251+0000 7f4fa8930880 1 waiting to reach incremental sync.. 2024-08-18T16:48:09.253 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:48:09.252+0000 7f4fa8930880 1 waiting to reach incremental sync.. 2024-08-18T16:48:14.254 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:48:14.253+0000 7f4fa8930880 1 waiting to reach incremental sync.. 2024-08-18T16:48:19.255 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:48:19.253+0000 7f4fa8930880 1 waiting to reach incremental sync.. 2024-08-18T16:48:24.256 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:48:24.254+0000 7f4fa8930880 1 waiting to reach incremental sync.. 2024-08-18T16:48:29.256 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:48:29.255+0000 7f4fa8930880 1 waiting to reach incremental sync.. 2024-08-18T16:48:34.257 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:48:34.256+0000 7f4fa8930880 1 waiting to reach incremental sync.. 2024-08-18T16:48:39.257 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:48:39.256+0000 7f4fa8930880 1 waiting to reach incremental sync.. 2024-08-18T16:48:44.258 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:48:44.257+0000 7f4fa8930880 1 waiting to reach incremental sync.. 2024-08-18T16:48:49.259 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:48:49.258+0000 7f4fa8930880 1 waiting to reach incremental sync.. 2024-08-18T16:48:54.259 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:48:54.258+0000 7f4fa8930880 1 waiting to reach incremental sync.. 2024-08-18T16:48:59.260 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:48:59.259+0000 7f4fa8930880 1 waiting to reach incremental sync.. 2024-08-18T16:49:04.261 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:49:04.260+0000 7f4fa8930880 1 waiting to reach incremental sync.. 2024-08-18T16:49:09.261 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:49:09.260+0000 7f4fa8930880 1 waiting to reach incremental sync.. 2024-08-18T16:49:14.262 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:49:14.261+0000 7f4fa8930880 1 waiting to reach incremental sync.. 2024-08-18T16:49:19.262 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:49:19.261+0000 7f4fa8930880 1 waiting to reach incremental sync.. 2024-08-18T16:49:24.263 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:49:24.262+0000 7f4fa8930880 1 waiting to reach incremental sync.. 2024-08-18T16:49:29.264 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:49:29.262+0000 7f4fa8930880 1 waiting to reach incremental sync.. 2024-08-18T16:49:34.265 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:49:34.264+0000 7f4fa8930880 1 waiting to reach incremental sync.. 2024-08-18T16:49:39.266 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:49:39.265+0000 7f4fa8930880 1 waiting to reach incremental sync.. 2024-08-18T16:49:44.266 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:49:44.265+0000 7f4fa8930880 1 waiting to reach incremental sync.. 2024-08-18T16:49:49.267 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:49:49.266+0000 7f4fa8930880 1 waiting to reach incremental sync.. 2024-08-18T16:49:54.268 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:49:54.267+0000 7f4fa8930880 1 waiting to reach incremental sync.. 2024-08-18T16:49:59.268 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:49:59.267+0000 7f4fa8930880 1 waiting to reach incremental sync.. 2024-08-18T16:50:04.269 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:50:04.268+0000 7f4fa8930880 1 waiting to reach incremental sync.. 2024-08-18T16:50:09.269 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:50:09.268+0000 7f4fa8930880 1 waiting to reach incremental sync.. 2024-08-18T16:50:14.270 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:50:14.269+0000 7f4fa8930880 1 waiting to reach incremental sync.. 2024-08-18T16:50:19.271 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:50:19.270+0000 7f4fa8930880 1 waiting to reach incremental sync.. 2024-08-18T16:50:24.273 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:50:24.270+0000 7f4fa8930880 1 waiting to reach incremental sync.. 2024-08-18T16:50:29.272 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:50:29.271+0000 7f4fa8930880 1 waiting to reach incremental sync.. 2024-08-18T16:50:34.273 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:50:34.271+0000 7f4fa8930880 1 waiting to reach incremental sync.. 2024-08-18T16:50:39.273 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:50:39.272+0000 7f4fa8930880 1 waiting to reach incremental sync.. 2024-08-18T16:50:44.274 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:50:44.273+0000 7f4fa8930880 1 waiting to reach incremental sync.. 2024-08-18T16:50:49.274 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:50:49.273+0000 7f4fa8930880 1 waiting to reach incremental sync.. 2024-08-18T16:50:54.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:50:54.274+0000 7f4fa8930880 1 waiting to reach incremental sync.. 2024-08-18T16:50:59.277 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:50:59.276+0000 7f4fa8930880 1 waiting to reach incremental sync.. 2024-08-18T16:51:04.278 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:51:04.276+0000 7f4fa8930880 1 waiting to reach incremental sync.. 2024-08-18T16:51:09.279 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:51:09.277+0000 7f4fa8930880 1 waiting to reach incremental sync.. 2024-08-18T16:51:14.279 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:51:14.278+0000 7f4fa8930880 1 waiting to reach incremental sync.. 2024-08-18T16:51:19.280 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:51:19.278+0000 7f4fa8930880 1 waiting to reach incremental sync.. 2024-08-18T16:51:24.280 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:51:24.279+0000 7f4fa8930880 1 waiting to reach incremental sync.. 2024-08-18T16:51:29.281 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:51:29.280+0000 7f4fa8930880 1 waiting to reach incremental sync.. 2024-08-18T16:51:34.281 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:51:34.280+0000 7f4fa8930880 1 waiting to reach incremental sync.. 2024-08-18T16:51:39.283 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:51:39.280+0000 7f4fa8930880 1 waiting to reach incremental sync.. 2024-08-18T16:51:44.283 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:51:44.282+0000 7f4fa8930880 1 waiting to reach incremental sync.. 2024-08-18T16:51:49.284 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:51:49.283+0000 7f4fa8930880 1 waiting to reach incremental sync.. 2024-08-18T16:51:54.285 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:51:54.283+0000 7f4fa8930880 1 waiting to reach incremental sync.. 2024-08-18T16:51:59.285 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:51:59.284+0000 7f4fa8930880 1 waiting to reach incremental sync.. 2024-08-18T16:52:04.286 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:52:04.285+0000 7f4fa8930880 1 waiting to reach incremental sync.. 2024-08-18T16:52:09.286 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:52:09.285+0000 7f4fa8930880 1 waiting to reach incremental sync.. 2024-08-18T16:52:14.287 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:52:14.286+0000 7f4fa8930880 1 waiting to reach incremental sync.. 2024-08-18T16:52:19.287 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:52:19.286+0000 7f4fa8930880 1 waiting to reach incremental sync.. 2024-08-18T16:52:24.288 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:52:24.287+0000 7f4fa8930880 1 waiting to reach incremental sync.. 2024-08-18T16:52:29.288 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:52:29.287+0000 7f4fa8930880 1 waiting to reach incremental sync.. 2024-08-18T16:52:34.289 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:52:34.288+0000 7f4fa8930880 1 waiting to reach incremental sync.. 2024-08-18T16:52:39.290 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:52:39.289+0000 7f4fa8930880 1 waiting to reach incremental sync.. 2024-08-18T16:52:44.291 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:52:44.289+0000 7f4fa8930880 1 waiting to reach incremental sync.. 2024-08-18T16:52:49.291 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:52:49.290+0000 7f4fa8930880 1 waiting to reach incremental sync.. 2024-08-18T16:52:54.292 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:52:54.291+0000 7f4fa8930880 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-08-18T16:52:54.292 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:52:54.291+0000 7f4fa8930880 0 bucket sync checkpoint failed: (110) Connection timed out 2024-08-18T16:52:54.292 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:52:54.291+0000 7f4fa8930880 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-08-18T16:52:54.310 DEBUG:teuthology.orchestra.run:got remote process result: 110 2024-08-18T16:52:54.312 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_resharded_bucket_full_sync_after_data_sync_init ... FAIL 2024-08-18T16:52:54.335 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:52:54.336 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:52:54.336 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:52:54.389 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:52:54.389 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:52:54.421 INFO:teuthology.orchestra.run.smithi134.stderr:No --period given, using current period=01050e30-179b-43be-bdbb-3bb23cf37ae8 2024-08-18T16:52:54.435 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T16:52:54.435 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:52:54.436 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T16:52:54.436 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T16:52:54.436 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:52:54.436 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:52:54.436 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T16:52:54.436 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T16:52:54.436 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:52:54.436 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:52:54.436 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T16:52:54.436 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:42:47.672461Z" 2024-08-18T16:52:54.436 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:52:54.436 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:52:54.437 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723999974.333303_1250.1", 2024-08-18T16:52:54.437 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:52:54.333303Z" 2024-08-18T16:52:54.437 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:52:54.437 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T16:52:54.438 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-08-18T16:52:54.438 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:52:54.438 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:52:54.439 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:52:54.495 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:52:54.495 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:52:54.533 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:52:54.532+0000 7f52dda33880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:52:54.534 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:52:54.533+0000 7f52dda33880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:52:54.548 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T16:52:54.548 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T16:52:54.549 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T16:52:54.549 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T16:52:54.549 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T16:52:54.549 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T16:52:54.549 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T16:52:54.549 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:52:54.549 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T16:52:54.549 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:52:54.549 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T16:52:54.549 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:52:54.549 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:52:54.550 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T16:52:54.550 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:52:54.550 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T16:52:54.550 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:52:54.550 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T16:52:54.550 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T16:52:54.550 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:52:54.551 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:52:54.551 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:52:54.551 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T16:52:54.551 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:52:54.551 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:52:54.551 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T16:52:54.552 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:52:54.552 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:52:54.552 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:52:54.552 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T16:52:54.552 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T16:52:54.552 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:52:54.553 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:52:54.553 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:52:54.553 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T16:52:54.553 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:52:54.553 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:52:54.553 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T16:52:54.554 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:52:54.554 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:52:54.554 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:52:54.554 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T16:52:54.554 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T16:52:54.554 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:52:54.555 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:52:54.555 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:52:54.555 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T16:52:54.555 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:52:54.555 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:52:54.555 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999974.333303_1250.1", 2024-08-18T16:52:54.556 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:52:54.556 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:52:54.556 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:52:54.556 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:52:54.333303Z", 2024-08-18T16:52:54.556 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T16:52:54.556 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:52:54.557 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:52:54.557 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T16:52:54.557 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:52:54.557 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T16:52:54.558 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T16:52:54.558 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T16:52:54.558 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:52:54.559 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T16:52:54Z" 2024-08-18T16:52:54.559 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T16:52:54.561 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-08-18T16:52:54.561 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-08-18T16:52:54.561 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:52:54.561 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:52:54.562 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:52:54.611 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:52:54.611 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:52:54.641 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:52:54.640+0000 7fbb0680f880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:52:54.641 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:52:54.641+0000 7fbb0680f880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:52:54.655 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T16:52:54.655 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T16:52:54.655 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T16:52:54.655 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T16:52:54.655 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T16:52:54.655 INFO:teuthology.orchestra.run.smithi134.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T16:52:54.655 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T16:52:54.655 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:52:54.655 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T16:52:54.655 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:52:54.655 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T16:52:54.656 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:52:54.656 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T16:52:54.656 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T16:52:54.656 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:52:54.656 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 1, 2024-08-18T16:52:54.656 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:52:54.656 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T16:52:54.656 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T16:52:54.656 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:52:54.656 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:52:54.656 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:52:54.656 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T16:52:54.656 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:52:54.657 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T16:52:54.657 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T16:52:54.657 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:52:54.657 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:52:54.657 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:52:54.657 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T16:52:54.657 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T16:52:54.657 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:52:54.657 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:52:54.657 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:52:54.657 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T16:52:54.657 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:52:54.657 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T16:52:54.657 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T16:52:54.658 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:52:54.658 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:52:54.658 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:52:54.658 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T16:52:54.658 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T16:52:54.658 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:52:54.658 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:52:54.658 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:52:54.658 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T16:52:54.658 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:52:54.658 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T16:52:54.658 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723999974.333303_1250.1", 2024-08-18T16:52:54.658 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:52:54.659 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:52:54.659 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:52:54.659 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:52:54.333303Z", 2024-08-18T16:52:54.659 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T16:52:54.659 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:52:54.659 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:52:54.659 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T16:52:54.659 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:52:54.659 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T16:52:54.659 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 1, 2024-08-18T16:52:54.659 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 1 2024-08-18T16:52:54.659 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:52:54.659 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T16:52:54Z" 2024-08-18T16:52:54.660 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T16:52:54.660 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-08-18T16:52:54.669 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-81', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:52:54.669 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-81', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:52:54.669 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-81 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T16:52:54.725 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:52:54.725 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:52:54.760 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:52:54.760+0000 7f712260f880 1 waiting to reach incremental sync.. 2024-08-18T16:52:59.762 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:52:59.761+0000 7f712260f880 1 bucket sync caught up with source: 2024-08-18T16:52:59.763 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000001.100.6, , , , , ] 2024-08-18T16:52:59.763 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000001.100.6, , , , , ] 2024-08-18T16:52:59.763 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:52:59.761+0000 7f712260f880 0 bucket checkpoint complete 2024-08-18T16:52:59.782 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-81', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:52:59.782 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-81', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:52:59.782 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-81 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:52:59.837 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:52:59.838 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:52:59.874 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:52:59.873+0000 7faddd411880 1 bucket sync caught up with source: 2024-08-18T16:52:59.874 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , 00000000001.100.6, , , , , ] 2024-08-18T16:52:59.874 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , 00000000001.100.6, , , , , ] 2024-08-18T16:52:59.874 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:52:59.873+0000 7faddd411880 0 bucket checkpoint complete 2024-08-18T16:52:59.888 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-81', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:52:59.888 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-81', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:52:59.888 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-81 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:52:59.942 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:52:59.942 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:52:59.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:52:59.976+0000 7f3aa8920880 1 waiting to reach incremental sync.. 2024-08-18T16:53:04.978 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:53:04.977+0000 7f3aa8920880 1 waiting to reach incremental sync.. 2024-08-18T16:53:09.978 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:53:09.977+0000 7f3aa8920880 1 waiting to reach incremental sync.. 2024-08-18T16:53:14.980 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:53:14.979+0000 7f3aa8920880 1 bucket sync caught up with source: 2024-08-18T16:53:14.980 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , 00000000001.337.6, , , , , ] 2024-08-18T16:53:14.980 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , 00000000001.337.6, , , , , ] 2024-08-18T16:53:14.980 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:53:14.979+0000 7f3aa8920880 0 bucket checkpoint complete 2024-08-18T16:53:14.997 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-81', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:53:14.997 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-81', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:53:14.997 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-81 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:53:15.054 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:53:15.054 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:53:15.092 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:53:15.091+0000 7fcd7b40f880 1 bucket sync caught up with source: 2024-08-18T16:53:15.092 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , 00000000001.337.6, , , , , ] 2024-08-18T16:53:15.092 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , 00000000001.337.6, , , , , ] 2024-08-18T16:53:15.092 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:53:15.091+0000 7fcd7b40f880 0 bucket checkpoint complete 2024-08-18T16:53:15.107 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-81', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:53:15.107 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-81', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:53:15.108 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-81 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:53:15.162 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:53:15.162 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:53:15.197 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:53:15.196+0000 7f0f8a419880 1 bucket sync caught up with source: 2024-08-18T16:53:15.197 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , 00000000001.516.6, , , , , ] 2024-08-18T16:53:15.197 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , 00000000001.516.6, , , , , ] 2024-08-18T16:53:15.197 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:53:15.196+0000 7f0f8a419880 0 bucket checkpoint complete 2024-08-18T16:53:15.212 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-81', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:53:15.212 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-81', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:53:15.213 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-81 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T16:53:15.274 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:53:15.274 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:53:15.313 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:53:15.313+0000 7f4a08610880 1 bucket sync caught up with source: 2024-08-18T16:53:15.314 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000001.516.6, , , , , ] 2024-08-18T16:53:15.314 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000001.516.6, , , , , ] 2024-08-18T16:53:15.314 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:53:15.313+0000 7f4a08610880 0 bucket checkpoint complete 2024-08-18T16:53:15.329 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-81 zones={test-zone2, test-zone1} 2024-08-18T16:53:15.338 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:53:15.341 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-08-18T16:53:15.341 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:53:15.344 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-08-18T16:53:15.348 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-08-18T16:53:15.355 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-08-18T16:53:15.415 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-08-18T16:53:15.463 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-81 zones={test-zone2, test-zone1} 2024-08-18T16:53:15.463 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-81 zones={test-zone3, test-zone1} 2024-08-18T16:53:15.469 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:53:15.472 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-08-18T16:53:15.472 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:53:15.474 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-08-18T16:53:15.478 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-08-18T16:53:15.484 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-08-18T16:53:15.536 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-08-18T16:53:15.635 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-81 zones={test-zone3, test-zone1} 2024-08-18T16:53:15.635 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-81 zones={test-zone3, test-zone2} 2024-08-18T16:53:15.639 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T16:53:15.642 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-08-18T16:53:15.642 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T16:53:15.645 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-08-18T16:53:15.650 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-08-18T16:53:15.656 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-08-18T16:53:15.709 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-08-18T16:53:15.765 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-81 zones={test-zone3, test-zone2} 2024-08-18T16:53:15.766 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2024-08-18T16:53:15.766 INFO:teuthology.orchestra.run:waiting for 300 2024-08-18T16:53:15.767 INFO:tasks.rgw.c2.client.0.smithi193.stdout:2024-08-18T16:53:15.766+0000 7ff534fec640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 83443) UID: 0 2024-08-18T16:53:15.767 INFO:tasks.rgw.c2.client.0.smithi193.stdout:2024-08-18T16:53:15.766+0000 7ff53b412880 -1 shutting down 2024-08-18T16:53:19.013 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2024-08-18T16:53:21.866 INFO:tasks.rgw.c2.client.0:Stopped 2024-08-18T16:53:21.945 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:53:21.945 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'init', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:53:21.946 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync init --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-08-18T16:53:22.005 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:53:22.005 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:53:22.043 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:53:22.042+0000 7f0d44015880 5 RGW-SYNC:data: waiting on data sync status lease 2024-08-18T16:53:22.044 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:53:22.043+0000 7f0d44015880 5 RGW-SYNC:data: acquired data sync status lease 2024-08-18T16:53:22.069 INFO:tasks.rgw.c2.client.0:Restarting daemon 2024-08-18T16:53:22.070 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2024-08-18T16:53:22.072 INFO:tasks.rgw.c2.client.0:Started 2024-08-18T16:53:22.073 DEBUG:teuthology.orchestra.run.smithi193:> curl http://smithi193.front.sepia.ceph.com:8000 2024-08-18T16:53:22.094 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-08-18T16:53:22.094 INFO:teuthology.orchestra.run.smithi193.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-08-18T16:53:22.094 INFO:teuthology.orchestra.run.smithi193.stderr: Dload Upload Total Spent Left Speed 2024-08-18T16:53:22.095 INFO:teuthology.orchestra.run.smithi193.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2024-08-18T16:53:22.095 INFO:teuthology.orchestra.run.smithi193.stderr:curl: (7) Failed to connect to smithi193.front.sepia.ceph.com port 8000: Connection refused 2024-08-18T16:53:23.096 DEBUG:teuthology.orchestra.run.smithi193:> curl http://smithi193.front.sepia.ceph.com:8000 2024-08-18T16:53:23.117 INFO:teuthology.orchestra.run.smithi193.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-08-18T16:53:23.118 INFO:teuthology.orchestra.run.smithi193.stderr: Dload Upload Total Spent Left Speed 2024-08-18T16:53:23.119 INFO:teuthology.orchestra.run.smithi193.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-08-18T16:53:23.119 INFO:teuthology.orchestra.run.smithi193.stdout:anonymous 2024-08-18T16:53:23.119 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-81', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:53:23.120 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-81', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:53:23.120 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-81 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T16:53:23.213 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:53:23.213 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:53:23.251 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:53:23.250+0000 7f407901f880 1 waiting for incremental sync to catch up: 2024-08-18T16:53:23.251 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000001.100.6, , , , , ] 2024-08-18T16:53:23.251 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000001.100.6, , , 00000000001.133.6, 00000000002.150.6, 00000000002.146.6] 2024-08-18T16:53:24.718 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2024-08-18T16:53:28.252 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:53:28.250+0000 7f407901f880 1 waiting for incremental sync to catch up: 2024-08-18T16:53:28.253 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000001.100.6, , , , , ] 2024-08-18T16:53:28.253 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000001.100.6, , , 00000000001.133.6, 00000000002.150.6, 00000000002.146.6] 2024-08-18T16:53:33.254 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:53:33.252+0000 7f407901f880 1 waiting for incremental sync to catch up: 2024-08-18T16:53:33.254 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000001.100.6, , , , , ] 2024-08-18T16:53:33.254 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000001.100.6, , , 00000000001.133.6, 00000000002.150.6, 00000000002.146.6] 2024-08-18T16:53:38.256 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:53:38.254+0000 7f407901f880 1 waiting for incremental sync to catch up: 2024-08-18T16:53:38.256 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000001.100.6, , , , , ] 2024-08-18T16:53:38.256 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000001.100.6, , , 00000000001.133.6, 00000000002.150.6, 00000000002.146.6] 2024-08-18T16:53:43.256 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:53:43.256+0000 7f407901f880 1 waiting for incremental sync to catch up: 2024-08-18T16:53:43.257 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000001.100.6, , , , , ] 2024-08-18T16:53:43.257 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000001.100.6, , , 00000000001.133.6, 00000000002.150.6, 00000000002.146.6] 2024-08-18T16:53:48.258 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:53:48.257+0000 7f407901f880 1 waiting for incremental sync to catch up: 2024-08-18T16:53:48.258 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000001.100.6, , , , , ] 2024-08-18T16:53:48.258 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000001.100.6, , , 00000000001.133.6, 00000000002.150.6, 00000000002.146.6] 2024-08-18T16:53:53.259 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:53:53.258+0000 7f407901f880 1 waiting for incremental sync to catch up: 2024-08-18T16:53:53.259 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000001.100.6, , , , , ] 2024-08-18T16:53:53.260 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000001.100.6, , , 00000000001.133.6, 00000000002.150.6, 00000000002.146.6] 2024-08-18T16:53:58.261 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:53:58.259+0000 7f407901f880 1 waiting for incremental sync to catch up: 2024-08-18T16:53:58.261 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000001.100.6, , , , , ] 2024-08-18T16:53:58.261 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000001.100.6, , , 00000000001.133.6, 00000000002.150.6, 00000000002.146.6] 2024-08-18T16:54:03.261 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:54:03.260+0000 7f407901f880 1 waiting for incremental sync to catch up: 2024-08-18T16:54:03.261 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000001.100.6, , , , , ] 2024-08-18T16:54:03.261 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000001.100.6, , , 00000000001.133.6, 00000000002.150.6, 00000000002.146.6] 2024-08-18T16:54:08.262 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:54:08.261+0000 7f407901f880 1 waiting for incremental sync to catch up: 2024-08-18T16:54:08.262 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000001.100.6, , , , , ] 2024-08-18T16:54:08.262 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000001.100.6, , , 00000000001.133.6, 00000000002.150.6, 00000000002.146.6] 2024-08-18T16:54:13.263 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:54:13.262+0000 7f407901f880 1 waiting for incremental sync to catch up: 2024-08-18T16:54:13.263 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000001.100.6, , , , , ] 2024-08-18T16:54:13.264 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000001.100.6, , , 00000000001.133.6, 00000000002.150.6, 00000000002.146.6] 2024-08-18T16:54:18.265 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:54:18.264+0000 7f407901f880 1 waiting for incremental sync to catch up: 2024-08-18T16:54:18.265 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000001.100.6, , , , , ] 2024-08-18T16:54:18.265 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000001.100.6, , , 00000000001.133.6, 00000000002.150.6, 00000000002.146.6] 2024-08-18T16:54:23.266 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:54:23.265+0000 7f407901f880 1 waiting for incremental sync to catch up: 2024-08-18T16:54:23.266 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000001.100.6, , , , , ] 2024-08-18T16:54:23.266 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000001.100.6, , , 00000000001.133.6, 00000000002.150.6, 00000000002.146.6] 2024-08-18T16:54:28.267 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:54:28.266+0000 7f407901f880 1 waiting for incremental sync to catch up: 2024-08-18T16:54:28.267 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000001.100.6, , , , , ] 2024-08-18T16:54:28.267 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000001.100.6, , , 00000000001.133.6, 00000000002.150.6, 00000000002.146.6] 2024-08-18T16:54:33.268 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:54:33.266+0000 7f407901f880 1 waiting for incremental sync to catch up: 2024-08-18T16:54:33.268 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000001.100.6, , , , , ] 2024-08-18T16:54:33.269 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000001.100.6, , , 00000000001.133.6, 00000000002.150.6, 00000000002.146.6] 2024-08-18T16:54:38.270 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:54:38.269+0000 7f407901f880 1 waiting for incremental sync to catch up: 2024-08-18T16:54:38.270 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000001.100.6, , , , , ] 2024-08-18T16:54:38.270 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000001.100.6, , , 00000000001.133.6, 00000000002.150.6, 00000000002.146.6] 2024-08-18T16:54:43.271 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:54:43.270+0000 7f407901f880 1 waiting for incremental sync to catch up: 2024-08-18T16:54:43.271 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000001.100.6, , , , , ] 2024-08-18T16:54:43.271 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000001.100.6, , , 00000000001.133.6, 00000000002.150.6, 00000000002.146.6] 2024-08-18T16:54:48.272 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:54:48.271+0000 7f407901f880 1 waiting for incremental sync to catch up: 2024-08-18T16:54:48.272 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000001.100.6, , , , , ] 2024-08-18T16:54:48.273 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000001.100.6, , , 00000000001.133.6, 00000000002.150.6, 00000000002.146.6] 2024-08-18T16:54:53.274 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:54:53.273+0000 7f407901f880 1 waiting for incremental sync to catch up: 2024-08-18T16:54:53.275 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000001.100.6, , , , , ] 2024-08-18T16:54:53.275 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000001.100.6, , , 00000000001.133.6, 00000000002.150.6, 00000000002.146.6] 2024-08-18T16:54:58.276 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:54:58.275+0000 7f407901f880 1 waiting for incremental sync to catch up: 2024-08-18T16:54:58.276 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000001.100.6, , , , , ] 2024-08-18T16:54:58.276 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000001.100.6, , , 00000000001.133.6, 00000000002.150.6, 00000000002.146.6] 2024-08-18T16:55:03.277 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:55:03.276+0000 7f407901f880 1 waiting for incremental sync to catch up: 2024-08-18T16:55:03.277 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000001.100.6, , , , , ] 2024-08-18T16:55:03.277 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000001.100.6, , , 00000000001.133.6, 00000000002.150.6, 00000000002.146.6] 2024-08-18T16:55:08.278 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:55:08.277+0000 7f407901f880 1 waiting for incremental sync to catch up: 2024-08-18T16:55:08.279 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000001.100.6, , , , , ] 2024-08-18T16:55:08.279 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000001.100.6, , , 00000000001.133.6, 00000000002.150.6, 00000000002.146.6] 2024-08-18T16:55:13.279 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:55:13.278+0000 7f407901f880 1 waiting for incremental sync to catch up: 2024-08-18T16:55:13.280 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000001.100.6, , , , , ] 2024-08-18T16:55:13.280 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000001.100.6, , , 00000000001.133.6, 00000000002.150.6, 00000000002.146.6] 2024-08-18T16:55:18.281 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:55:18.280+0000 7f407901f880 1 waiting for incremental sync to catch up: 2024-08-18T16:55:18.281 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000001.100.6, , , , , ] 2024-08-18T16:55:18.281 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000001.100.6, , , 00000000001.133.6, 00000000002.150.6, 00000000002.146.6] 2024-08-18T16:55:23.282 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:55:23.281+0000 7f407901f880 1 waiting for incremental sync to catch up: 2024-08-18T16:55:23.282 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000001.100.6, , , , , ] 2024-08-18T16:55:23.282 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000001.100.6, , , 00000000001.133.6, 00000000002.150.6, 00000000002.146.6] 2024-08-18T16:55:28.283 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:55:28.282+0000 7f407901f880 1 waiting for incremental sync to catch up: 2024-08-18T16:55:28.283 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000001.100.6, , , , 00000000002.150.6, 00000000001.144.6] 2024-08-18T16:55:28.284 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000001.100.6, , , 00000000001.133.6, 00000000002.150.6, 00000000002.146.6] 2024-08-18T16:55:33.284 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:55:33.283+0000 7f407901f880 1 waiting for incremental sync to catch up: 2024-08-18T16:55:33.285 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000001.100.6, , , , 00000000002.150.6, 00000000001.144.6] 2024-08-18T16:55:33.285 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000001.100.6, , , 00000000001.133.6, 00000000002.150.6, 00000000002.146.6] 2024-08-18T16:55:38.286 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:55:38.284+0000 7f407901f880 1 waiting for incremental sync to catch up: 2024-08-18T16:55:38.286 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000001.100.6, , , , 00000000002.150.6, 00000000001.144.6] 2024-08-18T16:55:38.286 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000001.100.6, , , 00000000001.133.6, 00000000002.150.6, 00000000002.146.6] 2024-08-18T16:55:43.287 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:55:43.286+0000 7f407901f880 1 waiting for incremental sync to catch up: 2024-08-18T16:55:43.287 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000001.100.6, , , , 00000000002.150.6, 00000000001.144.6] 2024-08-18T16:55:43.287 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000001.100.6, , , 00000000001.133.6, 00000000002.150.6, 00000000002.146.6] 2024-08-18T16:55:48.288 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:55:48.287+0000 7f407901f880 1 waiting for incremental sync to catch up: 2024-08-18T16:55:48.288 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000001.100.6, , , , 00000000002.150.6, 00000000001.144.6] 2024-08-18T16:55:48.289 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000001.100.6, , , 00000000001.133.6, 00000000002.150.6, 00000000002.146.6] 2024-08-18T16:55:53.290 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:55:53.289+0000 7f407901f880 1 waiting for incremental sync to catch up: 2024-08-18T16:55:53.290 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000001.100.6, , , , 00000000002.150.6, 00000000001.144.6] 2024-08-18T16:55:53.290 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000001.100.6, , , 00000000001.133.6, 00000000002.150.6, 00000000002.146.6] 2024-08-18T16:55:58.291 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:55:58.290+0000 7f407901f880 1 waiting for incremental sync to catch up: 2024-08-18T16:55:58.291 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000001.100.6, , , , 00000000002.150.6, 00000000001.144.6] 2024-08-18T16:55:58.291 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000001.100.6, , , 00000000001.133.6, 00000000002.150.6, 00000000002.146.6] 2024-08-18T16:56:03.292 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:56:03.291+0000 7f407901f880 1 waiting for incremental sync to catch up: 2024-08-18T16:56:03.292 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000001.100.6, , , , 00000000002.150.6, 00000000001.144.6] 2024-08-18T16:56:03.292 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000001.100.6, , , 00000000001.133.6, 00000000002.150.6, 00000000002.146.6] 2024-08-18T16:56:08.293 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:56:08.293+0000 7f407901f880 1 waiting for incremental sync to catch up: 2024-08-18T16:56:08.294 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000001.100.6, , , , 00000000002.150.6, 00000000001.144.6] 2024-08-18T16:56:08.294 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000001.100.6, , , 00000000001.133.6, 00000000002.150.6, 00000000002.146.6] 2024-08-18T16:56:13.295 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:56:13.294+0000 7f407901f880 1 waiting for incremental sync to catch up: 2024-08-18T16:56:13.295 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000001.100.6, , , , 00000000002.150.6, 00000000001.144.6] 2024-08-18T16:56:13.295 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000001.100.6, , , 00000000001.133.6, 00000000002.150.6, 00000000002.146.6] 2024-08-18T16:56:18.297 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:56:18.295+0000 7f407901f880 1 waiting for incremental sync to catch up: 2024-08-18T16:56:18.297 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000001.100.6, , , , 00000000002.150.6, 00000000001.144.6] 2024-08-18T16:56:18.297 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000001.100.6, , , 00000000001.133.6, 00000000002.150.6, 00000000002.146.6] 2024-08-18T16:56:23.298 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:56:23.297+0000 7f407901f880 1 waiting for incremental sync to catch up: 2024-08-18T16:56:23.298 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000001.100.6, , , , 00000000002.150.6, 00000000002.146.6] 2024-08-18T16:56:23.298 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000001.100.6, , , 00000000001.133.6, 00000000002.150.6, 00000000002.146.6] 2024-08-18T16:56:28.302 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:56:28.298+0000 7f407901f880 1 waiting for incremental sync to catch up: 2024-08-18T16:56:28.302 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000001.100.6, , , , 00000000002.150.6, 00000000002.146.6] 2024-08-18T16:56:28.303 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000001.100.6, , , 00000000001.133.6, 00000000002.150.6, 00000000002.146.6] 2024-08-18T16:56:33.300 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:56:33.299+0000 7f407901f880 1 waiting for incremental sync to catch up: 2024-08-18T16:56:33.301 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000001.100.6, , , , 00000000002.150.6, 00000000002.146.6] 2024-08-18T16:56:33.301 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000001.100.6, , , 00000000001.133.6, 00000000002.150.6, 00000000002.146.6] 2024-08-18T16:56:38.302 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:56:38.300+0000 7f407901f880 1 waiting for incremental sync to catch up: 2024-08-18T16:56:38.302 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000001.100.6, , , , 00000000002.150.6, 00000000002.146.6] 2024-08-18T16:56:38.302 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000001.100.6, , , 00000000001.133.6, 00000000002.150.6, 00000000002.146.6] 2024-08-18T16:56:43.303 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:56:43.301+0000 7f407901f880 1 waiting for incremental sync to catch up: 2024-08-18T16:56:43.303 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000001.100.6, , , , 00000000002.150.6, 00000000002.146.6] 2024-08-18T16:56:43.303 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000001.100.6, , , 00000000001.133.6, 00000000002.150.6, 00000000002.146.6] 2024-08-18T16:56:48.305 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:56:48.304+0000 7f407901f880 1 waiting for incremental sync to catch up: 2024-08-18T16:56:48.305 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000001.100.6, , , , 00000000002.150.6, 00000000002.146.6] 2024-08-18T16:56:48.305 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000001.100.6, , , 00000000001.133.6, 00000000002.150.6, 00000000002.146.6] 2024-08-18T16:56:53.306 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:56:53.305+0000 7f407901f880 1 waiting for incremental sync to catch up: 2024-08-18T16:56:53.306 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000001.100.6, , , , 00000000002.150.6, 00000000002.146.6] 2024-08-18T16:56:53.306 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000001.100.6, , , 00000000001.133.6, 00000000002.150.6, 00000000002.146.6] 2024-08-18T16:56:58.307 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:56:58.306+0000 7f407901f880 1 waiting for incremental sync to catch up: 2024-08-18T16:56:58.307 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000001.100.6, , , , 00000000002.150.6, 00000000002.146.6] 2024-08-18T16:56:58.307 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000001.100.6, , , 00000000001.133.6, 00000000002.150.6, 00000000002.146.6] 2024-08-18T16:57:03.308 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:57:03.307+0000 7f407901f880 1 waiting for incremental sync to catch up: 2024-08-18T16:57:03.308 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000001.100.6, , , , 00000000002.150.6, 00000000002.146.6] 2024-08-18T16:57:03.308 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000001.100.6, , , 00000000001.133.6, 00000000002.150.6, 00000000002.146.6] 2024-08-18T16:57:08.309 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:57:08.308+0000 7f407901f880 1 waiting for incremental sync to catch up: 2024-08-18T16:57:08.310 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000001.100.6, , , , 00000000002.150.6, 00000000002.146.6] 2024-08-18T16:57:08.310 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000001.100.6, , , 00000000001.133.6, 00000000002.150.6, 00000000002.146.6] 2024-08-18T16:57:13.310 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:57:13.309+0000 7f407901f880 1 waiting for incremental sync to catch up: 2024-08-18T16:57:13.310 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000001.100.6, , , , 00000000002.150.6, 00000000002.146.6] 2024-08-18T16:57:13.311 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000001.100.6, , , 00000000001.133.6, 00000000002.150.6, 00000000002.146.6] 2024-08-18T16:57:18.312 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:57:18.311+0000 7f407901f880 1 waiting for incremental sync to catch up: 2024-08-18T16:57:18.312 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000001.100.6, , , , 00000000002.150.6, 00000000002.146.6] 2024-08-18T16:57:18.312 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000001.100.6, , , 00000000001.133.6, 00000000002.150.6, 00000000002.146.6] 2024-08-18T16:57:23.313 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:57:23.312+0000 7f407901f880 1 waiting for incremental sync to catch up: 2024-08-18T16:57:23.313 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000001.100.6, , , , 00000000002.150.6, 00000000002.146.6] 2024-08-18T16:57:23.313 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000001.100.6, , , 00000000001.133.6, 00000000002.150.6, 00000000002.146.6] 2024-08-18T16:57:28.314 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:57:28.313+0000 7f407901f880 1 waiting for incremental sync to catch up: 2024-08-18T16:57:28.314 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000001.100.6, , , , 00000000002.150.6, 00000000002.146.6] 2024-08-18T16:57:28.314 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000001.100.6, , , 00000000001.133.6, 00000000002.150.6, 00000000002.146.6] 2024-08-18T16:57:33.315 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:57:33.314+0000 7f407901f880 1 waiting for incremental sync to catch up: 2024-08-18T16:57:33.316 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000001.100.6, , , , 00000000002.150.6, 00000000002.146.6] 2024-08-18T16:57:33.316 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000001.100.6, , , 00000000001.133.6, 00000000002.150.6, 00000000002.146.6] 2024-08-18T16:57:38.317 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:57:38.316+0000 7f407901f880 1 waiting for incremental sync to catch up: 2024-08-18T16:57:38.317 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000001.100.6, , , , 00000000002.150.6, 00000000002.146.6] 2024-08-18T16:57:38.317 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000001.100.6, , , 00000000001.133.6, 00000000002.150.6, 00000000002.146.6] 2024-08-18T16:57:43.318 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:57:43.316+0000 7f407901f880 1 waiting for incremental sync to catch up: 2024-08-18T16:57:43.318 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000001.100.6, , , , 00000000002.150.6, 00000000002.146.6] 2024-08-18T16:57:43.318 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000001.100.6, , , 00000000001.133.6, 00000000002.150.6, 00000000002.146.6] 2024-08-18T16:57:48.319 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:57:48.318+0000 7f407901f880 1 waiting for incremental sync to catch up: 2024-08-18T16:57:48.319 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000001.100.6, , , , 00000000002.150.6, 00000000002.146.6] 2024-08-18T16:57:48.319 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000001.100.6, , , 00000000001.133.6, 00000000002.150.6, 00000000002.146.6] 2024-08-18T16:57:53.320 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:57:53.319+0000 7f407901f880 1 waiting for incremental sync to catch up: 2024-08-18T16:57:53.321 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000001.100.6, , , , 00000000002.150.6, 00000000002.146.6] 2024-08-18T16:57:53.321 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000001.100.6, , , 00000000001.133.6, 00000000002.150.6, 00000000002.146.6] 2024-08-18T16:57:58.322 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:57:58.321+0000 7f407901f880 1 waiting for incremental sync to catch up: 2024-08-18T16:57:58.322 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000001.100.6, , , , 00000000002.150.6, 00000000002.146.6] 2024-08-18T16:57:58.322 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000001.100.6, , , 00000000001.133.6, 00000000002.150.6, 00000000002.146.6] 2024-08-18T16:58:03.323 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:58:03.322+0000 7f407901f880 1 waiting for incremental sync to catch up: 2024-08-18T16:58:03.323 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000001.100.6, , , , 00000000002.150.6, 00000000002.146.6] 2024-08-18T16:58:03.323 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000001.100.6, , , 00000000001.133.6, 00000000002.150.6, 00000000002.146.6] 2024-08-18T16:58:08.324 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:58:08.323+0000 7f407901f880 1 waiting for incremental sync to catch up: 2024-08-18T16:58:08.324 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000001.100.6, , , , 00000000002.150.6, 00000000002.146.6] 2024-08-18T16:58:08.324 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000001.100.6, , , 00000000001.133.6, 00000000002.150.6, 00000000002.146.6] 2024-08-18T16:58:13.325 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:58:13.324+0000 7f407901f880 1 waiting for incremental sync to catch up: 2024-08-18T16:58:13.325 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , 00000000001.100.6, , , , 00000000002.150.6, 00000000002.146.6] 2024-08-18T16:58:13.326 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , 00000000001.100.6, , , 00000000001.133.6, 00000000002.150.6, 00000000002.146.6] 2024-08-18T16:58:18.326 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:58:18.325+0000 7f407901f880 0 bucket checkpoint timed out waiting for incremental sync to catch up 2024-08-18T16:58:18.326 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:58:18.325+0000 7f407901f880 0 bucket sync checkpoint failed: (110) Connection timed out 2024-08-18T16:58:18.326 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:58:18.325+0000 7f407901f880 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-08-18T16:58:18.344 DEBUG:teuthology.orchestra.run:got remote process result: 110 2024-08-18T16:58:18.346 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_incremental_sync_after_data_sync_init ... FAIL 2024-08-18T16:58:18.370 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:58:18.371 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:58:18.371 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:58:18.422 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:58:18.422 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:58:18.458 INFO:teuthology.orchestra.run.smithi134.stderr:No --period given, using current period=01050e30-179b-43be-bdbb-3bb23cf37ae8 2024-08-18T16:58:18.472 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T16:58:18.472 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:58:18.472 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T16:58:18.473 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T16:58:18.473 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:58:18.473 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:58:18.473 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T16:58:18.473 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T16:58:18.473 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:58:18.473 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:58:18.473 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T16:58:18.473 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:42:47.672461Z" 2024-08-18T16:58:18.473 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:58:18.473 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:58:18.473 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1724000298.368251_1274.1", 2024-08-18T16:58:18.474 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:58:18.368251Z" 2024-08-18T16:58:18.474 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:58:18.474 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T16:58:18.475 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-08-18T16:58:18.475 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:58:18.475 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:58:18.475 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:58:18.530 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:58:18.530 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:58:18.564 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:58:18.563+0000 7f442de35880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:58:18.565 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:58:18.564+0000 7f442de35880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:58:18.579 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T16:58:18.580 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T16:58:18.580 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T16:58:18.580 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T16:58:18.580 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T16:58:18.580 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T16:58:18.580 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T16:58:18.580 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:58:18.580 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T16:58:18.580 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:58:18.580 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T16:58:18.580 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:58:18.580 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:58:18.581 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T16:58:18.581 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:58:18.581 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T16:58:18.581 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:58:18.581 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T16:58:18.581 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T16:58:18.581 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:58:18.581 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:58:18.581 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:58:18.581 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T16:58:18.581 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:58:18.581 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:58:18.582 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T16:58:18.582 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:58:18.582 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:58:18.582 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:58:18.582 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T16:58:18.582 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T16:58:18.582 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:58:18.582 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:58:18.582 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:58:18.583 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T16:58:18.583 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:58:18.583 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:58:18.583 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T16:58:18.583 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:58:18.583 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:58:18.583 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:58:18.583 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T16:58:18.583 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T16:58:18.583 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:58:18.583 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:58:18.583 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:58:18.583 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T16:58:18.584 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:58:18.584 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:58:18.584 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724000298.360801_1271.1", 2024-08-18T16:58:18.584 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:58:18.584 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:58:18.584 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:58:18.584 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:58:18.360801Z", 2024-08-18T16:58:18.584 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T16:58:18.584 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:58:18.584 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:58:18.584 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T16:58:18.584 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:58:18.584 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T16:58:18.585 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T16:58:18.585 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T16:58:18.585 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:58:18.585 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T16:58:18Z" 2024-08-18T16:58:18.585 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T16:58:18.586 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724000298.368251_1274.1 target=1_1724000298.360801_1271.1 2024-08-18T16:58:23.587 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:58:23.587 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:58:23.587 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:58:23.642 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:58:23.642 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:58:23.676 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:58:23.675+0000 7f113b538880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:58:23.677 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:58:23.676+0000 7f113b538880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:58:23.691 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T16:58:23.692 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T16:58:23.692 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T16:58:23.692 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T16:58:23.692 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T16:58:23.692 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T16:58:23.692 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T16:58:23.692 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:58:23.692 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T16:58:23.692 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:58:23.692 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T16:58:23.692 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:58:23.693 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:58:23.693 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T16:58:23.693 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:58:23.693 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T16:58:23.693 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:58:23.693 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T16:58:23.693 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T16:58:23.693 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:58:23.693 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:58:23.693 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:58:23.693 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T16:58:23.693 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:58:23.693 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:58:23.694 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T16:58:23.694 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:58:23.694 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:58:23.694 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:58:23.694 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T16:58:23.694 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T16:58:23.694 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:58:23.694 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:58:23.694 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:58:23.694 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T16:58:23.694 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:58:23.694 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:58:23.694 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T16:58:23.695 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:58:23.695 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:58:23.695 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:58:23.695 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T16:58:23.695 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T16:58:23.695 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:58:23.695 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:58:23.695 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T16:58:23.695 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T16:58:23.695 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T16:58:23.695 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T16:58:23.695 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724000298.368251_1274.1", 2024-08-18T16:58:23.695 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T16:58:23.696 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T16:58:23.696 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T16:58:23.696 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:58:18.368251Z", 2024-08-18T16:58:23.696 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T16:58:23.696 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:58:23.696 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T16:58:23.696 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T16:58:23.696 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:58:23.696 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T16:58:23.696 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T16:58:23.696 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T16:58:23.696 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T16:58:23.696 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T16:58:23Z" 2024-08-18T16:58:23.697 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T16:58:23.697 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-08-18T16:58:23.698 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-08-18T16:58:23.698 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:58:23.698 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T16:58:23.698 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T16:58:23.747 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:58:23.747 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:58:23.776 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:58:23.776+0000 7f1f63f36880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:58:23.777 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:58:23.776+0000 7f1f63f36880 20 RGW-SYNC:meta: read sync status 2024-08-18T16:58:23.791 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T16:58:23.791 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T16:58:23.791 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T16:58:23.791 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T16:58:23.791 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T16:58:23.791 INFO:teuthology.orchestra.run.smithi134.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T16:58:23.791 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T16:58:23.791 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:58:23.791 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T16:58:23.791 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:58:23.792 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T16:58:23.792 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:58:23.792 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T16:58:23.792 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T16:58:23.792 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:58:23.792 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 1, 2024-08-18T16:58:23.792 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:58:23.792 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T16:58:23.792 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T16:58:23.792 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:58:23.792 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:58:23.792 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:58:23.792 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T16:58:23.793 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:58:23.793 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T16:58:23.793 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T16:58:23.793 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:58:23.793 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:58:23.793 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:58:23.793 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T16:58:23.793 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T16:58:23.793 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:58:23.793 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:58:23.793 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:58:23.793 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T16:58:23.793 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:58:23.794 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T16:58:23.794 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T16:58:23.794 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:58:23.794 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:58:23.794 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:58:23.794 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T16:58:23.794 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T16:58:23.794 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:58:23.794 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:58:23.794 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T16:58:23.794 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T16:58:23.795 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T16:58:23.795 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T16:58:23.795 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1724000298.368251_1274.1", 2024-08-18T16:58:23.795 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T16:58:23.795 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T16:58:23.795 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T16:58:23.795 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:58:18.368251Z", 2024-08-18T16:58:23.795 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T16:58:23.795 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:58:23.795 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T16:58:23.795 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T16:58:23.795 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:58:23.796 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T16:58:23.796 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 1, 2024-08-18T16:58:23.796 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 1 2024-08-18T16:58:23.796 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T16:58:23.796 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T16:58:23Z" 2024-08-18T16:58:23.796 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T16:58:23.797 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-08-18T16:58:23.919 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-82', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:58:23.919 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-82', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:58:23.919 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-82 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:58:23.970 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:58:23.970 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:58:24.013 INFO:teuthology.orchestra.run.smithi134.stdout:tenant: 2024-08-18T16:58:24.013 INFO:teuthology.orchestra.run.smithi134.stdout:bucket name: kibgmb-82 2024-08-18T16:58:24.016 INFO:teuthology.orchestra.run.smithi134.stdout:total entries: 5 2024-08-18T16:58:24.130 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-82', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:58:24.130 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-82', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:58:24.130 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-82 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:58:24.180 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:58:24.180 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:58:24.225 INFO:teuthology.orchestra.run.smithi134.stdout:tenant: 2024-08-18T16:58:24.225 INFO:teuthology.orchestra.run.smithi134.stdout:bucket name: kibgmb-82 2024-08-18T16:58:24.230 INFO:teuthology.orchestra.run.smithi134.stdout:total entries: 10 2024-08-18T16:58:24.388 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-82', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:58:24.388 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-82', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:58:24.388 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-82 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:58:24.437 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:58:24.437 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:58:24.482 INFO:teuthology.orchestra.run.smithi134.stdout:tenant: 2024-08-18T16:58:24.482 INFO:teuthology.orchestra.run.smithi134.stdout:bucket name: kibgmb-82 2024-08-18T16:58:24.488 INFO:teuthology.orchestra.run.smithi134.stdout:total entries: 15 2024-08-18T16:58:24.651 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-82', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:58:24.651 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-82', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:58:24.651 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-82 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:58:24.701 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:58:24.701 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:58:24.749 INFO:teuthology.orchestra.run.smithi134.stdout:tenant: 2024-08-18T16:58:24.749 INFO:teuthology.orchestra.run.smithi134.stdout:bucket name: kibgmb-82 2024-08-18T16:58:24.757 INFO:teuthology.orchestra.run.smithi134.stdout:total entries: 20 2024-08-18T16:58:24.887 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-82', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:58:24.887 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-82', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:58:24.888 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-82 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:58:24.937 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:58:24.937 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:58:24.986 INFO:teuthology.orchestra.run.smithi134.stdout:tenant: 2024-08-18T16:58:24.986 INFO:teuthology.orchestra.run.smithi134.stdout:bucket name: kibgmb-82 2024-08-18T16:58:24.996 INFO:teuthology.orchestra.run.smithi134.stdout:total entries: 25 2024-08-18T16:58:25.171 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-82', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:58:25.171 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-82', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T16:58:25.171 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-82 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T16:58:25.223 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:58:25.223 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:58:25.272 INFO:teuthology.orchestra.run.smithi134.stdout:tenant: 2024-08-18T16:58:25.272 INFO:teuthology.orchestra.run.smithi134.stdout:bucket name: kibgmb-82 2024-08-18T16:58:25.284 INFO:teuthology.orchestra.run.smithi134.stdout:total entries: 30 2024-08-18T16:58:25.467 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-82', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:58:25.467 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-82', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:58:25.468 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-82 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T16:58:25.531 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:58:25.532 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:58:25.572 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:58:25.571+0000 7f0e3b132880 1 waiting to reach incremental sync.. 2024-08-18T16:58:30.576 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:58:30.575+0000 7f0e3b132880 1 bucket sync caught up with source: 2024-08-18T16:58:30.576 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , 00000000002.146.6, , , , , , 00000000002.94.6, , , , , 00000000002.138.6, , , , , , , , , , , , , , , 00000000001.96.6, , , , 00000000001.129.6] 2024-08-18T16:58:30.576 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , 00000000002.146.6, , , , , , 00000000002.94.6, , , , , 00000000002.138.6, , , , , , , , , , , , , , , 00000000001.96.6, , , , 00000000001.129.6] 2024-08-18T16:58:30.576 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T16:58:30.575+0000 7f0e3b132880 0 bucket checkpoint complete 2024-08-18T16:58:30.594 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-82', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:58:30.595 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-82', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:58:30.595 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-82 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:58:30.650 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:58:30.650 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:58:30.685 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:58:30.684+0000 7ffa8c929880 1 waiting to reach incremental sync.. 2024-08-18T16:58:35.687 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:58:35.685+0000 7ffa8c929880 1 waiting to reach incremental sync.. 2024-08-18T16:58:40.687 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:58:40.686+0000 7ffa8c929880 1 waiting to reach incremental sync.. 2024-08-18T16:58:45.688 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:58:45.686+0000 7ffa8c929880 1 waiting to reach incremental sync.. 2024-08-18T16:58:50.688 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:58:50.687+0000 7ffa8c929880 1 waiting to reach incremental sync.. 2024-08-18T16:58:55.689 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:58:55.688+0000 7ffa8c929880 1 waiting to reach incremental sync.. 2024-08-18T16:59:00.690 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:59:00.689+0000 7ffa8c929880 1 waiting to reach incremental sync.. 2024-08-18T16:59:05.690 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:59:05.689+0000 7ffa8c929880 1 waiting to reach incremental sync.. 2024-08-18T16:59:10.691 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:59:10.690+0000 7ffa8c929880 1 waiting to reach incremental sync.. 2024-08-18T16:59:15.692 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:59:15.691+0000 7ffa8c929880 1 waiting to reach incremental sync.. 2024-08-18T16:59:20.693 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:59:20.691+0000 7ffa8c929880 1 waiting to reach incremental sync.. 2024-08-18T16:59:25.693 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:59:25.692+0000 7ffa8c929880 1 waiting to reach incremental sync.. 2024-08-18T16:59:30.694 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:59:30.693+0000 7ffa8c929880 1 waiting to reach incremental sync.. 2024-08-18T16:59:35.695 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:59:35.694+0000 7ffa8c929880 1 waiting to reach incremental sync.. 2024-08-18T16:59:40.695 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:59:40.694+0000 7ffa8c929880 1 waiting to reach incremental sync.. 2024-08-18T16:59:45.699 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:59:45.698+0000 7ffa8c929880 1 bucket sync caught up with source: 2024-08-18T16:59:45.700 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , 00000000002.146.6, , , , , , 00000000002.94.6, , , , , 00000000002.138.6, , , , , , , , , , , , , , , 00000000001.96.6, , , , 00000000001.129.6] 2024-08-18T16:59:45.700 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , , 00000000002.146.6, , , , , , 00000000002.94.6, , , , , 00000000002.138.6, , , , , , , , , , , , , , , 00000000001.96.6, , , , 00000000001.129.6] 2024-08-18T16:59:45.700 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:59:45.698+0000 7ffa8c929880 0 bucket checkpoint complete 2024-08-18T16:59:45.723 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-82', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:59:45.723 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-82', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T16:59:45.724 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-82 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T16:59:45.783 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T16:59:45.783 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T16:59:45.823 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:59:45.822+0000 7f8be7919880 1 waiting to reach incremental sync.. 2024-08-18T16:59:50.824 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:59:50.822+0000 7f8be7919880 1 waiting to reach incremental sync.. 2024-08-18T16:59:55.825 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T16:59:55.823+0000 7f8be7919880 1 waiting to reach incremental sync.. 2024-08-18T17:00:00.825 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:00:00.824+0000 7f8be7919880 1 waiting to reach incremental sync.. 2024-08-18T17:00:05.826 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:00:05.825+0000 7f8be7919880 1 waiting to reach incremental sync.. 2024-08-18T17:00:10.827 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:00:10.826+0000 7f8be7919880 1 waiting to reach incremental sync.. 2024-08-18T17:00:15.827 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:00:15.826+0000 7f8be7919880 1 waiting to reach incremental sync.. 2024-08-18T17:00:20.828 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:00:20.827+0000 7f8be7919880 1 waiting to reach incremental sync.. 2024-08-18T17:00:25.828 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:00:25.827+0000 7f8be7919880 1 waiting to reach incremental sync.. 2024-08-18T17:00:30.829 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:00:30.828+0000 7f8be7919880 1 waiting to reach incremental sync.. 2024-08-18T17:00:35.830 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:00:35.829+0000 7f8be7919880 1 waiting to reach incremental sync.. 2024-08-18T17:00:40.831 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:00:40.830+0000 7f8be7919880 1 waiting to reach incremental sync.. 2024-08-18T17:00:45.832 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:00:45.831+0000 7f8be7919880 1 waiting to reach incremental sync.. 2024-08-18T17:00:50.832 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:00:50.831+0000 7f8be7919880 1 waiting to reach incremental sync.. 2024-08-18T17:00:55.833 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:00:55.832+0000 7f8be7919880 1 waiting to reach incremental sync.. 2024-08-18T17:01:00.834 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:01:00.832+0000 7f8be7919880 1 waiting to reach incremental sync.. 2024-08-18T17:01:05.834 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:01:05.833+0000 7f8be7919880 1 waiting to reach incremental sync.. 2024-08-18T17:01:10.835 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:01:10.834+0000 7f8be7919880 1 waiting to reach incremental sync.. 2024-08-18T17:01:15.835 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:01:15.834+0000 7f8be7919880 1 waiting to reach incremental sync.. 2024-08-18T17:01:20.836 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:01:20.835+0000 7f8be7919880 1 waiting to reach incremental sync.. 2024-08-18T17:01:25.837 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:01:25.836+0000 7f8be7919880 1 waiting to reach incremental sync.. 2024-08-18T17:01:30.838 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:01:30.836+0000 7f8be7919880 1 waiting to reach incremental sync.. 2024-08-18T17:01:35.838 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:01:35.837+0000 7f8be7919880 1 waiting to reach incremental sync.. 2024-08-18T17:01:40.839 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:01:40.838+0000 7f8be7919880 1 waiting to reach incremental sync.. 2024-08-18T17:01:45.840 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:01:45.839+0000 7f8be7919880 1 waiting to reach incremental sync.. 2024-08-18T17:01:50.840 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:01:50.839+0000 7f8be7919880 1 waiting to reach incremental sync.. 2024-08-18T17:01:55.841 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:01:55.840+0000 7f8be7919880 1 waiting to reach incremental sync.. 2024-08-18T17:02:00.842 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:02:00.841+0000 7f8be7919880 1 waiting to reach incremental sync.. 2024-08-18T17:02:05.842 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:02:05.841+0000 7f8be7919880 1 waiting to reach incremental sync.. 2024-08-18T17:02:10.843 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:02:10.842+0000 7f8be7919880 1 waiting to reach incremental sync.. 2024-08-18T17:02:15.844 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:02:15.842+0000 7f8be7919880 1 waiting to reach incremental sync.. 2024-08-18T17:02:20.844 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:02:20.843+0000 7f8be7919880 1 waiting to reach incremental sync.. 2024-08-18T17:02:25.845 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:02:25.844+0000 7f8be7919880 1 waiting to reach incremental sync.. 2024-08-18T17:02:30.846 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:02:30.844+0000 7f8be7919880 1 waiting to reach incremental sync.. 2024-08-18T17:02:35.846 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:02:35.845+0000 7f8be7919880 1 waiting to reach incremental sync.. 2024-08-18T17:02:40.846 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:02:40.846+0000 7f8be7919880 1 waiting to reach incremental sync.. 2024-08-18T17:02:45.847 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:02:45.846+0000 7f8be7919880 1 waiting to reach incremental sync.. 2024-08-18T17:02:50.848 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:02:50.847+0000 7f8be7919880 1 waiting to reach incremental sync.. 2024-08-18T17:02:55.848 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:02:55.847+0000 7f8be7919880 1 waiting to reach incremental sync.. 2024-08-18T17:03:00.849 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:03:00.848+0000 7f8be7919880 1 waiting to reach incremental sync.. 2024-08-18T17:03:05.849 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:03:05.848+0000 7f8be7919880 1 waiting to reach incremental sync.. 2024-08-18T17:03:10.850 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:03:10.849+0000 7f8be7919880 1 waiting to reach incremental sync.. 2024-08-18T17:03:15.852 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:03:15.851+0000 7f8be7919880 1 waiting to reach incremental sync.. 2024-08-18T17:03:20.854 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:03:20.853+0000 7f8be7919880 1 waiting to reach incremental sync.. 2024-08-18T17:03:25.855 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:03:25.853+0000 7f8be7919880 1 waiting to reach incremental sync.. 2024-08-18T17:03:30.855 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:03:30.854+0000 7f8be7919880 1 waiting to reach incremental sync.. 2024-08-18T17:03:35.856 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:03:35.855+0000 7f8be7919880 1 waiting to reach incremental sync.. 2024-08-18T17:03:40.856 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:03:40.855+0000 7f8be7919880 1 waiting to reach incremental sync.. 2024-08-18T17:03:45.857 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:03:45.856+0000 7f8be7919880 1 waiting to reach incremental sync.. 2024-08-18T17:03:50.857 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:03:50.856+0000 7f8be7919880 1 waiting to reach incremental sync.. 2024-08-18T17:03:55.858 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:03:55.857+0000 7f8be7919880 1 waiting to reach incremental sync.. 2024-08-18T17:04:00.858 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:04:00.857+0000 7f8be7919880 1 waiting to reach incremental sync.. 2024-08-18T17:04:05.859 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:04:05.858+0000 7f8be7919880 1 waiting to reach incremental sync.. 2024-08-18T17:04:10.860 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:04:10.859+0000 7f8be7919880 1 waiting to reach incremental sync.. 2024-08-18T17:04:15.861 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:04:15.860+0000 7f8be7919880 1 waiting to reach incremental sync.. 2024-08-18T17:04:20.861 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:04:20.860+0000 7f8be7919880 1 waiting to reach incremental sync.. 2024-08-18T17:04:25.862 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:04:25.861+0000 7f8be7919880 1 waiting to reach incremental sync.. 2024-08-18T17:04:30.863 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:04:30.862+0000 7f8be7919880 1 waiting to reach incremental sync.. 2024-08-18T17:04:35.863 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:04:35.862+0000 7f8be7919880 1 waiting to reach incremental sync.. 2024-08-18T17:04:40.864 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:04:40.863+0000 7f8be7919880 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-08-18T17:04:40.865 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:04:40.863+0000 7f8be7919880 0 bucket sync checkpoint failed: (110) Connection timed out 2024-08-18T17:04:40.865 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:04:40.863+0000 7f8be7919880 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-08-18T17:04:40.882 DEBUG:teuthology.orchestra.run:got remote process result: 110 2024-08-18T17:04:40.883 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_resharded_bucket_incremental_sync_latest_after_data_sync_init ... FAIL 2024-08-18T17:04:40.919 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:04:40.919 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:04:40.920 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:04:40.967 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:04:40.968 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:04:41.013 INFO:teuthology.orchestra.run.smithi134.stderr:No --period given, using current period=01050e30-179b-43be-bdbb-3bb23cf37ae8 2024-08-18T17:04:41.027 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T17:04:41.027 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:04:41.027 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:04:41.027 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T17:04:41.027 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:04:41.028 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:04:41.028 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:04:41.028 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T17:04:41.028 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:04:41.028 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:04:41.028 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:04:41.029 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:42:47.672461Z" 2024-08-18T17:04:41.029 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:04:41.030 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:04:41.030 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1724000680.916142_1346.1", 2024-08-18T17:04:41.030 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T17:04:40.916142Z" 2024-08-18T17:04:41.030 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:04:41.030 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T17:04:41.032 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-08-18T17:04:41.032 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:04:41.032 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:04:41.033 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:04:41.092 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:04:41.092 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:04:41.126 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:04:41.125+0000 7f6f9ac11880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:04:41.127 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:04:41.127+0000 7f6f9ac11880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:04:41.142 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:04:41.142 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:04:41.142 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:04:41.142 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:04:41.142 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:04:41.143 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:04:41.143 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:04:41.143 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:04:41.143 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:04:41.143 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:04:41.143 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:04:41.143 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:04:41.143 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:04:41.143 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:04:41.143 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:04:41.144 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:04:41.144 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:04:41.144 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:04:41.144 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:04:41.144 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:04:41.144 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:04:41.144 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:04:41.144 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:04:41.144 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:04:41.144 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:04:41.144 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:04:41.144 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:04:41.144 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:04:41.145 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:04:41.145 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:04:41.145 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:04:41.145 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:04:41.145 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:04:41.145 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:04:41.145 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:04:41.145 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:04:41.145 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:04:41.145 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:04:41.145 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:04:41.145 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:04:41.145 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:04:41.146 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:04:41.146 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:04:41.146 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:04:41.146 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:04:41.146 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:04:41.146 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:04:41.146 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:04:41.146 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:04:41.146 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724000305.292694_1321.1", 2024-08-18T17:04:41.146 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:04:41.146 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:04:41.146 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:04:41.146 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:58:25.292694Z", 2024-08-18T17:04:41.147 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:04:41.147 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:04:41.147 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:04:41.147 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:04:41.147 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:04:41.147 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:04:41.147 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:04:41.147 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:04:41.147 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:04:41.147 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:04:41Z" 2024-08-18T17:04:41.147 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:04:41.148 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724000680.916142_1346.1 target=1_1724000305.292694_1321.1 2024-08-18T17:04:46.149 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:04:46.149 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:04:46.149 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:04:46.203 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:04:46.204 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:04:46.237 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:04:46.236+0000 7f96c3812880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:04:46.238 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:04:46.237+0000 7f96c3812880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:04:46.252 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:04:46.252 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:04:46.252 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:04:46.252 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:04:46.253 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:04:46.253 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:04:46.253 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:04:46.253 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:04:46.253 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:04:46.253 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:04:46.253 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:04:46.253 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:04:46.253 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:04:46.253 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:04:46.253 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:04:46.253 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:04:46.253 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:04:46.254 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:04:46.254 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:04:46.254 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:04:46.254 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:04:46.254 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:04:46.254 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:04:46.254 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:04:46.254 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:04:46.254 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:04:46.254 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:04:46.254 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:04:46.254 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:04:46.255 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:04:46.255 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:04:46.255 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:04:46.255 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:04:46.255 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:04:46.255 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:04:46.255 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:04:46.255 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:04:46.255 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:04:46.255 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:04:46.255 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:04:46.255 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:04:46.255 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:04:46.255 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:04:46.256 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:04:46.256 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:04:46.256 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:04:46.256 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:04:46.256 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:04:46.256 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:04:46.256 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724000680.916142_1346.1", 2024-08-18T17:04:46.256 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:04:46.256 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:04:46.256 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:04:46.256 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:04:40.916142Z", 2024-08-18T17:04:46.256 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:04:46.256 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:04:46.257 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:04:46.257 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:04:46.257 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:04:46.257 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:04:46.257 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:04:46.257 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:04:46.257 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:04:46.257 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:04:46Z" 2024-08-18T17:04:46.257 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:04:46.258 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-08-18T17:04:46.259 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-08-18T17:04:46.259 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:04:46.259 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:04:46.259 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:04:46.313 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:04:46.314 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:04:46.350 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:04:46.349+0000 7f13aa60f880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:04:46.351 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:04:46.350+0000 7f13aa60f880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:04:46.367 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T17:04:46.367 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T17:04:46.367 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T17:04:46.367 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T17:04:46.367 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T17:04:46.367 INFO:teuthology.orchestra.run.smithi134.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:04:46.367 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:04:46.367 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:04:46.368 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T17:04:46.368 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:04:46.368 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T17:04:46.368 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:04:46.368 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:04:46.368 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:04:46.368 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:04:46.368 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 1, 2024-08-18T17:04:46.368 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:04:46.368 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:04:46.368 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T17:04:46.368 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:04:46.369 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:04:46.369 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:04:46.369 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T17:04:46.369 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:04:46.369 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:04:46.369 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:04:46.369 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:04:46.369 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:04:46.369 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:04:46.369 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T17:04:46.369 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T17:04:46.369 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:04:46.369 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:04:46.370 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:04:46.370 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T17:04:46.370 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:04:46.370 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:04:46.370 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:04:46.370 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:04:46.370 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:04:46.370 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:04:46.370 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:04:46.370 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:04:46.370 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:04:46.370 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:04:46.370 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:04:46.371 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T17:04:46.371 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:04:46.371 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:04:46.371 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1724000680.916142_1346.1", 2024-08-18T17:04:46.371 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:04:46.371 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:04:46.371 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:04:46.371 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T17:04:40.916142Z", 2024-08-18T17:04:46.371 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:04:46.371 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:04:46.371 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:04:46.371 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:04:46.371 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:04:46.372 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T17:04:46.372 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 1, 2024-08-18T17:04:46.372 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 1 2024-08-18T17:04:46.372 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:04:46.372 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T17:04:46Z" 2024-08-18T17:04:46.372 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T17:04:46.373 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-08-18T17:04:46.491 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-83', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T17:04:46.491 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-83', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T17:04:46.491 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-83 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T17:04:46.549 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:04:46.550 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:04:46.586 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:04:46.585+0000 7ff950017880 1 waiting to reach incremental sync.. 2024-08-18T17:04:51.588 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:04:51.587+0000 7ff950017880 1 bucket sync caught up with source: 2024-08-18T17:04:51.588 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , 00000000001.148.6, 00000000002.126.6, 00000000002.145.6] 2024-08-18T17:04:51.588 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , , , 00000000001.148.6, 00000000002.126.6, 00000000002.145.6] 2024-08-18T17:04:51.588 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:04:51.587+0000 7ff950017880 0 bucket checkpoint complete 2024-08-18T17:04:51.607 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-83', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T17:04:51.607 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-83', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T17:04:51.607 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-83 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T17:04:51.669 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:04:51.669 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:04:51.710 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:04:51.709+0000 7f58b4c12880 1 waiting to reach incremental sync.. 2024-08-18T17:04:56.710 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:04:56.709+0000 7f58b4c12880 1 waiting to reach incremental sync.. 2024-08-18T17:05:01.712 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:05:01.711+0000 7f58b4c12880 1 bucket sync caught up with source: 2024-08-18T17:05:01.712 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , , , 00000000001.148.6, 00000000002.126.6, 00000000002.145.6] 2024-08-18T17:05:01.713 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , , , , 00000000001.148.6, 00000000002.126.6, 00000000002.145.6] 2024-08-18T17:05:01.713 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:05:01.711+0000 7f58b4c12880 0 bucket checkpoint complete 2024-08-18T17:05:01.729 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-83', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T17:05:01.729 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-83', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T17:05:01.730 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-83 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T17:05:01.786 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:05:01.786 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:05:01.824 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:05:01.823+0000 7f3624527880 1 bucket sync caught up with source: 2024-08-18T17:05:01.824 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , , , 00000000001.363.6, 00000000002.320.6, 00000000002.367.6] 2024-08-18T17:05:01.824 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , , , , 00000000001.363.6, 00000000002.320.6, 00000000002.367.6] 2024-08-18T17:05:01.824 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:05:01.823+0000 7f3624527880 0 bucket checkpoint complete 2024-08-18T17:05:01.838 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-83', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T17:05:01.839 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-83', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T17:05:01.839 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-83 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T17:05:01.893 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:05:01.893 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:05:01.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:05:01.943+0000 7fe5a9122880 1 waiting to reach incremental sync.. 2024-08-18T17:05:06.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:05:06.945+0000 7fe5a9122880 1 bucket sync caught up with source: 2024-08-18T17:05:06.946 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , , , 00000000001.363.6, 00000000002.320.6, 00000000002.367.6] 2024-08-18T17:05:06.947 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , , , , 00000000001.363.6, 00000000002.320.6, 00000000002.367.6] 2024-08-18T17:05:06.947 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:05:06.945+0000 7fe5a9122880 0 bucket checkpoint complete 2024-08-18T17:05:06.968 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-83', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T17:05:06.968 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-83', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T17:05:06.968 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kibgmb-83 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-18T17:05:07.028 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:05:07.029 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:05:07.067 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:05:07.066+0000 7f76ae322880 1 waiting to reach incremental sync.. 2024-08-18T17:05:12.067 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:05:12.066+0000 7f76ae322880 1 waiting to reach incremental sync.. 2024-08-18T17:05:17.069 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:05:17.068+0000 7f76ae322880 1 bucket sync caught up with source: 2024-08-18T17:05:17.070 INFO:teuthology.orchestra.run.smithi134.stderr: local status: [, , , , , , , , 00000000001.206.6, 00000000002.206.6, 00000000002.213.6] 2024-08-18T17:05:17.070 INFO:teuthology.orchestra.run.smithi134.stderr: remote markers: [, , , , , , , , 00000000001.206.6, 00000000002.206.6, 00000000002.213.6] 2024-08-18T17:05:17.070 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:05:17.068+0000 7f76ae322880 0 bucket checkpoint complete 2024-08-18T17:05:17.091 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-83', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T17:05:17.091 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-83', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T17:05:17.091 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-83 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T17:05:17.152 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:05:17.152 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:05:17.192 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:05:17.191+0000 7f8b0f722880 1 bucket sync caught up with source: 2024-08-18T17:05:17.192 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , 00000000001.206.6, 00000000002.206.6, 00000000002.213.6] 2024-08-18T17:05:17.192 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , , , 00000000001.206.6, 00000000002.206.6, 00000000002.213.6] 2024-08-18T17:05:17.192 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:05:17.191+0000 7f8b0f722880 0 bucket checkpoint complete 2024-08-18T17:05:17.208 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-83 zones={test-zone2, test-zone1} 2024-08-18T17:05:17.217 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T17:05:17.220 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-08-18T17:05:17.220 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-08-18T17:05:17.220 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-08-18T17:05:17.220 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-08-18T17:05:17.220 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-08-18T17:05:17.221 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T17:05:17.224 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-08-18T17:05:17.224 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-08-18T17:05:17.224 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-08-18T17:05:17.224 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-08-18T17:05:17.224 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-08-18T17:05:17.229 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T17:05:17.237 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T17:05:17.293 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T17:05:17.346 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T17:05:17.356 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T17:05:17.415 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T17:05:17.423 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T17:05:17.478 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T17:05:17.492 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T17:05:17.544 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T17:05:17.552 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T17:05:17.605 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T17:05:17.663 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T17:05:18.007 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T17:05:18.090 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T17:05:18.110 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-83 zones={test-zone2, test-zone1} 2024-08-18T17:05:18.110 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-83 zones={test-zone3, test-zone1} 2024-08-18T17:05:18.124 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T17:05:18.128 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-08-18T17:05:18.128 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-08-18T17:05:18.128 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-08-18T17:05:18.128 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-08-18T17:05:18.128 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-08-18T17:05:18.128 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T17:05:18.133 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-08-18T17:05:18.133 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-08-18T17:05:18.133 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-08-18T17:05:18.133 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-08-18T17:05:18.133 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-08-18T17:05:18.144 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T17:05:18.198 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T17:05:18.213 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T17:05:18.270 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T17:05:18.322 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T17:05:18.334 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T17:05:18.385 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T17:05:18.437 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T17:05:18.497 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T17:05:18.506 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T17:05:18.556 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T17:05:18.567 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T17:05:18.616 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T17:05:18.622 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T17:05:18.674 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T17:05:18.682 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-83 zones={test-zone3, test-zone1} 2024-08-18T17:05:18.683 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kibgmb-83 zones={test-zone3, test-zone2} 2024-08-18T17:05:18.688 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-18T17:05:18.691 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-08-18T17:05:18.691 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-08-18T17:05:18.691 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-08-18T17:05:18.691 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-08-18T17:05:18.691 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-08-18T17:05:18.691 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-18T17:05:18.694 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-08-18T17:05:18.694 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-08-18T17:05:18.694 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-08-18T17:05:18.694 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-08-18T17:05:18.694 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-08-18T17:05:18.700 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T17:05:18.748 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T17:05:18.760 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-08-18T17:05:18.769 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T17:05:18.776 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T17:05:18.831 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-08-18T17:05:18.883 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T17:05:18.937 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T17:05:18.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-08-18T17:05:19.003 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T17:05:19.011 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T17:05:19.064 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-08-18T17:05:19.115 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T17:05:19.125 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T17:05:19.183 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-08-18T17:05:19.235 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kibgmb-83 zones={test-zone3, test-zone2} 2024-08-18T17:05:19.235 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2024-08-18T17:05:19.235 INFO:teuthology.orchestra.run:waiting for 300 2024-08-18T17:05:19.237 INFO:tasks.rgw.c2.client.0.smithi193.stdout:2024-08-18T17:05:19.236+0000 7f3339333640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 84659) UID: 0 2024-08-18T17:05:19.237 INFO:tasks.rgw.c2.client.0.smithi193.stdout:2024-08-18T17:05:19.236+0000 7f333f434880 -1 shutting down 2024-08-18T17:05:23.636 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2024-08-18T17:05:25.335 INFO:tasks.rgw.c2.client.0:Stopped 2024-08-18T17:05:25.336 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-83', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:05:25.336 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-83', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:05:25.336 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-83 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:05:25.394 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:05:25.395 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:05:25.443 INFO:teuthology.orchestra.run.smithi134.stdout:tenant: 2024-08-18T17:05:25.443 INFO:teuthology.orchestra.run.smithi134.stdout:bucket name: kibgmb-83 2024-08-18T17:05:25.447 INFO:teuthology.orchestra.run.smithi134.stdout:total entries: 5 2024-08-18T17:05:25.626 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-83', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:05:25.626 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-83', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:05:25.627 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-83 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:05:25.677 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:05:25.677 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:05:25.726 INFO:teuthology.orchestra.run.smithi134.stdout:tenant: 2024-08-18T17:05:25.726 INFO:teuthology.orchestra.run.smithi134.stdout:bucket name: kibgmb-83 2024-08-18T17:05:25.731 INFO:teuthology.orchestra.run.smithi134.stdout:total entries: 10 2024-08-18T17:05:25.911 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-83', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:05:25.911 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-83', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:05:25.912 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-83 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:05:25.962 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:05:25.962 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:05:26.038 INFO:teuthology.orchestra.run.smithi134.stdout:tenant: 2024-08-18T17:05:26.038 INFO:teuthology.orchestra.run.smithi134.stdout:bucket name: kibgmb-83 2024-08-18T17:05:26.045 INFO:teuthology.orchestra.run.smithi134.stdout:total entries: 15 2024-08-18T17:05:26.189 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-83', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:05:26.189 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-83', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:05:26.189 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-83 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:05:26.240 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:05:26.240 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:05:26.295 INFO:teuthology.orchestra.run.smithi134.stdout:tenant: 2024-08-18T17:05:26.295 INFO:teuthology.orchestra.run.smithi134.stdout:bucket name: kibgmb-83 2024-08-18T17:05:26.304 INFO:teuthology.orchestra.run.smithi134.stdout:total entries: 20 2024-08-18T17:05:26.483 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-83', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:05:26.483 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-83', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:05:26.484 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-83 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:05:26.533 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:05:26.533 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:05:26.586 INFO:teuthology.orchestra.run.smithi134.stdout:tenant: 2024-08-18T17:05:26.586 INFO:teuthology.orchestra.run.smithi134.stdout:bucket name: kibgmb-83 2024-08-18T17:05:26.597 INFO:teuthology.orchestra.run.smithi134.stdout:total entries: 25 2024-08-18T17:05:26.867 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-83', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:05:26.867 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'kibgmb-83', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:05:26.868 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket kibgmb-83 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:05:26.919 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:05:26.919 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:05:26.973 INFO:teuthology.orchestra.run.smithi134.stdout:tenant: 2024-08-18T17:05:26.973 INFO:teuthology.orchestra.run.smithi134.stdout:bucket name: kibgmb-83 2024-08-18T17:05:26.984 INFO:teuthology.orchestra.run.smithi134.stdout:total entries: 30 2024-08-18T17:05:27.213 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:05:27.213 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'init', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:05:27.213 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync init --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-08-18T17:05:27.269 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:05:27.269 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:05:27.305 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:05:27.305+0000 7fa496c3f880 5 RGW-SYNC:data: waiting on data sync status lease 2024-08-18T17:05:27.308 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:05:27.307+0000 7fa496c3f880 5 RGW-SYNC:data: acquired data sync status lease 2024-08-18T17:05:27.333 INFO:tasks.rgw.c2.client.0:Restarting daemon 2024-08-18T17:05:27.334 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2024-08-18T17:05:27.336 INFO:tasks.rgw.c2.client.0:Started 2024-08-18T17:05:27.336 DEBUG:teuthology.orchestra.run.smithi193:> curl http://smithi193.front.sepia.ceph.com:8000 2024-08-18T17:05:27.354 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-08-18T17:05:27.355 INFO:teuthology.orchestra.run.smithi193.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-08-18T17:05:27.355 INFO:teuthology.orchestra.run.smithi193.stderr: Dload Upload Total Spent Left Speed 2024-08-18T17:05:27.355 INFO:teuthology.orchestra.run.smithi193.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2024-08-18T17:05:27.355 INFO:teuthology.orchestra.run.smithi193.stderr:curl: (7) Failed to connect to smithi193.front.sepia.ceph.com port 8000: Connection refused 2024-08-18T17:05:28.356 DEBUG:teuthology.orchestra.run.smithi193:> curl http://smithi193.front.sepia.ceph.com:8000 2024-08-18T17:05:28.376 INFO:teuthology.orchestra.run.smithi193.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-08-18T17:05:28.377 INFO:teuthology.orchestra.run.smithi193.stderr: Dload Upload Total Spent Left Speed 2024-08-18T17:05:28.377 INFO:teuthology.orchestra.run.smithi193.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-08-18T17:05:28.378 INFO:teuthology.orchestra.run.smithi193.stdout:anonymous 2024-08-18T17:05:28.378 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-83', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T17:05:28.378 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kibgmb-83', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-18T17:05:28.378 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kibgmb-83 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-08-18T17:05:28.473 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:05:28.473 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:05:28.511 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:05:28.510+0000 7f2028d32880 1 waiting to reach latest gen 6, on 0.. 2024-08-18T17:05:29.343 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2024-08-18T17:05:33.512 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:05:33.511+0000 7f2028d32880 1 waiting to reach latest gen 6, on 0.. 2024-08-18T17:05:38.512 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:05:38.511+0000 7f2028d32880 1 waiting to reach latest gen 6, on 0.. 2024-08-18T17:05:43.513 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:05:43.512+0000 7f2028d32880 1 waiting to reach latest gen 6, on 0.. 2024-08-18T17:05:48.514 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:05:48.513+0000 7f2028d32880 1 waiting to reach latest gen 6, on 0.. 2024-08-18T17:05:53.514 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:05:53.513+0000 7f2028d32880 1 waiting to reach latest gen 6, on 0.. 2024-08-18T17:05:58.515 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:05:58.514+0000 7f2028d32880 1 waiting to reach latest gen 6, on 0.. 2024-08-18T17:06:03.516 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:06:03.515+0000 7f2028d32880 1 waiting to reach latest gen 6, on 0.. 2024-08-18T17:06:08.516 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:06:08.514+0000 7f2028d32880 1 waiting to reach latest gen 6, on 0.. 2024-08-18T17:06:13.517 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:06:13.515+0000 7f2028d32880 1 waiting to reach latest gen 6, on 0.. 2024-08-18T17:06:18.518 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:06:18.517+0000 7f2028d32880 1 waiting to reach latest gen 6, on 0.. 2024-08-18T17:06:23.518 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:06:23.517+0000 7f2028d32880 1 waiting to reach latest gen 6, on 0.. 2024-08-18T17:06:28.519 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:06:28.518+0000 7f2028d32880 1 waiting to reach latest gen 6, on 0.. 2024-08-18T17:06:33.521 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:06:33.520+0000 7f2028d32880 1 waiting for incremental sync to catch up: 2024-08-18T17:06:33.522 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , , , , , 00000000001.137.6] 2024-08-18T17:06:33.522 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , 00000000002.134.6, , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , 00000000001.224.6, , , , 00000000001.137.6] 2024-08-18T17:06:38.524 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:06:38.523+0000 7f2028d32880 1 waiting for incremental sync to catch up: 2024-08-18T17:06:38.524 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , , , , , 00000000001.137.6] 2024-08-18T17:06:38.524 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , 00000000002.134.6, , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , 00000000001.224.6, , , , 00000000001.137.6] 2024-08-18T17:06:43.527 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:06:43.526+0000 7f2028d32880 1 waiting for incremental sync to catch up: 2024-08-18T17:06:43.527 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , , , , , 00000000001.137.6] 2024-08-18T17:06:43.527 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , 00000000002.134.6, , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , 00000000001.224.6, , , , 00000000001.137.6] 2024-08-18T17:06:48.529 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:06:48.528+0000 7f2028d32880 1 waiting for incremental sync to catch up: 2024-08-18T17:06:48.530 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , , , , , 00000000001.137.6] 2024-08-18T17:06:48.530 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , 00000000002.134.6, , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , 00000000001.224.6, , , , 00000000001.137.6] 2024-08-18T17:06:53.532 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:06:53.531+0000 7f2028d32880 1 waiting for incremental sync to catch up: 2024-08-18T17:06:53.532 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , , , , , 00000000001.137.6] 2024-08-18T17:06:53.533 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , 00000000002.134.6, , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , 00000000001.224.6, , , , 00000000001.137.6] 2024-08-18T17:06:58.535 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:06:58.534+0000 7f2028d32880 1 waiting for incremental sync to catch up: 2024-08-18T17:06:58.535 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , , , , , 00000000001.137.6] 2024-08-18T17:06:58.535 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , 00000000002.134.6, , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , 00000000001.224.6, , , , 00000000001.137.6] 2024-08-18T17:07:03.537 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:07:03.536+0000 7f2028d32880 1 waiting for incremental sync to catch up: 2024-08-18T17:07:03.538 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , , , , , 00000000001.137.6] 2024-08-18T17:07:03.538 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , 00000000002.134.6, , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , 00000000001.224.6, , , , 00000000001.137.6] 2024-08-18T17:07:08.539 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:07:08.538+0000 7f2028d32880 1 waiting for incremental sync to catch up: 2024-08-18T17:07:08.540 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , , , , , 00000000001.137.6] 2024-08-18T17:07:08.540 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , 00000000002.134.6, , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , 00000000001.224.6, , , , 00000000001.137.6] 2024-08-18T17:07:13.542 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:07:13.540+0000 7f2028d32880 1 waiting for incremental sync to catch up: 2024-08-18T17:07:13.542 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , , , , , 00000000001.137.6] 2024-08-18T17:07:13.542 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , 00000000002.134.6, , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , 00000000001.224.6, , , , 00000000001.137.6] 2024-08-18T17:07:18.544 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:07:18.544+0000 7f2028d32880 1 waiting for incremental sync to catch up: 2024-08-18T17:07:18.545 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , , , , , 00000000001.137.6] 2024-08-18T17:07:18.545 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , 00000000002.134.6, , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , 00000000001.224.6, , , , 00000000001.137.6] 2024-08-18T17:07:23.547 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:07:23.546+0000 7f2028d32880 1 waiting for incremental sync to catch up: 2024-08-18T17:07:23.548 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , , , , , 00000000001.137.6] 2024-08-18T17:07:23.548 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , 00000000002.134.6, , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , 00000000001.224.6, , , , 00000000001.137.6] 2024-08-18T17:07:28.549 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:07:28.549+0000 7f2028d32880 1 waiting for incremental sync to catch up: 2024-08-18T17:07:28.550 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , , , , , 00000000001.137.6] 2024-08-18T17:07:28.550 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , 00000000002.134.6, , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , 00000000001.224.6, , , , 00000000001.137.6] 2024-08-18T17:07:33.552 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:07:33.551+0000 7f2028d32880 1 waiting for incremental sync to catch up: 2024-08-18T17:07:33.552 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , , , , , 00000000001.137.6] 2024-08-18T17:07:33.552 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , 00000000002.134.6, , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , 00000000001.224.6, , , , 00000000001.137.6] 2024-08-18T17:07:38.555 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:07:38.554+0000 7f2028d32880 1 waiting for incremental sync to catch up: 2024-08-18T17:07:38.555 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , , , , , 00000000001.137.6] 2024-08-18T17:07:38.555 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , 00000000002.134.6, , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , 00000000001.224.6, , , , 00000000001.137.6] 2024-08-18T17:07:43.557 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:07:43.556+0000 7f2028d32880 1 waiting for incremental sync to catch up: 2024-08-18T17:07:43.557 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , , , , , 00000000001.137.6] 2024-08-18T17:07:43.558 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , 00000000002.134.6, , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , 00000000001.224.6, , , , 00000000001.137.6] 2024-08-18T17:07:48.560 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:07:48.559+0000 7f2028d32880 1 waiting for incremental sync to catch up: 2024-08-18T17:07:48.560 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , , , , , 00000000001.137.6] 2024-08-18T17:07:48.560 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , 00000000002.134.6, , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , 00000000001.224.6, , , , 00000000001.137.6] 2024-08-18T17:07:53.563 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:07:53.561+0000 7f2028d32880 1 waiting for incremental sync to catch up: 2024-08-18T17:07:53.564 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , , , , , 00000000001.137.6] 2024-08-18T17:07:53.564 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , 00000000002.134.6, , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , 00000000001.224.6, , , , 00000000001.137.6] 2024-08-18T17:07:58.565 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:07:58.563+0000 7f2028d32880 1 waiting for incremental sync to catch up: 2024-08-18T17:07:58.565 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , , , , , 00000000001.137.6] 2024-08-18T17:07:58.565 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , 00000000002.134.6, , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , 00000000001.224.6, , , , 00000000001.137.6] 2024-08-18T17:08:03.567 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:08:03.566+0000 7f2028d32880 1 waiting for incremental sync to catch up: 2024-08-18T17:08:03.567 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , , , , , 00000000001.137.6] 2024-08-18T17:08:03.567 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , 00000000002.134.6, , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , 00000000001.224.6, , , , 00000000001.137.6] 2024-08-18T17:08:08.569 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:08:08.568+0000 7f2028d32880 1 waiting for incremental sync to catch up: 2024-08-18T17:08:08.569 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , , , , , 00000000001.137.6] 2024-08-18T17:08:08.570 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , 00000000002.134.6, , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , 00000000001.224.6, , , , 00000000001.137.6] 2024-08-18T17:08:13.572 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:08:13.571+0000 7f2028d32880 1 waiting for incremental sync to catch up: 2024-08-18T17:08:13.572 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , , , , , 00000000001.137.6] 2024-08-18T17:08:13.572 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , 00000000002.134.6, , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , 00000000001.224.6, , , , 00000000001.137.6] 2024-08-18T17:08:18.575 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:08:18.573+0000 7f2028d32880 1 waiting for incremental sync to catch up: 2024-08-18T17:08:18.575 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , , , , , 00000000001.137.6] 2024-08-18T17:08:18.575 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , 00000000002.134.6, , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , 00000000001.224.6, , , , 00000000001.137.6] 2024-08-18T17:08:23.577 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:08:23.576+0000 7f2028d32880 1 waiting for incremental sync to catch up: 2024-08-18T17:08:23.577 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , , , , , 00000000001.137.6] 2024-08-18T17:08:23.577 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , 00000000002.134.6, , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , 00000000001.224.6, , , , 00000000001.137.6] 2024-08-18T17:08:28.579 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:08:28.578+0000 7f2028d32880 1 waiting for incremental sync to catch up: 2024-08-18T17:08:28.580 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , , , , , 00000000001.137.6] 2024-08-18T17:08:28.580 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , 00000000002.134.6, , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , 00000000001.224.6, , , , 00000000001.137.6] 2024-08-18T17:08:33.582 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:08:33.581+0000 7f2028d32880 1 waiting for incremental sync to catch up: 2024-08-18T17:08:33.582 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , , , , , 00000000001.137.6] 2024-08-18T17:08:33.582 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , 00000000002.134.6, , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , 00000000001.224.6, , , , 00000000001.137.6] 2024-08-18T17:08:38.584 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:08:38.583+0000 7f2028d32880 1 waiting for incremental sync to catch up: 2024-08-18T17:08:38.585 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , , , , , 00000000001.137.6] 2024-08-18T17:08:38.585 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , 00000000002.134.6, , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , 00000000001.224.6, , , , 00000000001.137.6] 2024-08-18T17:08:43.586 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:08:43.585+0000 7f2028d32880 1 waiting for incremental sync to catch up: 2024-08-18T17:08:43.587 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , , , , , 00000000001.137.6] 2024-08-18T17:08:43.587 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , 00000000002.134.6, , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , 00000000001.224.6, , , , 00000000001.137.6] 2024-08-18T17:08:48.589 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:08:48.588+0000 7f2028d32880 1 waiting for incremental sync to catch up: 2024-08-18T17:08:48.589 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , , , , , 00000000001.137.6] 2024-08-18T17:08:48.589 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , 00000000002.134.6, , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , 00000000001.224.6, , , , 00000000001.137.6] 2024-08-18T17:08:53.591 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:08:53.590+0000 7f2028d32880 1 waiting for incremental sync to catch up: 2024-08-18T17:08:53.591 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , , , , , 00000000001.137.6] 2024-08-18T17:08:53.592 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , 00000000002.134.6, , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , 00000000001.224.6, , , , 00000000001.137.6] 2024-08-18T17:08:58.594 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:08:58.593+0000 7f2028d32880 1 waiting for incremental sync to catch up: 2024-08-18T17:08:58.594 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , , , , , 00000000001.137.6] 2024-08-18T17:08:58.594 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , 00000000002.134.6, , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , 00000000001.224.6, , , , 00000000001.137.6] 2024-08-18T17:09:03.597 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:09:03.596+0000 7f2028d32880 1 waiting for incremental sync to catch up: 2024-08-18T17:09:03.597 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , , , , , 00000000001.137.6] 2024-08-18T17:09:03.597 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , 00000000002.134.6, , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , 00000000001.224.6, , , , 00000000001.137.6] 2024-08-18T17:09:08.599 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:09:08.598+0000 7f2028d32880 1 waiting for incremental sync to catch up: 2024-08-18T17:09:08.599 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , , , , , 00000000001.137.6] 2024-08-18T17:09:08.600 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , 00000000002.134.6, , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , 00000000001.224.6, , , , 00000000001.137.6] 2024-08-18T17:09:13.602 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:09:13.601+0000 7f2028d32880 1 waiting for incremental sync to catch up: 2024-08-18T17:09:13.602 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , , , , , 00000000001.137.6] 2024-08-18T17:09:13.602 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , 00000000002.134.6, , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , 00000000001.224.6, , , , 00000000001.137.6] 2024-08-18T17:09:18.604 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:09:18.602+0000 7f2028d32880 1 waiting for incremental sync to catch up: 2024-08-18T17:09:18.605 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , , , , , 00000000001.137.6] 2024-08-18T17:09:18.605 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , 00000000002.134.6, , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , 00000000001.224.6, , , , 00000000001.137.6] 2024-08-18T17:09:23.607 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:09:23.605+0000 7f2028d32880 1 waiting for incremental sync to catch up: 2024-08-18T17:09:23.607 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , , , , , 00000000001.137.6] 2024-08-18T17:09:23.608 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , 00000000002.134.6, , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , 00000000001.224.6, , , , 00000000001.137.6] 2024-08-18T17:09:28.610 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:09:28.609+0000 7f2028d32880 1 waiting for incremental sync to catch up: 2024-08-18T17:09:28.610 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , , , , , 00000000001.137.6] 2024-08-18T17:09:28.611 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , 00000000002.134.6, , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , 00000000001.224.6, , , , 00000000001.137.6] 2024-08-18T17:09:33.613 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:09:33.612+0000 7f2028d32880 1 waiting for incremental sync to catch up: 2024-08-18T17:09:33.613 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , , , , , 00000000001.137.6] 2024-08-18T17:09:33.613 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , 00000000002.134.6, , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , 00000000001.224.6, , , , 00000000001.137.6] 2024-08-18T17:09:38.615 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:09:38.614+0000 7f2028d32880 1 waiting for incremental sync to catch up: 2024-08-18T17:09:38.615 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , , , , , 00000000001.137.6] 2024-08-18T17:09:38.615 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , 00000000002.134.6, , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , 00000000001.224.6, , , , 00000000001.137.6] 2024-08-18T17:09:43.617 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:09:43.617+0000 7f2028d32880 1 waiting for incremental sync to catch up: 2024-08-18T17:09:43.618 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , , , , , 00000000001.137.6] 2024-08-18T17:09:43.618 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , 00000000002.134.6, , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , 00000000001.224.6, , , , 00000000001.137.6] 2024-08-18T17:09:48.621 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:09:48.619+0000 7f2028d32880 1 waiting for incremental sync to catch up: 2024-08-18T17:09:48.621 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , , , , , 00000000001.137.6] 2024-08-18T17:09:48.621 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , 00000000002.134.6, , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , 00000000001.224.6, , , , 00000000001.137.6] 2024-08-18T17:09:53.624 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:09:53.622+0000 7f2028d32880 1 waiting for incremental sync to catch up: 2024-08-18T17:09:53.624 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , , , , , 00000000001.137.6] 2024-08-18T17:09:53.624 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , 00000000002.134.6, , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , 00000000001.224.6, , , , 00000000001.137.6] 2024-08-18T17:09:58.625 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:09:58.624+0000 7f2028d32880 1 waiting for incremental sync to catch up: 2024-08-18T17:09:58.626 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , , , , , 00000000001.137.6] 2024-08-18T17:09:58.626 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , 00000000002.134.6, , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , 00000000001.224.6, , , , 00000000001.137.6] 2024-08-18T17:10:03.628 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:10:03.627+0000 7f2028d32880 1 waiting for incremental sync to catch up: 2024-08-18T17:10:03.628 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , , , , , 00000000001.137.6] 2024-08-18T17:10:03.629 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , 00000000002.134.6, , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , 00000000001.224.6, , , , 00000000001.137.6] 2024-08-18T17:10:08.631 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:10:08.630+0000 7f2028d32880 1 waiting for incremental sync to catch up: 2024-08-18T17:10:08.631 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , , , , , 00000000001.137.6] 2024-08-18T17:10:08.631 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , 00000000002.134.6, , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , 00000000001.224.6, , , , 00000000001.137.6] 2024-08-18T17:10:13.633 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:10:13.633+0000 7f2028d32880 1 waiting for incremental sync to catch up: 2024-08-18T17:10:13.634 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , , , , , 00000000001.137.6] 2024-08-18T17:10:13.634 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , 00000000002.134.6, , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , 00000000001.224.6, , , , 00000000001.137.6] 2024-08-18T17:10:18.636 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:10:18.635+0000 7f2028d32880 1 waiting for incremental sync to catch up: 2024-08-18T17:10:18.637 INFO:teuthology.orchestra.run.smithi193.stderr: local status: [, , , , , , , , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , , , , , 00000000001.137.6] 2024-08-18T17:10:18.637 INFO:teuthology.orchestra.run.smithi193.stderr: remote markers: [, , , , , , 00000000002.134.6, , , , , , 00000000002.163.6, , , , , 00000000002.156.6, , , , , , , , , , , , , , , 00000000001.224.6, , , , 00000000001.137.6] 2024-08-18T17:10:23.639 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:10:23.637+0000 7f2028d32880 0 bucket checkpoint timed out waiting for incremental sync to catch up 2024-08-18T17:10:23.639 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:10:23.637+0000 7f2028d32880 0 bucket sync checkpoint failed: (110) Connection timed out 2024-08-18T17:10:23.639 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:10:23.637+0000 7f2028d32880 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-08-18T17:10:23.658 DEBUG:teuthology.orchestra.run:got remote process result: 110 2024-08-18T17:10:23.659 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_resharded_bucket_incremental_sync_oldest_after_data_sync_init ... FAIL 2024-08-18T17:10:23.660 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:10:23.660 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:10:23.661 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:10:23.715 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:10:23.716 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:10:23.753 INFO:teuthology.orchestra.run.smithi134.stderr:No --period given, using current period=01050e30-179b-43be-bdbb-3bb23cf37ae8 2024-08-18T17:10:23.768 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T17:10:23.768 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:10:23.768 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:10:23.768 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T17:10:23.769 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:10:23.769 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:10:23.769 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:10:23.769 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T17:10:23.769 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:10:23.769 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:10:23.769 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:10:23.769 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:42:47.672461Z" 2024-08-18T17:10:23.769 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:10:23.769 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:10:23.769 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1724000726.998407_1395.1", 2024-08-18T17:10:23.769 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T17:05:26.998407Z" 2024-08-18T17:10:23.769 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:10:23.770 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T17:10:23.770 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-08-18T17:10:23.771 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:10:23.771 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:10:23.771 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:10:23.826 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:10:23.826 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:10:23.860 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:10:23.858+0000 7fea75132880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:10:23.860 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:10:23.859+0000 7fea75132880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:10:23.875 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:10:23.876 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:10:23.876 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:10:23.876 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:10:23.876 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:10:23.876 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:10:23.876 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:10:23.876 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:10:23.876 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:10:23.876 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:10:23.876 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:10:23.876 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:10:23.876 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:10:23.876 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:10:23.877 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:10:23.877 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:10:23.877 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:10:23.877 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:10:23.877 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:10:23.877 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:10:23.877 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:10:23.877 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:10:23.877 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:10:23.877 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:10:23.877 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:10:23.877 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:10:23.877 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:10:23.877 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:10:23.878 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:10:23.878 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:10:23.878 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:10:23.878 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:10:23.878 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:10:23.878 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:10:23.878 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:10:23.878 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:10:23.878 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:10:23.878 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:10:23.878 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:10:23.878 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:10:23.878 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:10:23.879 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:10:23.879 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:10:23.879 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:10:23.879 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:10:23.879 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:10:23.879 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:10:23.879 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:10:23.879 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:10:23.879 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724000726.998407_1395.1", 2024-08-18T17:10:23.879 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:10:23.879 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:10:23.879 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:10:23.879 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:05:26.998407Z", 2024-08-18T17:10:23.879 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:10:23.880 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:10:23.880 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:10:23.880 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:10:23.880 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:10:23.880 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:10:23.880 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:10:23.880 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:10:23.880 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:10:23.880 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:10:23Z" 2024-08-18T17:10:23.880 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:10:23.881 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-08-18T17:10:23.881 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-08-18T17:10:23.881 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:10:23.881 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:10:23.882 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:10:23.930 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:10:23.930 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:10:23.966 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:10:23.965+0000 7f6a16126880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:10:23.967 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:10:23.966+0000 7f6a16126880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:10:23.987 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T17:10:23.987 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T17:10:23.987 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T17:10:23.987 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T17:10:23.987 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T17:10:23.987 INFO:teuthology.orchestra.run.smithi134.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:10:23.987 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:10:23.987 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:10:23.987 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T17:10:23.987 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:10:23.988 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T17:10:23.988 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:10:23.988 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:10:23.988 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:10:23.988 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:10:23.988 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 1, 2024-08-18T17:10:23.988 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:10:23.988 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:10:23.988 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T17:10:23.988 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:10:23.988 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:10:23.988 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:10:23.988 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T17:10:23.988 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:10:23.989 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:10:23.989 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:10:23.989 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:10:23.989 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:10:23.989 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:10:23.989 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T17:10:23.989 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T17:10:23.989 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:10:23.989 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:10:23.989 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:10:23.989 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T17:10:23.989 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:10:23.989 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:10:23.990 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:10:23.990 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:10:23.990 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:10:23.990 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:10:23.990 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:10:23.990 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:10:23.990 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:10:23.990 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:10:23.990 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:10:23.990 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T17:10:23.990 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:10:23.990 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:10:23.991 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1724000726.998407_1395.1", 2024-08-18T17:10:23.991 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:10:23.991 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:10:23.991 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:10:23.991 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T17:05:26.998407Z", 2024-08-18T17:10:23.991 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:10:23.991 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:10:23.991 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:10:23.991 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:10:23.991 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:10:23.992 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T17:10:23.992 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 1, 2024-08-18T17:10:23.992 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 1 2024-08-18T17:10:23.992 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:10:23.992 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T17:10:23Z" 2024-08-18T17:10:23.992 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T17:10:23.993 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-08-18T17:10:23.993 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:10:23.993 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:10:23.994 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:10:24.037 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:10:24.037 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:10:24.114 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[]} 2024-08-18T17:10:24.114 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:10:24.114 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:10:24.115 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:10:24.202 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:10:24.202 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:10:24.253 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2024-08-18T17:10:24.254 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:10:24.254 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:10:24.254 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:10:24.301 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:10:24.301 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:10:24.378 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"enabled"}]} 2024-08-18T17:10:24.378 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'get', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:10:24.379 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'get', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:10:24.379 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group get --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:10:24.425 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:10:24.425 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:10:24.503 INFO:teuthology.orchestra.run.smithi134.stdout:{"id":"sync-group","data_flow":{},"pipes":[],"status":"enabled"} 2024-08-18T17:10:24.503 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:10:24.503 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:10:24.504 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:10:24.547 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:10:24.548 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:10:24.598 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"enabled"}]} 2024-08-18T17:10:24.599 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:10:24.599 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:10:24.599 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=test-zone1,test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:10:24.690 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:10:24.690 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:10:24.744 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}]},"pipes":[],"status":"enabled"}]} 2024-08-18T17:10:24.744 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=test-zone1', '--dest-zone=test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:10:24.744 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=test-zone1', '--dest-zone=test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:10:24.744 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow2 --flow-type directional --source-zone=test-zone1 --dest-zone=test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:10:24.796 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:10:24.796 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:10:24.864 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[],"status":"enabled"}]} 2024-08-18T17:10:24.865 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1,test-zone2', '--dest-zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:10:24.865 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1,test-zone2', '--dest-zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:10:24.865 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=test-zone1,test-zone2 --dest-zones=test-zone1,test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:10:24.916 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:10:24.916 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:10:24.968 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-08-18T17:10:24.969 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'get', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:10:24.969 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'get', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:10:24.969 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group get --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:10:25.014 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:10:25.015 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:10:25.063 INFO:teuthology.orchestra.run.smithi134.stdout:{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"} 2024-08-18T17:10:25.064 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:10:25.064 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:10:25.064 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:10:25.110 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:10:25.110 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:10:25.159 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T17:10:25.159 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:10:25.160 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 10, 2024-08-18T17:10:25.160 INFO:teuthology.orchestra.run.smithi134.stdout: "predecessor_uuid": "64309209-7567-46e5-8881-65783d84cf3c", 2024-08-18T17:10:25.160 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": [ 2024-08-18T17:10:25.160 INFO:teuthology.orchestra.run.smithi134.stdout: "", 2024-08-18T17:10:25.160 INFO:teuthology.orchestra.run.smithi134.stdout: "", 2024-08-18T17:10:25.160 INFO:teuthology.orchestra.run.smithi134.stdout: "", 2024-08-18T17:10:25.160 INFO:teuthology.orchestra.run.smithi134.stdout: "1_1723995848.930494_187.1" 2024-08-18T17:10:25.160 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:10:25.160 INFO:teuthology.orchestra.run.smithi134.stdout: "period_map": { 2024-08-18T17:10:25.160 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:10:25.160 INFO:teuthology.orchestra.run.smithi134.stdout: "zonegroups": [ 2024-08-18T17:10:25.160 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:10:25.160 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "test-zonegroup", 2024-08-18T17:10:25.161 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "test-zonegroup", 2024-08-18T17:10:25.161 INFO:teuthology.orchestra.run.smithi134.stdout: "api_name": "", 2024-08-18T17:10:25.161 INFO:teuthology.orchestra.run.smithi134.stdout: "is_master": true, 2024-08-18T17:10:25.161 INFO:teuthology.orchestra.run.smithi134.stdout: "endpoints": [ 2024-08-18T17:10:25.161 INFO:teuthology.orchestra.run.smithi134.stdout: "http://smithi134.front.sepia.ceph.com:8000" 2024-08-18T17:10:25.161 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:10:25.161 INFO:teuthology.orchestra.run.smithi134.stdout: "hostnames": [], 2024-08-18T17:10:25.161 INFO:teuthology.orchestra.run.smithi134.stdout: "hostnames_s3website": [], 2024-08-18T17:10:25.161 INFO:teuthology.orchestra.run.smithi134.stdout: "master_zone": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T17:10:25.161 INFO:teuthology.orchestra.run.smithi134.stdout: "zones": [ 2024-08-18T17:10:25.161 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:10:25.161 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T17:10:25.161 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "test-zone3", 2024-08-18T17:10:25.162 INFO:teuthology.orchestra.run.smithi134.stdout: "endpoints": [ 2024-08-18T17:10:25.162 INFO:teuthology.orchestra.run.smithi134.stdout: "http://smithi134.front.sepia.ceph.com:8001" 2024-08-18T17:10:25.162 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:10:25.162 INFO:teuthology.orchestra.run.smithi134.stdout: "log_meta": false, 2024-08-18T17:10:25.162 INFO:teuthology.orchestra.run.smithi134.stdout: "log_data": true, 2024-08-18T17:10:25.162 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_index_max_shards": 11, 2024-08-18T17:10:25.162 INFO:teuthology.orchestra.run.smithi134.stdout: "read_only": false, 2024-08-18T17:10:25.162 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_type": "", 2024-08-18T17:10:25.162 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from_all": true, 2024-08-18T17:10:25.162 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from": [], 2024-08-18T17:10:25.162 INFO:teuthology.orchestra.run.smithi134.stdout: "redirect_zone": "", 2024-08-18T17:10:25.162 INFO:teuthology.orchestra.run.smithi134.stdout: "supported_features": [ 2024-08-18T17:10:25.163 INFO:teuthology.orchestra.run.smithi134.stdout: "compress-encrypted", 2024-08-18T17:10:25.163 INFO:teuthology.orchestra.run.smithi134.stdout: "resharding" 2024-08-18T17:10:25.163 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:10:25.163 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:10:25.163 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:10:25.163 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "10090ccf-aad6-416d-9775-df8b7b27345a", 2024-08-18T17:10:25.163 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "test-zone2", 2024-08-18T17:10:25.163 INFO:teuthology.orchestra.run.smithi134.stdout: "endpoints": [ 2024-08-18T17:10:25.163 INFO:teuthology.orchestra.run.smithi134.stdout: "http://smithi193.front.sepia.ceph.com:8000" 2024-08-18T17:10:25.163 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:10:25.163 INFO:teuthology.orchestra.run.smithi134.stdout: "log_meta": false, 2024-08-18T17:10:25.163 INFO:teuthology.orchestra.run.smithi134.stdout: "log_data": true, 2024-08-18T17:10:25.163 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_index_max_shards": 11, 2024-08-18T17:10:25.164 INFO:teuthology.orchestra.run.smithi134.stdout: "read_only": false, 2024-08-18T17:10:25.164 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_type": "", 2024-08-18T17:10:25.164 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from_all": true, 2024-08-18T17:10:25.164 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from": [], 2024-08-18T17:10:25.164 INFO:teuthology.orchestra.run.smithi134.stdout: "redirect_zone": "", 2024-08-18T17:10:25.164 INFO:teuthology.orchestra.run.smithi134.stdout: "supported_features": [ 2024-08-18T17:10:25.164 INFO:teuthology.orchestra.run.smithi134.stdout: "compress-encrypted", 2024-08-18T17:10:25.164 INFO:teuthology.orchestra.run.smithi134.stdout: "resharding" 2024-08-18T17:10:25.165 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:10:25.165 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:10:25.165 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:10:25.165 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T17:10:25.165 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "test-zone1", 2024-08-18T17:10:25.165 INFO:teuthology.orchestra.run.smithi134.stdout: "endpoints": [ 2024-08-18T17:10:25.165 INFO:teuthology.orchestra.run.smithi134.stdout: "http://smithi134.front.sepia.ceph.com:8000" 2024-08-18T17:10:25.165 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:10:25.165 INFO:teuthology.orchestra.run.smithi134.stdout: "log_meta": false, 2024-08-18T17:10:25.165 INFO:teuthology.orchestra.run.smithi134.stdout: "log_data": true, 2024-08-18T17:10:25.165 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_index_max_shards": 11, 2024-08-18T17:10:25.165 INFO:teuthology.orchestra.run.smithi134.stdout: "read_only": false, 2024-08-18T17:10:25.165 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_type": "", 2024-08-18T17:10:25.166 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from_all": true, 2024-08-18T17:10:25.166 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from": [], 2024-08-18T17:10:25.166 INFO:teuthology.orchestra.run.smithi134.stdout: "redirect_zone": "", 2024-08-18T17:10:25.166 INFO:teuthology.orchestra.run.smithi134.stdout: "supported_features": [ 2024-08-18T17:10:25.166 INFO:teuthology.orchestra.run.smithi134.stdout: "compress-encrypted", 2024-08-18T17:10:25.166 INFO:teuthology.orchestra.run.smithi134.stdout: "resharding" 2024-08-18T17:10:25.166 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:10:25.166 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:10:25.166 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:10:25.166 INFO:teuthology.orchestra.run.smithi134.stdout: "placement_targets": [ 2024-08-18T17:10:25.166 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:10:25.166 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "default-placement", 2024-08-18T17:10:25.167 INFO:teuthology.orchestra.run.smithi134.stdout: "tags": [], 2024-08-18T17:10:25.167 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_classes": [ 2024-08-18T17:10:25.167 INFO:teuthology.orchestra.run.smithi134.stdout: "STANDARD" 2024-08-18T17:10:25.167 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:10:25.167 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:10:25.167 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:10:25.167 INFO:teuthology.orchestra.run.smithi134.stdout: "default_placement": "default-placement", 2024-08-18T17:10:25.167 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_id": "2827c6f0-bf0d-4836-944c-9e7fcf22457b", 2024-08-18T17:10:25.167 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_policy": { 2024-08-18T17:10:25.167 INFO:teuthology.orchestra.run.smithi134.stdout: "groups": [ 2024-08-18T17:10:25.167 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:10:25.167 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "sync-group", 2024-08-18T17:10:25.167 INFO:teuthology.orchestra.run.smithi134.stdout: "data_flow": { 2024-08-18T17:10:25.168 INFO:teuthology.orchestra.run.smithi134.stdout: "symmetrical": [ 2024-08-18T17:10:25.168 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:10:25.168 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "sync-flow1", 2024-08-18T17:10:25.168 INFO:teuthology.orchestra.run.smithi134.stdout: "zones": [ 2024-08-18T17:10:25.186 INFO:teuthology.orchestra.run.smithi134.stdout: "10090ccf-aad6-416d-9775-df8b7b27345a", 2024-08-18T17:10:25.186 INFO:teuthology.orchestra.run.smithi134.stdout: "96bf8502-2f39-4d53-8196-3378c28da3b4" 2024-08-18T17:10:25.186 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:10:25.186 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:10:25.186 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:10:25.186 INFO:teuthology.orchestra.run.smithi134.stdout: "directional": [ 2024-08-18T17:10:25.187 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:10:25.187 INFO:teuthology.orchestra.run.smithi134.stdout: "source_zone": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T17:10:25.187 INFO:teuthology.orchestra.run.smithi134.stdout: "dest_zone": "10090ccf-aad6-416d-9775-df8b7b27345a" 2024-08-18T17:10:25.187 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:10:25.187 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:10:25.187 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:10:25.187 INFO:teuthology.orchestra.run.smithi134.stdout: "pipes": [ 2024-08-18T17:10:25.187 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:10:25.187 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "sync-pipe", 2024-08-18T17:10:25.187 INFO:teuthology.orchestra.run.smithi134.stdout: "source": { 2024-08-18T17:10:25.187 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket": "*", 2024-08-18T17:10:25.187 INFO:teuthology.orchestra.run.smithi134.stdout: "zones": [ 2024-08-18T17:10:25.187 INFO:teuthology.orchestra.run.smithi134.stdout: "10090ccf-aad6-416d-9775-df8b7b27345a", 2024-08-18T17:10:25.188 INFO:teuthology.orchestra.run.smithi134.stdout: "96bf8502-2f39-4d53-8196-3378c28da3b4" 2024-08-18T17:10:25.188 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:10:25.188 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:10:25.188 INFO:teuthology.orchestra.run.smithi134.stdout: "dest": { 2024-08-18T17:10:25.188 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket": "*", 2024-08-18T17:10:25.188 INFO:teuthology.orchestra.run.smithi134.stdout: "zones": [ 2024-08-18T17:10:25.188 INFO:teuthology.orchestra.run.smithi134.stdout: "10090ccf-aad6-416d-9775-df8b7b27345a", 2024-08-18T17:10:25.188 INFO:teuthology.orchestra.run.smithi134.stdout: "96bf8502-2f39-4d53-8196-3378c28da3b4" 2024-08-18T17:10:25.188 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:10:25.188 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:10:25.188 INFO:teuthology.orchestra.run.smithi134.stdout: "params": { 2024-08-18T17:10:25.188 INFO:teuthology.orchestra.run.smithi134.stdout: "source": { 2024-08-18T17:10:25.188 INFO:teuthology.orchestra.run.smithi134.stdout: "filter": { 2024-08-18T17:10:25.188 INFO:teuthology.orchestra.run.smithi134.stdout: "tags": [] 2024-08-18T17:10:25.189 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:10:25.189 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:10:25.189 INFO:teuthology.orchestra.run.smithi134.stdout: "dest": {}, 2024-08-18T17:10:25.189 INFO:teuthology.orchestra.run.smithi134.stdout: "priority": 0, 2024-08-18T17:10:25.189 INFO:teuthology.orchestra.run.smithi134.stdout: "mode": "system", 2024-08-18T17:10:25.189 INFO:teuthology.orchestra.run.smithi134.stdout: "user": "" 2024-08-18T17:10:25.189 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:10:25.189 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:10:25.189 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:10:25.189 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "enabled" 2024-08-18T17:10:25.189 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:10:25.189 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:10:25.189 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:10:25.189 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled_features": [ 2024-08-18T17:10:25.190 INFO:teuthology.orchestra.run.smithi134.stdout: "resharding" 2024-08-18T17:10:25.190 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:10:25.190 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:10:25.190 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:10:25.190 INFO:teuthology.orchestra.run.smithi134.stdout: "short_zone_ids": [ 2024-08-18T17:10:25.190 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:10:25.190 INFO:teuthology.orchestra.run.smithi134.stdout: "key": "018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T17:10:25.190 INFO:teuthology.orchestra.run.smithi134.stdout: "val": 462036791 2024-08-18T17:10:25.190 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:10:25.190 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:10:25.190 INFO:teuthology.orchestra.run.smithi134.stdout: "key": "10090ccf-aad6-416d-9775-df8b7b27345a", 2024-08-18T17:10:25.190 INFO:teuthology.orchestra.run.smithi134.stdout: "val": 874289624 2024-08-18T17:10:25.190 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:10:25.191 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:10:25.191 INFO:teuthology.orchestra.run.smithi134.stdout: "key": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T17:10:25.191 INFO:teuthology.orchestra.run.smithi134.stdout: "val": 1721399886 2024-08-18T17:10:25.191 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:10:25.191 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:10:25.191 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:10:25.191 INFO:teuthology.orchestra.run.smithi134.stdout: "master_zonegroup": "test-zonegroup", 2024-08-18T17:10:25.191 INFO:teuthology.orchestra.run.smithi134.stdout: "master_zone": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T17:10:25.191 INFO:teuthology.orchestra.run.smithi134.stdout: "period_config": { 2024-08-18T17:10:25.191 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_quota": { 2024-08-18T17:10:25.191 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false, 2024-08-18T17:10:25.191 INFO:teuthology.orchestra.run.smithi134.stdout: "check_on_raw": false, 2024-08-18T17:10:25.191 INFO:teuthology.orchestra.run.smithi134.stdout: "max_size": -1, 2024-08-18T17:10:25.191 INFO:teuthology.orchestra.run.smithi134.stdout: "max_size_kb": 0, 2024-08-18T17:10:25.192 INFO:teuthology.orchestra.run.smithi134.stdout: "max_objects": -1 2024-08-18T17:10:25.192 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:10:25.192 INFO:teuthology.orchestra.run.smithi134.stdout: "user_quota": { 2024-08-18T17:10:25.192 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false, 2024-08-18T17:10:25.192 INFO:teuthology.orchestra.run.smithi134.stdout: "check_on_raw": false, 2024-08-18T17:10:25.192 INFO:teuthology.orchestra.run.smithi134.stdout: "max_size": -1, 2024-08-18T17:10:25.192 INFO:teuthology.orchestra.run.smithi134.stdout: "max_size_kb": 0, 2024-08-18T17:10:25.192 INFO:teuthology.orchestra.run.smithi134.stdout: "max_objects": -1 2024-08-18T17:10:25.193 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:10:25.193 INFO:teuthology.orchestra.run.smithi134.stdout: "user_ratelimit": { 2024-08-18T17:10:25.193 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_ops": 0, 2024-08-18T17:10:25.193 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_ops": 0, 2024-08-18T17:10:25.193 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_bytes": 0, 2024-08-18T17:10:25.193 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_bytes": 0, 2024-08-18T17:10:25.193 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false 2024-08-18T17:10:25.193 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:10:25.193 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_ratelimit": { 2024-08-18T17:10:25.193 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_ops": 0, 2024-08-18T17:10:25.193 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_ops": 0, 2024-08-18T17:10:25.193 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_bytes": 0, 2024-08-18T17:10:25.194 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_bytes": 0, 2024-08-18T17:10:25.194 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false 2024-08-18T17:10:25.194 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:10:25.194 INFO:teuthology.orchestra.run.smithi134.stdout: "anonymous_ratelimit": { 2024-08-18T17:10:25.194 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_ops": 0, 2024-08-18T17:10:25.194 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_ops": 0, 2024-08-18T17:10:25.194 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_bytes": 0, 2024-08-18T17:10:25.194 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_bytes": 0, 2024-08-18T17:10:25.194 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false 2024-08-18T17:10:25.194 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:10:25.194 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:10:25.194 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_id": "2827c6f0-bf0d-4836-944c-9e7fcf22457b", 2024-08-18T17:10:25.194 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_name": "test-realm", 2024-08-18T17:10:25.194 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:10:25.195 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T17:10:25.196 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=kibgmb-84 2024-08-18T17:10:25.669 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'kibgmb-84', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:10:25.669 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'kibgmb-84', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:10:25.669 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-bucket --status allowed --bucket kibgmb-84 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:10:25.732 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:10:25.732 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:10:25.879 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"allowed"}]} 2024-08-18T17:10:25.880 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'kibgmb-84', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:10:25.880 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'kibgmb-84', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:10:25.880 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-bucket --status enabled --bucket kibgmb-84 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:10:25.940 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:10:25.940 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:10:26.063 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"enabled"}]} 2024-08-18T17:10:26.063 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'get', '--group-id', 'sync-bucket', '--bucket', 'kibgmb-84', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:10:26.063 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'get', '--group-id', 'sync-bucket', '--bucket', 'kibgmb-84', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:10:26.063 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group get --group-id sync-bucket --bucket kibgmb-84 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:10:26.125 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:10:26.125 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:10:26.235 INFO:teuthology.orchestra.run.smithi134.stdout:{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"enabled"} 2024-08-18T17:10:26.235 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--bucket', 'kibgmb-84', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:10:26.236 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--bucket', 'kibgmb-84', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:10:26.236 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --bucket kibgmb-84 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:10:26.286 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:10:26.286 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:10:26.387 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"enabled"}]} 2024-08-18T17:10:26.387 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--bucket', 'kibgmb-84', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:10:26.388 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--bucket', 'kibgmb-84', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:10:26.388 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flow1 --flow-type symmetrical --zones=test-zone1,test-zone2 --bucket kibgmb-84 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:10:26.444 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:10:26.445 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:10:26.533 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}]},"pipes":[],"status":"enabled"}]} 2024-08-18T17:10:26.533 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=test-zone1', '--dest-zone=test-zone2', '--bucket', 'kibgmb-84', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:10:26.534 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=test-zone1', '--dest-zone=test-zone2', '--bucket', 'kibgmb-84', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:10:26.534 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flow2 --flow-type directional --source-zone=test-zone1 --dest-zone=test-zone2 --bucket kibgmb-84 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:10:26.588 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:10:26.588 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:10:26.643 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[],"status":"enabled"}]} 2024-08-18T17:10:26.644 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1,test-zone2', '--dest-zones=test-zone1,test-zone2', '--bucket=kibgmb-84', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:10:26.644 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1,test-zone2', '--dest-zones=test-zone1,test-zone2', '--bucket=kibgmb-84', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:10:26.644 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe --source-zones=test-zone1,test-zone2 --dest-zones=test-zone1,test-zone2 --bucket=kibgmb-84 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:10:26.695 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:10:26.695 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:10:26.771 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2024-08-18T17:10:26.771 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'get', '--group-id', 'sync-bucket', '--bucket', 'kibgmb-84', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:10:26.772 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'get', '--group-id', 'sync-bucket', '--bucket', 'kibgmb-84', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:10:26.772 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group get --group-id sync-bucket --bucket kibgmb-84 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:10:26.819 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:10:26.819 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:10:26.899 INFO:teuthology.orchestra.run.smithi134.stdout:{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"} 2024-08-18T17:10:26.899 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:10:26.900 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:10:26.900 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:10:26.945 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:10:26.945 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:10:26.983 INFO:teuthology.orchestra.run.smithi134.stderr:No --period given, using current period=01050e30-179b-43be-bdbb-3bb23cf37ae8 2024-08-18T17:10:26.998 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T17:10:26.999 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:10:26.999 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:10:26.999 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T17:10:26.999 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:10:26.999 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:10:26.999 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:10:26.999 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T17:10:26.999 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:10:26.999 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:10:26.999 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:10:26.999 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:42:47.672461Z" 2024-08-18T17:10:27.000 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:10:27.000 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:10:27.000 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1724001026.744398_1436.1", 2024-08-18T17:10:27.000 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T17:10:26.744398Z" 2024-08-18T17:10:27.000 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:10:27.000 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T17:10:27.001 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-08-18T17:10:27.001 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:10:27.001 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:10:27.002 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:10:27.057 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:10:27.057 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:10:27.091 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:10:27.090+0000 7f70ea722880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:10:27.092 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:10:27.091+0000 7f70ea722880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:10:27.107 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:10:27.107 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:10:27.107 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:10:27.107 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:10:27.107 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:10:27.107 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:10:27.107 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:10:27.108 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:10:27.108 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:10:27.108 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:10:27.108 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:10:27.108 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:10:27.108 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:10:27.108 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:10:27.108 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:10:27.108 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:10:27.108 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:10:27.108 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:10:27.108 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:10:27.108 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:10:27.109 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:10:27.109 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:10:27.109 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:10:27.109 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:10:27.109 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:10:27.109 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:10:27.109 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:10:27.109 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:10:27.109 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:10:27.109 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:10:27.109 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:10:27.109 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:10:27.109 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:10:27.109 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:10:27.110 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:10:27.110 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:10:27.110 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:10:27.110 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:10:27.110 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:10:27.110 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:10:27.110 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:10:27.110 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:10:27.110 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:10:27.110 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:10:27.110 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:10:27.110 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:10:27.110 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:10:27.111 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:10:27.111 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:10:27.111 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724000726.998407_1395.1", 2024-08-18T17:10:27.111 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:10:27.111 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:10:27.111 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:10:27.111 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:05:26.998407Z", 2024-08-18T17:10:27.111 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:10:27.111 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:10:27.111 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:10:27.111 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:10:27.111 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:10:27.111 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:10:27.111 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:10:27.112 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:10:27.112 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:10:27.112 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:10:27Z" 2024-08-18T17:10:27.112 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:10:27.113 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001026.744398_1436.1 target=1_1724000726.998407_1395.1 2024-08-18T17:10:32.114 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:10:32.114 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:10:32.115 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:10:32.170 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:10:32.171 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:10:32.204 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:10:32.203+0000 7ff287117880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:10:32.205 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:10:32.204+0000 7ff287117880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:10:32.220 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:10:32.220 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:10:32.220 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:10:32.220 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:10:32.220 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:10:32.221 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:10:32.221 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:10:32.221 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:10:32.221 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:10:32.221 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:10:32.221 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:10:32.221 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:10:32.221 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:10:32.221 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:10:32.221 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:10:32.221 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:10:32.221 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:10:32.221 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:10:32.222 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:10:32.222 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:10:32.222 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:10:32.222 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:10:32.222 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:10:32.222 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:10:32.222 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:10:32.222 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:10:32.222 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:10:32.222 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:10:32.222 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:10:32.222 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:10:32.222 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:10:32.223 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:10:32.223 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:10:32.223 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:10:32.223 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:10:32.223 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:10:32.223 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:10:32.223 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:10:32.223 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:10:32.223 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:10:32.223 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:10:32.223 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:10:32.223 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:10:32.223 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:10:32.223 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:10:32.224 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:10:32.224 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:10:32.224 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:10:32.224 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:10:32.224 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724000726.998407_1395.1", 2024-08-18T17:10:32.224 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:10:32.224 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:10:32.224 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:10:32.224 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:05:26.998407Z", 2024-08-18T17:10:32.224 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:10:32.224 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:10:32.224 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:10:32.224 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:10:32.225 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:10:32.225 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:10:32.225 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:10:32.225 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:10:32.225 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:10:32.225 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:10:32Z" 2024-08-18T17:10:32.225 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:10:32.226 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001026.744398_1436.1 target=1_1724000726.998407_1395.1 2024-08-18T17:10:37.227 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:10:37.227 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:10:37.228 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:10:37.283 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:10:37.283 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:10:37.318 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:10:37.317+0000 7f8ba0833880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:10:37.319 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:10:37.318+0000 7f8ba0833880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:10:37.334 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:10:37.334 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:10:37.334 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:10:37.334 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:10:37.334 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:10:37.335 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:10:37.335 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:10:37.335 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:10:37.335 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:10:37.335 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:10:37.335 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:10:37.335 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:10:37.335 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:10:37.335 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:10:37.335 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:10:37.335 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:10:37.335 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:10:37.335 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:10:37.335 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:10:37.336 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:10:37.336 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:10:37.336 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:10:37.336 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:10:37.336 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:10:37.336 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:10:37.336 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:10:37.336 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:10:37.336 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:10:37.336 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:10:37.336 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:10:37.337 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:10:37.337 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:10:37.337 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:10:37.337 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:10:37.337 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:10:37.337 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:10:37.337 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:10:37.337 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:10:37.337 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:10:37.337 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:10:37.337 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:10:37.337 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:10:37.337 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:10:37.337 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:10:37.338 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:10:37.338 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:10:37.338 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:10:37.338 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:10:37.338 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:10:37.338 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724000726.998407_1395.1", 2024-08-18T17:10:37.338 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:10:37.338 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:10:37.338 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:10:37.338 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:05:26.998407Z", 2024-08-18T17:10:37.338 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:10:37.338 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:10:37.338 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:10:37.339 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:10:37.339 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:10:37.339 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:10:37.339 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:10:37.339 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:10:37.339 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:10:37.339 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:10:37Z" 2024-08-18T17:10:37.339 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:10:37.340 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001026.744398_1436.1 target=1_1724000726.998407_1395.1 2024-08-18T17:10:42.341 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:10:42.341 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:10:42.342 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:10:42.398 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:10:42.399 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:10:42.432 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:10:42.431+0000 7f4e59723880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:10:42.432 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:10:42.432+0000 7f4e59723880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:10:42.447 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:10:42.448 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:10:42.448 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:10:42.448 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:10:42.448 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:10:42.448 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:10:42.449 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:10:42.449 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:10:42.449 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:10:42.449 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:10:42.449 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:10:42.450 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:10:42.450 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:10:42.450 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:10:42.450 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:10:42.450 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:10:42.450 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:10:42.451 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:10:42.451 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:10:42.451 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:10:42.451 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:10:42.451 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:10:42.451 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:10:42.452 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:10:42.452 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:10:42.452 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:10:42.452 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:10:42.452 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:10:42.452 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:10:42.453 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:10:42.453 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:10:42.453 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:10:42.453 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:10:42.453 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:10:42.453 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:10:42.453 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:10:42.454 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:10:42.454 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:10:42.454 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:10:42.454 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:10:42.454 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:10:42.454 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:10:42.454 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:10:42.455 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:10:42.455 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:10:42.455 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:10:42.455 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:10:42.456 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:10:42.456 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:10:42.456 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724000726.998407_1395.1", 2024-08-18T17:10:42.456 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:10:42.456 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:10:42.456 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:10:42.457 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:05:26.998407Z", 2024-08-18T17:10:42.457 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:10:42.457 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:10:42.457 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:10:42.457 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:10:42.457 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:10:42.457 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:10:42.457 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:10:42.458 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:10:42.458 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:10:42.458 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:10:42Z" 2024-08-18T17:10:42.458 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:10:42.460 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001026.744398_1436.1 target=1_1724000726.998407_1395.1 2024-08-18T17:10:47.460 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:10:47.461 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:10:47.461 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:10:47.517 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:10:47.517 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:10:47.551 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:10:47.550+0000 7f74b2010880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:10:47.551 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:10:47.551+0000 7f74b2010880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:10:47.566 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:10:47.567 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:10:47.567 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:10:47.567 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:10:47.567 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:10:47.567 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:10:47.567 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:10:47.567 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:10:47.567 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:10:47.567 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:10:47.567 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:10:47.567 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:10:47.567 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:10:47.567 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:10:47.568 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:10:47.568 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:10:47.568 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:10:47.568 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:10:47.568 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:10:47.568 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:10:47.568 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:10:47.568 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:10:47.568 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:10:47.568 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:10:47.568 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:10:47.568 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:10:47.568 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:10:47.568 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:10:47.569 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:10:47.569 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:10:47.569 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:10:47.569 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:10:47.569 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:10:47.569 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:10:47.569 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:10:47.569 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:10:47.569 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:10:47.569 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:10:47.569 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:10:47.569 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:10:47.569 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:10:47.570 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:10:47.570 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:10:47.570 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:10:47.570 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:10:47.570 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:10:47.570 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:10:47.570 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:10:47.570 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:10:47.570 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724000726.998407_1395.1", 2024-08-18T17:10:47.570 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:10:47.570 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:10:47.570 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:10:47.570 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:05:26.998407Z", 2024-08-18T17:10:47.571 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:10:47.571 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:10:47.571 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:10:47.571 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:10:47.571 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:10:47.571 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:10:47.571 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:10:47.571 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:10:47.571 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:10:47.571 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:10:47Z" 2024-08-18T17:10:47.571 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:10:47.572 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001026.744398_1436.1 target=1_1724000726.998407_1395.1 2024-08-18T17:10:52.574 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:10:52.574 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:10:52.575 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:10:52.630 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:10:52.630 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:10:52.667 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:10:52.666+0000 7ff2c7c11880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:10:52.667 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:10:52.667+0000 7ff2c7c11880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:10:52.682 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:10:52.682 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:10:52.682 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:10:52.683 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:10:52.683 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:10:52.683 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:10:52.683 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:10:52.683 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:10:52.683 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:10:52.683 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:10:52.683 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:10:52.683 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:10:52.683 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:10:52.683 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:10:52.683 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:10:52.683 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:10:52.684 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:10:52.684 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:10:52.684 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:10:52.684 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:10:52.684 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:10:52.684 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:10:52.684 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:10:52.684 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:10:52.684 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:10:52.684 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:10:52.684 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:10:52.684 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:10:52.684 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:10:52.684 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:10:52.685 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:10:52.685 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:10:52.685 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:10:52.685 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:10:52.685 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:10:52.685 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:10:52.685 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:10:52.685 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:10:52.685 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:10:52.685 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:10:52.685 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:10:52.685 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:10:52.685 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:10:52.686 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:10:52.686 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:10:52.686 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:10:52.686 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:10:52.686 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:10:52.686 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:10:52.686 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724000726.998407_1395.1", 2024-08-18T17:10:52.686 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:10:52.686 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:10:52.686 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:10:52.686 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:05:26.998407Z", 2024-08-18T17:10:52.686 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:10:52.686 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:10:52.687 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:10:52.687 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:10:52.687 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:10:52.687 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:10:52.687 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:10:52.687 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:10:52.687 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:10:52.687 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:10:52Z" 2024-08-18T17:10:52.687 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:10:52.688 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001026.744398_1436.1 target=1_1724000726.998407_1395.1 2024-08-18T17:10:57.689 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:10:57.689 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:10:57.689 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:10:57.747 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:10:57.747 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:10:57.781 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:10:57.780+0000 7f2d6fe10880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:10:57.781 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:10:57.781+0000 7f2d6fe10880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:10:57.796 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:10:57.796 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:10:57.796 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:10:57.796 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:10:57.797 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:10:57.797 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:10:57.797 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:10:57.797 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:10:57.797 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:10:57.797 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:10:57.797 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:10:57.797 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:10:57.797 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:10:57.797 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:10:57.798 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:10:57.798 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:10:57.799 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:10:57.799 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:10:57.799 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:10:57.799 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:10:57.799 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:10:57.799 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:10:57.799 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:10:57.799 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:10:57.799 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:10:57.799 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:10:57.799 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:10:57.799 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:10:57.799 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:10:57.800 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:10:57.800 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:10:57.800 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:10:57.800 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:10:57.800 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:10:57.800 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:10:57.800 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:10:57.800 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:10:57.800 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:10:57.800 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:10:57.800 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:10:57.800 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:10:57.800 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:10:57.801 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:10:57.801 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:10:57.801 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:10:57.801 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:10:57.801 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:10:57.801 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:10:57.801 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:10:57.801 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724000726.998407_1395.1", 2024-08-18T17:10:57.801 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:10:57.801 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:10:57.801 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:10:57.801 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:05:26.998407Z", 2024-08-18T17:10:57.801 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:10:57.801 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:10:57.802 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:10:57.802 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:10:57.802 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:10:57.802 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:10:57.802 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:10:57.802 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:10:57.802 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:10:57.802 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:10:57Z" 2024-08-18T17:10:57.802 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:10:57.803 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001026.744398_1436.1 target=1_1724000726.998407_1395.1 2024-08-18T17:11:02.804 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:11:02.804 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:11:02.804 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:11:02.861 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:11:02.861 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:11:02.894 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:11:02.893+0000 7ff864b23880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:11:02.895 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:11:02.894+0000 7ff864b23880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:11:02.911 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:11:02.912 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:11:02.912 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:11:02.912 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:11:02.912 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:11:02.912 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:11:02.912 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:11:02.912 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:11:02.912 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:11:02.912 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:11:02.912 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:11:02.912 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:11:02.913 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:11:02.913 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:11:02.913 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:11:02.913 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:11:02.913 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:11:02.913 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:11:02.913 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:11:02.913 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:11:02.913 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:11:02.913 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:11:02.913 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:11:02.913 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:11:02.913 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:11:02.914 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:11:02.914 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:11:02.914 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:11:02.914 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:11:02.914 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:11:02.914 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:11:02.914 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:11:02.914 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:11:02.914 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:11:02.914 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:11:02.914 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:11:02.914 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:11:02.915 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:11:02.915 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:11:02.915 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:11:02.915 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:11:02.915 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:11:02.915 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:11:02.915 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:11:02.915 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:11:02.915 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:11:02.915 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:11:02.915 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:11:02.915 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:11:02.915 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724000726.998407_1395.1", 2024-08-18T17:11:02.916 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:11:02.916 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:11:02.916 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:11:02.916 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:05:26.998407Z", 2024-08-18T17:11:02.916 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:11:02.916 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:11:02.916 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:11:02.916 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:11:02.916 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:11:02.916 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:11:02.916 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:11:02.916 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:11:02.916 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:11:02.917 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:11:02Z" 2024-08-18T17:11:02.917 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:11:02.918 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001026.744398_1436.1 target=1_1724000726.998407_1395.1 2024-08-18T17:11:07.919 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:11:07.919 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:11:07.919 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:11:07.974 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:11:07.974 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:11:08.008 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:11:08.007+0000 7fe958d34880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:11:08.008 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:11:08.008+0000 7fe958d34880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:11:08.023 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:11:08.024 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:11:08.024 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:11:08.024 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:11:08.024 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:11:08.024 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:11:08.025 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:11:08.025 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:11:08.025 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:11:08.025 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:11:08.025 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:11:08.025 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:11:08.026 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:11:08.026 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:11:08.026 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:11:08.026 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:11:08.026 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:11:08.026 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:11:08.026 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:11:08.026 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:11:08.027 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:11:08.027 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:11:08.027 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:11:08.027 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:11:08.027 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:11:08.027 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:11:08.027 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:11:08.028 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:11:08.028 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:11:08.028 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:11:08.028 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:11:08.028 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:11:08.028 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:11:08.028 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:11:08.028 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:11:08.028 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:11:08.029 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:11:08.029 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:11:08.029 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:11:08.029 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:11:08.029 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:11:08.029 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:11:08.029 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:11:08.030 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:11:08.030 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:11:08.030 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:11:08.031 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:11:08.031 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:11:08.031 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:11:08.031 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724000726.998407_1395.1", 2024-08-18T17:11:08.031 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:11:08.031 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:11:08.031 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:11:08.031 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:05:26.998407Z", 2024-08-18T17:11:08.031 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:11:08.031 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:11:08.031 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:11:08.032 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:11:08.032 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:11:08.032 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:11:08.032 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:11:08.032 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:11:08.032 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:11:08.032 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:11:08Z" 2024-08-18T17:11:08.032 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:11:08.033 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001026.744398_1436.1 target=1_1724000726.998407_1395.1 2024-08-18T17:11:13.034 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:11:13.034 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:11:13.035 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:11:13.089 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:11:13.090 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:11:13.123 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:11:13.123+0000 7fb091286880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:11:13.125 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:11:13.124+0000 7fb091286880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:11:13.139 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:11:13.139 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:11:13.139 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:11:13.139 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:11:13.140 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:11:13.140 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:11:13.140 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:11:13.140 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:11:13.140 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:11:13.140 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:11:13.140 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:11:13.140 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:11:13.140 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:11:13.140 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:11:13.140 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:11:13.140 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:11:13.140 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:11:13.141 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:11:13.141 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:11:13.141 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:11:13.141 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:11:13.141 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:11:13.141 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:11:13.141 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:11:13.141 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:11:13.141 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:11:13.142 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:11:13.142 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:11:13.142 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:11:13.142 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:11:13.142 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:11:13.142 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:11:13.142 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:11:13.142 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:11:13.142 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:11:13.142 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:11:13.142 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:11:13.142 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:11:13.143 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:11:13.143 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:11:13.143 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:11:13.143 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:11:13.143 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:11:13.143 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:11:13.143 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:11:13.143 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:11:13.143 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:11:13.143 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:11:13.143 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:11:13.143 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724000726.998407_1395.1", 2024-08-18T17:11:13.144 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:11:13.144 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:11:13.146 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:11:13.146 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:05:26.998407Z", 2024-08-18T17:11:13.146 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:11:13.146 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:11:13.146 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:11:13.147 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:11:13.147 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:11:13.147 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:11:13.147 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:11:13.147 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:11:13.147 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:11:13.147 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:11:13Z" 2024-08-18T17:11:13.147 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:11:13.149 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001026.744398_1436.1 target=1_1724000726.998407_1395.1 2024-08-18T17:11:18.149 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:11:18.149 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:11:18.150 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:11:18.206 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:11:18.206 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:11:18.240 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:11:18.239+0000 7f3780131880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:11:18.240 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:11:18.240+0000 7f3780131880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:11:18.255 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:11:18.255 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:11:18.255 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:11:18.256 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:11:18.256 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:11:18.256 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:11:18.256 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:11:18.256 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:11:18.256 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:11:18.256 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:11:18.256 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:11:18.256 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:11:18.256 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:11:18.256 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:11:18.256 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:11:18.256 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:11:18.257 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:11:18.257 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:11:18.257 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:11:18.257 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:11:18.257 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:11:18.257 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:11:18.257 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:11:18.257 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:11:18.257 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:11:18.257 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:11:18.257 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:11:18.257 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:11:18.258 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:11:18.258 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:11:18.258 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:11:18.258 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:11:18.258 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:11:18.258 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:11:18.258 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:11:18.258 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:11:18.258 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:11:18.258 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:11:18.258 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:11:18.259 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:11:18.259 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:11:18.259 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:11:18.259 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:11:18.259 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:11:18.259 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:11:18.259 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:11:18.259 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:11:18.259 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:11:18.260 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:11:18.260 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724000726.998407_1395.1", 2024-08-18T17:11:18.260 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:11:18.260 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:11:18.260 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:11:18.260 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:05:26.998407Z", 2024-08-18T17:11:18.260 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:11:18.260 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:11:18.260 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:11:18.260 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:11:18.260 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:11:18.261 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:11:18.261 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:11:18.261 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:11:18.261 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:11:18.261 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:11:18Z" 2024-08-18T17:11:18.261 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:11:18.262 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001026.744398_1436.1 target=1_1724000726.998407_1395.1 2024-08-18T17:11:23.263 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:11:23.264 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:11:23.264 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:11:23.319 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:11:23.320 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:11:23.353 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:11:23.352+0000 7f6c3f146880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:11:23.354 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:11:23.353+0000 7f6c3f146880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:11:23.368 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:11:23.369 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:11:23.369 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:11:23.369 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:11:23.369 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:11:23.369 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:11:23.369 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:11:23.369 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:11:23.369 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:11:23.369 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:11:23.369 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:11:23.369 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:11:23.369 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:11:23.369 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:11:23.370 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:11:23.370 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:11:23.370 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:11:23.370 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:11:23.370 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:11:23.370 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:11:23.370 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:11:23.370 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:11:23.370 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:11:23.370 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:11:23.370 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:11:23.370 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:11:23.371 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:11:23.371 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:11:23.371 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:11:23.371 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:11:23.371 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:11:23.371 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:11:23.371 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:11:23.371 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:11:23.371 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:11:23.371 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:11:23.371 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:11:23.371 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:11:23.371 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:11:23.371 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:11:23.372 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:11:23.372 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:11:23.372 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:11:23.372 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:11:23.372 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:11:23.372 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:11:23.372 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:11:23.372 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:11:23.372 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:11:23.372 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724000726.998407_1395.1", 2024-08-18T17:11:23.372 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:11:23.372 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:11:23.372 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:11:23.373 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:05:26.998407Z", 2024-08-18T17:11:23.373 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:11:23.373 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:11:23.373 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:11:23.373 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:11:23.373 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:11:23.373 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:11:23.373 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:11:23.373 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:11:23.373 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:11:23.373 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:11:23Z" 2024-08-18T17:11:23.373 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:11:23.374 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001026.744398_1436.1 target=1_1724000726.998407_1395.1 2024-08-18T17:11:28.375 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:11:28.375 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:11:28.376 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:11:28.430 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:11:28.431 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:11:28.464 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:11:28.463+0000 7ff5b8d23880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:11:28.465 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:11:28.463+0000 7ff5b8d23880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:11:28.480 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:11:28.480 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:11:28.480 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:11:28.480 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:11:28.480 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:11:28.480 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:11:28.481 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:11:28.481 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:11:28.481 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:11:28.481 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:11:28.481 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:11:28.481 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:11:28.481 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:11:28.481 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:11:28.481 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:11:28.481 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:11:28.481 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:11:28.481 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:11:28.481 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:11:28.481 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:11:28.482 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:11:28.482 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:11:28.482 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:11:28.482 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:11:28.482 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:11:28.482 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:11:28.482 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:11:28.482 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:11:28.482 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:11:28.482 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:11:28.482 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:11:28.482 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:11:28.482 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:11:28.483 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:11:28.483 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:11:28.483 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:11:28.483 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:11:28.483 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:11:28.483 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:11:28.483 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:11:28.483 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:11:28.483 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:11:28.483 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:11:28.483 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:11:28.483 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:11:28.483 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:11:28.484 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:11:28.484 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:11:28.484 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:11:28.484 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724000726.998407_1395.1", 2024-08-18T17:11:28.484 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:11:28.484 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:11:28.484 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:11:28.484 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:05:26.998407Z", 2024-08-18T17:11:28.484 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:11:28.484 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:11:28.484 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:11:28.484 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:11:28.484 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:11:28.485 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:11:28.485 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:11:28.485 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:11:28.485 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:11:28.485 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:11:28Z" 2024-08-18T17:11:28.485 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:11:28.486 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001026.744398_1436.1 target=1_1724000726.998407_1395.1 2024-08-18T17:11:33.487 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:11:33.487 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:11:33.487 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:11:33.543 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:11:33.543 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:11:33.578 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:11:33.578+0000 7fe034898880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:11:33.579 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:11:33.579+0000 7fe034898880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:11:33.594 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:11:33.594 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:11:33.594 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:11:33.594 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:11:33.594 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:11:33.595 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:11:33.595 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:11:33.595 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:11:33.595 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:11:33.595 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:11:33.595 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:11:33.595 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:11:33.595 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:11:33.595 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:11:33.595 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:11:33.596 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:11:33.596 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:11:33.596 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:11:33.596 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:11:33.596 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:11:33.596 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:11:33.596 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:11:33.596 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:11:33.596 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:11:33.596 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:11:33.596 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:11:33.597 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:11:33.597 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:11:33.597 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:11:33.597 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:11:33.597 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:11:33.597 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:11:33.597 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:11:33.597 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:11:33.597 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:11:33.597 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:11:33.597 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:11:33.598 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:11:33.598 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:11:33.598 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:11:33.598 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:11:33.598 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:11:33.598 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:11:33.598 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:11:33.598 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:11:33.598 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:11:33.599 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:11:33.599 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:11:33.599 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:11:33.599 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724000726.998407_1395.1", 2024-08-18T17:11:33.599 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:11:33.599 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:11:33.599 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:11:33.599 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:05:26.998407Z", 2024-08-18T17:11:33.599 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:11:33.599 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:11:33.599 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:11:33.599 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:11:33.600 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:11:33.600 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:11:33.600 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:11:33.600 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:11:33.600 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:11:33.600 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:11:33Z" 2024-08-18T17:11:33.600 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:11:33.601 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001026.744398_1436.1 target=1_1724000726.998407_1395.1 2024-08-18T17:11:38.602 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:11:38.602 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:11:38.602 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:11:38.656 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:11:38.656 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:11:38.690 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:11:38.689+0000 7f80f9538880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:11:38.691 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:11:38.690+0000 7f80f9538880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:11:38.705 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:11:38.705 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:11:38.706 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:11:38.706 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:11:38.706 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:11:38.706 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:11:38.706 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:11:38.707 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:11:38.707 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:11:38.707 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:11:38.707 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:11:38.707 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:11:38.708 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:11:38.708 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:11:38.708 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:11:38.708 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:11:38.708 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:11:38.708 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:11:38.709 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:11:38.709 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:11:38.709 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:11:38.709 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:11:38.709 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:11:38.710 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:11:38.710 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:11:38.710 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:11:38.710 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:11:38.710 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:11:38.710 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:11:38.711 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:11:38.711 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:11:38.711 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:11:38.712 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:11:38.712 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:11:38.712 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:11:38.712 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:11:38.712 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:11:38.713 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:11:38.713 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:11:38.713 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:11:38.714 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:11:38.714 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:11:38.714 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:11:38.714 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:11:38.714 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:11:38.715 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:11:38.715 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:11:38.715 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:11:38.715 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:11:38.715 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724000726.998407_1395.1", 2024-08-18T17:11:38.716 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:11:38.716 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:11:38.716 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:11:38.716 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:05:26.998407Z", 2024-08-18T17:11:38.716 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:11:38.716 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:11:38.717 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:11:38.717 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:11:38.717 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:11:38.717 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:11:38.717 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:11:38.718 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:11:38.718 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:11:38.718 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:11:38Z" 2024-08-18T17:11:38.718 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:11:38.720 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001026.744398_1436.1 target=1_1724000726.998407_1395.1 2024-08-18T17:11:43.721 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:11:43.721 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:11:43.721 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:11:43.774 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:11:43.775 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:11:43.807 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:11:43.806+0000 7f7d19212880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:11:43.808 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:11:43.807+0000 7f7d19212880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:11:43.822 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:11:43.822 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:11:43.822 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:11:43.822 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:11:43.822 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:11:43.823 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:11:43.823 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:11:43.823 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:11:43.823 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:11:43.823 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:11:43.823 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:11:43.823 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:11:43.823 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:11:43.823 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:11:43.823 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:11:43.823 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:11:43.823 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:11:43.824 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:11:43.824 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:11:43.824 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:11:43.824 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:11:43.824 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:11:43.824 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:11:43.824 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:11:43.824 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:11:43.824 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:11:43.824 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:11:43.824 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:11:43.825 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:11:43.825 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:11:43.825 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:11:43.825 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:11:43.825 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:11:43.825 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:11:43.825 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:11:43.825 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:11:43.825 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:11:43.825 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:11:43.825 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:11:43.826 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:11:43.826 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:11:43.826 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:11:43.826 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:11:43.826 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:11:43.826 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:11:43.826 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:11:43.826 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:11:43.826 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:11:43.826 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:11:43.826 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724000726.998407_1395.1", 2024-08-18T17:11:43.827 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:11:43.827 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:11:43.827 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:11:43.827 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:05:26.998407Z", 2024-08-18T17:11:43.827 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:11:43.827 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:11:43.827 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:11:43.827 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:11:43.827 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:11:43.827 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:11:43.827 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:11:43.828 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:11:43.828 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:11:43.828 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:11:43Z" 2024-08-18T17:11:43.828 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:11:43.829 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001026.744398_1436.1 target=1_1724000726.998407_1395.1 2024-08-18T17:11:48.830 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:11:48.830 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:11:48.830 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:11:48.887 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:11:48.887 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:11:48.920 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:11:48.919+0000 7f3119923880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:11:48.921 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:11:48.920+0000 7f3119923880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:11:48.937 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:11:48.937 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:11:48.937 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:11:48.937 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:11:48.937 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:11:48.937 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:11:48.937 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:11:48.938 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:11:48.938 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:11:48.938 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:11:48.938 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:11:48.938 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:11:48.938 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:11:48.938 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:11:48.938 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:11:48.939 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:11:48.939 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:11:48.939 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:11:48.939 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:11:48.939 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:11:48.939 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:11:48.939 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:11:48.939 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:11:48.939 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:11:48.939 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:11:48.940 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:11:48.940 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:11:48.940 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:11:48.940 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:11:48.940 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:11:48.940 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:11:48.940 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:11:48.940 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:11:48.940 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:11:48.940 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:11:48.940 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:11:48.941 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:11:48.941 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:11:48.941 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:11:48.941 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:11:48.941 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:11:48.941 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:11:48.941 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:11:48.941 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:11:48.941 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:11:48.941 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:11:48.941 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:11:48.941 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:11:48.941 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:11:48.941 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724000726.998407_1395.1", 2024-08-18T17:11:48.942 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:11:48.942 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:11:48.942 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:11:48.942 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:05:26.998407Z", 2024-08-18T17:11:48.942 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:11:48.942 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:11:48.942 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:11:48.942 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:11:48.942 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:11:48.942 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:11:48.942 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:11:48.942 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:11:48.943 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:11:48.943 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:11:48Z" 2024-08-18T17:11:48.943 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:11:48.944 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001026.744398_1436.1 target=1_1724000726.998407_1395.1 2024-08-18T17:11:53.945 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:11:53.946 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:11:53.946 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:11:54.001 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:11:54.001 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:11:54.035 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:11:54.034+0000 7fc70f146880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:11:54.035 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:11:54.035+0000 7fc70f146880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:11:54.050 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:11:54.050 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:11:54.050 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:11:54.051 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:11:54.051 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:11:54.051 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:11:54.051 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:11:54.051 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:11:54.051 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:11:54.051 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:11:54.051 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:11:54.051 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:11:54.051 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:11:54.051 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:11:54.051 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:11:54.051 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:11:54.052 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:11:54.052 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:11:54.052 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:11:54.052 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:11:54.052 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:11:54.052 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:11:54.052 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:11:54.052 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:11:54.052 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:11:54.052 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:11:54.052 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:11:54.052 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:11:54.052 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:11:54.053 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:11:54.053 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:11:54.053 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:11:54.053 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:11:54.053 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:11:54.053 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:11:54.053 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:11:54.053 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:11:54.053 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:11:54.053 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:11:54.053 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:11:54.053 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:11:54.053 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:11:54.053 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:11:54.054 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:11:54.054 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:11:54.054 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:11:54.054 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:11:54.054 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:11:54.054 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:11:54.054 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724000726.998407_1395.1", 2024-08-18T17:11:54.054 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:11:54.054 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:11:54.054 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:11:54.054 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:05:26.998407Z", 2024-08-18T17:11:54.054 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:11:54.054 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:11:54.055 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:11:54.055 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:11:54.055 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:11:54.055 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:11:54.055 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:11:54.055 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:11:54.055 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:11:54.055 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:11:54Z" 2024-08-18T17:11:54.055 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:11:54.056 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001026.744398_1436.1 target=1_1724000726.998407_1395.1 2024-08-18T17:11:56.653 INFO:tasks.rgw.c1.client.0.smithi134.stdout:2024-08-18T17:11:56.652+0000 7f13cd01b640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-08-18T17:11:59.057 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:11:59.057 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:11:59.058 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:11:59.113 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:11:59.113 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:11:59.147 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:11:59.146+0000 7fd626736880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:11:59.148 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:11:59.147+0000 7fd626736880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:11:59.162 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:11:59.163 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:11:59.163 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:11:59.163 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:11:59.163 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:11:59.163 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:11:59.163 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:11:59.163 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:11:59.163 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:11:59.163 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:11:59.163 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:11:59.163 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:11:59.164 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:11:59.164 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:11:59.164 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:11:59.164 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:11:59.164 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:11:59.164 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:11:59.164 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:11:59.164 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:11:59.164 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:11:59.164 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:11:59.164 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:11:59.164 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:11:59.164 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:11:59.165 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:11:59.165 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:11:59.165 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:11:59.165 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:11:59.165 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:11:59.165 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:11:59.165 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:11:59.165 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:11:59.165 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:11:59.165 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:11:59.165 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:11:59.165 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:11:59.166 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:11:59.166 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:11:59.166 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:11:59.166 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:11:59.166 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:11:59.166 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:11:59.166 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:11:59.166 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:11:59.166 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:11:59.166 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:11:59.166 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:11:59.166 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:11:59.167 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724000726.998407_1395.1", 2024-08-18T17:11:59.167 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:11:59.167 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:11:59.167 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:11:59.167 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:05:26.998407Z", 2024-08-18T17:11:59.167 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:11:59.167 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:11:59.167 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:11:59.167 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:11:59.167 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:11:59.167 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:11:59.167 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:11:59.167 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:11:59.167 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:11:59.168 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:11:59Z" 2024-08-18T17:11:59.168 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:11:59.169 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001026.744398_1436.1 target=1_1724000726.998407_1395.1 2024-08-18T17:12:04.170 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:12:04.170 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:12:04.170 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:12:04.226 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:12:04.226 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:12:04.260 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:12:04.259+0000 7fb5de032880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:12:04.261 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:12:04.260+0000 7fb5de032880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:12:04.275 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:12:04.275 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:12:04.276 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:12:04.276 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:12:04.276 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:12:04.276 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:12:04.276 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:12:04.276 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:12:04.276 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:12:04.276 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:12:04.276 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:12:04.276 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:12:04.276 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:12:04.276 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:12:04.276 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:12:04.277 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:12:04.277 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:12:04.277 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:12:04.277 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:12:04.277 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:12:04.277 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:12:04.277 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:12:04.277 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:12:04.277 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:12:04.277 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:12:04.277 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:12:04.277 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:12:04.277 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:12:04.278 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:12:04.278 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:12:04.278 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:12:04.278 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:12:04.278 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:12:04.278 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:12:04.278 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:12:04.278 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:12:04.278 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:12:04.278 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:12:04.278 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:12:04.278 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:12:04.278 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:12:04.279 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:12:04.279 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:12:04.279 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:12:04.279 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:12:04.279 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:12:04.279 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:12:04.279 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:12:04.279 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:12:04.279 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724000726.998407_1395.1", 2024-08-18T17:12:04.279 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:12:04.279 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:12:04.279 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:12:04.280 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:05:26.998407Z", 2024-08-18T17:12:04.280 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:12:04.280 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:12:04.280 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:12:04.280 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:12:04.280 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:12:04.280 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:12:04.280 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:12:04.280 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:12:04.280 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:12:04.280 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:12:04Z" 2024-08-18T17:12:04.280 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:12:04.281 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001026.744398_1436.1 target=1_1724000726.998407_1395.1 2024-08-18T17:12:09.282 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:12:09.283 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:12:09.283 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:12:09.340 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:12:09.340 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:12:09.376 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:12:09.375+0000 7f245dd19880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:12:09.377 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:12:09.376+0000 7f245dd19880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:12:09.392 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:12:09.392 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:12:09.392 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:12:09.393 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:12:09.393 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:12:09.393 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:12:09.393 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:12:09.393 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:12:09.393 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:12:09.394 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:12:09.394 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:12:09.394 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:12:09.394 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:12:09.394 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:12:09.395 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:12:09.395 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:12:09.395 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:12:09.395 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:12:09.395 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:12:09.395 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:12:09.396 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:12:09.396 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:12:09.396 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:12:09.396 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:12:09.396 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:12:09.396 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:12:09.397 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:12:09.397 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:12:09.397 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:12:09.397 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:12:09.397 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:12:09.397 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:12:09.397 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:12:09.398 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:12:09.398 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:12:09.398 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:12:09.398 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:12:09.398 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:12:09.399 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:12:09.399 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:12:09.399 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:12:09.399 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:12:09.400 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:12:09.400 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:12:09.400 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:12:09.400 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:12:09.400 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:12:09.400 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:12:09.400 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:12:09.401 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001026.744398_1436.1", 2024-08-18T17:12:09.401 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:12:09.401 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:12:09.401 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:12:09.401 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:10:26.744398Z", 2024-08-18T17:12:09.401 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:12:09.402 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:12:09.402 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:12:09.402 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:12:09.402 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:12:09.402 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:12:09.402 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:12:09.403 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:12:09.403 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:12:09.403 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:12:09Z" 2024-08-18T17:12:09.403 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:12:09.405 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-08-18T17:12:09.405 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-08-18T17:12:09.405 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:12:09.406 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:12:09.406 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:12:09.461 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:12:09.461 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:12:09.502 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:12:09.500+0000 7f363840e880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:12:09.502 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:12:09.501+0000 7f363840e880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:12:09.521 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T17:12:09.522 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T17:12:09.522 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T17:12:09.522 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T17:12:09.522 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T17:12:09.522 INFO:teuthology.orchestra.run.smithi134.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:12:09.522 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:12:09.522 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:12:09.522 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T17:12:09.522 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:12:09.522 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T17:12:09.522 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:12:09.523 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:12:09.523 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:12:09.523 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:12:09.523 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 1, 2024-08-18T17:12:09.523 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:12:09.523 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:12:09.523 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T17:12:09.523 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:12:09.523 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:12:09.523 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:12:09.523 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T17:12:09.523 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:12:09.523 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:12:09.524 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:12:09.524 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:12:09.524 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:12:09.524 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:12:09.524 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T17:12:09.524 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T17:12:09.524 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:12:09.524 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:12:09.524 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:12:09.524 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T17:12:09.524 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:12:09.524 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:12:09.524 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:12:09.525 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:12:09.525 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:12:09.525 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:12:09.525 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:12:09.525 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:12:09.525 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:12:09.525 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:12:09.525 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:12:09.525 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T17:12:09.526 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:12:09.526 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:12:09.526 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1724001026.744398_1436.1", 2024-08-18T17:12:09.526 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:12:09.526 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:12:09.526 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:12:09.526 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T17:10:26.744398Z", 2024-08-18T17:12:09.526 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:12:09.526 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:12:09.526 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:12:09.526 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:12:09.526 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:12:09.527 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T17:12:09.527 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 1, 2024-08-18T17:12:09.527 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 1 2024-08-18T17:12:09.527 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:12:09.527 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T17:12:09Z" 2024-08-18T17:12:09.527 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T17:12:09.528 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-08-18T17:12:09.528 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--bucket=kibgmb-84', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:12:09.528 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--bucket=kibgmb-84', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:12:09.528 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-bucket --pipe-id sync-pipe --bucket=kibgmb-84 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:12:09.580 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:12:09.580 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:12:09.647 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[],"status":"enabled"}]} 2024-08-18T17:12:09.648 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'remove', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=test-zone1', '--dest-zone=test-zone2', '--bucket', 'kibgmb-84', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:12:09.648 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'remove', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=test-zone1', '--dest-zone=test-zone2', '--bucket', 'kibgmb-84', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:12:09.648 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow remove --group-id sync-bucket --flow-id sync-flow2 --flow-type directional --source-zone=test-zone1 --dest-zone=test-zone2 --bucket kibgmb-84 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:12:09.701 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:12:09.702 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:12:09.775 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}]},"pipes":[],"status":"enabled"}]} 2024-08-18T17:12:09.775 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'remove', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--bucket', 'kibgmb-84', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:12:09.775 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'remove', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--bucket', 'kibgmb-84', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:12:09.775 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow remove --group-id sync-bucket --flow-id sync-flow1 --flow-type symmetrical --zones=test-zone1,test-zone2 --bucket kibgmb-84 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:12:09.829 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:12:09.830 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:12:09.914 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"enabled"}]} 2024-08-18T17:12:09.915 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'kibgmb-84', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:12:09.915 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'kibgmb-84', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:12:09.915 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-bucket --bucket kibgmb-84 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:12:09.963 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:12:09.963 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:12:10.023 INFO:teuthology.orchestra.run.smithi134.stdout:{"sync_policy":{"groups":[]}} 2024-08-18T17:12:10.024 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--bucket', 'kibgmb-84', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:12:10.024 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--bucket', 'kibgmb-84', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:12:10.024 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --bucket kibgmb-84 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:12:10.070 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:12:10.070 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:12:10.141 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[]} 2024-08-18T17:12:10.141 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:12:10.141 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:12:10.142 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:12:10.187 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:12:10.188 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:12:10.241 INFO:teuthology.orchestra.run.smithi134.stderr:No --period given, using current period=01050e30-179b-43be-bdbb-3bb23cf37ae8 2024-08-18T17:12:10.259 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T17:12:10.259 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:12:10.259 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:12:10.259 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T17:12:10.259 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:12:10.260 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:12:10.260 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:12:10.260 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T17:12:10.260 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:12:10.260 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:12:10.260 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:12:10.260 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:42:47.672461Z" 2024-08-18T17:12:10.260 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:12:10.260 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:12:10.260 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1724001130.000617_1445.1", 2024-08-18T17:12:10.260 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T17:12:10.000617Z" 2024-08-18T17:12:10.260 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:12:10.261 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T17:12:10.261 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-08-18T17:12:10.262 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:12:10.262 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:12:10.262 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:12:10.318 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:12:10.318 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:12:10.352 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:12:10.352+0000 7f7ef3146880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:12:10.353 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:12:10.353+0000 7f7ef3146880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:12:10.368 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:12:10.368 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:12:10.368 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:12:10.369 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:12:10.369 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:12:10.369 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:12:10.369 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:12:10.369 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:12:10.369 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:12:10.369 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:12:10.369 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:12:10.369 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:12:10.369 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:12:10.369 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:12:10.369 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:12:10.369 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:12:10.370 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:12:10.370 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:12:10.370 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:12:10.370 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:12:10.370 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:12:10.370 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:12:10.370 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:12:10.370 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:12:10.370 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:12:10.370 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:12:10.370 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:12:10.370 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:12:10.370 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:12:10.371 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:12:10.371 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:12:10.371 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:12:10.371 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:12:10.371 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:12:10.371 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:12:10.371 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:12:10.371 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:12:10.371 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:12:10.371 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:12:10.371 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:12:10.371 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:12:10.371 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:12:10.372 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:12:10.372 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:12:10.372 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:12:10.372 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:12:10.372 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:12:10.372 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:12:10.372 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:12:10.372 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001026.744398_1436.1", 2024-08-18T17:12:10.372 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:12:10.372 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:12:10.372 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:12:10.372 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:10:26.744398Z", 2024-08-18T17:12:10.372 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:12:10.372 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:12:10.373 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:12:10.373 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:12:10.373 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:12:10.373 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:12:10.373 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:12:10.373 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:12:10.373 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:12:10.373 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:12:10Z" 2024-08-18T17:12:10.373 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:12:10.374 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001130.000617_1445.1 target=1_1724001026.744398_1436.1 2024-08-18T17:12:15.375 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:12:15.375 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:12:15.375 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:12:15.430 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:12:15.431 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:12:15.464 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:12:15.463+0000 7fe0fca0e880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:12:15.465 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:12:15.464+0000 7fe0fca0e880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:12:15.480 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:12:15.480 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:12:15.480 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:12:15.480 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:12:15.480 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:12:15.480 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:12:15.481 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:12:15.481 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:12:15.481 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:12:15.481 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:12:15.481 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:12:15.481 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:12:15.481 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:12:15.481 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:12:15.481 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:12:15.481 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:12:15.481 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:12:15.481 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:12:15.481 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:12:15.481 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:12:15.482 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:12:15.482 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:12:15.482 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:12:15.482 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:12:15.482 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:12:15.482 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:12:15.482 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:12:15.482 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:12:15.482 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:12:15.482 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:12:15.482 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:12:15.482 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:12:15.482 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:12:15.483 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:12:15.483 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:12:15.483 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:12:15.483 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:12:15.483 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:12:15.483 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:12:15.483 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:12:15.483 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:12:15.483 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:12:15.483 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:12:15.483 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:12:15.483 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:12:15.484 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:12:15.484 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:12:15.484 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:12:15.484 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:12:15.484 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001026.744398_1436.1", 2024-08-18T17:12:15.484 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:12:15.484 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:12:15.484 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:12:15.484 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:10:26.744398Z", 2024-08-18T17:12:15.484 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:12:15.484 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:12:15.484 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:12:15.484 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:12:15.485 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:12:15.485 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:12:15.485 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:12:15.485 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:12:15.485 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:12:15.485 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:12:15Z" 2024-08-18T17:12:15.485 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:12:15.487 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001130.000617_1445.1 target=1_1724001026.744398_1436.1 2024-08-18T17:12:20.488 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:12:20.488 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:12:20.488 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:12:20.543 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:12:20.543 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:12:20.578 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:12:20.577+0000 7f29abc0f880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:12:20.578 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:12:20.578+0000 7f29abc0f880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:12:20.594 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:12:20.594 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:12:20.594 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:12:20.594 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:12:20.594 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:12:20.594 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:12:20.594 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:12:20.594 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:12:20.594 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:12:20.595 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:12:20.595 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:12:20.595 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:12:20.595 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:12:20.595 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:12:20.595 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:12:20.595 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:12:20.595 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:12:20.595 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:12:20.595 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:12:20.595 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:12:20.595 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:12:20.596 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:12:20.596 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:12:20.596 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:12:20.596 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:12:20.596 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:12:20.596 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:12:20.596 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:12:20.596 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:12:20.596 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:12:20.596 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:12:20.596 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:12:20.596 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:12:20.596 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:12:20.597 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:12:20.597 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:12:20.597 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:12:20.597 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:12:20.597 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:12:20.597 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:12:20.597 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:12:20.597 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:12:20.597 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:12:20.597 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:12:20.597 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:12:20.597 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:12:20.597 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:12:20.597 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:12:20.598 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:12:20.598 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001130.000617_1445.1", 2024-08-18T17:12:20.598 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:12:20.598 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:12:20.598 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:12:20.598 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:12:10.000617Z", 2024-08-18T17:12:20.598 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:12:20.598 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:12:20.598 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:12:20.598 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:12:20.598 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:12:20.599 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:12:20.599 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:12:20.599 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:12:20.599 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:12:20.599 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:12:20Z" 2024-08-18T17:12:20.599 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:12:20.600 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-08-18T17:12:20.600 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-08-18T17:12:20.600 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:12:20.600 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:12:20.601 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:12:20.656 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:12:20.656 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:12:20.703 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:12:20.702+0000 7f52ad924880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:12:20.703 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:12:20.703+0000 7f52ad924880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:12:20.727 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T17:12:20.727 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T17:12:20.727 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T17:12:20.727 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T17:12:20.727 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T17:12:20.727 INFO:teuthology.orchestra.run.smithi134.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:12:20.727 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:12:20.727 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:12:20.728 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T17:12:20.728 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:12:20.728 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T17:12:20.728 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:12:20.728 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:12:20.728 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:12:20.728 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:12:20.728 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 1, 2024-08-18T17:12:20.728 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:12:20.728 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:12:20.728 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T17:12:20.728 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:12:20.728 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:12:20.728 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:12:20.729 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T17:12:20.729 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:12:20.729 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:12:20.729 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:12:20.729 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:12:20.729 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:12:20.729 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:12:20.729 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T17:12:20.729 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T17:12:20.729 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:12:20.729 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:12:20.729 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:12:20.729 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T17:12:20.730 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:12:20.730 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:12:20.730 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:12:20.730 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:12:20.730 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:12:20.730 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:12:20.730 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:12:20.730 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:12:20.730 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:12:20.730 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:12:20.730 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:12:20.730 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T17:12:20.730 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:12:20.731 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:12:20.731 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1724001130.000617_1445.1", 2024-08-18T17:12:20.731 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:12:20.731 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:12:20.731 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:12:20.731 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T17:12:10.000617Z", 2024-08-18T17:12:20.731 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:12:20.731 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:12:20.731 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:12:20.731 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:12:20.731 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:12:20.731 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T17:12:20.731 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 1, 2024-08-18T17:12:20.731 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 1 2024-08-18T17:12:20.732 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:12:20.732 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T17:12:20Z" 2024-08-18T17:12:20.732 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T17:12:20.732 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-08-18T17:12:20.732 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:12:20.732 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:12:20.732 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-group --pipe-id sync-pipe --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:12:20.782 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:12:20.782 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:12:20.857 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[],"status":"enabled"}]} 2024-08-18T17:12:20.858 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=test-zone1', '--dest-zone=test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:12:20.858 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=test-zone1', '--dest-zone=test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:12:20.858 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow remove --group-id sync-group --flow-id sync-flow2 --flow-type directional --source-zone=test-zone1 --dest-zone=test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:12:20.910 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:12:20.910 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:12:20.981 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}]},"pipes":[],"status":"enabled"}]} 2024-08-18T17:12:20.982 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:12:20.982 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:12:20.982 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow remove --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:12:21.032 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:12:21.032 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:12:21.102 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"enabled"}]} 2024-08-18T17:12:21.103 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:12:21.103 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:12:21.103 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:12:21.149 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:12:21.149 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:12:21.219 INFO:teuthology.orchestra.run.smithi134.stdout:{"sync_policy":{"groups":[]}} 2024-08-18T17:12:21.220 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:12:21.220 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:12:21.220 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:12:21.263 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:12:21.264 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:12:21.322 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[]} 2024-08-18T17:12:21.323 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:12:21.323 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:12:21.323 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:12:21.370 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:12:21.370 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:12:21.421 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T17:12:21.421 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:12:21.422 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 11, 2024-08-18T17:12:21.422 INFO:teuthology.orchestra.run.smithi134.stdout: "predecessor_uuid": "64309209-7567-46e5-8881-65783d84cf3c", 2024-08-18T17:12:21.422 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": [ 2024-08-18T17:12:21.422 INFO:teuthology.orchestra.run.smithi134.stdout: "", 2024-08-18T17:12:21.422 INFO:teuthology.orchestra.run.smithi134.stdout: "", 2024-08-18T17:12:21.422 INFO:teuthology.orchestra.run.smithi134.stdout: "", 2024-08-18T17:12:21.423 INFO:teuthology.orchestra.run.smithi134.stdout: "1_1723995848.930494_187.1" 2024-08-18T17:12:21.423 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:12:21.423 INFO:teuthology.orchestra.run.smithi134.stdout: "period_map": { 2024-08-18T17:12:21.423 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:12:21.423 INFO:teuthology.orchestra.run.smithi134.stdout: "zonegroups": [ 2024-08-18T17:12:21.423 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:12:21.423 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "test-zonegroup", 2024-08-18T17:12:21.423 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "test-zonegroup", 2024-08-18T17:12:21.423 INFO:teuthology.orchestra.run.smithi134.stdout: "api_name": "", 2024-08-18T17:12:21.423 INFO:teuthology.orchestra.run.smithi134.stdout: "is_master": true, 2024-08-18T17:12:21.423 INFO:teuthology.orchestra.run.smithi134.stdout: "endpoints": [ 2024-08-18T17:12:21.423 INFO:teuthology.orchestra.run.smithi134.stdout: "http://smithi134.front.sepia.ceph.com:8000" 2024-08-18T17:12:21.423 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:12:21.423 INFO:teuthology.orchestra.run.smithi134.stdout: "hostnames": [], 2024-08-18T17:12:21.424 INFO:teuthology.orchestra.run.smithi134.stdout: "hostnames_s3website": [], 2024-08-18T17:12:21.424 INFO:teuthology.orchestra.run.smithi134.stdout: "master_zone": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T17:12:21.424 INFO:teuthology.orchestra.run.smithi134.stdout: "zones": [ 2024-08-18T17:12:21.424 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:12:21.424 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T17:12:21.424 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "test-zone3", 2024-08-18T17:12:21.424 INFO:teuthology.orchestra.run.smithi134.stdout: "endpoints": [ 2024-08-18T17:12:21.424 INFO:teuthology.orchestra.run.smithi134.stdout: "http://smithi134.front.sepia.ceph.com:8001" 2024-08-18T17:12:21.424 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:12:21.424 INFO:teuthology.orchestra.run.smithi134.stdout: "log_meta": false, 2024-08-18T17:12:21.424 INFO:teuthology.orchestra.run.smithi134.stdout: "log_data": true, 2024-08-18T17:12:21.424 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_index_max_shards": 11, 2024-08-18T17:12:21.425 INFO:teuthology.orchestra.run.smithi134.stdout: "read_only": false, 2024-08-18T17:12:21.425 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_type": "", 2024-08-18T17:12:21.425 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from_all": true, 2024-08-18T17:12:21.425 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from": [], 2024-08-18T17:12:21.425 INFO:teuthology.orchestra.run.smithi134.stdout: "redirect_zone": "", 2024-08-18T17:12:21.425 INFO:teuthology.orchestra.run.smithi134.stdout: "supported_features": [ 2024-08-18T17:12:21.425 INFO:teuthology.orchestra.run.smithi134.stdout: "compress-encrypted", 2024-08-18T17:12:21.425 INFO:teuthology.orchestra.run.smithi134.stdout: "resharding" 2024-08-18T17:12:21.425 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:12:21.425 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:12:21.425 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:12:21.425 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "10090ccf-aad6-416d-9775-df8b7b27345a", 2024-08-18T17:12:21.425 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "test-zone2", 2024-08-18T17:12:21.426 INFO:teuthology.orchestra.run.smithi134.stdout: "endpoints": [ 2024-08-18T17:12:21.426 INFO:teuthology.orchestra.run.smithi134.stdout: "http://smithi193.front.sepia.ceph.com:8000" 2024-08-18T17:12:21.426 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:12:21.426 INFO:teuthology.orchestra.run.smithi134.stdout: "log_meta": false, 2024-08-18T17:12:21.426 INFO:teuthology.orchestra.run.smithi134.stdout: "log_data": true, 2024-08-18T17:12:21.426 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_index_max_shards": 11, 2024-08-18T17:12:21.426 INFO:teuthology.orchestra.run.smithi134.stdout: "read_only": false, 2024-08-18T17:12:21.426 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_type": "", 2024-08-18T17:12:21.426 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from_all": true, 2024-08-18T17:12:21.426 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from": [], 2024-08-18T17:12:21.426 INFO:teuthology.orchestra.run.smithi134.stdout: "redirect_zone": "", 2024-08-18T17:12:21.426 INFO:teuthology.orchestra.run.smithi134.stdout: "supported_features": [ 2024-08-18T17:12:21.426 INFO:teuthology.orchestra.run.smithi134.stdout: "compress-encrypted", 2024-08-18T17:12:21.427 INFO:teuthology.orchestra.run.smithi134.stdout: "resharding" 2024-08-18T17:12:21.427 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:12:21.427 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:12:21.427 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:12:21.427 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T17:12:21.427 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "test-zone1", 2024-08-18T17:12:21.427 INFO:teuthology.orchestra.run.smithi134.stdout: "endpoints": [ 2024-08-18T17:12:21.427 INFO:teuthology.orchestra.run.smithi134.stdout: "http://smithi134.front.sepia.ceph.com:8000" 2024-08-18T17:12:21.427 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:12:21.427 INFO:teuthology.orchestra.run.smithi134.stdout: "log_meta": false, 2024-08-18T17:12:21.427 INFO:teuthology.orchestra.run.smithi134.stdout: "log_data": true, 2024-08-18T17:12:21.427 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_index_max_shards": 11, 2024-08-18T17:12:21.427 INFO:teuthology.orchestra.run.smithi134.stdout: "read_only": false, 2024-08-18T17:12:21.427 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_type": "", 2024-08-18T17:12:21.428 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from_all": true, 2024-08-18T17:12:21.428 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from": [], 2024-08-18T17:12:21.428 INFO:teuthology.orchestra.run.smithi134.stdout: "redirect_zone": "", 2024-08-18T17:12:21.428 INFO:teuthology.orchestra.run.smithi134.stdout: "supported_features": [ 2024-08-18T17:12:21.428 INFO:teuthology.orchestra.run.smithi134.stdout: "compress-encrypted", 2024-08-18T17:12:21.428 INFO:teuthology.orchestra.run.smithi134.stdout: "resharding" 2024-08-18T17:12:21.428 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:12:21.428 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:12:21.428 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:12:21.428 INFO:teuthology.orchestra.run.smithi134.stdout: "placement_targets": [ 2024-08-18T17:12:21.428 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:12:21.428 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "default-placement", 2024-08-18T17:12:21.428 INFO:teuthology.orchestra.run.smithi134.stdout: "tags": [], 2024-08-18T17:12:21.429 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_classes": [ 2024-08-18T17:12:21.429 INFO:teuthology.orchestra.run.smithi134.stdout: "STANDARD" 2024-08-18T17:12:21.429 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:12:21.429 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:12:21.429 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:12:21.429 INFO:teuthology.orchestra.run.smithi134.stdout: "default_placement": "default-placement", 2024-08-18T17:12:21.429 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_id": "2827c6f0-bf0d-4836-944c-9e7fcf22457b", 2024-08-18T17:12:21.429 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_policy": { 2024-08-18T17:12:21.429 INFO:teuthology.orchestra.run.smithi134.stdout: "groups": [] 2024-08-18T17:12:21.429 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:12:21.429 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled_features": [ 2024-08-18T17:12:21.430 INFO:teuthology.orchestra.run.smithi134.stdout: "resharding" 2024-08-18T17:12:21.430 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:12:21.430 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:12:21.430 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:12:21.430 INFO:teuthology.orchestra.run.smithi134.stdout: "short_zone_ids": [ 2024-08-18T17:12:21.430 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:12:21.431 INFO:teuthology.orchestra.run.smithi134.stdout: "key": "018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T17:12:21.431 INFO:teuthology.orchestra.run.smithi134.stdout: "val": 462036791 2024-08-18T17:12:21.431 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:12:21.431 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:12:21.438 INFO:teuthology.orchestra.run.smithi134.stdout: "key": "10090ccf-aad6-416d-9775-df8b7b27345a", 2024-08-18T17:12:21.439 INFO:teuthology.orchestra.run.smithi134.stdout: "val": 874289624 2024-08-18T17:12:21.439 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:12:21.439 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:12:21.439 INFO:teuthology.orchestra.run.smithi134.stdout: "key": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T17:12:21.439 INFO:teuthology.orchestra.run.smithi134.stdout: "val": 1721399886 2024-08-18T17:12:21.439 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:12:21.439 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:12:21.439 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:12:21.439 INFO:teuthology.orchestra.run.smithi134.stdout: "master_zonegroup": "test-zonegroup", 2024-08-18T17:12:21.439 INFO:teuthology.orchestra.run.smithi134.stdout: "master_zone": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T17:12:21.440 INFO:teuthology.orchestra.run.smithi134.stdout: "period_config": { 2024-08-18T17:12:21.440 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_quota": { 2024-08-18T17:12:21.440 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false, 2024-08-18T17:12:21.440 INFO:teuthology.orchestra.run.smithi134.stdout: "check_on_raw": false, 2024-08-18T17:12:21.440 INFO:teuthology.orchestra.run.smithi134.stdout: "max_size": -1, 2024-08-18T17:12:21.440 INFO:teuthology.orchestra.run.smithi134.stdout: "max_size_kb": 0, 2024-08-18T17:12:21.440 INFO:teuthology.orchestra.run.smithi134.stdout: "max_objects": -1 2024-08-18T17:12:21.440 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:12:21.440 INFO:teuthology.orchestra.run.smithi134.stdout: "user_quota": { 2024-08-18T17:12:21.440 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false, 2024-08-18T17:12:21.440 INFO:teuthology.orchestra.run.smithi134.stdout: "check_on_raw": false, 2024-08-18T17:12:21.440 INFO:teuthology.orchestra.run.smithi134.stdout: "max_size": -1, 2024-08-18T17:12:21.440 INFO:teuthology.orchestra.run.smithi134.stdout: "max_size_kb": 0, 2024-08-18T17:12:21.441 INFO:teuthology.orchestra.run.smithi134.stdout: "max_objects": -1 2024-08-18T17:12:21.441 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:12:21.441 INFO:teuthology.orchestra.run.smithi134.stdout: "user_ratelimit": { 2024-08-18T17:12:21.441 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_ops": 0, 2024-08-18T17:12:21.441 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_ops": 0, 2024-08-18T17:12:21.441 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_bytes": 0, 2024-08-18T17:12:21.441 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_bytes": 0, 2024-08-18T17:12:21.441 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false 2024-08-18T17:12:21.441 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:12:21.441 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_ratelimit": { 2024-08-18T17:12:21.441 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_ops": 0, 2024-08-18T17:12:21.441 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_ops": 0, 2024-08-18T17:12:21.441 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_bytes": 0, 2024-08-18T17:12:21.441 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_bytes": 0, 2024-08-18T17:12:21.442 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false 2024-08-18T17:12:21.442 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:12:21.442 INFO:teuthology.orchestra.run.smithi134.stdout: "anonymous_ratelimit": { 2024-08-18T17:12:21.442 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_ops": 0, 2024-08-18T17:12:21.442 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_ops": 0, 2024-08-18T17:12:21.442 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_bytes": 0, 2024-08-18T17:12:21.442 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_bytes": 0, 2024-08-18T17:12:21.442 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false 2024-08-18T17:12:21.442 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:12:21.442 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:12:21.443 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_id": "2827c6f0-bf0d-4836-944c-9e7fcf22457b", 2024-08-18T17:12:21.443 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_name": "test-realm", 2024-08-18T17:12:21.443 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:12:21.443 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T17:12:21.444 INFO:tasks.rgw_multisite_tests:test_sync_policy_config_zonegroup: ... ok 2024-08-18T17:12:21.445 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:12:21.445 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:12:21.445 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:12:21.497 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:12:21.497 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:12:21.564 INFO:teuthology.orchestra.run.smithi134.stderr:No --period given, using current period=01050e30-179b-43be-bdbb-3bb23cf37ae8 2024-08-18T17:12:21.582 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T17:12:21.582 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:12:21.582 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:12:21.582 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T17:12:21.582 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:12:21.582 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:12:21.582 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:12:21.582 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T17:12:21.583 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:12:21.583 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:12:21.583 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:12:21.583 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:42:47.672461Z" 2024-08-18T17:12:21.583 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:12:21.583 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:12:21.583 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1724001130.000617_1445.1", 2024-08-18T17:12:21.583 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T17:12:10.000617Z" 2024-08-18T17:12:21.583 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:12:21.583 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T17:12:21.584 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-08-18T17:12:21.584 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:12:21.584 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:12:21.585 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:12:21.646 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:12:21.646 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:12:21.685 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:12:21.684+0000 7f3a0fa0f880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:12:21.686 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:12:21.685+0000 7f3a0fa0f880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:12:21.703 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:12:21.703 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:12:21.703 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:12:21.703 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:12:21.703 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:12:21.703 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:12:21.703 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:12:21.703 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:12:21.704 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:12:21.704 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:12:21.704 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:12:21.704 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:12:21.704 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:12:21.704 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:12:21.704 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:12:21.704 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:12:21.704 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:12:21.704 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:12:21.704 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:12:21.704 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:12:21.705 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:12:21.705 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:12:21.705 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:12:21.705 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:12:21.705 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:12:21.705 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:12:21.705 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:12:21.705 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:12:21.705 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:12:21.705 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:12:21.705 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:12:21.705 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:12:21.705 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:12:21.705 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:12:21.706 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:12:21.706 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:12:21.706 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:12:21.706 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:12:21.706 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:12:21.706 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:12:21.706 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:12:21.706 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:12:21.706 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:12:21.706 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:12:21.706 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:12:21.706 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:12:21.706 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:12:21.707 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:12:21.707 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:12:21.707 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001130.000617_1445.1", 2024-08-18T17:12:21.707 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:12:21.707 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:12:21.707 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:12:21.707 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:12:10.000617Z", 2024-08-18T17:12:21.707 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:12:21.707 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:12:21.707 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:12:21.707 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:12:21.707 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:12:21.707 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:12:21.708 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:12:21.708 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:12:21.708 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:12:21.708 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:12:21Z" 2024-08-18T17:12:21.708 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:12:21.709 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-08-18T17:12:21.709 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-08-18T17:12:21.709 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:12:21.709 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:12:21.710 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:12:21.784 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:12:21.784 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:12:21.870 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:12:21.869+0000 7fb8f4731880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:12:21.871 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:12:21.871+0000 7fb8f4731880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:12:21.910 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T17:12:21.910 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T17:12:21.910 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T17:12:21.910 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T17:12:21.910 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T17:12:21.910 INFO:teuthology.orchestra.run.smithi134.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:12:21.911 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:12:21.911 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:12:21.911 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T17:12:21.911 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:12:21.911 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T17:12:21.911 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:12:21.911 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:12:21.911 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:12:21.911 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:12:21.911 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 1, 2024-08-18T17:12:21.911 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:12:21.911 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:12:21.912 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T17:12:21.912 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:12:21.912 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:12:21.912 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:12:21.912 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T17:12:21.912 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:12:21.912 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:12:21.912 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:12:21.912 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:12:21.912 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:12:21.912 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:12:21.912 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T17:12:21.912 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T17:12:21.913 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:12:21.913 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:12:21.913 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:12:21.913 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T17:12:21.913 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:12:21.913 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:12:21.913 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:12:21.913 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:12:21.913 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:12:21.913 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:12:21.913 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:12:21.913 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:12:21.913 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:12:21.914 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:12:21.914 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:12:21.914 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T17:12:21.914 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:12:21.914 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:12:21.914 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1724001130.000617_1445.1", 2024-08-18T17:12:21.914 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:12:21.914 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:12:21.914 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:12:21.914 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T17:12:10.000617Z", 2024-08-18T17:12:21.914 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:12:21.914 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:12:21.914 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:12:21.915 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:12:21.915 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:12:21.915 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T17:12:21.915 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 1, 2024-08-18T17:12:21.915 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 1 2024-08-18T17:12:21.915 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:12:21.915 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T17:12:21Z" 2024-08-18T17:12:21.915 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T17:12:21.916 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-08-18T17:12:21.916 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:12:21.916 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:12:21.917 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:12:21.975 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:12:21.975 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:12:22.088 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[]} 2024-08-18T17:12:22.088 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:12:22.088 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:12:22.088 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:12:22.145 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:12:22.145 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:12:22.274 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2024-08-18T17:12:22.274 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:12:22.274 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:12:22.275 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=test-zone1,test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:12:22.333 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:12:22.334 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:12:22.458 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}]},"pipes":[],"status":"allowed"}]} 2024-08-18T17:12:22.459 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1,test-zone2', '--dest-zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:12:22.459 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1,test-zone2', '--dest-zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:12:22.459 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=test-zone1,test-zone2 --dest-zones=test-zone1,test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:12:22.518 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:12:22.518 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:12:22.619 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-08-18T17:12:22.619 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:12:22.619 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:12:22.619 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:12:22.667 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:12:22.668 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:12:22.758 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-08-18T17:12:22.758 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:12:22.758 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:12:22.759 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:12:22.812 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:12:22.812 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:12:22.875 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T17:12:22.876 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:12:22.876 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 12, 2024-08-18T17:12:22.876 INFO:teuthology.orchestra.run.smithi134.stdout: "predecessor_uuid": "64309209-7567-46e5-8881-65783d84cf3c", 2024-08-18T17:12:22.876 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": [ 2024-08-18T17:12:22.876 INFO:teuthology.orchestra.run.smithi134.stdout: "", 2024-08-18T17:12:22.876 INFO:teuthology.orchestra.run.smithi134.stdout: "", 2024-08-18T17:12:22.876 INFO:teuthology.orchestra.run.smithi134.stdout: "", 2024-08-18T17:12:22.876 INFO:teuthology.orchestra.run.smithi134.stdout: "1_1723995848.930494_187.1" 2024-08-18T17:12:22.876 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:12:22.876 INFO:teuthology.orchestra.run.smithi134.stdout: "period_map": { 2024-08-18T17:12:22.876 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:12:22.877 INFO:teuthology.orchestra.run.smithi134.stdout: "zonegroups": [ 2024-08-18T17:12:22.877 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:12:22.877 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "test-zonegroup", 2024-08-18T17:12:22.877 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "test-zonegroup", 2024-08-18T17:12:22.877 INFO:teuthology.orchestra.run.smithi134.stdout: "api_name": "", 2024-08-18T17:12:22.877 INFO:teuthology.orchestra.run.smithi134.stdout: "is_master": true, 2024-08-18T17:12:22.877 INFO:teuthology.orchestra.run.smithi134.stdout: "endpoints": [ 2024-08-18T17:12:22.877 INFO:teuthology.orchestra.run.smithi134.stdout: "http://smithi134.front.sepia.ceph.com:8000" 2024-08-18T17:12:22.877 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:12:22.877 INFO:teuthology.orchestra.run.smithi134.stdout: "hostnames": [], 2024-08-18T17:12:22.877 INFO:teuthology.orchestra.run.smithi134.stdout: "hostnames_s3website": [], 2024-08-18T17:12:22.877 INFO:teuthology.orchestra.run.smithi134.stdout: "master_zone": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T17:12:22.878 INFO:teuthology.orchestra.run.smithi134.stdout: "zones": [ 2024-08-18T17:12:22.878 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:12:22.878 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T17:12:22.878 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "test-zone3", 2024-08-18T17:12:22.878 INFO:teuthology.orchestra.run.smithi134.stdout: "endpoints": [ 2024-08-18T17:12:22.878 INFO:teuthology.orchestra.run.smithi134.stdout: "http://smithi134.front.sepia.ceph.com:8001" 2024-08-18T17:12:22.878 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:12:22.878 INFO:teuthology.orchestra.run.smithi134.stdout: "log_meta": false, 2024-08-18T17:12:22.878 INFO:teuthology.orchestra.run.smithi134.stdout: "log_data": true, 2024-08-18T17:12:22.878 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_index_max_shards": 11, 2024-08-18T17:12:22.878 INFO:teuthology.orchestra.run.smithi134.stdout: "read_only": false, 2024-08-18T17:12:22.878 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_type": "", 2024-08-18T17:12:22.878 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from_all": true, 2024-08-18T17:12:22.879 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from": [], 2024-08-18T17:12:22.879 INFO:teuthology.orchestra.run.smithi134.stdout: "redirect_zone": "", 2024-08-18T17:12:22.879 INFO:teuthology.orchestra.run.smithi134.stdout: "supported_features": [ 2024-08-18T17:12:22.879 INFO:teuthology.orchestra.run.smithi134.stdout: "compress-encrypted", 2024-08-18T17:12:22.879 INFO:teuthology.orchestra.run.smithi134.stdout: "resharding" 2024-08-18T17:12:22.879 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:12:22.879 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:12:22.879 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:12:22.879 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "10090ccf-aad6-416d-9775-df8b7b27345a", 2024-08-18T17:12:22.879 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "test-zone2", 2024-08-18T17:12:22.879 INFO:teuthology.orchestra.run.smithi134.stdout: "endpoints": [ 2024-08-18T17:12:22.879 INFO:teuthology.orchestra.run.smithi134.stdout: "http://smithi193.front.sepia.ceph.com:8000" 2024-08-18T17:12:22.880 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:12:22.880 INFO:teuthology.orchestra.run.smithi134.stdout: "log_meta": false, 2024-08-18T17:12:22.880 INFO:teuthology.orchestra.run.smithi134.stdout: "log_data": true, 2024-08-18T17:12:22.880 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_index_max_shards": 11, 2024-08-18T17:12:22.880 INFO:teuthology.orchestra.run.smithi134.stdout: "read_only": false, 2024-08-18T17:12:22.880 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_type": "", 2024-08-18T17:12:22.880 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from_all": true, 2024-08-18T17:12:22.880 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from": [], 2024-08-18T17:12:22.880 INFO:teuthology.orchestra.run.smithi134.stdout: "redirect_zone": "", 2024-08-18T17:12:22.880 INFO:teuthology.orchestra.run.smithi134.stdout: "supported_features": [ 2024-08-18T17:12:22.880 INFO:teuthology.orchestra.run.smithi134.stdout: "compress-encrypted", 2024-08-18T17:12:22.880 INFO:teuthology.orchestra.run.smithi134.stdout: "resharding" 2024-08-18T17:12:22.880 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:12:22.881 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:12:22.881 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:12:22.881 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T17:12:22.881 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "test-zone1", 2024-08-18T17:12:22.881 INFO:teuthology.orchestra.run.smithi134.stdout: "endpoints": [ 2024-08-18T17:12:22.881 INFO:teuthology.orchestra.run.smithi134.stdout: "http://smithi134.front.sepia.ceph.com:8000" 2024-08-18T17:12:22.881 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:12:22.881 INFO:teuthology.orchestra.run.smithi134.stdout: "log_meta": false, 2024-08-18T17:12:22.881 INFO:teuthology.orchestra.run.smithi134.stdout: "log_data": true, 2024-08-18T17:12:22.881 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_index_max_shards": 11, 2024-08-18T17:12:22.881 INFO:teuthology.orchestra.run.smithi134.stdout: "read_only": false, 2024-08-18T17:12:22.881 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_type": "", 2024-08-18T17:12:22.881 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from_all": true, 2024-08-18T17:12:22.881 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from": [], 2024-08-18T17:12:22.882 INFO:teuthology.orchestra.run.smithi134.stdout: "redirect_zone": "", 2024-08-18T17:12:22.882 INFO:teuthology.orchestra.run.smithi134.stdout: "supported_features": [ 2024-08-18T17:12:22.882 INFO:teuthology.orchestra.run.smithi134.stdout: "compress-encrypted", 2024-08-18T17:12:22.882 INFO:teuthology.orchestra.run.smithi134.stdout: "resharding" 2024-08-18T17:12:22.882 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:12:22.882 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:12:22.882 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:12:22.882 INFO:teuthology.orchestra.run.smithi134.stdout: "placement_targets": [ 2024-08-18T17:12:22.882 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:12:22.882 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "default-placement", 2024-08-18T17:12:22.882 INFO:teuthology.orchestra.run.smithi134.stdout: "tags": [], 2024-08-18T17:12:22.882 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_classes": [ 2024-08-18T17:12:22.882 INFO:teuthology.orchestra.run.smithi134.stdout: "STANDARD" 2024-08-18T17:12:22.883 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:12:22.883 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:12:22.883 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:12:22.883 INFO:teuthology.orchestra.run.smithi134.stdout: "default_placement": "default-placement", 2024-08-18T17:12:22.883 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_id": "2827c6f0-bf0d-4836-944c-9e7fcf22457b", 2024-08-18T17:12:22.883 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_policy": { 2024-08-18T17:12:22.883 INFO:teuthology.orchestra.run.smithi134.stdout: "groups": [ 2024-08-18T17:12:22.883 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:12:22.883 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "sync-group", 2024-08-18T17:12:22.883 INFO:teuthology.orchestra.run.smithi134.stdout: "data_flow": { 2024-08-18T17:12:22.883 INFO:teuthology.orchestra.run.smithi134.stdout: "symmetrical": [ 2024-08-18T17:12:22.883 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:12:22.883 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "sync-flow1", 2024-08-18T17:12:22.884 INFO:teuthology.orchestra.run.smithi134.stdout: "zones": [ 2024-08-18T17:12:22.891 INFO:teuthology.orchestra.run.smithi134.stdout: "10090ccf-aad6-416d-9775-df8b7b27345a", 2024-08-18T17:12:22.891 INFO:teuthology.orchestra.run.smithi134.stdout: "96bf8502-2f39-4d53-8196-3378c28da3b4" 2024-08-18T17:12:22.891 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:12:22.891 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:12:22.891 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:12:22.891 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:12:22.891 INFO:teuthology.orchestra.run.smithi134.stdout: "pipes": [ 2024-08-18T17:12:22.892 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:12:22.892 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "sync-pipe", 2024-08-18T17:12:22.892 INFO:teuthology.orchestra.run.smithi134.stdout: "source": { 2024-08-18T17:12:22.892 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket": "*", 2024-08-18T17:12:22.892 INFO:teuthology.orchestra.run.smithi134.stdout: "zones": [ 2024-08-18T17:12:22.892 INFO:teuthology.orchestra.run.smithi134.stdout: "10090ccf-aad6-416d-9775-df8b7b27345a", 2024-08-18T17:12:22.892 INFO:teuthology.orchestra.run.smithi134.stdout: "96bf8502-2f39-4d53-8196-3378c28da3b4" 2024-08-18T17:12:22.892 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:12:22.892 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:12:22.892 INFO:teuthology.orchestra.run.smithi134.stdout: "dest": { 2024-08-18T17:12:22.892 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket": "*", 2024-08-18T17:12:22.892 INFO:teuthology.orchestra.run.smithi134.stdout: "zones": [ 2024-08-18T17:12:22.892 INFO:teuthology.orchestra.run.smithi134.stdout: "10090ccf-aad6-416d-9775-df8b7b27345a", 2024-08-18T17:12:22.893 INFO:teuthology.orchestra.run.smithi134.stdout: "96bf8502-2f39-4d53-8196-3378c28da3b4" 2024-08-18T17:12:22.893 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:12:22.893 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:12:22.893 INFO:teuthology.orchestra.run.smithi134.stdout: "params": { 2024-08-18T17:12:22.893 INFO:teuthology.orchestra.run.smithi134.stdout: "source": { 2024-08-18T17:12:22.893 INFO:teuthology.orchestra.run.smithi134.stdout: "filter": { 2024-08-18T17:12:22.893 INFO:teuthology.orchestra.run.smithi134.stdout: "tags": [] 2024-08-18T17:12:22.893 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:12:22.893 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:12:22.893 INFO:teuthology.orchestra.run.smithi134.stdout: "dest": {}, 2024-08-18T17:12:22.893 INFO:teuthology.orchestra.run.smithi134.stdout: "priority": 0, 2024-08-18T17:12:22.893 INFO:teuthology.orchestra.run.smithi134.stdout: "mode": "system", 2024-08-18T17:12:22.893 INFO:teuthology.orchestra.run.smithi134.stdout: "user": "" 2024-08-18T17:12:22.894 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:12:22.894 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:12:22.894 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:12:22.894 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "enabled" 2024-08-18T17:12:22.894 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:12:22.894 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:12:22.894 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:12:22.894 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled_features": [ 2024-08-18T17:12:22.894 INFO:teuthology.orchestra.run.smithi134.stdout: "resharding" 2024-08-18T17:12:22.894 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:12:22.894 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:12:22.895 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:12:22.895 INFO:teuthology.orchestra.run.smithi134.stdout: "short_zone_ids": [ 2024-08-18T17:12:22.895 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:12:22.895 INFO:teuthology.orchestra.run.smithi134.stdout: "key": "018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T17:12:22.895 INFO:teuthology.orchestra.run.smithi134.stdout: "val": 462036791 2024-08-18T17:12:22.895 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:12:22.895 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:12:22.895 INFO:teuthology.orchestra.run.smithi134.stdout: "key": "10090ccf-aad6-416d-9775-df8b7b27345a", 2024-08-18T17:12:22.895 INFO:teuthology.orchestra.run.smithi134.stdout: "val": 874289624 2024-08-18T17:12:22.895 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:12:22.895 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:12:22.895 INFO:teuthology.orchestra.run.smithi134.stdout: "key": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T17:12:22.895 INFO:teuthology.orchestra.run.smithi134.stdout: "val": 1721399886 2024-08-18T17:12:22.895 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:12:22.896 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:12:22.896 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:12:22.896 INFO:teuthology.orchestra.run.smithi134.stdout: "master_zonegroup": "test-zonegroup", 2024-08-18T17:12:22.896 INFO:teuthology.orchestra.run.smithi134.stdout: "master_zone": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T17:12:22.896 INFO:teuthology.orchestra.run.smithi134.stdout: "period_config": { 2024-08-18T17:12:22.896 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_quota": { 2024-08-18T17:12:22.896 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false, 2024-08-18T17:12:22.896 INFO:teuthology.orchestra.run.smithi134.stdout: "check_on_raw": false, 2024-08-18T17:12:22.896 INFO:teuthology.orchestra.run.smithi134.stdout: "max_size": -1, 2024-08-18T17:12:22.896 INFO:teuthology.orchestra.run.smithi134.stdout: "max_size_kb": 0, 2024-08-18T17:12:22.896 INFO:teuthology.orchestra.run.smithi134.stdout: "max_objects": -1 2024-08-18T17:12:22.896 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:12:22.896 INFO:teuthology.orchestra.run.smithi134.stdout: "user_quota": { 2024-08-18T17:12:22.897 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false, 2024-08-18T17:12:22.897 INFO:teuthology.orchestra.run.smithi134.stdout: "check_on_raw": false, 2024-08-18T17:12:22.897 INFO:teuthology.orchestra.run.smithi134.stdout: "max_size": -1, 2024-08-18T17:12:22.897 INFO:teuthology.orchestra.run.smithi134.stdout: "max_size_kb": 0, 2024-08-18T17:12:22.897 INFO:teuthology.orchestra.run.smithi134.stdout: "max_objects": -1 2024-08-18T17:12:22.897 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:12:22.897 INFO:teuthology.orchestra.run.smithi134.stdout: "user_ratelimit": { 2024-08-18T17:12:22.897 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_ops": 0, 2024-08-18T17:12:22.897 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_ops": 0, 2024-08-18T17:12:22.897 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_bytes": 0, 2024-08-18T17:12:22.897 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_bytes": 0, 2024-08-18T17:12:22.897 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false 2024-08-18T17:12:22.897 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:12:22.897 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_ratelimit": { 2024-08-18T17:12:22.898 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_ops": 0, 2024-08-18T17:12:22.898 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_ops": 0, 2024-08-18T17:12:22.898 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_bytes": 0, 2024-08-18T17:12:22.898 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_bytes": 0, 2024-08-18T17:12:22.898 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false 2024-08-18T17:12:22.898 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:12:22.898 INFO:teuthology.orchestra.run.smithi134.stdout: "anonymous_ratelimit": { 2024-08-18T17:12:22.898 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_ops": 0, 2024-08-18T17:12:22.898 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_ops": 0, 2024-08-18T17:12:22.898 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_bytes": 0, 2024-08-18T17:12:22.898 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_bytes": 0, 2024-08-18T17:12:22.898 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false 2024-08-18T17:12:22.898 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:12:22.899 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:12:22.899 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_id": "2827c6f0-bf0d-4836-944c-9e7fcf22457b", 2024-08-18T17:12:22.899 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_name": "test-realm", 2024-08-18T17:12:22.899 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:12:22.899 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T17:12:22.900 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:12:22.900 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:12:22.900 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:12:22.944 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:12:22.944 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:12:22.998 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-08-18T17:12:22.998 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=kibgmb-85 2024-08-18T17:12:23.356 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=kibgmb-86 2024-08-18T17:12:23.423 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:12:23.423 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:12:23.423 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:12:23.477 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:12:23.477 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:12:23.542 INFO:teuthology.orchestra.run.smithi134.stderr:No --period given, using current period=01050e30-179b-43be-bdbb-3bb23cf37ae8 2024-08-18T17:12:23.596 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T17:12:23.596 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:12:23.596 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:12:23.596 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T17:12:23.596 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:12:23.596 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:12:23.596 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:12:23.596 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T17:12:23.596 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:12:23.597 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:12:23.597 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:12:23.597 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:42:47.672461Z" 2024-08-18T17:12:23.597 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:12:23.597 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:12:23.597 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1724001143.400665_1473.1", 2024-08-18T17:12:23.597 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T17:12:23.400665Z" 2024-08-18T17:12:23.597 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:12:23.597 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T17:12:23.598 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-08-18T17:12:23.598 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:12:23.598 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:12:23.599 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:12:23.646 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:12:23.646 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:12:23.676 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:12:23.676+0000 7fe564e0f880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:12:23.677 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:12:23.677+0000 7fe564e0f880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:12:23.691 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:12:23.691 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:12:23.691 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:12:23.692 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:12:23.692 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:12:23.692 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:12:23.692 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:12:23.692 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:12:23.692 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:12:23.692 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:12:23.692 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:12:23.692 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:12:23.692 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:12:23.692 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:12:23.692 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:12:23.693 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:12:23.693 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:12:23.693 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:12:23.693 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:12:23.693 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:12:23.693 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:12:23.693 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:12:23.693 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:12:23.693 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:12:23.693 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:12:23.693 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:12:23.693 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:12:23.693 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:12:23.694 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:12:23.694 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:12:23.694 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:12:23.694 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:12:23.694 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:12:23.694 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:12:23.694 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:12:23.694 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:12:23.694 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:12:23.694 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:12:23.694 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:12:23.694 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:12:23.694 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:12:23.695 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:12:23.695 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:12:23.695 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:12:23.695 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:12:23.695 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:12:23.695 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:12:23.695 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:12:23.695 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:12:23.695 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001130.000617_1445.1", 2024-08-18T17:12:23.695 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:12:23.695 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:12:23.695 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:12:23.695 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:12:10.000617Z", 2024-08-18T17:12:23.696 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:12:23.696 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:12:23.696 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:12:23.696 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:12:23.696 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:12:23.696 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:12:23.696 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:12:23.696 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:12:23.696 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:12:23.696 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:12:23Z" 2024-08-18T17:12:23.696 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:12:23.697 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001143.400665_1473.1 target=1_1724001130.000617_1445.1 2024-08-18T17:12:28.699 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:12:28.699 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:12:28.699 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:12:28.754 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:12:28.754 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:12:28.788 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:12:28.786+0000 7f3559898880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:12:28.788 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:12:28.787+0000 7f3559898880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:12:28.803 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:12:28.803 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:12:28.804 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:12:28.804 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:12:28.804 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:12:28.804 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:12:28.804 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:12:28.804 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:12:28.804 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:12:28.804 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:12:28.804 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:12:28.804 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:12:28.804 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:12:28.804 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:12:28.804 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:12:28.805 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:12:28.805 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:12:28.805 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:12:28.805 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:12:28.805 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:12:28.805 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:12:28.805 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:12:28.805 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:12:28.805 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:12:28.805 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:12:28.805 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:12:28.805 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:12:28.805 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:12:28.806 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:12:28.806 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:12:28.806 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:12:28.806 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:12:28.806 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:12:28.806 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:12:28.806 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:12:28.806 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:12:28.806 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:12:28.806 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:12:28.806 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:12:28.806 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:12:28.806 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:12:28.807 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:12:28.807 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:12:28.807 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:12:28.807 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:12:28.807 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:12:28.807 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:12:28.807 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:12:28.807 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:12:28.807 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001130.000617_1445.1", 2024-08-18T17:12:28.807 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:12:28.807 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:12:28.807 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:12:28.807 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:12:10.000617Z", 2024-08-18T17:12:28.808 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:12:28.808 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:12:28.808 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:12:28.808 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:12:28.808 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:12:28.808 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:12:28.808 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:12:28.808 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:12:28.808 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:12:28.808 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:12:28Z" 2024-08-18T17:12:28.808 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:12:28.809 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001143.400665_1473.1 target=1_1724001130.000617_1445.1 2024-08-18T17:12:33.810 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:12:33.810 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:12:33.811 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:12:33.867 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:12:33.867 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:12:33.902 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:12:33.900+0000 7fc734942880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:12:33.903 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:12:33.901+0000 7fc734942880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:12:33.917 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:12:33.918 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:12:33.918 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:12:33.918 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:12:33.918 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:12:33.918 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:12:33.918 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:12:33.918 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:12:33.918 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:12:33.918 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:12:33.918 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:12:33.918 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:12:33.919 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:12:33.919 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:12:33.919 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:12:33.919 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:12:33.919 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:12:33.919 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:12:33.919 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:12:33.919 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:12:33.919 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:12:33.919 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:12:33.919 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:12:33.919 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:12:33.919 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:12:33.919 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:12:33.920 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:12:33.920 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:12:33.920 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:12:33.920 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:12:33.920 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:12:33.920 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:12:33.920 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:12:33.920 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:12:33.920 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:12:33.920 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:12:33.920 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:12:33.920 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:12:33.920 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:12:33.921 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:12:33.921 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:12:33.921 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:12:33.921 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:12:33.921 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:12:33.921 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:12:33.921 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:12:33.921 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:12:33.921 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:12:33.921 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:12:33.921 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001130.000617_1445.1", 2024-08-18T17:12:33.921 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:12:33.921 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:12:33.922 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:12:33.922 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:12:10.000617Z", 2024-08-18T17:12:33.922 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:12:33.922 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:12:33.922 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:12:33.922 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:12:33.922 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:12:33.922 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:12:33.922 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:12:33.922 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:12:33.923 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:12:33.923 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:12:33Z" 2024-08-18T17:12:33.923 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:12:33.924 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001143.400665_1473.1 target=1_1724001130.000617_1445.1 2024-08-18T17:12:38.925 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:12:38.925 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:12:38.926 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:12:38.982 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:12:38.982 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:12:39.015 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:12:39.015+0000 7f7f1bc3f880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:12:39.016 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:12:39.016+0000 7f7f1bc3f880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:12:39.031 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:12:39.031 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:12:39.031 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:12:39.031 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:12:39.031 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:12:39.031 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:12:39.031 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:12:39.032 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:12:39.032 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:12:39.032 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:12:39.032 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:12:39.032 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:12:39.032 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:12:39.032 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:12:39.032 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:12:39.032 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:12:39.032 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:12:39.032 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:12:39.032 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:12:39.032 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:12:39.033 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:12:39.033 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:12:39.033 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:12:39.033 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:12:39.033 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:12:39.033 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:12:39.033 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:12:39.033 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:12:39.033 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:12:39.033 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:12:39.033 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:12:39.033 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:12:39.033 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:12:39.034 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:12:39.034 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:12:39.034 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:12:39.034 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:12:39.034 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:12:39.034 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:12:39.034 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:12:39.034 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:12:39.034 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:12:39.034 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:12:39.034 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:12:39.034 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:12:39.034 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:12:39.034 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:12:39.035 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:12:39.035 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:12:39.035 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001130.000617_1445.1", 2024-08-18T17:12:39.035 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:12:39.035 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:12:39.035 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:12:39.035 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:12:10.000617Z", 2024-08-18T17:12:39.035 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:12:39.035 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:12:39.035 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:12:39.035 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:12:39.035 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:12:39.035 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:12:39.036 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:12:39.036 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:12:39.036 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:12:39.036 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:12:39Z" 2024-08-18T17:12:39.036 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:12:39.037 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001143.400665_1473.1 target=1_1724001130.000617_1445.1 2024-08-18T17:12:44.038 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:12:44.038 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:12:44.038 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:12:44.094 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:12:44.094 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:12:44.129 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:12:44.129+0000 7f0feac10880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:12:44.130 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:12:44.129+0000 7f0feac10880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:12:44.145 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:12:44.145 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:12:44.145 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:12:44.145 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:12:44.145 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:12:44.146 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:12:44.146 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:12:44.146 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:12:44.146 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:12:44.146 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:12:44.146 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:12:44.146 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:12:44.146 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:12:44.146 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:12:44.146 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:12:44.146 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:12:44.147 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:12:44.147 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:12:44.147 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:12:44.147 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:12:44.147 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:12:44.147 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:12:44.147 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:12:44.147 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:12:44.147 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:12:44.147 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:12:44.147 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:12:44.148 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:12:44.148 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:12:44.148 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:12:44.148 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:12:44.148 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:12:44.148 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:12:44.148 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:12:44.148 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:12:44.148 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:12:44.148 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:12:44.148 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:12:44.148 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:12:44.148 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:12:44.149 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:12:44.149 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:12:44.149 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:12:44.149 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:12:44.149 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:12:44.149 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:12:44.149 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:12:44.149 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:12:44.149 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:12:44.149 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001130.000617_1445.1", 2024-08-18T17:12:44.149 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:12:44.149 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:12:44.149 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:12:44.150 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:12:10.000617Z", 2024-08-18T17:12:44.150 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:12:44.150 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:12:44.150 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:12:44.150 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:12:44.150 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:12:44.150 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:12:44.150 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:12:44.150 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:12:44.150 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:12:44.150 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:12:44Z" 2024-08-18T17:12:44.150 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:12:44.151 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001143.400665_1473.1 target=1_1724001130.000617_1445.1 2024-08-18T17:12:49.152 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:12:49.153 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:12:49.153 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:12:49.209 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:12:49.209 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:12:49.242 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:12:49.241+0000 7f3145d42880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:12:49.243 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:12:49.242+0000 7f3145d42880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:12:49.258 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:12:49.258 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:12:49.258 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:12:49.258 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:12:49.258 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:12:49.258 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:12:49.259 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:12:49.259 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:12:49.259 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:12:49.259 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:12:49.259 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:12:49.259 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:12:49.259 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:12:49.259 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:12:49.259 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:12:49.259 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:12:49.259 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:12:49.259 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:12:49.260 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:12:49.260 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:12:49.260 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:12:49.260 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:12:49.260 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:12:49.260 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:12:49.260 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:12:49.260 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:12:49.260 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:12:49.260 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:12:49.261 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:12:49.261 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:12:49.261 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:12:49.261 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:12:49.261 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:12:49.261 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:12:49.261 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:12:49.261 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:12:49.261 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:12:49.261 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:12:49.261 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:12:49.261 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:12:49.261 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:12:49.262 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:12:49.262 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:12:49.262 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:12:49.262 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:12:49.262 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:12:49.262 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:12:49.262 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:12:49.262 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:12:49.262 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001130.000617_1445.1", 2024-08-18T17:12:49.262 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:12:49.262 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:12:49.263 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:12:49.263 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:12:10.000617Z", 2024-08-18T17:12:49.263 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:12:49.263 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:12:49.263 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:12:49.263 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:12:49.263 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:12:49.263 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:12:49.263 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:12:49.263 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:12:49.263 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:12:49.263 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:12:49Z" 2024-08-18T17:12:49.263 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:12:49.264 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001143.400665_1473.1 target=1_1724001130.000617_1445.1 2024-08-18T17:12:54.266 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:12:54.266 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:12:54.266 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:12:54.323 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:12:54.323 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:12:54.356 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:12:54.356+0000 7f3396522880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:12:54.357 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:12:54.357+0000 7f3396522880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:12:54.373 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:12:54.373 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:12:54.373 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:12:54.373 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:12:54.373 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:12:54.374 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:12:54.374 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:12:54.374 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:12:54.374 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:12:54.374 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:12:54.374 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:12:54.374 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:12:54.374 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:12:54.374 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:12:54.374 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:12:54.375 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:12:54.375 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:12:54.375 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:12:54.375 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:12:54.375 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:12:54.375 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:12:54.375 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:12:54.375 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:12:54.375 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:12:54.375 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:12:54.375 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:12:54.375 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:12:54.375 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:12:54.376 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:12:54.376 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:12:54.376 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:12:54.376 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:12:54.376 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:12:54.376 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:12:54.376 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:12:54.376 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:12:54.376 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:12:54.376 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:12:54.376 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:12:54.376 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:12:54.376 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:12:54.377 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:12:54.377 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:12:54.377 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:12:54.377 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:12:54.377 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:12:54.377 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:12:54.377 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:12:54.377 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:12:54.377 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001130.000617_1445.1", 2024-08-18T17:12:54.377 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:12:54.377 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:12:54.377 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:12:54.377 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:12:10.000617Z", 2024-08-18T17:12:54.377 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:12:54.378 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:12:54.378 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:12:54.378 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:12:54.378 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:12:54.378 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:12:54.378 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:12:54.378 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:12:54.378 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:12:54.378 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:12:54Z" 2024-08-18T17:12:54.378 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:12:54.379 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001143.400665_1473.1 target=1_1724001130.000617_1445.1 2024-08-18T17:12:59.380 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:12:59.380 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:12:59.381 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:12:59.436 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:12:59.436 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:12:59.469 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:12:59.468+0000 7ffb43011880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:12:59.470 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:12:59.469+0000 7ffb43011880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:12:59.484 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:12:59.484 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:12:59.484 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:12:59.484 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:12:59.485 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:12:59.485 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:12:59.485 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:12:59.485 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:12:59.485 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:12:59.485 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:12:59.485 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:12:59.485 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:12:59.485 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:12:59.485 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:12:59.485 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:12:59.485 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:12:59.485 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:12:59.486 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:12:59.486 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:12:59.486 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:12:59.486 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:12:59.486 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:12:59.486 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:12:59.486 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:12:59.486 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:12:59.486 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:12:59.486 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:12:59.486 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:12:59.486 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:12:59.486 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:12:59.487 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:12:59.487 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:12:59.487 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:12:59.487 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:12:59.487 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:12:59.487 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:12:59.487 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:12:59.487 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:12:59.487 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:12:59.487 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:12:59.487 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:12:59.487 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:12:59.487 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:12:59.488 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:12:59.488 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:12:59.488 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:12:59.488 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:12:59.488 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:12:59.488 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:12:59.488 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001130.000617_1445.1", 2024-08-18T17:12:59.488 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:12:59.488 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:12:59.488 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:12:59.488 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:12:10.000617Z", 2024-08-18T17:12:59.488 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:12:59.488 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:12:59.488 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:12:59.489 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:12:59.489 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:12:59.489 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:12:59.489 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:12:59.489 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:12:59.489 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:12:59.489 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:12:59Z" 2024-08-18T17:12:59.489 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:12:59.490 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001143.400665_1473.1 target=1_1724001130.000617_1445.1 2024-08-18T17:13:04.491 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:13:04.491 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:13:04.492 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:13:04.549 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:13:04.549 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:13:04.583 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:13:04.582+0000 7f8475f42880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:13:04.583 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:13:04.583+0000 7f8475f42880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:13:04.598 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:13:04.599 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:13:04.599 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:13:04.599 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:13:04.599 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:13:04.599 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:13:04.599 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:13:04.599 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:13:04.599 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:13:04.599 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:13:04.599 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:13:04.599 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:13:04.599 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:13:04.600 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:13:04.600 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:13:04.600 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:13:04.600 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:13:04.600 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:13:04.600 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:13:04.600 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:13:04.600 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:13:04.600 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:13:04.600 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:13:04.600 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:13:04.600 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:13:04.600 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:13:04.600 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:13:04.601 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:13:04.601 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:13:04.601 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:13:04.601 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:13:04.601 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:13:04.601 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:13:04.601 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:13:04.601 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:13:04.601 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:13:04.601 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:13:04.601 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:13:04.601 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:13:04.601 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:13:04.602 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:13:04.602 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:13:04.602 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:13:04.602 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:13:04.602 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:13:04.602 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:13:04.602 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:13:04.602 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:13:04.602 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:13:04.602 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001130.000617_1445.1", 2024-08-18T17:13:04.602 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:13:04.602 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:13:04.602 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:13:04.603 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:12:10.000617Z", 2024-08-18T17:13:04.603 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:13:04.603 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:13:04.603 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:13:04.603 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:13:04.603 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:13:04.603 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:13:04.603 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:13:04.603 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:13:04.603 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:13:04.603 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:13:04Z" 2024-08-18T17:13:04.603 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:13:04.604 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001143.400665_1473.1 target=1_1724001130.000617_1445.1 2024-08-18T17:13:09.605 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:13:09.606 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:13:09.606 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:13:09.661 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:13:09.661 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:13:09.695 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:13:09.695+0000 7fcad7c11880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:13:09.696 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:13:09.696+0000 7fcad7c11880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:13:09.712 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:13:09.712 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:13:09.712 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:13:09.712 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:13:09.712 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:13:09.712 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:13:09.712 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:13:09.712 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:13:09.712 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:13:09.712 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:13:09.712 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:13:09.713 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:13:09.713 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:13:09.713 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:13:09.713 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:13:09.713 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:13:09.713 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:13:09.713 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:13:09.713 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:13:09.713 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:13:09.713 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:13:09.713 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:13:09.713 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:13:09.713 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:13:09.714 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:13:09.714 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:13:09.714 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:13:09.714 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:13:09.714 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:13:09.714 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:13:09.714 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:13:09.714 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:13:09.714 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:13:09.714 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:13:09.714 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:13:09.714 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:13:09.714 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:13:09.714 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:13:09.715 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:13:09.715 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:13:09.715 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:13:09.715 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:13:09.715 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:13:09.715 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:13:09.715 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:13:09.715 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:13:09.715 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:13:09.715 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:13:09.715 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:13:09.715 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001130.000617_1445.1", 2024-08-18T17:13:09.715 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:13:09.715 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:13:09.715 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:13:09.716 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:12:10.000617Z", 2024-08-18T17:13:09.716 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:13:09.716 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:13:09.716 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:13:09.716 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:13:09.716 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:13:09.716 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:13:09.716 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:13:09.716 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:13:09.716 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:13:09.716 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:13:09Z" 2024-08-18T17:13:09.716 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:13:09.717 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001143.400665_1473.1 target=1_1724001130.000617_1445.1 2024-08-18T17:13:14.718 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:13:14.719 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:13:14.719 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:13:14.774 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:13:14.774 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:13:14.808 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:13:14.808+0000 7f7401a13880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:13:14.809 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:13:14.809+0000 7f7401a13880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:13:14.824 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:13:14.824 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:13:14.824 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:13:14.824 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:13:14.824 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:13:14.824 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:13:14.824 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:13:14.825 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:13:14.825 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:13:14.825 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:13:14.825 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:13:14.825 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:13:14.825 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:13:14.825 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:13:14.825 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:13:14.825 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:13:14.825 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:13:14.825 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:13:14.825 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:13:14.825 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:13:14.825 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:13:14.826 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:13:14.826 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:13:14.826 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:13:14.826 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:13:14.826 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:13:14.826 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:13:14.826 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:13:14.826 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:13:14.826 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:13:14.826 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:13:14.826 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:13:14.826 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:13:14.826 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:13:14.827 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:13:14.827 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:13:14.827 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:13:14.827 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:13:14.827 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:13:14.827 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:13:14.827 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:13:14.827 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:13:14.827 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:13:14.827 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:13:14.827 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:13:14.827 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:13:14.827 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:13:14.827 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:13:14.828 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:13:14.828 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001130.000617_1445.1", 2024-08-18T17:13:14.828 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:13:14.828 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:13:14.828 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:13:14.828 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:12:10.000617Z", 2024-08-18T17:13:14.828 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:13:14.828 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:13:14.828 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:13:14.828 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:13:14.828 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:13:14.828 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:13:14.828 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:13:14.828 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:13:14.829 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:13:14.829 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:13:14Z" 2024-08-18T17:13:14.829 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:13:14.830 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001143.400665_1473.1 target=1_1724001130.000617_1445.1 2024-08-18T17:13:19.831 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:13:19.831 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:13:19.831 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:13:19.887 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:13:19.887 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:13:19.922 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:13:19.921+0000 7ff949696880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:13:19.923 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:13:19.922+0000 7ff949696880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:13:19.937 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:13:19.937 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:13:19.937 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:13:19.938 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:13:19.938 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:13:19.938 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:13:19.938 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:13:19.938 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:13:19.938 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:13:19.938 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:13:19.938 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:13:19.938 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:13:19.938 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:13:19.938 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:13:19.938 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:13:19.938 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:13:19.939 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:13:19.939 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:13:19.939 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:13:19.939 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:13:19.939 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:13:19.939 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:13:19.939 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:13:19.939 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:13:19.939 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:13:19.939 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:13:19.939 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:13:19.939 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:13:19.939 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:13:19.939 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:13:19.940 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:13:19.940 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:13:19.940 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:13:19.940 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:13:19.940 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:13:19.940 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:13:19.940 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:13:19.940 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:13:19.940 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:13:19.940 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:13:19.940 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:13:19.940 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:13:19.940 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:13:19.941 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:13:19.941 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:13:19.941 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:13:19.941 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:13:19.941 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:13:19.941 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:13:19.941 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001130.000617_1445.1", 2024-08-18T17:13:19.941 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:13:19.941 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:13:19.941 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:13:19.941 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:12:10.000617Z", 2024-08-18T17:13:19.941 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:13:19.941 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:13:19.942 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:13:19.942 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:13:19.942 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:13:19.942 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:13:19.942 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:13:19.942 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:13:19.942 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:13:19.942 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:13:19Z" 2024-08-18T17:13:19.942 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:13:19.943 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001143.400665_1473.1 target=1_1724001130.000617_1445.1 2024-08-18T17:13:24.944 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:13:24.944 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:13:24.945 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:13:25.001 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:13:25.001 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:13:25.035 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:13:25.034+0000 7f6502d42880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:13:25.036 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:13:25.035+0000 7f6502d42880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:13:25.051 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:13:25.051 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:13:25.051 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:13:25.051 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:13:25.051 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:13:25.051 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:13:25.051 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:13:25.052 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:13:25.052 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:13:25.052 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:13:25.052 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:13:25.052 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:13:25.052 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:13:25.052 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:13:25.052 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:13:25.052 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:13:25.052 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:13:25.052 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:13:25.052 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:13:25.053 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:13:25.053 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:13:25.053 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:13:25.053 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:13:25.053 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:13:25.053 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:13:25.053 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:13:25.053 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:13:25.053 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:13:25.053 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:13:25.053 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:13:25.054 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:13:25.054 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:13:25.054 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:13:25.054 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:13:25.054 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:13:25.054 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:13:25.054 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:13:25.054 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:13:25.054 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:13:25.054 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:13:25.054 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:13:25.055 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:13:25.055 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:13:25.055 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:13:25.055 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:13:25.055 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:13:25.055 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:13:25.055 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:13:25.055 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:13:25.055 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001130.000617_1445.1", 2024-08-18T17:13:25.055 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:13:25.055 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:13:25.056 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:13:25.056 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:12:10.000617Z", 2024-08-18T17:13:25.056 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:13:25.056 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:13:25.056 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:13:25.056 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:13:25.056 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:13:25.056 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:13:25.056 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:13:25.056 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:13:25.056 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:13:25.057 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:13:25Z" 2024-08-18T17:13:25.057 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:13:25.058 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001143.400665_1473.1 target=1_1724001130.000617_1445.1 2024-08-18T17:13:30.058 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:13:30.059 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:13:30.059 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:13:30.116 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:13:30.116 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:13:30.150 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:13:30.149+0000 7faad9010880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:13:30.150 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:13:30.150+0000 7faad9010880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:13:30.165 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:13:30.165 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:13:30.165 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:13:30.165 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:13:30.165 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:13:30.165 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:13:30.165 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:13:30.165 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:13:30.165 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:13:30.166 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:13:30.166 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:13:30.166 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:13:30.166 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:13:30.166 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:13:30.166 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:13:30.166 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:13:30.166 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:13:30.166 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:13:30.166 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:13:30.166 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:13:30.166 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:13:30.166 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:13:30.167 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:13:30.167 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:13:30.167 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:13:30.167 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:13:30.167 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:13:30.167 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:13:30.167 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:13:30.167 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:13:30.167 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:13:30.167 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:13:30.167 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:13:30.167 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:13:30.168 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:13:30.168 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:13:30.168 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:13:30.168 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:13:30.168 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:13:30.168 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:13:30.168 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:13:30.168 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:13:30.168 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:13:30.168 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:13:30.168 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:13:30.168 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:13:30.168 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:13:30.169 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:13:30.169 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:13:30.169 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001130.000617_1445.1", 2024-08-18T17:13:30.169 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:13:30.169 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:13:30.169 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:13:30.169 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:12:10.000617Z", 2024-08-18T17:13:30.169 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:13:30.169 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:13:30.169 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:13:30.169 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:13:30.169 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:13:30.169 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:13:30.169 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:13:30.170 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:13:30.170 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:13:30.170 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:13:30Z" 2024-08-18T17:13:30.170 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:13:30.171 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001143.400665_1473.1 target=1_1724001130.000617_1445.1 2024-08-18T17:13:35.172 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:13:35.172 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:13:35.173 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:13:35.227 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:13:35.227 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:13:35.261 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:13:35.259+0000 7feb1a924880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:13:35.261 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:13:35.260+0000 7feb1a924880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:13:35.276 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:13:35.276 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:13:35.276 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:13:35.277 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:13:35.277 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:13:35.277 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:13:35.277 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:13:35.277 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:13:35.277 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:13:35.277 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:13:35.277 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:13:35.277 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:13:35.277 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:13:35.277 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:13:35.277 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:13:35.277 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:13:35.278 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:13:35.278 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:13:35.278 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:13:35.278 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:13:35.278 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:13:35.278 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:13:35.278 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:13:35.278 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:13:35.278 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:13:35.278 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:13:35.278 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:13:35.278 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:13:35.279 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:13:35.279 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:13:35.279 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:13:35.279 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:13:35.279 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:13:35.279 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:13:35.279 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:13:35.279 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:13:35.279 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:13:35.279 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:13:35.279 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:13:35.279 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:13:35.279 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:13:35.280 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:13:35.280 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:13:35.280 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:13:35.280 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:13:35.280 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:13:35.280 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:13:35.280 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:13:35.280 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:13:35.280 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001130.000617_1445.1", 2024-08-18T17:13:35.280 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:13:35.280 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:13:35.280 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:13:35.280 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:12:10.000617Z", 2024-08-18T17:13:35.281 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:13:35.281 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:13:35.281 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:13:35.281 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:13:35.281 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:13:35.281 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:13:35.281 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:13:35.281 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:13:35.281 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:13:35.281 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:13:35Z" 2024-08-18T17:13:35.281 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:13:35.282 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001143.400665_1473.1 target=1_1724001130.000617_1445.1 2024-08-18T17:13:40.283 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:13:40.283 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:13:40.284 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:13:40.339 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:13:40.339 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:13:40.373 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:13:40.372+0000 7ff4d2a15880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:13:40.374 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:13:40.373+0000 7ff4d2a15880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:13:40.389 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:13:40.389 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:13:40.389 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:13:40.389 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:13:40.389 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:13:40.389 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:13:40.389 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:13:40.389 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:13:40.389 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:13:40.389 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:13:40.389 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:13:40.390 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:13:40.390 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:13:40.390 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:13:40.390 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:13:40.390 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:13:40.390 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:13:40.390 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:13:40.390 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:13:40.390 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:13:40.390 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:13:40.390 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:13:40.390 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:13:40.391 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:13:40.391 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:13:40.391 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:13:40.391 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:13:40.391 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:13:40.391 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:13:40.391 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:13:40.391 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:13:40.391 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:13:40.391 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:13:40.391 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:13:40.391 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:13:40.391 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:13:40.392 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:13:40.392 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:13:40.392 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:13:40.392 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:13:40.392 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:13:40.392 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:13:40.392 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:13:40.392 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:13:40.392 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:13:40.392 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:13:40.392 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:13:40.392 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:13:40.393 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:13:40.393 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001130.000617_1445.1", 2024-08-18T17:13:40.393 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:13:40.393 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:13:40.393 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:13:40.393 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:12:10.000617Z", 2024-08-18T17:13:40.393 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:13:40.393 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:13:40.393 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:13:40.393 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:13:40.393 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:13:40.393 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:13:40.393 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:13:40.393 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:13:40.394 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:13:40.394 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:13:40Z" 2024-08-18T17:13:40.394 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:13:40.395 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001143.400665_1473.1 target=1_1724001130.000617_1445.1 2024-08-18T17:13:45.395 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:13:45.396 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:13:45.396 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:13:45.451 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:13:45.451 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:13:45.485 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:13:45.484+0000 7fd51cc0f880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:13:45.485 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:13:45.485+0000 7fd51cc0f880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:13:45.500 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:13:45.500 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:13:45.500 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:13:45.500 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:13:45.500 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:13:45.500 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:13:45.500 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:13:45.501 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:13:45.501 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:13:45.501 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:13:45.501 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:13:45.501 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:13:45.501 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:13:45.501 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:13:45.501 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:13:45.501 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:13:45.501 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:13:45.501 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:13:45.501 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:13:45.501 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:13:45.501 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:13:45.502 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:13:45.502 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:13:45.502 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:13:45.502 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:13:45.502 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:13:45.502 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:13:45.502 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:13:45.502 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:13:45.502 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:13:45.502 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:13:45.502 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:13:45.502 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:13:45.503 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:13:45.503 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:13:45.503 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:13:45.503 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:13:45.503 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:13:45.503 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:13:45.503 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:13:45.503 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:13:45.503 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:13:45.503 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:13:45.503 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:13:45.503 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:13:45.503 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:13:45.503 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:13:45.504 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:13:45.504 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:13:45.504 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001130.000617_1445.1", 2024-08-18T17:13:45.504 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:13:45.504 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:13:45.504 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:13:45.504 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:12:10.000617Z", 2024-08-18T17:13:45.504 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:13:45.504 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:13:45.504 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:13:45.504 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:13:45.504 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:13:45.504 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:13:45.504 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:13:45.505 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:13:45.505 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:13:45.505 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:13:45Z" 2024-08-18T17:13:45.505 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:13:45.506 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001143.400665_1473.1 target=1_1724001130.000617_1445.1 2024-08-18T17:13:50.507 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:13:50.507 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:13:50.508 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:13:50.563 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:13:50.563 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:13:50.597 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:13:50.597+0000 7fad9d938880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:13:50.598 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:13:50.597+0000 7fad9d938880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:13:50.613 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:13:50.613 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:13:50.613 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:13:50.613 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:13:50.613 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:13:50.613 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:13:50.613 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:13:50.613 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:13:50.613 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:13:50.613 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:13:50.613 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:13:50.613 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:13:50.614 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:13:50.614 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:13:50.614 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:13:50.614 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:13:50.614 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:13:50.614 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:13:50.614 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:13:50.614 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:13:50.614 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:13:50.614 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:13:50.614 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:13:50.614 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:13:50.615 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:13:50.615 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:13:50.615 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:13:50.615 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:13:50.615 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:13:50.615 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:13:50.615 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:13:50.615 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:13:50.615 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:13:50.615 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:13:50.615 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:13:50.615 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:13:50.615 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:13:50.616 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:13:50.616 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:13:50.616 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:13:50.616 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:13:50.616 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:13:50.616 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:13:50.616 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:13:50.616 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:13:50.616 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:13:50.616 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:13:50.616 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:13:50.616 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:13:50.616 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001130.000617_1445.1", 2024-08-18T17:13:50.616 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:13:50.617 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:13:50.617 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:13:50.617 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:12:10.000617Z", 2024-08-18T17:13:50.617 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:13:50.617 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:13:50.617 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:13:50.617 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:13:50.617 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:13:50.617 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:13:50.617 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:13:50.617 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:13:50.617 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:13:50.617 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:13:50Z" 2024-08-18T17:13:50.617 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:13:50.618 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001143.400665_1473.1 target=1_1724001130.000617_1445.1 2024-08-18T17:13:55.619 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:13:55.620 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:13:55.620 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:13:55.675 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:13:55.675 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:13:55.709 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:13:55.708+0000 7f813c40e880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:13:55.710 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:13:55.709+0000 7f813c40e880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:13:55.724 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:13:55.725 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:13:55.725 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:13:55.725 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:13:55.725 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:13:55.725 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:13:55.725 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:13:55.725 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:13:55.725 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:13:55.725 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:13:55.725 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:13:55.725 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:13:55.726 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:13:55.726 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:13:55.726 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:13:55.726 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:13:55.726 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:13:55.726 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:13:55.726 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:13:55.726 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:13:55.726 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:13:55.726 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:13:55.726 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:13:55.726 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:13:55.726 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:13:55.727 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:13:55.727 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:13:55.727 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:13:55.727 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:13:55.727 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:13:55.727 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:13:55.727 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:13:55.727 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:13:55.727 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:13:55.727 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:13:55.727 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:13:55.727 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:13:55.727 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:13:55.728 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:13:55.728 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:13:55.728 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:13:55.728 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:13:55.728 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:13:55.728 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:13:55.728 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:13:55.728 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:13:55.728 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:13:55.729 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:13:55.729 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:13:55.729 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001130.000617_1445.1", 2024-08-18T17:13:55.729 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:13:55.729 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:13:55.729 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:13:55.729 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:12:10.000617Z", 2024-08-18T17:13:55.729 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:13:55.729 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:13:55.729 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:13:55.729 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:13:55.729 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:13:55.730 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:13:55.730 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:13:55.730 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:13:55.730 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:13:55.730 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:13:55Z" 2024-08-18T17:13:55.730 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:13:55.732 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001143.400665_1473.1 target=1_1724001130.000617_1445.1 2024-08-18T17:14:00.732 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:14:00.733 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:14:00.733 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:14:00.789 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:14:00.790 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:14:00.823 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:14:00.822+0000 7fda16322880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:14:00.824 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:14:00.823+0000 7fda16322880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:14:00.838 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:14:00.838 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:14:00.838 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:14:00.839 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:14:00.839 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:14:00.839 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:14:00.839 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:14:00.839 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:00.839 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:14:00.839 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:14:00.839 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:14:00.839 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:14:00.839 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:14:00.839 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:14:00.839 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:14:00.839 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:14:00.840 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:14:00.840 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:14:00.840 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:14:00.840 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:14:00.840 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:00.840 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:14:00.840 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:14:00.840 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:14:00.840 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:14:00.840 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:14:00.840 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:14:00.840 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:14:00.841 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:14:00.841 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:14:00.841 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:14:00.841 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:14:00.841 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:00.841 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:14:00.841 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:14:00.841 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:14:00.841 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:14:00.841 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:14:00.841 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:14:00.841 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:14:00.841 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:14:00.841 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:14:00.842 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:14:00.842 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:14:00.842 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:00.842 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:14:00.842 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:14:00.842 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:14:00.842 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:14:00.842 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001130.000617_1445.1", 2024-08-18T17:14:00.842 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:14:00.842 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:14:00.842 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:14:00.842 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:12:10.000617Z", 2024-08-18T17:14:00.842 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:14:00.843 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:14:00.843 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:14:00.843 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:14:00.843 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:00.843 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:14:00.843 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:14:00.843 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:14:00.843 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:00.843 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:14:00Z" 2024-08-18T17:14:00.844 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:14:00.845 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001143.400665_1473.1 target=1_1724001130.000617_1445.1 2024-08-18T17:14:05.845 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:14:05.846 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:14:05.846 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:14:05.901 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:14:05.901 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:14:05.935 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:14:05.934+0000 7fcda7612880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:14:05.935 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:14:05.935+0000 7fcda7612880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:14:05.950 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:14:05.950 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:14:05.950 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:14:05.950 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:14:05.951 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:14:05.951 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:14:05.951 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:14:05.951 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:05.951 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:14:05.951 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:14:05.951 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:14:05.951 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:14:05.951 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:14:05.951 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:14:05.951 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:14:05.952 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:14:05.952 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:14:05.952 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:14:05.952 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:14:05.952 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:14:05.952 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:05.952 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:14:05.952 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:14:05.952 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:14:05.952 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:14:05.952 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:14:05.952 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:14:05.952 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:14:05.953 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:14:05.953 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:14:05.953 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:14:05.953 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:14:05.953 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:05.953 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:14:05.953 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:14:05.953 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:14:05.953 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:14:05.953 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:14:05.953 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:14:05.953 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:14:05.953 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:14:05.954 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:14:05.954 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:14:05.954 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:14:05.954 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:05.954 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:14:05.954 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:14:05.954 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:14:05.954 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:14:05.954 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001130.000617_1445.1", 2024-08-18T17:14:05.954 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:14:05.954 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:14:05.955 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:14:05.955 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:12:10.000617Z", 2024-08-18T17:14:05.955 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:14:05.955 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:14:05.955 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:14:05.955 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:14:05.955 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:05.955 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:14:05.956 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:14:05.956 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:14:05.956 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:05.956 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:14:05Z" 2024-08-18T17:14:05.956 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:14:05.957 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001143.400665_1473.1 target=1_1724001130.000617_1445.1 2024-08-18T17:14:10.959 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:14:10.959 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:14:10.959 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:14:11.018 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:14:11.018 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:14:11.048 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:14:11.047+0000 7fd6b6626880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:14:11.049 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:14:11.048+0000 7fd6b6626880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:14:11.064 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:14:11.064 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:14:11.064 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:14:11.064 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:14:11.064 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:14:11.064 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:14:11.064 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:14:11.064 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:11.064 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:14:11.064 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:14:11.065 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:14:11.065 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:14:11.065 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:14:11.065 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:14:11.065 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:14:11.065 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:14:11.065 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:14:11.065 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:14:11.065 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:14:11.065 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:14:11.065 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:11.065 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:14:11.065 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:14:11.065 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:14:11.066 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:14:11.066 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:14:11.066 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:14:11.066 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:14:11.066 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:14:11.066 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:14:11.066 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:14:11.066 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:14:11.066 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:11.066 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:14:11.066 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:14:11.066 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:14:11.066 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:14:11.067 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:14:11.067 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:14:11.067 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:14:11.067 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:14:11.067 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:14:11.067 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:14:11.067 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:14:11.067 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:11.067 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:14:11.067 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:14:11.068 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:14:11.068 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:14:11.068 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001130.000617_1445.1", 2024-08-18T17:14:11.068 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:14:11.068 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:14:11.068 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:14:11.068 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:12:10.000617Z", 2024-08-18T17:14:11.068 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:14:11.068 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:14:11.068 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:14:11.068 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:14:11.068 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:11.068 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:14:11.068 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:14:11.069 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:14:11.069 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:11.069 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:14:11Z" 2024-08-18T17:14:11.069 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:14:11.070 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001143.400665_1473.1 target=1_1724001130.000617_1445.1 2024-08-18T17:14:16.071 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:14:16.071 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:14:16.071 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:14:16.127 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:14:16.128 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:14:16.161 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:14:16.160+0000 7f48d283f880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:14:16.162 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:14:16.161+0000 7f48d283f880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:14:16.178 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:14:16.178 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:14:16.178 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:14:16.178 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:14:16.178 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:14:16.178 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:14:16.178 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:14:16.178 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:16.178 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:14:16.178 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:14:16.178 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:14:16.179 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:14:16.179 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:14:16.179 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:14:16.179 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:14:16.179 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:14:16.179 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:14:16.179 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:14:16.179 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:14:16.179 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:14:16.179 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:16.179 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:14:16.179 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:14:16.179 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:14:16.179 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:14:16.180 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:14:16.180 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:14:16.180 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:14:16.180 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:14:16.180 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:14:16.180 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:14:16.180 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:14:16.180 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:16.180 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:14:16.180 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:14:16.180 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:14:16.180 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:14:16.180 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:14:16.181 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:14:16.181 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:14:16.181 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:14:16.181 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:14:16.181 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:14:16.181 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:14:16.181 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:16.181 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:14:16.181 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:14:16.181 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:14:16.181 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:14:16.181 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001143.400665_1473.1", 2024-08-18T17:14:16.181 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:14:16.182 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:14:16.182 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:14:16.182 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:12:23.400665Z", 2024-08-18T17:14:16.182 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:14:16.182 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:14:16.182 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:14:16.182 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:14:16.182 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:16.182 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:14:16.182 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:14:16.182 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:14:16.182 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:16.182 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:14:16Z" 2024-08-18T17:14:16.183 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:14:16.184 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-08-18T17:14:16.184 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-08-18T17:14:16.184 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:14:16.184 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:14:16.184 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:14:16.239 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:14:16.239 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:14:16.285 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:14:16.284+0000 7f16e5212880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:14:16.286 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:14:16.285+0000 7f16e5212880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:14:16.301 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T17:14:16.301 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T17:14:16.301 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T17:14:16.301 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T17:14:16.301 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T17:14:16.301 INFO:teuthology.orchestra.run.smithi134.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:14:16.301 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:14:16.301 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:14:16.302 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T17:14:16.302 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:14:16.302 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T17:14:16.302 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:14:16.302 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:14:16.302 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:14:16.302 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:14:16.302 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 1, 2024-08-18T17:14:16.302 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:14:16.302 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:14:16.302 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T17:14:16.302 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:14:16.303 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:14:16.303 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:14:16.303 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T17:14:16.303 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:14:16.303 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:14:16.303 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:14:16.303 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:14:16.303 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:14:16.303 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:14:16.303 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T17:14:16.303 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T17:14:16.303 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:14:16.303 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:14:16.304 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:14:16.304 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T17:14:16.304 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:14:16.304 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:14:16.304 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:14:16.304 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:14:16.304 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:14:16.304 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:14:16.304 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:14:16.304 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:14:16.304 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:14:16.304 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:14:16.305 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:14:16.305 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T17:14:16.305 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:14:16.305 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:14:16.305 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1724001143.400665_1473.1", 2024-08-18T17:14:16.305 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:14:16.305 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:14:16.305 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:14:16.305 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T17:12:23.400665Z", 2024-08-18T17:14:16.305 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:14:16.305 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:14:16.305 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:14:16.305 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:14:16.306 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:14:16.306 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T17:14:16.306 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 1, 2024-08-18T17:14:16.306 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 1 2024-08-18T17:14:16.306 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:14:16.306 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T17:14:16Z" 2024-08-18T17:14:16.306 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T17:14:16.307 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-08-18T17:14:16.307 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:14:16.307 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:14:16.307 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:14:16.355 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:14:16.355 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:14:16.411 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T17:14:16.412 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:14:16.412 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000051033", 2024-08-18T17:14:16.412 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T17:05:27.155053Z" 2024-08-18T17:14:16.412 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:14:16.412 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:14:16.412 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000037274", 2024-08-18T17:14:16.412 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T17:05:35.911460Z" 2024-08-18T17:14:16.412 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:14:16.412 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:14:16.412 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000058157", 2024-08-18T17:14:16.412 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T17:12:23.352239Z" 2024-08-18T17:14:16.413 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:14:16.413 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:14:16.413 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000052068", 2024-08-18T17:14:16.413 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T17:05:27.099520Z" 2024-08-18T17:14:16.413 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:14:16.413 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T17:14:16.414 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-08-18T17:14:16.414 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:14:16.414 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:14:16.414 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:14:16.467 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:14:16.467 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:14:16.514 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:14:16.515 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:14:16.515 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:14:16.515 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:14:16.515 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:14:16.515 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:14:16.515 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:16.515 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:14:16.516 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:14:16.516 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:14:16.516 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:14:16.516 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:14:16.516 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000051033", 2024-08-18T17:14:16.516 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:14:16.516 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:14:16.516 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:14:16.516 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:05:27.155053Z" 2024-08-18T17:14:16.516 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:14:16.516 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:16.517 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:14:16.517 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:14:16.517 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:14:16.517 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:14:16.517 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037274", 2024-08-18T17:14:16.517 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:14:16.517 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:14:16.517 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:14:16.517 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:05:35.911460Z" 2024-08-18T17:14:16.517 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:14:16.517 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:16.517 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:14:16.517 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:14:16.518 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:14:16.518 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:14:16.518 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000058015", 2024-08-18T17:14:16.518 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:14:16.518 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:14:16.518 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:14:16.518 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:05:27.212163Z" 2024-08-18T17:14:16.518 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:14:16.518 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:16.518 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:14:16.518 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:14:16.518 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:14:16.518 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:14:16.519 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052068", 2024-08-18T17:14:16.519 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:14:16.519 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:14:16.519 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:14:16.519 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:05:27.099520Z" 2024-08-18T17:14:16.519 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:14:16.519 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:14:16.519 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:14:16.519 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:16.519 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:14:16.519 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:14:16.519 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:14:16.519 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:16.519 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:14:16Z" 2024-08-18T17:14:16.520 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:14:16.520 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=2 master=00000000000000000000:00000000000000058157 target=00000000000000000000:00000000000000058015 2024-08-18T17:14:21.521 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:14:21.522 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:14:21.522 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:14:21.579 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:14:21.579 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:14:21.630 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:14:21.630 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:14:21.630 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:14:21.630 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:14:21.630 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:14:21.631 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:14:21.631 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:21.631 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:14:21.631 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:14:21.631 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:14:21.631 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:14:21.631 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:14:21.631 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000051033", 2024-08-18T17:14:21.631 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:14:21.631 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:14:21.631 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:14:21.631 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:05:27.155053Z" 2024-08-18T17:14:21.631 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:14:21.632 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:21.632 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:14:21.632 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:14:21.632 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:14:21.632 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:14:21.632 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037274", 2024-08-18T17:14:21.632 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:14:21.633 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:14:21.633 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:14:21.633 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:05:35.911460Z" 2024-08-18T17:14:21.633 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:14:21.633 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:21.633 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:14:21.634 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:14:21.634 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:14:21.634 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:14:21.634 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000058015", 2024-08-18T17:14:21.634 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:14:21.634 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:14:21.634 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:14:21.635 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:05:27.212163Z" 2024-08-18T17:14:21.635 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:14:21.635 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:21.635 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:14:21.635 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:14:21.635 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:14:21.636 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:14:21.636 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052068", 2024-08-18T17:14:21.636 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:14:21.636 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:14:21.636 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:14:21.636 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:05:27.099520Z" 2024-08-18T17:14:21.636 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:14:21.636 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:14:21.637 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:14:21.637 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:21.637 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:14:21.637 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:14:21.637 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:14:21.637 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:21.638 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:14:21Z" 2024-08-18T17:14:21.638 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:14:21.639 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=2 master=00000000000000000000:00000000000000058157 target=00000000000000000000:00000000000000058015 2024-08-18T17:14:26.640 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:14:26.641 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:14:26.641 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:14:26.698 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:14:26.698 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:14:26.748 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:14:26.749 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:14:26.749 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:14:26.749 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:14:26.749 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:14:26.749 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:14:26.749 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:26.749 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:14:26.749 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:14:26.749 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:14:26.749 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:14:26.749 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:14:26.749 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000051033", 2024-08-18T17:14:26.749 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:14:26.750 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:14:26.750 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:14:26.750 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:05:27.155053Z" 2024-08-18T17:14:26.750 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:14:26.750 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:26.750 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:14:26.750 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:14:26.750 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:14:26.750 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:14:26.750 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037274", 2024-08-18T17:14:26.750 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:14:26.751 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:14:26.751 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:14:26.751 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:05:35.911460Z" 2024-08-18T17:14:26.751 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:14:26.751 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:26.751 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:14:26.751 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:14:26.751 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:14:26.751 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:14:26.751 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000058157", 2024-08-18T17:14:26.751 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:14:26.751 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:14:26.751 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:14:26.751 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:12:23.352239Z" 2024-08-18T17:14:26.752 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:14:26.752 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:26.752 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:14:26.752 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:14:26.752 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:14:26.752 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:14:26.752 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052068", 2024-08-18T17:14:26.752 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:14:26.752 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:14:26.752 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:14:26.752 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:05:27.099520Z" 2024-08-18T17:14:26.752 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:14:26.752 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:14:26.752 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:14:26.753 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:26.753 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:14:26.753 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:14:26.753 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:14:26.753 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:26.753 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:14:26Z" 2024-08-18T17:14:26.753 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:14:26.754 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-08-18T17:14:26.772 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:14:26.772 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:14:26.772 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:14:26.825 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:14:26.825 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:14:26.890 INFO:teuthology.orchestra.run.smithi134.stdout:{"sync_policy":{"groups":[]}} 2024-08-18T17:14:26.891 INFO:tasks.rgw_multisite_tests:test_sync_flow_symmetrical_zonegroup_all: ... ok 2024-08-18T17:14:26.892 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:14:26.892 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:14:26.892 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:14:26.939 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:14:26.939 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:14:26.993 INFO:teuthology.orchestra.run.smithi134.stderr:No --period given, using current period=01050e30-179b-43be-bdbb-3bb23cf37ae8 2024-08-18T17:14:27.012 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T17:14:27.013 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:14:27.013 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:14:27.013 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T17:14:27.013 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:14:27.013 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:14:27.013 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:14:27.013 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T17:14:27.013 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:14:27.013 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:14:27.013 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:14:27.013 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:42:47.672461Z" 2024-08-18T17:14:27.013 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:14:27.013 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:14:27.014 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1724001143.400665_1473.1", 2024-08-18T17:14:27.014 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T17:12:23.400665Z" 2024-08-18T17:14:27.014 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:14:27.014 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T17:14:27.015 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-08-18T17:14:27.015 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:14:27.015 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:14:27.015 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:14:27.067 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:14:27.068 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:14:27.099 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:14:27.098+0000 7fd554b23880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:14:27.100 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:14:27.099+0000 7fd554b23880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:14:27.114 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:14:27.114 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:14:27.114 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:14:27.114 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:14:27.114 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:14:27.114 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:14:27.114 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:14:27.114 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:27.114 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:14:27.114 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:14:27.115 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:14:27.115 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:14:27.115 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:14:27.115 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:14:27.115 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:14:27.115 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:14:27.115 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:14:27.115 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:14:27.115 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:14:27.115 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:14:27.115 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:27.115 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:14:27.115 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:14:27.115 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:14:27.116 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:14:27.116 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:14:27.116 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:14:27.116 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:14:27.116 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:14:27.116 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:14:27.116 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:14:27.116 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:14:27.116 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:27.116 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:14:27.116 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:14:27.116 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:14:27.116 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:14:27.117 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:14:27.117 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:14:27.117 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:14:27.117 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:14:27.117 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:14:27.117 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:14:27.117 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:14:27.117 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:27.117 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:14:27.117 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:14:27.117 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:14:27.117 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:14:27.117 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001143.400665_1473.1", 2024-08-18T17:14:27.118 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:14:27.118 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:14:27.118 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:14:27.118 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:12:23.400665Z", 2024-08-18T17:14:27.118 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:14:27.118 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:14:27.118 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:14:27.118 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:14:27.118 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:27.118 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:14:27.118 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:14:27.118 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:14:27.118 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:27.119 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:14:27Z" 2024-08-18T17:14:27.119 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:14:27.120 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-08-18T17:14:27.120 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-08-18T17:14:27.120 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:14:27.120 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:14:27.120 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:14:27.168 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:14:27.168 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:14:27.204 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:14:27.203+0000 7fbcf7410880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:14:27.204 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:14:27.204+0000 7fbcf7410880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:14:27.218 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T17:14:27.218 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T17:14:27.219 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T17:14:27.219 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T17:14:27.219 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T17:14:27.219 INFO:teuthology.orchestra.run.smithi134.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:14:27.219 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:14:27.219 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:14:27.219 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T17:14:27.219 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:14:27.219 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T17:14:27.219 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:14:27.219 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:14:27.219 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:14:27.219 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:14:27.219 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 1, 2024-08-18T17:14:27.220 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:14:27.220 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:14:27.220 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T17:14:27.220 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:14:27.220 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:14:27.220 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:14:27.220 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T17:14:27.220 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:14:27.220 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:14:27.220 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:14:27.220 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:14:27.220 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:14:27.220 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:14:27.220 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T17:14:27.221 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T17:14:27.221 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:14:27.221 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:14:27.221 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:14:27.221 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T17:14:27.221 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:14:27.221 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:14:27.221 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:14:27.221 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:14:27.221 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:14:27.221 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:14:27.221 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:14:27.221 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:14:27.221 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:14:27.222 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:14:27.222 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:14:27.222 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T17:14:27.222 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:14:27.222 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:14:27.222 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1724001143.400665_1473.1", 2024-08-18T17:14:27.222 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:14:27.222 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:14:27.222 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:14:27.222 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T17:12:23.400665Z", 2024-08-18T17:14:27.222 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:14:27.222 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:14:27.222 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:14:27.223 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:14:27.223 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:14:27.223 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T17:14:27.223 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 1, 2024-08-18T17:14:27.223 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 1 2024-08-18T17:14:27.223 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:14:27.223 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T17:14:27Z" 2024-08-18T17:14:27.223 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T17:14:27.226 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-08-18T17:14:27.226 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:14:27.226 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:14:27.226 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:14:27.271 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:14:27.271 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:14:27.318 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[]} 2024-08-18T17:14:27.318 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:14:27.319 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:14:27.319 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:14:27.367 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:14:27.367 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:14:27.444 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2024-08-18T17:14:27.444 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:14:27.444 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:14:27.444 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow --flow-type symmetrical --zones=test-zone1,test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:14:27.495 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:14:27.495 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:14:27.569 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow","zones":["test-zone2","test-zone1"]}]},"pipes":[],"status":"allowed"}]} 2024-08-18T17:14:27.570 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1,test-zone2', '--dest-zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:14:27.570 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1,test-zone2', '--dest-zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:14:27.570 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=test-zone1,test-zone2 --dest-zones=test-zone1,test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:14:27.619 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:14:27.620 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:14:27.671 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow","zones":["test-zone2","test-zone1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-08-18T17:14:27.671 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:14:27.671 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:14:27.671 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:14:27.718 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:14:27.718 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:14:27.804 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow","zones":["test-zone2","test-zone1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-08-18T17:14:27.804 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:14:27.804 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:14:27.804 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:14:27.850 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:14:27.851 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:14:27.919 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T17:14:27.920 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:14:27.920 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 13, 2024-08-18T17:14:27.920 INFO:teuthology.orchestra.run.smithi134.stdout: "predecessor_uuid": "64309209-7567-46e5-8881-65783d84cf3c", 2024-08-18T17:14:27.920 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": [ 2024-08-18T17:14:27.920 INFO:teuthology.orchestra.run.smithi134.stdout: "", 2024-08-18T17:14:27.920 INFO:teuthology.orchestra.run.smithi134.stdout: "", 2024-08-18T17:14:27.920 INFO:teuthology.orchestra.run.smithi134.stdout: "", 2024-08-18T17:14:27.920 INFO:teuthology.orchestra.run.smithi134.stdout: "1_1723995848.930494_187.1" 2024-08-18T17:14:27.920 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:14:27.920 INFO:teuthology.orchestra.run.smithi134.stdout: "period_map": { 2024-08-18T17:14:27.920 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:14:27.920 INFO:teuthology.orchestra.run.smithi134.stdout: "zonegroups": [ 2024-08-18T17:14:27.921 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:14:27.921 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "test-zonegroup", 2024-08-18T17:14:27.921 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "test-zonegroup", 2024-08-18T17:14:27.921 INFO:teuthology.orchestra.run.smithi134.stdout: "api_name": "", 2024-08-18T17:14:27.921 INFO:teuthology.orchestra.run.smithi134.stdout: "is_master": true, 2024-08-18T17:14:27.921 INFO:teuthology.orchestra.run.smithi134.stdout: "endpoints": [ 2024-08-18T17:14:27.921 INFO:teuthology.orchestra.run.smithi134.stdout: "http://smithi134.front.sepia.ceph.com:8000" 2024-08-18T17:14:27.921 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:14:27.921 INFO:teuthology.orchestra.run.smithi134.stdout: "hostnames": [], 2024-08-18T17:14:27.921 INFO:teuthology.orchestra.run.smithi134.stdout: "hostnames_s3website": [], 2024-08-18T17:14:27.921 INFO:teuthology.orchestra.run.smithi134.stdout: "master_zone": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T17:14:27.921 INFO:teuthology.orchestra.run.smithi134.stdout: "zones": [ 2024-08-18T17:14:27.921 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:14:27.921 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T17:14:27.922 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "test-zone3", 2024-08-18T17:14:27.922 INFO:teuthology.orchestra.run.smithi134.stdout: "endpoints": [ 2024-08-18T17:14:27.922 INFO:teuthology.orchestra.run.smithi134.stdout: "http://smithi134.front.sepia.ceph.com:8001" 2024-08-18T17:14:27.922 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:14:27.922 INFO:teuthology.orchestra.run.smithi134.stdout: "log_meta": false, 2024-08-18T17:14:27.922 INFO:teuthology.orchestra.run.smithi134.stdout: "log_data": true, 2024-08-18T17:14:27.922 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_index_max_shards": 11, 2024-08-18T17:14:27.922 INFO:teuthology.orchestra.run.smithi134.stdout: "read_only": false, 2024-08-18T17:14:27.922 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_type": "", 2024-08-18T17:14:27.922 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from_all": true, 2024-08-18T17:14:27.922 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from": [], 2024-08-18T17:14:27.922 INFO:teuthology.orchestra.run.smithi134.stdout: "redirect_zone": "", 2024-08-18T17:14:27.923 INFO:teuthology.orchestra.run.smithi134.stdout: "supported_features": [ 2024-08-18T17:14:27.923 INFO:teuthology.orchestra.run.smithi134.stdout: "compress-encrypted", 2024-08-18T17:14:27.923 INFO:teuthology.orchestra.run.smithi134.stdout: "resharding" 2024-08-18T17:14:27.923 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:14:27.923 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:14:27.923 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:14:27.923 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "10090ccf-aad6-416d-9775-df8b7b27345a", 2024-08-18T17:14:27.923 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "test-zone2", 2024-08-18T17:14:27.923 INFO:teuthology.orchestra.run.smithi134.stdout: "endpoints": [ 2024-08-18T17:14:27.923 INFO:teuthology.orchestra.run.smithi134.stdout: "http://smithi193.front.sepia.ceph.com:8000" 2024-08-18T17:14:27.923 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:14:27.923 INFO:teuthology.orchestra.run.smithi134.stdout: "log_meta": false, 2024-08-18T17:14:27.923 INFO:teuthology.orchestra.run.smithi134.stdout: "log_data": true, 2024-08-18T17:14:27.924 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_index_max_shards": 11, 2024-08-18T17:14:27.924 INFO:teuthology.orchestra.run.smithi134.stdout: "read_only": false, 2024-08-18T17:14:27.924 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_type": "", 2024-08-18T17:14:27.924 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from_all": true, 2024-08-18T17:14:27.924 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from": [], 2024-08-18T17:14:27.924 INFO:teuthology.orchestra.run.smithi134.stdout: "redirect_zone": "", 2024-08-18T17:14:27.924 INFO:teuthology.orchestra.run.smithi134.stdout: "supported_features": [ 2024-08-18T17:14:27.924 INFO:teuthology.orchestra.run.smithi134.stdout: "compress-encrypted", 2024-08-18T17:14:27.924 INFO:teuthology.orchestra.run.smithi134.stdout: "resharding" 2024-08-18T17:14:27.924 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:14:27.924 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:14:27.924 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:14:27.924 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T17:14:27.924 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "test-zone1", 2024-08-18T17:14:27.925 INFO:teuthology.orchestra.run.smithi134.stdout: "endpoints": [ 2024-08-18T17:14:27.925 INFO:teuthology.orchestra.run.smithi134.stdout: "http://smithi134.front.sepia.ceph.com:8000" 2024-08-18T17:14:27.925 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:14:27.925 INFO:teuthology.orchestra.run.smithi134.stdout: "log_meta": false, 2024-08-18T17:14:27.925 INFO:teuthology.orchestra.run.smithi134.stdout: "log_data": true, 2024-08-18T17:14:27.925 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_index_max_shards": 11, 2024-08-18T17:14:27.925 INFO:teuthology.orchestra.run.smithi134.stdout: "read_only": false, 2024-08-18T17:14:27.925 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_type": "", 2024-08-18T17:14:27.925 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from_all": true, 2024-08-18T17:14:27.925 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from": [], 2024-08-18T17:14:27.925 INFO:teuthology.orchestra.run.smithi134.stdout: "redirect_zone": "", 2024-08-18T17:14:27.925 INFO:teuthology.orchestra.run.smithi134.stdout: "supported_features": [ 2024-08-18T17:14:27.925 INFO:teuthology.orchestra.run.smithi134.stdout: "compress-encrypted", 2024-08-18T17:14:27.926 INFO:teuthology.orchestra.run.smithi134.stdout: "resharding" 2024-08-18T17:14:27.926 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:14:27.926 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:14:27.926 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:14:27.926 INFO:teuthology.orchestra.run.smithi134.stdout: "placement_targets": [ 2024-08-18T17:14:27.926 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:14:27.926 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "default-placement", 2024-08-18T17:14:27.926 INFO:teuthology.orchestra.run.smithi134.stdout: "tags": [], 2024-08-18T17:14:27.926 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_classes": [ 2024-08-18T17:14:27.926 INFO:teuthology.orchestra.run.smithi134.stdout: "STANDARD" 2024-08-18T17:14:27.926 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:14:27.926 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:14:27.927 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:14:27.927 INFO:teuthology.orchestra.run.smithi134.stdout: "default_placement": "default-placement", 2024-08-18T17:14:27.927 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_id": "2827c6f0-bf0d-4836-944c-9e7fcf22457b", 2024-08-18T17:14:27.927 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_policy": { 2024-08-18T17:14:27.927 INFO:teuthology.orchestra.run.smithi134.stdout: "groups": [ 2024-08-18T17:14:27.927 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:14:27.927 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "sync-group", 2024-08-18T17:14:27.927 INFO:teuthology.orchestra.run.smithi134.stdout: "data_flow": { 2024-08-18T17:14:27.927 INFO:teuthology.orchestra.run.smithi134.stdout: "symmetrical": [ 2024-08-18T17:14:27.927 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:14:27.927 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "sync-flow", 2024-08-18T17:14:27.927 INFO:teuthology.orchestra.run.smithi134.stdout: "zones": [ 2024-08-18T17:14:27.936 INFO:teuthology.orchestra.run.smithi134.stdout: "10090ccf-aad6-416d-9775-df8b7b27345a", 2024-08-18T17:14:27.936 INFO:teuthology.orchestra.run.smithi134.stdout: "96bf8502-2f39-4d53-8196-3378c28da3b4" 2024-08-18T17:14:27.936 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:14:27.936 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:14:27.936 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:14:27.936 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:14:27.936 INFO:teuthology.orchestra.run.smithi134.stdout: "pipes": [ 2024-08-18T17:14:27.936 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:14:27.936 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "sync-pipe", 2024-08-18T17:14:27.937 INFO:teuthology.orchestra.run.smithi134.stdout: "source": { 2024-08-18T17:14:27.937 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket": "*", 2024-08-18T17:14:27.937 INFO:teuthology.orchestra.run.smithi134.stdout: "zones": [ 2024-08-18T17:14:27.937 INFO:teuthology.orchestra.run.smithi134.stdout: "10090ccf-aad6-416d-9775-df8b7b27345a", 2024-08-18T17:14:27.937 INFO:teuthology.orchestra.run.smithi134.stdout: "96bf8502-2f39-4d53-8196-3378c28da3b4" 2024-08-18T17:14:27.937 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:14:27.937 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:14:27.937 INFO:teuthology.orchestra.run.smithi134.stdout: "dest": { 2024-08-18T17:14:27.937 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket": "*", 2024-08-18T17:14:27.937 INFO:teuthology.orchestra.run.smithi134.stdout: "zones": [ 2024-08-18T17:14:27.937 INFO:teuthology.orchestra.run.smithi134.stdout: "10090ccf-aad6-416d-9775-df8b7b27345a", 2024-08-18T17:14:27.937 INFO:teuthology.orchestra.run.smithi134.stdout: "96bf8502-2f39-4d53-8196-3378c28da3b4" 2024-08-18T17:14:27.937 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:14:27.938 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:14:27.938 INFO:teuthology.orchestra.run.smithi134.stdout: "params": { 2024-08-18T17:14:27.938 INFO:teuthology.orchestra.run.smithi134.stdout: "source": { 2024-08-18T17:14:27.938 INFO:teuthology.orchestra.run.smithi134.stdout: "filter": { 2024-08-18T17:14:27.938 INFO:teuthology.orchestra.run.smithi134.stdout: "tags": [] 2024-08-18T17:14:27.938 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:14:27.938 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:14:27.938 INFO:teuthology.orchestra.run.smithi134.stdout: "dest": {}, 2024-08-18T17:14:27.938 INFO:teuthology.orchestra.run.smithi134.stdout: "priority": 0, 2024-08-18T17:14:27.939 INFO:teuthology.orchestra.run.smithi134.stdout: "mode": "system", 2024-08-18T17:14:27.939 INFO:teuthology.orchestra.run.smithi134.stdout: "user": "" 2024-08-18T17:14:27.939 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:14:27.939 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:14:27.939 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:14:27.939 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "enabled" 2024-08-18T17:14:27.939 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:14:27.939 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:14:27.939 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:14:27.939 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled_features": [ 2024-08-18T17:14:27.939 INFO:teuthology.orchestra.run.smithi134.stdout: "resharding" 2024-08-18T17:14:27.939 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:14:27.939 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:14:27.940 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:14:27.940 INFO:teuthology.orchestra.run.smithi134.stdout: "short_zone_ids": [ 2024-08-18T17:14:27.940 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:14:27.940 INFO:teuthology.orchestra.run.smithi134.stdout: "key": "018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T17:14:27.940 INFO:teuthology.orchestra.run.smithi134.stdout: "val": 462036791 2024-08-18T17:14:27.940 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:14:27.940 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:14:27.940 INFO:teuthology.orchestra.run.smithi134.stdout: "key": "10090ccf-aad6-416d-9775-df8b7b27345a", 2024-08-18T17:14:27.940 INFO:teuthology.orchestra.run.smithi134.stdout: "val": 874289624 2024-08-18T17:14:27.940 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:14:27.940 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:14:27.940 INFO:teuthology.orchestra.run.smithi134.stdout: "key": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T17:14:27.940 INFO:teuthology.orchestra.run.smithi134.stdout: "val": 1721399886 2024-08-18T17:14:27.941 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:14:27.941 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:14:27.941 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:14:27.941 INFO:teuthology.orchestra.run.smithi134.stdout: "master_zonegroup": "test-zonegroup", 2024-08-18T17:14:27.941 INFO:teuthology.orchestra.run.smithi134.stdout: "master_zone": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T17:14:27.941 INFO:teuthology.orchestra.run.smithi134.stdout: "period_config": { 2024-08-18T17:14:27.941 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_quota": { 2024-08-18T17:14:27.941 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false, 2024-08-18T17:14:27.941 INFO:teuthology.orchestra.run.smithi134.stdout: "check_on_raw": false, 2024-08-18T17:14:27.941 INFO:teuthology.orchestra.run.smithi134.stdout: "max_size": -1, 2024-08-18T17:14:27.941 INFO:teuthology.orchestra.run.smithi134.stdout: "max_size_kb": 0, 2024-08-18T17:14:27.941 INFO:teuthology.orchestra.run.smithi134.stdout: "max_objects": -1 2024-08-18T17:14:27.941 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:14:27.942 INFO:teuthology.orchestra.run.smithi134.stdout: "user_quota": { 2024-08-18T17:14:27.942 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false, 2024-08-18T17:14:27.942 INFO:teuthology.orchestra.run.smithi134.stdout: "check_on_raw": false, 2024-08-18T17:14:27.942 INFO:teuthology.orchestra.run.smithi134.stdout: "max_size": -1, 2024-08-18T17:14:27.942 INFO:teuthology.orchestra.run.smithi134.stdout: "max_size_kb": 0, 2024-08-18T17:14:27.942 INFO:teuthology.orchestra.run.smithi134.stdout: "max_objects": -1 2024-08-18T17:14:27.942 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:14:27.942 INFO:teuthology.orchestra.run.smithi134.stdout: "user_ratelimit": { 2024-08-18T17:14:27.942 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_ops": 0, 2024-08-18T17:14:27.942 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_ops": 0, 2024-08-18T17:14:27.942 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_bytes": 0, 2024-08-18T17:14:27.942 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_bytes": 0, 2024-08-18T17:14:27.942 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false 2024-08-18T17:14:27.943 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:14:27.943 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_ratelimit": { 2024-08-18T17:14:27.943 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_ops": 0, 2024-08-18T17:14:27.943 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_ops": 0, 2024-08-18T17:14:27.943 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_bytes": 0, 2024-08-18T17:14:27.943 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_bytes": 0, 2024-08-18T17:14:27.943 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false 2024-08-18T17:14:27.943 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:14:27.943 INFO:teuthology.orchestra.run.smithi134.stdout: "anonymous_ratelimit": { 2024-08-18T17:14:27.943 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_ops": 0, 2024-08-18T17:14:27.943 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_ops": 0, 2024-08-18T17:14:27.943 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_bytes": 0, 2024-08-18T17:14:27.943 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_bytes": 0, 2024-08-18T17:14:27.944 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false 2024-08-18T17:14:27.944 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:14:27.944 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:14:27.944 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_id": "2827c6f0-bf0d-4836-944c-9e7fcf22457b", 2024-08-18T17:14:27.944 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_name": "test-realm", 2024-08-18T17:14:27.944 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:14:27.944 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T17:14:27.945 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:14:27.945 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:14:27.946 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:14:27.993 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:14:27.993 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:14:28.119 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow","zones":["test-zone2","test-zone1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-08-18T17:14:28.119 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=kibgmb-87 2024-08-18T17:14:28.265 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=kibgmb-88 2024-08-18T17:14:28.417 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:14:28.418 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:14:28.418 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:14:28.486 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:14:28.486 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:14:28.576 INFO:teuthology.orchestra.run.smithi134.stderr:No --period given, using current period=01050e30-179b-43be-bdbb-3bb23cf37ae8 2024-08-18T17:14:28.622 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T17:14:28.622 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:14:28.622 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:14:28.622 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T17:14:28.622 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:14:28.622 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:14:28.622 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:14:28.622 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T17:14:28.622 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:14:28.622 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:14:28.623 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:14:28.623 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:42:47.672461Z" 2024-08-18T17:14:28.623 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:14:28.623 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:14:28.623 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1724001268.323120_1508.1", 2024-08-18T17:14:28.623 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T17:14:28.323120Z" 2024-08-18T17:14:28.623 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:14:28.623 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T17:14:28.624 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-08-18T17:14:28.624 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:14:28.624 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:14:28.624 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:14:28.673 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:14:28.673 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:14:28.704 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:14:28.703+0000 7efe70613880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:14:28.704 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:14:28.704+0000 7efe70613880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:14:28.719 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:14:28.719 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:14:28.719 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:14:28.719 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:14:28.719 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:14:28.719 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:14:28.719 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:14:28.719 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:28.719 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:14:28.720 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:14:28.720 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:14:28.720 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:14:28.720 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:14:28.720 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:14:28.720 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:14:28.720 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:14:28.720 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:14:28.720 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:14:28.720 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:14:28.720 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:14:28.720 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:28.720 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:14:28.721 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:14:28.721 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:14:28.721 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:14:28.721 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:14:28.721 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:14:28.721 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:14:28.721 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:14:28.721 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:14:28.721 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:14:28.721 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:14:28.721 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:28.721 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:14:28.721 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:14:28.721 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:14:28.722 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:14:28.722 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:14:28.722 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:14:28.722 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:14:28.722 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:14:28.722 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:14:28.722 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:14:28.722 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:14:28.722 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:28.722 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:14:28.722 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:14:28.722 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:14:28.723 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:14:28.723 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001143.400665_1473.1", 2024-08-18T17:14:28.723 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:14:28.723 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:14:28.723 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:14:28.723 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:12:23.400665Z", 2024-08-18T17:14:28.723 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:14:28.723 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:14:28.723 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:14:28.723 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:14:28.723 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:28.723 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:14:28.724 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:14:28.724 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:14:28.724 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:28.724 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:14:28Z" 2024-08-18T17:14:28.724 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:14:28.725 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001268.323120_1508.1 target=1_1724001143.400665_1473.1 2024-08-18T17:14:33.726 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:14:33.726 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:14:33.726 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:14:33.782 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:14:33.782 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:14:33.816 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:14:33.815+0000 7fc35c411880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:14:33.817 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:14:33.816+0000 7fc35c411880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:14:33.832 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:14:33.832 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:14:33.832 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:14:33.832 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:14:33.833 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:14:33.833 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:14:33.833 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:14:33.833 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:33.833 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:14:33.833 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:14:33.833 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:14:33.833 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:14:33.833 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:14:33.833 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:14:33.833 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:14:33.833 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:14:33.833 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:14:33.833 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:14:33.834 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:14:33.834 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:14:33.834 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:33.834 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:14:33.834 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:14:33.834 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:14:33.834 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:14:33.834 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:14:33.834 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:14:33.834 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:14:33.834 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:14:33.834 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:14:33.834 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:14:33.835 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:14:33.835 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:33.835 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:14:33.835 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:14:33.835 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:14:33.835 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:14:33.835 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:14:33.835 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:14:33.835 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:14:33.835 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:14:33.835 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:14:33.835 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:14:33.836 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:14:33.836 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:33.836 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:14:33.836 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:14:33.836 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:14:33.836 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:14:33.836 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001143.400665_1473.1", 2024-08-18T17:14:33.836 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:14:33.836 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:14:33.836 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:14:33.836 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:12:23.400665Z", 2024-08-18T17:14:33.836 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:14:33.836 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:14:33.837 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:14:33.837 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:14:33.837 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:33.837 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:14:33.837 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:14:33.837 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:14:33.837 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:33.837 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:14:33Z" 2024-08-18T17:14:33.837 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:14:33.838 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001268.323120_1508.1 target=1_1724001143.400665_1473.1 2024-08-18T17:14:38.839 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:14:38.839 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:14:38.839 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:14:38.894 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:14:38.894 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:14:38.928 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:14:38.926+0000 7fd88b01f880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:14:38.929 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:14:38.927+0000 7fd88b01f880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:14:38.944 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:14:38.944 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:14:38.944 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:14:38.944 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:14:38.944 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:14:38.945 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:14:38.945 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:14:38.945 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:38.945 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:14:38.945 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:14:38.945 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:14:38.945 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:14:38.945 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:14:38.945 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:14:38.946 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:14:38.946 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:14:38.946 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:14:38.946 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:14:38.946 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:14:38.946 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:14:38.946 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:38.946 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:14:38.946 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:14:38.946 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:14:38.946 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:14:38.946 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:14:38.947 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:14:38.947 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:14:38.947 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:14:38.947 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:14:38.947 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:14:38.947 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:14:38.947 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:38.947 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:14:38.947 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:14:38.947 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:14:38.947 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:14:38.947 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:14:38.947 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:14:38.947 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:14:38.948 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:14:38.948 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:14:38.948 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:14:38.948 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:14:38.948 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:38.948 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:14:38.948 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:14:38.948 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:14:38.948 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:14:38.948 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001143.400665_1473.1", 2024-08-18T17:14:38.948 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:14:38.948 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:14:38.948 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:14:38.949 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:12:23.400665Z", 2024-08-18T17:14:38.949 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:14:38.949 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:14:38.949 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:14:38.949 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:14:38.949 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:38.949 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:14:38.949 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:14:38.949 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:14:38.949 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:38.949 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:14:38Z" 2024-08-18T17:14:38.949 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:14:38.950 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001268.323120_1508.1 target=1_1724001143.400665_1473.1 2024-08-18T17:14:43.951 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:14:43.951 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:14:43.952 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:14:44.007 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:14:44.007 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:14:44.042 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:14:44.041+0000 7fd454942880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:14:44.043 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:14:44.042+0000 7fd454942880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:14:44.058 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:14:44.058 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:14:44.059 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:14:44.059 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:14:44.059 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:14:44.059 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:14:44.059 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:14:44.059 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:44.060 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:14:44.060 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:14:44.060 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:14:44.060 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:14:44.060 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:14:44.060 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:14:44.060 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:14:44.061 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:14:44.061 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:14:44.061 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:14:44.061 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:14:44.061 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:14:44.061 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:44.061 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:14:44.061 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:14:44.062 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:14:44.062 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:14:44.062 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:14:44.062 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:14:44.062 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:14:44.062 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:14:44.062 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:14:44.062 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:14:44.063 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:14:44.063 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:44.063 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:14:44.063 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:14:44.064 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:14:44.064 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:14:44.064 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:14:44.064 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:14:44.064 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:14:44.064 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:14:44.064 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:14:44.064 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:14:44.065 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:14:44.065 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:44.065 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:14:44.065 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:14:44.065 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:14:44.065 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:14:44.065 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001143.400665_1473.1", 2024-08-18T17:14:44.065 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:14:44.066 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:14:44.066 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:14:44.066 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:12:23.400665Z", 2024-08-18T17:14:44.066 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:14:44.066 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:14:44.066 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:14:44.067 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:14:44.067 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:44.067 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:14:44.067 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:14:44.067 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:14:44.067 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:44.067 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:14:44Z" 2024-08-18T17:14:44.067 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:14:44.068 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001268.323120_1508.1 target=1_1724001143.400665_1473.1 2024-08-18T17:14:49.069 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:14:49.070 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:14:49.070 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:14:49.126 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:14:49.127 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:14:49.160 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:14:49.160+0000 7f5e74813880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:14:49.161 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:14:49.161+0000 7f5e74813880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:14:49.176 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:14:49.176 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:14:49.176 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:14:49.176 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:14:49.176 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:14:49.177 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:14:49.177 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:14:49.177 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:49.177 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:14:49.177 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:14:49.177 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:14:49.177 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:14:49.177 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:14:49.177 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:14:49.177 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:14:49.177 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:14:49.177 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:14:49.178 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:14:49.178 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:14:49.178 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:14:49.178 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:49.178 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:14:49.178 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:14:49.178 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:14:49.178 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:14:49.178 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:14:49.178 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:14:49.178 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:14:49.179 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:14:49.179 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:14:49.179 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:14:49.179 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:14:49.179 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:49.179 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:14:49.179 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:14:49.179 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:14:49.179 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:14:49.179 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:14:49.179 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:14:49.179 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:14:49.179 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:14:49.180 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:14:49.180 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:14:49.180 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:14:49.180 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:49.180 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:14:49.180 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:14:49.180 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:14:49.180 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:14:49.180 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001143.400665_1473.1", 2024-08-18T17:14:49.180 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:14:49.180 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:14:49.180 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:14:49.181 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:12:23.400665Z", 2024-08-18T17:14:49.181 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:14:49.181 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:14:49.181 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:14:49.181 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:14:49.181 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:49.181 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:14:49.181 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:14:49.181 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:14:49.181 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:49.181 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:14:49Z" 2024-08-18T17:14:49.181 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:14:49.182 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001268.323120_1508.1 target=1_1724001143.400665_1473.1 2024-08-18T17:14:54.183 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:14:54.184 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:14:54.184 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:14:54.239 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:14:54.240 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:14:54.274 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:14:54.273+0000 7f1f85d34880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:14:54.274 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:14:54.274+0000 7f1f85d34880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:14:54.290 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:14:54.290 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:14:54.290 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:14:54.290 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:14:54.291 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:14:54.291 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:14:54.291 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:14:54.291 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:54.291 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:14:54.291 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:14:54.292 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:14:54.292 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:14:54.292 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:14:54.292 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:14:54.293 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:14:54.293 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:14:54.293 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:14:54.293 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:14:54.293 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:14:54.293 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:14:54.294 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:54.294 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:14:54.294 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:14:54.294 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:14:54.295 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:14:54.295 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:14:54.295 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:14:54.295 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:14:54.295 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:14:54.295 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:14:54.295 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:14:54.295 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:14:54.295 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:54.296 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:14:54.296 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:14:54.296 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:14:54.296 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:14:54.296 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:14:54.296 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:14:54.296 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:14:54.296 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:14:54.296 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:14:54.296 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:14:54.296 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:14:54.296 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:54.297 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:14:54.297 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:14:54.297 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:14:54.297 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:14:54.297 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001143.400665_1473.1", 2024-08-18T17:14:54.297 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:14:54.297 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:14:54.298 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:14:54.298 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:12:23.400665Z", 2024-08-18T17:14:54.298 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:14:54.298 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:14:54.298 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:14:54.298 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:14:54.298 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:54.298 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:14:54.298 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:14:54.298 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:14:54.298 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:54.299 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:14:54Z" 2024-08-18T17:14:54.299 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:14:54.300 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001268.323120_1508.1 target=1_1724001143.400665_1473.1 2024-08-18T17:14:59.301 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:14:59.301 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:14:59.301 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:14:59.357 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:14:59.357 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:14:59.391 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:14:59.390+0000 7efcd2c15880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:14:59.392 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:14:59.391+0000 7efcd2c15880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:14:59.407 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:14:59.407 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:14:59.407 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:14:59.407 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:14:59.407 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:14:59.407 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:14:59.407 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:14:59.408 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:59.408 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:14:59.408 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:14:59.408 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:14:59.408 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:14:59.408 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:14:59.408 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:14:59.408 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:14:59.408 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:14:59.408 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:14:59.408 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:14:59.408 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:14:59.408 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:14:59.408 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:59.409 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:14:59.409 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:14:59.409 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:14:59.409 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:14:59.409 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:14:59.409 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:14:59.409 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:14:59.409 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:14:59.409 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:14:59.409 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:14:59.409 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:14:59.409 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:59.409 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:14:59.409 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:14:59.410 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:14:59.410 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:14:59.410 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:14:59.410 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:14:59.410 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:14:59.410 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:14:59.410 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:14:59.410 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:14:59.410 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:14:59.410 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:59.410 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:14:59.410 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:14:59.411 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:14:59.411 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:14:59.411 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001143.400665_1473.1", 2024-08-18T17:14:59.411 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:14:59.411 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:14:59.411 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:14:59.411 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:12:23.400665Z", 2024-08-18T17:14:59.411 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:14:59.411 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:14:59.411 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:14:59.411 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:14:59.411 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:59.411 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:14:59.411 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:14:59.412 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:14:59.412 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:14:59.412 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:14:59Z" 2024-08-18T17:14:59.412 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:14:59.413 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001268.323120_1508.1 target=1_1724001143.400665_1473.1 2024-08-18T17:15:04.414 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:15:04.414 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:15:04.415 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:15:04.469 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:15:04.469 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:15:04.504 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:15:04.503+0000 7f0361d26880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:15:04.504 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:15:04.504+0000 7f0361d26880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:15:04.519 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:15:04.520 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:15:04.520 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:15:04.520 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:15:04.520 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:15:04.521 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:15:04.521 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:15:04.521 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:15:04.521 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:15:04.521 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:15:04.521 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:15:04.522 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:15:04.522 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:15:04.522 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:15:04.522 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:15:04.522 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:15:04.522 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:15:04.522 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:15:04.523 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:15:04.523 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:15:04.523 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:15:04.523 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:15:04.523 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:15:04.523 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:15:04.523 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:15:04.523 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:15:04.524 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:15:04.524 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:15:04.524 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:15:04.524 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:15:04.524 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:15:04.524 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:15:04.524 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:15:04.524 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:15:04.525 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:15:04.525 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:15:04.525 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:15:04.525 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:15:04.525 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:15:04.525 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:15:04.525 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:15:04.525 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:15:04.526 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:15:04.526 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:15:04.526 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:15:04.526 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:15:04.526 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:15:04.526 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:15:04.526 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:15:04.527 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001143.400665_1473.1", 2024-08-18T17:15:04.527 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:15:04.527 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:15:04.527 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:15:04.527 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:12:23.400665Z", 2024-08-18T17:15:04.527 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:15:04.528 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:15:04.528 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:15:04.528 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:15:04.528 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:15:04.528 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:15:04.528 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:15:04.528 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:15:04.528 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:15:04.528 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:15:04Z" 2024-08-18T17:15:04.528 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:15:04.530 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001268.323120_1508.1 target=1_1724001143.400665_1473.1 2024-08-18T17:15:09.530 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:15:09.531 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:15:09.531 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:15:09.588 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:15:09.588 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:15:09.622 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:15:09.622+0000 7f3af7b36880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:15:09.623 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:15:09.622+0000 7f3af7b36880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:15:09.637 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:15:09.638 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:15:09.638 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:15:09.638 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:15:09.638 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:15:09.638 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:15:09.638 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:15:09.638 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:15:09.638 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:15:09.638 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:15:09.638 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:15:09.639 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:15:09.639 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:15:09.639 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:15:09.639 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:15:09.639 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:15:09.639 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:15:09.639 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:15:09.639 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:15:09.639 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:15:09.639 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:15:09.639 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:15:09.639 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:15:09.639 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:15:09.640 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:15:09.640 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:15:09.640 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:15:09.640 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:15:09.640 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:15:09.640 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:15:09.640 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:15:09.640 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:15:09.640 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:15:09.640 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:15:09.640 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:15:09.640 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:15:09.640 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:15:09.641 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:15:09.641 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:15:09.641 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:15:09.641 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:15:09.641 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:15:09.641 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:15:09.641 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:15:09.641 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:15:09.641 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:15:09.641 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:15:09.641 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:15:09.641 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:15:09.642 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001143.400665_1473.1", 2024-08-18T17:15:09.642 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:15:09.642 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:15:09.642 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:15:09.642 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:12:23.400665Z", 2024-08-18T17:15:09.642 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:15:09.642 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:15:09.642 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:15:09.642 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:15:09.642 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:15:09.642 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:15:09.642 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:15:09.643 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:15:09.643 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:15:09.643 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:15:09Z" 2024-08-18T17:15:09.643 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:15:09.644 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001268.323120_1508.1 target=1_1724001143.400665_1473.1 2024-08-18T17:15:14.645 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:15:14.645 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:15:14.645 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:15:14.700 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:15:14.701 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:15:14.737 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:15:14.736+0000 7f9de1726880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:15:14.737 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:15:14.737+0000 7f9de1726880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:15:14.752 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:15:14.752 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:15:14.752 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:15:14.752 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:15:14.752 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:15:14.752 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:15:14.753 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:15:14.753 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:15:14.753 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:15:14.753 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:15:14.753 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:15:14.753 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:15:14.753 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:15:14.753 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:15:14.753 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:15:14.753 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:15:14.753 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:15:14.753 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:15:14.754 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:15:14.754 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:15:14.754 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:15:14.754 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:15:14.754 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:15:14.754 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:15:14.754 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:15:14.754 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:15:14.754 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:15:14.754 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:15:14.754 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:15:14.754 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:15:14.754 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:15:14.755 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:15:14.755 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:15:14.755 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:15:14.755 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:15:14.755 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:15:14.755 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:15:14.755 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:15:14.755 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:15:14.755 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:15:14.755 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:15:14.755 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:15:14.755 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:15:14.755 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:15:14.755 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:15:14.756 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:15:14.756 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:15:14.756 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:15:14.756 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:15:14.756 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001143.400665_1473.1", 2024-08-18T17:15:14.756 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:15:14.756 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:15:14.756 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:15:14.756 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:12:23.400665Z", 2024-08-18T17:15:14.756 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:15:14.756 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:15:14.756 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:15:14.756 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:15:14.757 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:15:14.757 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:15:14.757 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:15:14.757 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:15:14.757 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:15:14.757 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:15:14Z" 2024-08-18T17:15:14.757 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:15:14.758 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001268.323120_1508.1 target=1_1724001143.400665_1473.1 2024-08-18T17:15:19.759 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:15:19.760 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:15:19.760 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:15:19.815 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:15:19.815 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:15:19.849 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:15:19.848+0000 7f351a931880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:15:19.850 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:15:19.849+0000 7f351a931880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:15:19.864 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:15:19.865 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:15:19.865 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:15:19.865 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:15:19.865 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:15:19.865 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:15:19.865 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:15:19.865 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:15:19.865 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:15:19.865 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:15:19.865 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:15:19.865 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:15:19.866 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:15:19.866 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:15:19.866 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:15:19.866 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:15:19.866 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:15:19.866 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:15:19.866 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:15:19.866 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:15:19.866 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:15:19.866 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:15:19.866 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:15:19.866 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:15:19.867 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:15:19.867 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:15:19.867 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:15:19.867 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:15:19.867 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:15:19.867 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:15:19.867 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:15:19.867 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:15:19.867 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:15:19.867 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:15:19.867 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:15:19.867 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:15:19.867 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:15:19.867 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:15:19.868 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:15:19.868 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:15:19.868 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:15:19.868 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:15:19.868 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:15:19.868 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:15:19.868 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:15:19.868 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:15:19.868 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:15:19.868 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:15:19.868 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:15:19.868 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001143.400665_1473.1", 2024-08-18T17:15:19.869 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:15:19.869 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:15:19.869 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:15:19.869 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:12:23.400665Z", 2024-08-18T17:15:19.869 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:15:19.869 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:15:19.869 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:15:19.869 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:15:19.869 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:15:19.869 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:15:19.869 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:15:19.869 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:15:19.869 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:15:19.869 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:15:19Z" 2024-08-18T17:15:19.870 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:15:19.871 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001268.323120_1508.1 target=1_1724001143.400665_1473.1 2024-08-18T17:15:24.871 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:15:24.871 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:15:24.872 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:15:24.926 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:15:24.927 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:15:24.961 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:15:24.960+0000 7fbaa7035880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:15:24.961 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:15:24.961+0000 7fbaa7035880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:15:24.976 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:15:24.976 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:15:24.976 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:15:24.976 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:15:24.976 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:15:24.976 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:15:24.977 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:15:24.977 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:15:24.977 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:15:24.977 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:15:24.977 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:15:24.977 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:15:24.977 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:15:24.977 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:15:24.977 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:15:24.977 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:15:24.977 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:15:24.978 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:15:24.978 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:15:24.978 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:15:24.978 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:15:24.978 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:15:24.978 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:15:24.978 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:15:24.978 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:15:24.978 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:15:24.978 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:15:24.978 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:15:24.978 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:15:24.978 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:15:24.979 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:15:24.979 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:15:24.979 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:15:24.979 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:15:24.979 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:15:24.979 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:15:24.979 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:15:24.979 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:15:24.979 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:15:24.979 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:15:24.979 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:15:24.979 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:15:24.979 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:15:24.980 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:15:24.980 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:15:24.980 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:15:24.980 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:15:24.980 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:15:24.980 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:15:24.980 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001143.400665_1473.1", 2024-08-18T17:15:24.980 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:15:24.980 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:15:24.980 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:15:24.980 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:12:23.400665Z", 2024-08-18T17:15:24.980 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:15:24.980 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:15:24.980 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:15:24.981 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:15:24.981 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:15:24.981 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:15:24.981 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:15:24.981 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:15:24.981 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:15:24.981 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:15:24Z" 2024-08-18T17:15:24.981 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:15:24.982 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001268.323120_1508.1 target=1_1724001143.400665_1473.1 2024-08-18T17:15:29.983 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:15:29.984 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:15:29.984 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:15:30.038 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:15:30.038 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:15:30.072 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:15:30.072+0000 7f34b1920880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:15:30.073 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:15:30.073+0000 7f34b1920880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:15:30.091 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:15:30.091 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:15:30.091 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:15:30.091 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:15:30.091 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:15:30.091 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:15:30.092 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:15:30.092 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:15:30.092 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:15:30.092 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:15:30.092 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:15:30.092 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:15:30.092 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:15:30.092 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:15:30.092 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:15:30.092 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:15:30.092 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:15:30.092 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:15:30.092 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:15:30.093 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:15:30.093 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:15:30.093 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:15:30.093 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:15:30.093 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:15:30.093 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:15:30.093 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:15:30.093 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:15:30.093 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:15:30.093 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:15:30.093 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:15:30.093 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:15:30.093 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:15:30.094 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:15:30.094 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:15:30.094 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:15:30.094 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:15:30.094 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:15:30.094 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:15:30.094 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:15:30.094 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:15:30.094 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:15:30.094 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:15:30.094 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:15:30.094 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:15:30.094 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:15:30.095 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:15:30.095 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:15:30.095 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:15:30.095 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:15:30.095 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001143.400665_1473.1", 2024-08-18T17:15:30.095 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:15:30.095 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:15:30.095 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:15:30.095 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:12:23.400665Z", 2024-08-18T17:15:30.095 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:15:30.095 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:15:30.095 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:15:30.095 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:15:30.096 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:15:30.096 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:15:30.096 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:15:30.096 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:15:30.096 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:15:30.096 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:15:30Z" 2024-08-18T17:15:30.096 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:15:30.097 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001268.323120_1508.1 target=1_1724001143.400665_1473.1 2024-08-18T17:15:35.098 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:15:35.099 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:15:35.099 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:15:35.154 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:15:35.154 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:15:35.189 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:15:35.188+0000 7f85b5e32880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:15:35.190 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:15:35.189+0000 7f85b5e32880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:15:35.205 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:15:35.205 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:15:35.205 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:15:35.205 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:15:35.205 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:15:35.205 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:15:35.205 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:15:35.205 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:15:35.205 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:15:35.205 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:15:35.206 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:15:35.206 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:15:35.206 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:15:35.206 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:15:35.206 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:15:35.206 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:15:35.207 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:15:35.207 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:15:35.207 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:15:35.207 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:15:35.207 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:15:35.207 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:15:35.207 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:15:35.207 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:15:35.207 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:15:35.207 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:15:35.207 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:15:35.207 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:15:35.207 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:15:35.208 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:15:35.208 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:15:35.208 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:15:35.208 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:15:35.208 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:15:35.208 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:15:35.208 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:15:35.208 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:15:35.208 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:15:35.208 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:15:35.208 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:15:35.209 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:15:35.209 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:15:35.209 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:15:35.209 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:15:35.209 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:15:35.209 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:15:35.209 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:15:35.209 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:15:35.209 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:15:35.209 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001143.400665_1473.1", 2024-08-18T17:15:35.209 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:15:35.209 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:15:35.210 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:15:35.210 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:12:23.400665Z", 2024-08-18T17:15:35.210 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:15:35.210 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:15:35.210 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:15:35.210 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:15:35.210 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:15:35.210 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:15:35.210 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:15:35.210 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:15:35.211 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:15:35.211 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:15:35Z" 2024-08-18T17:15:35.211 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:15:35.212 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001268.323120_1508.1 target=1_1724001143.400665_1473.1 2024-08-18T17:15:40.213 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:15:40.213 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:15:40.213 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:15:40.269 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:15:40.269 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:15:40.303 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:15:40.301+0000 7fe00b63f880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:15:40.304 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:15:40.302+0000 7fe00b63f880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:15:40.319 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:15:40.319 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:15:40.319 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:15:40.319 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:15:40.319 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:15:40.320 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:15:40.320 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:15:40.320 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:15:40.320 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:15:40.320 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:15:40.321 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:15:40.321 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:15:40.321 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:15:40.321 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:15:40.321 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:15:40.322 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:15:40.322 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:15:40.322 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:15:40.322 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:15:40.322 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:15:40.322 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:15:40.323 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:15:40.323 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:15:40.323 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:15:40.323 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:15:40.323 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:15:40.324 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:15:40.324 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:15:40.324 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:15:40.324 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:15:40.324 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:15:40.324 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:15:40.324 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:15:40.324 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:15:40.325 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:15:40.325 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:15:40.325 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:15:40.325 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:15:40.325 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:15:40.326 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:15:40.326 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:15:40.326 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:15:40.326 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:15:40.326 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:15:40.326 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:15:40.327 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:15:40.327 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:15:40.327 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:15:40.327 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:15:40.327 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001143.400665_1473.1", 2024-08-18T17:15:40.328 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:15:40.328 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:15:40.328 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:15:40.328 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:12:23.400665Z", 2024-08-18T17:15:40.328 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:15:40.328 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:15:40.329 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:15:40.329 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:15:40.329 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:15:40.329 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:15:40.329 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:15:40.329 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:15:40.330 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:15:40.330 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:15:40Z" 2024-08-18T17:15:40.330 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:15:40.332 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001268.323120_1508.1 target=1_1724001143.400665_1473.1 2024-08-18T17:15:45.332 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:15:45.332 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:15:45.333 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:15:45.389 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:15:45.389 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:15:45.423 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:15:45.423+0000 7fcb96b32880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:15:45.424 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:15:45.424+0000 7fcb96b32880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:15:45.439 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:15:45.439 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:15:45.439 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:15:45.439 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:15:45.439 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:15:45.439 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:15:45.440 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:15:45.440 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:15:45.440 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:15:45.440 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:15:45.440 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:15:45.440 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:15:45.440 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:15:45.440 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:15:45.440 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:15:45.440 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:15:45.440 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:15:45.440 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:15:45.440 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:15:45.440 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:15:45.441 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:15:45.441 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:15:45.441 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:15:45.441 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:15:45.441 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:15:45.441 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:15:45.441 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:15:45.442 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:15:45.442 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:15:45.442 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:15:45.442 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:15:45.442 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:15:45.442 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:15:45.442 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:15:45.443 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:15:45.443 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:15:45.443 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:15:45.443 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:15:45.443 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:15:45.443 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:15:45.443 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:15:45.443 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:15:45.444 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:15:45.444 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:15:45.444 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:15:45.444 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:15:45.444 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:15:45.444 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:15:45.444 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:15:45.444 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001143.400665_1473.1", 2024-08-18T17:15:45.444 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:15:45.444 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:15:45.445 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:15:45.445 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:12:23.400665Z", 2024-08-18T17:15:45.445 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:15:45.445 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:15:45.445 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:15:45.445 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:15:45.445 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:15:45.445 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:15:45.445 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:15:45.445 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:15:45.445 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:15:45.445 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:15:45Z" 2024-08-18T17:15:45.445 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:15:45.446 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001268.323120_1508.1 target=1_1724001143.400665_1473.1 2024-08-18T17:15:50.448 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:15:50.448 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:15:50.448 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:15:50.503 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:15:50.504 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:15:50.537 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:15:50.537+0000 7ffa71810880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:15:50.538 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:15:50.538+0000 7ffa71810880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:15:50.556 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:15:50.556 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:15:50.556 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:15:50.556 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:15:50.556 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:15:50.557 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:15:50.557 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:15:50.557 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:15:50.557 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:15:50.557 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:15:50.557 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:15:50.557 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:15:50.557 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:15:50.557 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:15:50.557 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:15:50.557 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:15:50.558 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:15:50.558 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:15:50.558 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:15:50.558 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:15:50.558 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:15:50.558 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:15:50.558 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:15:50.558 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:15:50.558 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:15:50.558 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:15:50.558 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:15:50.558 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:15:50.558 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:15:50.559 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:15:50.559 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:15:50.559 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:15:50.559 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:15:50.559 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:15:50.559 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:15:50.559 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:15:50.559 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:15:50.559 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:15:50.559 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:15:50.559 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:15:50.559 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:15:50.559 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:15:50.560 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:15:50.560 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:15:50.560 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:15:50.560 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:15:50.560 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:15:50.560 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:15:50.560 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:15:50.560 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001143.400665_1473.1", 2024-08-18T17:15:50.560 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:15:50.560 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:15:50.560 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:15:50.560 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:12:23.400665Z", 2024-08-18T17:15:50.561 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:15:50.561 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:15:50.561 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:15:50.561 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:15:50.561 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:15:50.561 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:15:50.561 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:15:50.561 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:15:50.561 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:15:50.561 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:15:50Z" 2024-08-18T17:15:50.561 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:15:50.563 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001268.323120_1508.1 target=1_1724001143.400665_1473.1 2024-08-18T17:15:55.564 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:15:55.564 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:15:55.564 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:15:55.619 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:15:55.620 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:15:55.654 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:15:55.653+0000 7f6bbc129880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:15:55.655 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:15:55.654+0000 7f6bbc129880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:15:55.669 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:15:55.670 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:15:55.670 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:15:55.670 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:15:55.670 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:15:55.670 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:15:55.670 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:15:55.670 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:15:55.670 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:15:55.670 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:15:55.670 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:15:55.670 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:15:55.671 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:15:55.671 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:15:55.671 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:15:55.671 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:15:55.671 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:15:55.671 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:15:55.671 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:15:55.671 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:15:55.671 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:15:55.671 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:15:55.671 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:15:55.672 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:15:55.672 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:15:55.672 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:15:55.672 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:15:55.672 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:15:55.672 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:15:55.672 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:15:55.673 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:15:55.673 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:15:55.673 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:15:55.673 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:15:55.673 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:15:55.673 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:15:55.673 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:15:55.673 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:15:55.673 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:15:55.673 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:15:55.673 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:15:55.673 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:15:55.673 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:15:55.674 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:15:55.674 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:15:55.674 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:15:55.674 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:15:55.674 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:15:55.674 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:15:55.674 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001143.400665_1473.1", 2024-08-18T17:15:55.674 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:15:55.674 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:15:55.674 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:15:55.674 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:12:23.400665Z", 2024-08-18T17:15:55.674 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:15:55.675 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:15:55.675 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:15:55.675 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:15:55.675 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:15:55.675 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:15:55.675 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:15:55.675 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:15:55.675 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:15:55.675 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:15:55Z" 2024-08-18T17:15:55.675 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:15:55.676 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001268.323120_1508.1 target=1_1724001143.400665_1473.1 2024-08-18T17:16:00.677 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:16:00.677 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:16:00.677 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:16:00.732 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:16:00.732 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:16:00.769 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:16:00.768+0000 7f31c4b24880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:16:00.769 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:16:00.769+0000 7f31c4b24880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:16:00.784 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:16:00.784 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:16:00.784 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:16:00.784 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:16:00.784 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:16:00.784 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:16:00.785 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:16:00.785 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:16:00.785 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:16:00.785 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:16:00.785 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:16:00.785 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:16:00.785 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:16:00.785 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:16:00.785 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:16:00.785 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:16:00.785 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:16:00.785 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:16:00.786 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:16:00.786 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:16:00.786 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:16:00.786 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:16:00.786 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:16:00.786 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:16:00.786 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:16:00.786 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:16:00.786 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:16:00.786 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:16:00.786 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:16:00.786 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:16:00.787 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:16:00.787 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:16:00.787 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:16:00.787 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:16:00.787 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:16:00.787 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:16:00.787 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:16:00.787 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:16:00.787 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:16:00.787 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:16:00.787 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:16:00.787 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:16:00.787 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:16:00.788 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:16:00.788 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:16:00.788 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:16:00.788 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:16:00.789 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:16:00.789 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:16:00.789 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001143.400665_1473.1", 2024-08-18T17:16:00.789 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:16:00.789 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:16:00.789 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:16:00.789 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:12:23.400665Z", 2024-08-18T17:16:00.789 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:16:00.790 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:16:00.790 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:16:00.790 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:16:00.790 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:16:00.790 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:16:00.790 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:16:00.790 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:16:00.790 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:16:00.790 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:16:00Z" 2024-08-18T17:16:00.790 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:16:00.791 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001268.323120_1508.1 target=1_1724001143.400665_1473.1 2024-08-18T17:16:05.792 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:16:05.792 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:16:05.793 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:16:05.848 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:16:05.848 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:16:05.881 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:16:05.881+0000 7f97e5630880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:16:05.882 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:16:05.882+0000 7f97e5630880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:16:05.897 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:16:05.897 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:16:05.898 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:16:05.898 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:16:05.898 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:16:05.898 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:16:05.898 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:16:05.898 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:16:05.898 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:16:05.898 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:16:05.898 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:16:05.898 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:16:05.898 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:16:05.898 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:16:05.899 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:16:05.899 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:16:05.899 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:16:05.899 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:16:05.899 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:16:05.899 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:16:05.899 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:16:05.899 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:16:05.899 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:16:05.899 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:16:05.899 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:16:05.899 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:16:05.900 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:16:05.900 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:16:05.900 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:16:05.900 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:16:05.900 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:16:05.900 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:16:05.900 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:16:05.900 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:16:05.900 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:16:05.900 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:16:05.900 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:16:05.900 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:16:05.900 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:16:05.900 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:16:05.901 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:16:05.901 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:16:05.901 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:16:05.901 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:16:05.901 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:16:05.901 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:16:05.901 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:16:05.901 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:16:05.901 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:16:05.901 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001143.400665_1473.1", 2024-08-18T17:16:05.901 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:16:05.901 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:16:05.901 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:16:05.901 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:12:23.400665Z", 2024-08-18T17:16:05.901 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:16:05.902 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:16:05.902 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:16:05.902 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:16:05.902 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:16:05.902 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:16:05.902 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:16:05.902 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:16:05.902 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:16:05.902 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:16:05Z" 2024-08-18T17:16:05.902 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:16:05.903 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001268.323120_1508.1 target=1_1724001143.400665_1473.1 2024-08-18T17:16:10.904 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:16:10.904 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:16:10.904 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:16:10.959 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:16:10.960 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:16:10.993 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:16:10.992+0000 7f9b65212880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:16:10.994 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:16:10.993+0000 7f9b65212880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:16:11.009 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:16:11.010 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:16:11.010 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:16:11.010 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:16:11.010 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:16:11.010 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:16:11.010 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:16:11.010 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:16:11.010 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:16:11.010 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:16:11.010 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:16:11.010 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:16:11.011 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:16:11.011 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:16:11.011 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:16:11.011 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:16:11.011 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:16:11.011 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:16:11.011 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:16:11.011 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:16:11.011 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:16:11.011 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:16:11.011 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:16:11.011 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:16:11.011 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:16:11.011 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:16:11.012 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:16:11.012 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:16:11.012 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:16:11.012 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:16:11.012 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:16:11.012 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:16:11.012 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:16:11.012 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:16:11.012 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:16:11.012 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:16:11.012 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:16:11.012 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:16:11.012 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:16:11.012 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:16:11.013 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:16:11.013 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:16:11.013 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:16:11.013 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:16:11.013 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:16:11.013 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:16:11.013 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:16:11.013 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:16:11.013 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:16:11.013 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001143.400665_1473.1", 2024-08-18T17:16:11.013 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:16:11.013 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:16:11.013 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:16:11.013 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:12:23.400665Z", 2024-08-18T17:16:11.014 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:16:11.014 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:16:11.014 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:16:11.014 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:16:11.014 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:16:11.014 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:16:11.014 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:16:11.014 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:16:11.014 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:16:11.014 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:16:10Z" 2024-08-18T17:16:11.014 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:16:11.015 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001268.323120_1508.1 target=1_1724001143.400665_1473.1 2024-08-18T17:16:16.016 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:16:16.016 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:16:16.017 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:16:16.072 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:16:16.072 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:16:16.106 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:16:16.105+0000 7f16a3a0e880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:16:16.106 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:16:16.106+0000 7f16a3a0e880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:16:16.122 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:16:16.122 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:16:16.122 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:16:16.123 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:16:16.123 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:16:16.123 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:16:16.123 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:16:16.123 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:16:16.123 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:16:16.123 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:16:16.123 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:16:16.123 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:16:16.123 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:16:16.123 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:16:16.123 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:16:16.123 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:16:16.124 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:16:16.124 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:16:16.124 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:16:16.124 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:16:16.124 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:16:16.124 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:16:16.124 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:16:16.124 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:16:16.124 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:16:16.124 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:16:16.124 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:16:16.124 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:16:16.124 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:16:16.124 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:16:16.125 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:16:16.125 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:16:16.125 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:16:16.125 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:16:16.125 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:16:16.125 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:16:16.125 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:16:16.125 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:16:16.125 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:16:16.125 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:16:16.125 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:16:16.125 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:16:16.125 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:16:16.125 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:16:16.126 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:16:16.126 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:16:16.126 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:16:16.126 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:16:16.126 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:16:16.126 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001143.400665_1473.1", 2024-08-18T17:16:16.126 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:16:16.126 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:16:16.126 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:16:16.126 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:12:23.400665Z", 2024-08-18T17:16:16.126 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:16:16.126 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:16:16.126 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:16:16.127 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:16:16.127 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:16:16.127 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:16:16.127 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:16:16.127 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:16:16.127 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:16:16.127 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:16:16Z" 2024-08-18T17:16:16.127 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:16:16.128 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001268.323120_1508.1 target=1_1724001143.400665_1473.1 2024-08-18T17:16:21.129 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:16:21.129 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:16:21.130 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:16:21.185 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:16:21.185 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:16:21.219 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:16:21.218+0000 7f97b5a26880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:16:21.220 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:16:21.219+0000 7f97b5a26880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:16:21.235 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:16:21.235 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:16:21.235 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:16:21.235 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:16:21.235 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:16:21.235 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:16:21.235 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:16:21.235 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:16:21.235 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:16:21.235 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:16:21.235 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:16:21.236 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:16:21.236 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:16:21.236 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:16:21.236 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:16:21.236 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:16:21.236 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:16:21.236 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:16:21.236 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:16:21.236 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:16:21.236 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:16:21.236 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:16:21.236 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:16:21.236 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:16:21.237 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:16:21.237 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:16:21.237 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:16:21.237 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:16:21.237 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:16:21.237 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:16:21.237 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:16:21.237 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:16:21.237 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:16:21.237 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:16:21.237 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:16:21.237 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:16:21.237 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:16:21.238 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:16:21.238 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:16:21.238 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:16:21.238 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:16:21.238 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:16:21.238 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:16:21.238 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:16:21.238 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:16:21.238 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:16:21.238 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:16:21.238 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:16:21.238 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:16:21.238 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001268.323120_1508.1", 2024-08-18T17:16:21.239 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:16:21.239 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:16:21.239 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:16:21.239 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:14:28.323120Z", 2024-08-18T17:16:21.239 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:16:21.239 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:16:21.239 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:16:21.239 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:16:21.239 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:16:21.239 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:16:21.239 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:16:21.239 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:16:21.239 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:16:21.240 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:16:21Z" 2024-08-18T17:16:21.240 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:16:21.240 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-08-18T17:16:21.241 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-08-18T17:16:21.241 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:16:21.241 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:16:21.241 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:16:21.298 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:16:21.298 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:16:21.360 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:16:21.359+0000 7f40c01af880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:16:21.361 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:16:21.360+0000 7f40c01af880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:16:21.383 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T17:16:21.383 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T17:16:21.383 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T17:16:21.383 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T17:16:21.383 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T17:16:21.384 INFO:teuthology.orchestra.run.smithi134.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:16:21.384 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:16:21.384 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:16:21.384 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T17:16:21.384 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:16:21.384 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T17:16:21.384 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:16:21.384 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:16:21.384 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:16:21.384 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:16:21.384 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 1, 2024-08-18T17:16:21.384 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:16:21.384 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:16:21.385 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T17:16:21.385 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:16:21.385 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:16:21.385 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:16:21.385 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T17:16:21.385 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:16:21.385 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:16:21.385 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:16:21.385 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:16:21.385 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:16:21.386 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:16:21.386 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T17:16:21.386 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T17:16:21.386 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:16:21.386 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:16:21.386 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:16:21.386 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T17:16:21.386 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:16:21.386 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:16:21.386 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:16:21.386 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:16:21.386 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:16:21.387 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:16:21.387 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:16:21.387 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:16:21.387 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:16:21.387 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:16:21.387 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:16:21.387 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T17:16:21.387 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:16:21.387 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:16:21.387 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1724001268.323120_1508.1", 2024-08-18T17:16:21.388 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:16:21.388 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:16:21.388 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:16:21.388 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T17:14:28.323120Z", 2024-08-18T17:16:21.388 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:16:21.388 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:16:21.388 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:16:21.388 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:16:21.388 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:16:21.388 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T17:16:21.388 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 1, 2024-08-18T17:16:21.389 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 1 2024-08-18T17:16:21.389 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:16:21.389 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T17:16:21Z" 2024-08-18T17:16:21.389 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T17:16:21.390 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-08-18T17:16:21.390 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:16:21.390 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:16:21.390 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:16:21.438 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:16:21.439 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:16:21.507 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T17:16:21.507 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:16:21.508 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000051314", 2024-08-18T17:16:21.508 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T17:14:28.223219Z" 2024-08-18T17:16:21.508 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:16:21.508 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:16:21.508 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000037274", 2024-08-18T17:16:21.508 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T17:05:35.911460Z" 2024-08-18T17:16:21.508 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:16:21.508 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:16:21.508 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000058437", 2024-08-18T17:16:21.508 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T17:14:28.260736Z" 2024-08-18T17:16:21.508 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:16:21.508 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:16:21.509 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000052068", 2024-08-18T17:16:21.509 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T17:05:27.099520Z" 2024-08-18T17:16:21.509 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:16:21.509 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T17:16:21.510 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-08-18T17:16:21.510 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:16:21.510 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:16:21.510 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:16:21.563 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:16:21.563 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:16:21.611 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:16:21.636 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:16:21.636 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:16:21.636 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:16:21.636 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:16:21.636 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:16:21.637 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:16:21.637 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:16:21.637 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:16:21.637 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:16:21.637 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:16:21.637 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:16:21.637 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000051033", 2024-08-18T17:16:21.637 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:16:21.637 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:16:21.637 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:16:21.637 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:05:27.155053Z" 2024-08-18T17:16:21.637 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:16:21.637 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:16:21.638 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:16:21.638 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:16:21.638 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:16:21.638 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:16:21.638 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037274", 2024-08-18T17:16:21.638 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:16:21.638 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:16:21.638 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:16:21.638 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:05:35.911460Z" 2024-08-18T17:16:21.638 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:16:21.638 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:16:21.638 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:16:21.639 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:16:21.639 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:16:21.639 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:16:21.639 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000058157", 2024-08-18T17:16:21.639 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:16:21.639 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:16:21.639 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:16:21.639 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:12:23.352239Z" 2024-08-18T17:16:21.639 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:16:21.639 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:16:21.639 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:16:21.639 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:16:21.639 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:16:21.640 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:16:21.640 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052068", 2024-08-18T17:16:21.640 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:16:21.640 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:16:21.640 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:16:21.640 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:05:27.099520Z" 2024-08-18T17:16:21.640 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:16:21.640 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:16:21.640 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:16:21.640 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:16:21.640 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:16:21.640 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:16:21.640 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:16:21.640 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:16:21.641 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:16:21Z" 2024-08-18T17:16:21.641 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:16:21.642 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000051314 target=00000000000000000000:00000000000000051033, shard=2 master=00000000000000000000:00000000000000058437 target=00000000000000000000:00000000000000058157 2024-08-18T17:16:26.642 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:16:26.643 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:16:26.643 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:16:26.703 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:16:26.703 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:16:26.753 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:16:26.753 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:16:26.754 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:16:26.754 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:16:26.754 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:16:26.754 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:16:26.754 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:16:26.754 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:16:26.754 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:16:26.755 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:16:26.755 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:16:26.755 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:16:26.755 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000051033", 2024-08-18T17:16:26.755 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:16:26.755 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:16:26.755 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:16:26.755 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:05:27.155053Z" 2024-08-18T17:16:26.756 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:16:26.756 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:16:26.756 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:16:26.756 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:16:26.756 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:16:26.756 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:16:26.756 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037274", 2024-08-18T17:16:26.757 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:16:26.757 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:16:26.757 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:16:26.757 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:05:35.911460Z" 2024-08-18T17:16:26.757 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:16:26.757 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:16:26.757 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:16:26.757 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:16:26.758 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:16:26.758 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:16:26.758 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000058157", 2024-08-18T17:16:26.758 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:16:26.758 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:16:26.758 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:16:26.758 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:12:23.352239Z" 2024-08-18T17:16:26.758 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:16:26.758 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:16:26.758 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:16:26.759 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:16:26.759 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:16:26.759 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:16:26.759 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052068", 2024-08-18T17:16:26.759 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:16:26.759 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:16:26.759 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:16:26.759 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:05:27.099520Z" 2024-08-18T17:16:26.759 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:16:26.759 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:16:26.759 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:16:26.759 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:16:26.760 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:16:26.760 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:16:26.760 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:16:26.760 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:16:26.760 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:16:26Z" 2024-08-18T17:16:26.760 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:16:26.762 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000051314 target=00000000000000000000:00000000000000051033, shard=2 master=00000000000000000000:00000000000000058437 target=00000000000000000000:00000000000000058157 2024-08-18T17:16:31.763 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:16:31.763 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:16:31.764 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:16:31.820 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:16:31.820 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:16:31.870 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:16:31.870 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:16:31.870 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:16:31.870 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:16:31.871 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:16:31.871 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:16:31.871 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:16:31.871 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:16:31.871 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:16:31.871 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:16:31.871 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:16:31.871 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:16:31.871 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000051314", 2024-08-18T17:16:31.871 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:16:31.871 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:16:31.871 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:16:31.871 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:14:28.223219Z" 2024-08-18T17:16:31.872 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:16:31.872 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:16:31.872 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:16:31.872 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:16:31.872 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:16:31.872 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:16:31.872 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037274", 2024-08-18T17:16:31.872 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:16:31.872 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:16:31.872 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:16:31.872 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:05:35.911460Z" 2024-08-18T17:16:31.872 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:16:31.872 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:16:31.872 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:16:31.872 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:16:31.873 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:16:31.873 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:16:31.873 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000058437", 2024-08-18T17:16:31.873 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:16:31.873 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:16:31.873 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:16:31.873 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:14:28.260736Z" 2024-08-18T17:16:31.873 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:16:31.873 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:16:31.873 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:16:31.873 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:16:31.873 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:16:31.873 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:16:31.874 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052068", 2024-08-18T17:16:31.874 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:16:31.874 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:16:31.874 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:16:31.874 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:05:27.099520Z" 2024-08-18T17:16:31.874 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:16:31.874 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:16:31.874 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:16:31.874 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:16:31.874 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:16:31.874 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:16:31.874 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:16:31.874 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:16:31.874 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:16:31Z" 2024-08-18T17:16:31.875 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:16:31.875 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-08-18T17:16:31.876 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:16:31.876 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:16:31.876 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:16:31.924 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:16:31.924 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:16:31.975 INFO:teuthology.orchestra.run.smithi193.stdout:[ 2024-08-18T17:16:31.975 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:16:31.975 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000018040", 2024-08-18T17:16:31.975 INFO:teuthology.orchestra.run.smithi193.stdout: "last_update": "2024-08-18T17:16:29.306213Z" 2024-08-18T17:16:31.975 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:16:31.975 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:16:31.975 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000011786", 2024-08-18T17:16:31.976 INFO:teuthology.orchestra.run.smithi193.stdout: "last_update": "2024-08-18T17:14:28.358891Z" 2024-08-18T17:16:31.976 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:16:31.976 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:16:31.976 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000022916", 2024-08-18T17:16:31.976 INFO:teuthology.orchestra.run.smithi193.stdout: "last_update": "2024-08-18T17:16:29.307733Z" 2024-08-18T17:16:31.976 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:16:31.976 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:16:31.976 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000023344", 2024-08-18T17:16:31.976 INFO:teuthology.orchestra.run.smithi193.stdout: "last_update": "2024-08-18T17:06:33.445168Z" 2024-08-18T17:16:31.976 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:16:31.976 INFO:teuthology.orchestra.run.smithi193.stdout:] 2024-08-18T17:16:31.977 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2024-08-18T17:16:31.977 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:16:31.978 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:16:31.978 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:16:32.034 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:16:32.035 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:16:32.091 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T17:16:32.092 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T17:16:32.092 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T17:16:32.092 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T17:16:32.092 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T17:16:32.092 INFO:teuthology.orchestra.run.smithi134.stdout: "instance_id": 15510818030922206205 2024-08-18T17:16:32.092 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:16:32.092 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T17:16:32.092 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:16:32.092 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T17:16:32.092 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:16:32.092 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T17:16:32.093 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000017900", 2024-08-18T17:16:32.093 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:16:32.093 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:16:32.093 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:16:32.093 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T17:14:28.416680Z" 2024-08-18T17:16:32.093 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:16:32.093 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:16:32.093 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:16:32.093 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T17:16:32.093 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:16:32.093 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T17:16:32.093 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000011786", 2024-08-18T17:16:32.094 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:16:32.094 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:16:32.094 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:16:32.094 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T17:14:28.358891Z" 2024-08-18T17:16:32.094 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:16:32.094 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:16:32.094 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:16:32.094 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T17:16:32.094 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:16:32.094 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T17:16:32.094 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000022636", 2024-08-18T17:16:32.095 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:16:32.095 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:16:32.095 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:16:32.095 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T17:14:28.370324Z" 2024-08-18T17:16:32.095 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:16:32.095 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:16:32.095 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:16:32.095 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T17:16:32.095 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:16:32.095 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T17:16:32.095 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000023344", 2024-08-18T17:16:32.095 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:16:32.095 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:16:32.096 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:16:32.096 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T17:06:33.445168Z" 2024-08-18T17:16:32.096 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:16:32.096 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:16:32.096 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:16:32.096 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:16:32.096 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T17:16:32.096 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 0, 2024-08-18T17:16:32.096 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 0 2024-08-18T17:16:32.096 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:16:32.096 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T17:16:32Z" 2024-08-18T17:16:32.096 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T17:16:32.098 WARNING:rgw_multi.tests:data of zone test-zone1 behind zone test-zone2: shard=0 master=00000000000000000000:00000000000000018040 target=00000000000000000000:00000000000000017900, shard=2 master=00000000000000000000:00000000000000022916 target=00000000000000000000:00000000000000022636 2024-08-18T17:16:37.099 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:16:37.099 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:16:37.100 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:16:37.156 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:16:37.156 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:16:37.240 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T17:16:37.240 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T17:16:37.240 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T17:16:37.241 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T17:16:37.241 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T17:16:37.241 INFO:teuthology.orchestra.run.smithi134.stdout: "instance_id": 15510818030922206205 2024-08-18T17:16:37.241 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:16:37.241 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T17:16:37.241 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:16:37.241 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T17:16:37.241 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:16:37.241 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T17:16:37.241 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000017900", 2024-08-18T17:16:37.241 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:16:37.241 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:16:37.241 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:16:37.242 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T17:14:28.416680Z" 2024-08-18T17:16:37.242 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:16:37.242 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:16:37.242 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:16:37.242 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T17:16:37.242 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:16:37.242 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T17:16:37.242 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000011786", 2024-08-18T17:16:37.242 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:16:37.242 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:16:37.242 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:16:37.242 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T17:14:28.358891Z" 2024-08-18T17:16:37.242 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:16:37.243 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:16:37.243 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:16:37.243 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T17:16:37.243 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:16:37.243 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T17:16:37.243 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000022636", 2024-08-18T17:16:37.243 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:16:37.244 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:16:37.244 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:16:37.244 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T17:14:28.370324Z" 2024-08-18T17:16:37.244 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:16:37.244 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:16:37.244 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:16:37.244 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T17:16:37.244 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:16:37.244 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T17:16:37.244 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000023344", 2024-08-18T17:16:37.244 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:16:37.244 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:16:37.244 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:16:37.245 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T17:06:33.445168Z" 2024-08-18T17:16:37.245 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:16:37.245 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:16:37.245 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:16:37.245 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:16:37.245 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T17:16:37.245 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 0, 2024-08-18T17:16:37.245 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 0 2024-08-18T17:16:37.245 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:16:37.245 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T17:16:37Z" 2024-08-18T17:16:37.245 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T17:16:37.246 WARNING:rgw_multi.tests:data of zone test-zone1 behind zone test-zone2: shard=0 master=00000000000000000000:00000000000000018040 target=00000000000000000000:00000000000000017900, shard=2 master=00000000000000000000:00000000000000022916 target=00000000000000000000:00000000000000022636 2024-08-18T17:16:42.247 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:16:42.248 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:16:42.248 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:16:42.305 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:16:42.305 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:16:42.371 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T17:16:42.371 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T17:16:42.371 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T17:16:42.371 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T17:16:42.371 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T17:16:42.371 INFO:teuthology.orchestra.run.smithi134.stdout: "instance_id": 15510818030922206205 2024-08-18T17:16:42.371 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:16:42.371 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T17:16:42.372 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:16:42.372 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T17:16:42.372 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:16:42.372 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T17:16:42.372 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000017900", 2024-08-18T17:16:42.372 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:16:42.372 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:16:42.372 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:16:42.372 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T17:14:28.416680Z" 2024-08-18T17:16:42.372 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:16:42.372 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:16:42.372 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:16:42.373 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T17:16:42.373 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:16:42.373 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T17:16:42.373 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000011786", 2024-08-18T17:16:42.373 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:16:42.373 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:16:42.373 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:16:42.373 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T17:14:28.358891Z" 2024-08-18T17:16:42.373 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:16:42.373 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:16:42.373 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:16:42.373 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T17:16:42.374 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:16:42.374 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T17:16:42.374 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000022636", 2024-08-18T17:16:42.374 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:16:42.374 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:16:42.374 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:16:42.374 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T17:14:28.370324Z" 2024-08-18T17:16:42.374 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:16:42.374 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:16:42.374 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:16:42.374 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T17:16:42.374 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:16:42.375 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T17:16:42.375 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000023344", 2024-08-18T17:16:42.375 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:16:42.375 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:16:42.375 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:16:42.375 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T17:06:33.445168Z" 2024-08-18T17:16:42.375 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:16:42.375 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:16:42.375 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:16:42.375 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:16:42.375 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T17:16:42.375 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 0, 2024-08-18T17:16:42.375 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 0 2024-08-18T17:16:42.376 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:16:42.376 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T17:16:42Z" 2024-08-18T17:16:42.376 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T17:16:42.377 WARNING:rgw_multi.tests:data of zone test-zone1 behind zone test-zone2: shard=0 master=00000000000000000000:00000000000000018040 target=00000000000000000000:00000000000000017900, shard=2 master=00000000000000000000:00000000000000022916 target=00000000000000000000:00000000000000022636 2024-08-18T17:16:47.378 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:16:47.379 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:16:47.379 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:16:47.436 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:16:47.436 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:16:47.506 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T17:16:47.507 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T17:16:47.507 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T17:16:47.507 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T17:16:47.507 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T17:16:47.507 INFO:teuthology.orchestra.run.smithi134.stdout: "instance_id": 15510818030922206205 2024-08-18T17:16:47.507 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:16:47.507 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T17:16:47.507 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:16:47.507 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T17:16:47.507 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:16:47.507 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T17:16:47.508 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000017900", 2024-08-18T17:16:47.508 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:16:47.508 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:16:47.508 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:16:47.508 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T17:14:28.416680Z" 2024-08-18T17:16:47.508 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:16:47.508 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:16:47.508 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:16:47.508 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T17:16:47.508 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:16:47.508 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T17:16:47.508 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000011786", 2024-08-18T17:16:47.508 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:16:47.509 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:16:47.509 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:16:47.509 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T17:14:28.358891Z" 2024-08-18T17:16:47.509 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:16:47.509 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:16:47.509 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:16:47.509 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T17:16:47.509 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:16:47.509 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T17:16:47.509 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000022636", 2024-08-18T17:16:47.509 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:16:47.509 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:16:47.510 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:16:47.510 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T17:14:28.370324Z" 2024-08-18T17:16:47.510 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:16:47.510 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:16:47.510 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:16:47.510 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T17:16:47.510 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:16:47.510 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T17:16:47.510 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000023344", 2024-08-18T17:16:47.510 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:16:47.510 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:16:47.510 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:16:47.510 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T17:06:33.445168Z" 2024-08-18T17:16:47.511 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:16:47.511 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:16:47.511 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:16:47.511 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:16:47.511 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T17:16:47.511 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 0, 2024-08-18T17:16:47.511 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 0 2024-08-18T17:16:47.511 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:16:47.511 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T17:16:47Z" 2024-08-18T17:16:47.511 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T17:16:47.512 WARNING:rgw_multi.tests:data of zone test-zone1 behind zone test-zone2: shard=0 master=00000000000000000000:00000000000000018040 target=00000000000000000000:00000000000000017900, shard=2 master=00000000000000000000:00000000000000022916 target=00000000000000000000:00000000000000022636 2024-08-18T17:16:52.513 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:16:52.513 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:16:52.514 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:16:52.571 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:16:52.571 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:16:52.644 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T17:16:52.644 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T17:16:52.644 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T17:16:52.645 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T17:16:52.645 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T17:16:52.645 INFO:teuthology.orchestra.run.smithi134.stdout: "instance_id": 15510818030922206205 2024-08-18T17:16:52.645 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:16:52.645 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T17:16:52.645 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:16:52.645 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T17:16:52.645 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:16:52.645 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T17:16:52.645 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000018040", 2024-08-18T17:16:52.645 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:16:52.645 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:16:52.646 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:16:52.646 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T17:16:29.306213Z" 2024-08-18T17:16:52.646 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:16:52.646 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:16:52.646 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:16:52.646 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T17:16:52.646 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:16:52.646 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T17:16:52.646 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000011786", 2024-08-18T17:16:52.646 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:16:52.646 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:16:52.646 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:16:52.646 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T17:14:28.358891Z" 2024-08-18T17:16:52.647 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:16:52.647 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:16:52.647 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:16:52.647 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T17:16:52.647 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:16:52.647 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T17:16:52.647 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000022916", 2024-08-18T17:16:52.647 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:16:52.647 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:16:52.647 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:16:52.647 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T17:16:29.307733Z" 2024-08-18T17:16:52.647 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:16:52.648 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:16:52.648 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:16:52.648 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T17:16:52.648 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:16:52.648 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "incremental-sync", 2024-08-18T17:16:52.648 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000023344", 2024-08-18T17:16:52.648 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:16:52.648 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:16:52.648 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:16:52.648 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T17:06:33.445168Z" 2024-08-18T17:16:52.648 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:16:52.648 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:16:52.648 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:16:52.649 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:16:52.649 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T17:16:52.649 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 0, 2024-08-18T17:16:52.649 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 0 2024-08-18T17:16:52.649 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:16:52.649 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T17:16:52Z" 2024-08-18T17:16:52.649 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T17:16:52.650 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2024-08-18T17:16:52.742 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:16:52.742 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:16:52.743 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:16:52.789 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:16:52.789 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:16:52.839 INFO:teuthology.orchestra.run.smithi134.stdout:{"sync_policy":{"groups":[]}} 2024-08-18T17:16:52.839 INFO:tasks.rgw_multisite_tests:test_sync_flow_symmetrical_zonegroup_select: ... ok 2024-08-18T17:16:52.840 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:16:52.840 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:16:52.840 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:16:52.886 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:16:52.886 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:16:52.922 INFO:teuthology.orchestra.run.smithi134.stderr:No --period given, using current period=01050e30-179b-43be-bdbb-3bb23cf37ae8 2024-08-18T17:16:52.942 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T17:16:52.943 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:16:52.943 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:16:52.943 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T17:16:52.943 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:16:52.943 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:16:52.943 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:16:52.944 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T17:16:52.944 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:16:52.944 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:16:52.944 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:16:52.944 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:42:47.672461Z" 2024-08-18T17:16:52.944 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:16:52.945 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:16:52.945 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1724001268.323120_1508.1", 2024-08-18T17:16:52.945 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T17:14:28.323120Z" 2024-08-18T17:16:52.945 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:16:52.945 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T17:16:52.947 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-08-18T17:16:52.948 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:16:52.948 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:16:52.948 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:16:53.003 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:16:53.003 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:16:53.036 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:16:53.036+0000 7f0940e0f880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:16:53.037 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:16:53.037+0000 7f0940e0f880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:16:53.052 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:16:53.052 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:16:53.052 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:16:53.052 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:16:53.052 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:16:53.052 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:16:53.053 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:16:53.053 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:16:53.053 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:16:53.053 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:16:53.053 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:16:53.053 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:16:53.053 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:16:53.053 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:16:53.053 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:16:53.053 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:16:53.053 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:16:53.053 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:16:53.053 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:16:53.054 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:16:53.054 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:16:53.054 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:16:53.054 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:16:53.054 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:16:53.054 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:16:53.054 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:16:53.054 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:16:53.054 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:16:53.054 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:16:53.054 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:16:53.054 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:16:53.055 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:16:53.055 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:16:53.055 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:16:53.055 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:16:53.055 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:16:53.055 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:16:53.055 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:16:53.055 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:16:53.055 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:16:53.055 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:16:53.055 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:16:53.055 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:16:53.055 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:16:53.056 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:16:53.056 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:16:53.056 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:16:53.056 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:16:53.056 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:16:53.056 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001268.323120_1508.1", 2024-08-18T17:16:53.056 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:16:53.056 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:16:53.056 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:16:53.057 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:14:28.323120Z", 2024-08-18T17:16:53.057 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:16:53.057 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:16:53.057 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:16:53.057 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:16:53.057 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:16:53.057 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:16:53.057 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:16:53.057 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:16:53.057 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:16:53.058 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:16:53Z" 2024-08-18T17:16:53.058 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:16:53.059 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-08-18T17:16:53.059 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-08-18T17:16:53.059 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:16:53.059 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:16:53.059 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:16:53.107 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:16:53.107 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:16:53.161 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:16:53.160+0000 7f51d0142880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:16:53.161 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:16:53.161+0000 7f51d0142880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:16:53.186 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T17:16:53.186 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T17:16:53.186 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T17:16:53.186 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T17:16:53.186 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T17:16:53.187 INFO:teuthology.orchestra.run.smithi134.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:16:53.187 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:16:53.187 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:16:53.187 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T17:16:53.187 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:16:53.187 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T17:16:53.187 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:16:53.187 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:16:53.187 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:16:53.187 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:16:53.187 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 1, 2024-08-18T17:16:53.187 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:16:53.187 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:16:53.187 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T17:16:53.188 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:16:53.188 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:16:53.188 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:16:53.188 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T17:16:53.188 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:16:53.188 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:16:53.188 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:16:53.188 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:16:53.188 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:16:53.188 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:16:53.188 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T17:16:53.188 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T17:16:53.188 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:16:53.189 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:16:53.189 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:16:53.189 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T17:16:53.189 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:16:53.189 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:16:53.189 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:16:53.189 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:16:53.189 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:16:53.189 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:16:53.189 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:16:53.189 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:16:53.189 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:16:53.189 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:16:53.190 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:16:53.190 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T17:16:53.190 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:16:53.190 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:16:53.190 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1724001268.323120_1508.1", 2024-08-18T17:16:53.190 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:16:53.190 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:16:53.190 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:16:53.190 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T17:14:28.323120Z", 2024-08-18T17:16:53.190 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:16:53.190 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:16:53.190 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:16:53.190 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:16:53.191 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:16:53.191 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T17:16:53.191 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 1, 2024-08-18T17:16:53.191 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 1 2024-08-18T17:16:53.191 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:16:53.191 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T17:16:53Z" 2024-08-18T17:16:53.191 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T17:16:53.192 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-08-18T17:16:53.192 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:16:53.192 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:16:53.192 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:16:53.236 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:16:53.236 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:16:53.293 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[]} 2024-08-18T17:16:53.293 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:16:53.293 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:16:53.293 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:16:53.382 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:16:53.382 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:16:53.454 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2024-08-18T17:16:53.455 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'directional', '--source-zone=test-zone1', '--dest-zone=test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:16:53.455 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'directional', '--source-zone=test-zone1', '--dest-zone=test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:16:53.456 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow --flow-type directional --source-zone=test-zone1 --dest-zone=test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:16:53.507 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:16:53.508 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:16:53.576 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[],"status":"allowed"}]} 2024-08-18T17:16:53.576 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:16:53.576 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:16:53.576 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=test-zone1 --dest-zones=test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:16:53.625 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:16:53.626 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:16:53.684 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-08-18T17:16:53.684 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:16:53.684 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:16:53.684 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:16:53.731 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:16:53.732 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:16:53.788 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-08-18T17:16:53.788 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:16:53.788 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:16:53.788 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:16:53.836 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:16:53.836 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:16:53.904 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T17:16:53.904 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:16:53.904 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 14, 2024-08-18T17:16:53.904 INFO:teuthology.orchestra.run.smithi134.stdout: "predecessor_uuid": "64309209-7567-46e5-8881-65783d84cf3c", 2024-08-18T17:16:53.904 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": [ 2024-08-18T17:16:53.904 INFO:teuthology.orchestra.run.smithi134.stdout: "", 2024-08-18T17:16:53.904 INFO:teuthology.orchestra.run.smithi134.stdout: "", 2024-08-18T17:16:53.904 INFO:teuthology.orchestra.run.smithi134.stdout: "", 2024-08-18T17:16:53.904 INFO:teuthology.orchestra.run.smithi134.stdout: "1_1723995848.930494_187.1" 2024-08-18T17:16:53.904 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:16:53.905 INFO:teuthology.orchestra.run.smithi134.stdout: "period_map": { 2024-08-18T17:16:53.905 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:16:53.905 INFO:teuthology.orchestra.run.smithi134.stdout: "zonegroups": [ 2024-08-18T17:16:53.905 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:16:53.905 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "test-zonegroup", 2024-08-18T17:16:53.905 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "test-zonegroup", 2024-08-18T17:16:53.905 INFO:teuthology.orchestra.run.smithi134.stdout: "api_name": "", 2024-08-18T17:16:53.905 INFO:teuthology.orchestra.run.smithi134.stdout: "is_master": true, 2024-08-18T17:16:53.905 INFO:teuthology.orchestra.run.smithi134.stdout: "endpoints": [ 2024-08-18T17:16:53.905 INFO:teuthology.orchestra.run.smithi134.stdout: "http://smithi134.front.sepia.ceph.com:8000" 2024-08-18T17:16:53.905 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:16:53.905 INFO:teuthology.orchestra.run.smithi134.stdout: "hostnames": [], 2024-08-18T17:16:53.905 INFO:teuthology.orchestra.run.smithi134.stdout: "hostnames_s3website": [], 2024-08-18T17:16:53.906 INFO:teuthology.orchestra.run.smithi134.stdout: "master_zone": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T17:16:53.906 INFO:teuthology.orchestra.run.smithi134.stdout: "zones": [ 2024-08-18T17:16:53.906 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:16:53.906 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T17:16:53.906 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "test-zone3", 2024-08-18T17:16:53.906 INFO:teuthology.orchestra.run.smithi134.stdout: "endpoints": [ 2024-08-18T17:16:53.906 INFO:teuthology.orchestra.run.smithi134.stdout: "http://smithi134.front.sepia.ceph.com:8001" 2024-08-18T17:16:53.906 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:16:53.906 INFO:teuthology.orchestra.run.smithi134.stdout: "log_meta": false, 2024-08-18T17:16:53.906 INFO:teuthology.orchestra.run.smithi134.stdout: "log_data": true, 2024-08-18T17:16:53.906 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_index_max_shards": 11, 2024-08-18T17:16:53.906 INFO:teuthology.orchestra.run.smithi134.stdout: "read_only": false, 2024-08-18T17:16:53.907 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_type": "", 2024-08-18T17:16:53.907 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from_all": true, 2024-08-18T17:16:53.907 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from": [], 2024-08-18T17:16:53.907 INFO:teuthology.orchestra.run.smithi134.stdout: "redirect_zone": "", 2024-08-18T17:16:53.907 INFO:teuthology.orchestra.run.smithi134.stdout: "supported_features": [ 2024-08-18T17:16:53.907 INFO:teuthology.orchestra.run.smithi134.stdout: "compress-encrypted", 2024-08-18T17:16:53.907 INFO:teuthology.orchestra.run.smithi134.stdout: "resharding" 2024-08-18T17:16:53.907 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:16:53.907 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:16:53.908 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:16:53.908 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "10090ccf-aad6-416d-9775-df8b7b27345a", 2024-08-18T17:16:53.908 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "test-zone2", 2024-08-18T17:16:53.908 INFO:teuthology.orchestra.run.smithi134.stdout: "endpoints": [ 2024-08-18T17:16:53.908 INFO:teuthology.orchestra.run.smithi134.stdout: "http://smithi193.front.sepia.ceph.com:8000" 2024-08-18T17:16:53.908 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:16:53.908 INFO:teuthology.orchestra.run.smithi134.stdout: "log_meta": false, 2024-08-18T17:16:53.908 INFO:teuthology.orchestra.run.smithi134.stdout: "log_data": true, 2024-08-18T17:16:53.908 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_index_max_shards": 11, 2024-08-18T17:16:53.908 INFO:teuthology.orchestra.run.smithi134.stdout: "read_only": false, 2024-08-18T17:16:53.908 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_type": "", 2024-08-18T17:16:53.909 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from_all": true, 2024-08-18T17:16:53.909 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from": [], 2024-08-18T17:16:53.909 INFO:teuthology.orchestra.run.smithi134.stdout: "redirect_zone": "", 2024-08-18T17:16:53.909 INFO:teuthology.orchestra.run.smithi134.stdout: "supported_features": [ 2024-08-18T17:16:53.909 INFO:teuthology.orchestra.run.smithi134.stdout: "compress-encrypted", 2024-08-18T17:16:53.909 INFO:teuthology.orchestra.run.smithi134.stdout: "resharding" 2024-08-18T17:16:53.909 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:16:53.909 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:16:53.909 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:16:53.910 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T17:16:53.910 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "test-zone1", 2024-08-18T17:16:53.910 INFO:teuthology.orchestra.run.smithi134.stdout: "endpoints": [ 2024-08-18T17:16:53.910 INFO:teuthology.orchestra.run.smithi134.stdout: "http://smithi134.front.sepia.ceph.com:8000" 2024-08-18T17:16:53.910 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:16:53.910 INFO:teuthology.orchestra.run.smithi134.stdout: "log_meta": false, 2024-08-18T17:16:53.910 INFO:teuthology.orchestra.run.smithi134.stdout: "log_data": true, 2024-08-18T17:16:53.910 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_index_max_shards": 11, 2024-08-18T17:16:53.910 INFO:teuthology.orchestra.run.smithi134.stdout: "read_only": false, 2024-08-18T17:16:53.912 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_type": "", 2024-08-18T17:16:53.912 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from_all": true, 2024-08-18T17:16:53.912 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from": [], 2024-08-18T17:16:53.912 INFO:teuthology.orchestra.run.smithi134.stdout: "redirect_zone": "", 2024-08-18T17:16:53.912 INFO:teuthology.orchestra.run.smithi134.stdout: "supported_features": [ 2024-08-18T17:16:53.912 INFO:teuthology.orchestra.run.smithi134.stdout: "compress-encrypted", 2024-08-18T17:16:53.912 INFO:teuthology.orchestra.run.smithi134.stdout: "resharding" 2024-08-18T17:16:53.912 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:16:53.912 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:16:53.912 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:16:53.913 INFO:teuthology.orchestra.run.smithi134.stdout: "placement_targets": [ 2024-08-18T17:16:53.913 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:16:53.913 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "default-placement", 2024-08-18T17:16:53.913 INFO:teuthology.orchestra.run.smithi134.stdout: "tags": [], 2024-08-18T17:16:53.913 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_classes": [ 2024-08-18T17:16:53.913 INFO:teuthology.orchestra.run.smithi134.stdout: "STANDARD" 2024-08-18T17:16:53.913 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:16:53.913 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:16:53.913 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:16:53.913 INFO:teuthology.orchestra.run.smithi134.stdout: "default_placement": "default-placement", 2024-08-18T17:16:53.913 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_id": "2827c6f0-bf0d-4836-944c-9e7fcf22457b", 2024-08-18T17:16:53.913 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_policy": { 2024-08-18T17:16:53.913 INFO:teuthology.orchestra.run.smithi134.stdout: "groups": [ 2024-08-18T17:16:53.914 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:16:53.914 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "sync-group", 2024-08-18T17:16:53.914 INFO:teuthology.orchestra.run.smithi134.stdout: "data_flow": { 2024-08-18T17:16:53.914 INFO:teuthology.orchestra.run.smithi134.stdout: "directional": [ 2024-08-18T17:16:53.914 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:16:53.914 INFO:teuthology.orchestra.run.smithi134.stdout: "source_zone": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T17:16:53.920 INFO:teuthology.orchestra.run.smithi134.stdout: "dest_zone": "10090ccf-aad6-416d-9775-df8b7b27345a" 2024-08-18T17:16:53.920 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:16:53.920 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:16:53.920 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:16:53.921 INFO:teuthology.orchestra.run.smithi134.stdout: "pipes": [ 2024-08-18T17:16:53.921 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:16:53.921 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "sync-pipe", 2024-08-18T17:16:53.921 INFO:teuthology.orchestra.run.smithi134.stdout: "source": { 2024-08-18T17:16:53.921 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket": "*", 2024-08-18T17:16:53.921 INFO:teuthology.orchestra.run.smithi134.stdout: "zones": [ 2024-08-18T17:16:53.921 INFO:teuthology.orchestra.run.smithi134.stdout: "96bf8502-2f39-4d53-8196-3378c28da3b4" 2024-08-18T17:16:53.921 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:16:53.921 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:16:53.921 INFO:teuthology.orchestra.run.smithi134.stdout: "dest": { 2024-08-18T17:16:53.921 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket": "*", 2024-08-18T17:16:53.921 INFO:teuthology.orchestra.run.smithi134.stdout: "zones": [ 2024-08-18T17:16:53.921 INFO:teuthology.orchestra.run.smithi134.stdout: "10090ccf-aad6-416d-9775-df8b7b27345a" 2024-08-18T17:16:53.921 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:16:53.922 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:16:53.922 INFO:teuthology.orchestra.run.smithi134.stdout: "params": { 2024-08-18T17:16:53.922 INFO:teuthology.orchestra.run.smithi134.stdout: "source": { 2024-08-18T17:16:53.922 INFO:teuthology.orchestra.run.smithi134.stdout: "filter": { 2024-08-18T17:16:53.922 INFO:teuthology.orchestra.run.smithi134.stdout: "tags": [] 2024-08-18T17:16:53.922 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:16:53.922 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:16:53.922 INFO:teuthology.orchestra.run.smithi134.stdout: "dest": {}, 2024-08-18T17:16:53.922 INFO:teuthology.orchestra.run.smithi134.stdout: "priority": 0, 2024-08-18T17:16:53.922 INFO:teuthology.orchestra.run.smithi134.stdout: "mode": "system", 2024-08-18T17:16:53.922 INFO:teuthology.orchestra.run.smithi134.stdout: "user": "" 2024-08-18T17:16:53.923 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:16:53.923 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:16:53.923 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:16:53.923 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "enabled" 2024-08-18T17:16:53.923 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:16:53.923 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:16:53.923 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:16:53.923 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled_features": [ 2024-08-18T17:16:53.923 INFO:teuthology.orchestra.run.smithi134.stdout: "resharding" 2024-08-18T17:16:53.923 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:16:53.923 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:16:53.923 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:16:53.923 INFO:teuthology.orchestra.run.smithi134.stdout: "short_zone_ids": [ 2024-08-18T17:16:53.924 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:16:53.924 INFO:teuthology.orchestra.run.smithi134.stdout: "key": "018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T17:16:53.924 INFO:teuthology.orchestra.run.smithi134.stdout: "val": 462036791 2024-08-18T17:16:53.924 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:16:53.924 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:16:53.924 INFO:teuthology.orchestra.run.smithi134.stdout: "key": "10090ccf-aad6-416d-9775-df8b7b27345a", 2024-08-18T17:16:53.924 INFO:teuthology.orchestra.run.smithi134.stdout: "val": 874289624 2024-08-18T17:16:53.924 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:16:53.924 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:16:53.924 INFO:teuthology.orchestra.run.smithi134.stdout: "key": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T17:16:53.924 INFO:teuthology.orchestra.run.smithi134.stdout: "val": 1721399886 2024-08-18T17:16:53.924 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:16:53.924 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:16:53.925 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:16:53.925 INFO:teuthology.orchestra.run.smithi134.stdout: "master_zonegroup": "test-zonegroup", 2024-08-18T17:16:53.925 INFO:teuthology.orchestra.run.smithi134.stdout: "master_zone": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T17:16:53.925 INFO:teuthology.orchestra.run.smithi134.stdout: "period_config": { 2024-08-18T17:16:53.925 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_quota": { 2024-08-18T17:16:53.925 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false, 2024-08-18T17:16:53.925 INFO:teuthology.orchestra.run.smithi134.stdout: "check_on_raw": false, 2024-08-18T17:16:53.925 INFO:teuthology.orchestra.run.smithi134.stdout: "max_size": -1, 2024-08-18T17:16:53.925 INFO:teuthology.orchestra.run.smithi134.stdout: "max_size_kb": 0, 2024-08-18T17:16:53.925 INFO:teuthology.orchestra.run.smithi134.stdout: "max_objects": -1 2024-08-18T17:16:53.925 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:16:53.925 INFO:teuthology.orchestra.run.smithi134.stdout: "user_quota": { 2024-08-18T17:16:53.925 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false, 2024-08-18T17:16:53.926 INFO:teuthology.orchestra.run.smithi134.stdout: "check_on_raw": false, 2024-08-18T17:16:53.926 INFO:teuthology.orchestra.run.smithi134.stdout: "max_size": -1, 2024-08-18T17:16:53.926 INFO:teuthology.orchestra.run.smithi134.stdout: "max_size_kb": 0, 2024-08-18T17:16:53.926 INFO:teuthology.orchestra.run.smithi134.stdout: "max_objects": -1 2024-08-18T17:16:53.926 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:16:53.926 INFO:teuthology.orchestra.run.smithi134.stdout: "user_ratelimit": { 2024-08-18T17:16:53.926 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_ops": 0, 2024-08-18T17:16:53.926 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_ops": 0, 2024-08-18T17:16:53.926 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_bytes": 0, 2024-08-18T17:16:53.926 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_bytes": 0, 2024-08-18T17:16:53.926 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false 2024-08-18T17:16:53.926 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:16:53.926 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_ratelimit": { 2024-08-18T17:16:53.927 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_ops": 0, 2024-08-18T17:16:53.927 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_ops": 0, 2024-08-18T17:16:53.927 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_bytes": 0, 2024-08-18T17:16:53.927 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_bytes": 0, 2024-08-18T17:16:53.927 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false 2024-08-18T17:16:53.927 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:16:53.927 INFO:teuthology.orchestra.run.smithi134.stdout: "anonymous_ratelimit": { 2024-08-18T17:16:53.927 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_ops": 0, 2024-08-18T17:16:53.927 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_ops": 0, 2024-08-18T17:16:53.927 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_bytes": 0, 2024-08-18T17:16:53.927 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_bytes": 0, 2024-08-18T17:16:53.927 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false 2024-08-18T17:16:53.927 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:16:53.927 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:16:53.928 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_id": "2827c6f0-bf0d-4836-944c-9e7fcf22457b", 2024-08-18T17:16:53.928 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_name": "test-realm", 2024-08-18T17:16:53.928 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:16:53.928 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T17:16:53.929 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:16:53.929 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:16:53.929 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:16:53.974 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:16:53.974 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:16:54.083 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-08-18T17:16:54.083 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=kibgmb-89 2024-08-18T17:16:54.291 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=kibgmb-90 2024-08-18T17:16:54.432 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:16:54.432 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:16:54.432 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:16:54.491 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:16:54.491 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:16:54.561 INFO:teuthology.orchestra.run.smithi134.stderr:No --period given, using current period=01050e30-179b-43be-bdbb-3bb23cf37ae8 2024-08-18T17:16:54.593 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T17:16:54.594 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:16:54.594 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:16:54.594 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T17:16:54.594 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:16:54.594 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:16:54.594 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:16:54.594 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T17:16:54.594 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:16:54.594 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:16:54.594 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:16:54.594 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:42:47.672461Z" 2024-08-18T17:16:54.594 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:16:54.595 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:16:54.595 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1724001414.343331_1533.1", 2024-08-18T17:16:54.595 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T17:16:54.343331Z" 2024-08-18T17:16:54.595 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:16:54.595 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T17:16:54.596 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-08-18T17:16:54.596 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:16:54.596 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:16:54.596 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:16:54.645 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:16:54.646 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:16:54.675 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:16:54.675+0000 7fddb0e0f880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:16:54.676 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:16:54.676+0000 7fddb0e0f880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:16:54.690 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:16:54.690 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:16:54.691 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:16:54.691 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:16:54.692 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:16:54.692 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:16:54.692 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:16:54.692 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:16:54.692 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:16:54.692 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:16:54.692 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:16:54.692 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:16:54.692 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:16:54.692 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:16:54.693 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:16:54.693 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:16:54.693 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:16:54.693 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:16:54.693 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:16:54.693 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:16:54.693 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:16:54.693 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:16:54.693 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:16:54.693 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:16:54.693 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:16:54.693 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:16:54.693 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:16:54.694 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:16:54.694 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:16:54.694 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:16:54.694 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:16:54.694 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:16:54.694 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:16:54.695 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:16:54.695 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:16:54.695 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:16:54.695 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:16:54.695 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:16:54.695 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:16:54.695 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:16:54.696 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:16:54.696 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:16:54.696 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:16:54.696 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:16:54.696 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:16:54.696 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:16:54.696 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:16:54.696 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:16:54.696 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:16:54.696 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001268.323120_1508.1", 2024-08-18T17:16:54.696 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:16:54.697 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:16:54.697 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:16:54.697 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:14:28.323120Z", 2024-08-18T17:16:54.697 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:16:54.697 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:16:54.697 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:16:54.697 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:16:54.697 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:16:54.697 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:16:54.698 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:16:54.698 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:16:54.698 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:16:54.698 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:16:54Z" 2024-08-18T17:16:54.698 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:16:54.699 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001414.343331_1533.1 target=1_1724001268.323120_1508.1 2024-08-18T17:16:59.700 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:16:59.700 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:16:59.700 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:16:59.755 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:16:59.756 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:16:59.790 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:16:59.789+0000 7f4cf6a62880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:16:59.790 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:16:59.790+0000 7f4cf6a62880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:16:59.805 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:16:59.805 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:16:59.805 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:16:59.805 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:16:59.806 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:16:59.806 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:16:59.806 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:16:59.806 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:16:59.806 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:16:59.806 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:16:59.806 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:16:59.806 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:16:59.806 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:16:59.806 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:16:59.806 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:16:59.807 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:16:59.807 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:16:59.807 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:16:59.807 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:16:59.807 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:16:59.807 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:16:59.807 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:16:59.807 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:16:59.807 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:16:59.807 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:16:59.807 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:16:59.807 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:16:59.807 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:16:59.808 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:16:59.808 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:16:59.808 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:16:59.808 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:16:59.808 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:16:59.808 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:16:59.808 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:16:59.808 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:16:59.808 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:16:59.808 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:16:59.808 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:16:59.808 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:16:59.808 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:16:59.809 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:16:59.809 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:16:59.809 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:16:59.809 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:16:59.809 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:16:59.809 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:16:59.809 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:16:59.809 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:16:59.809 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001268.323120_1508.1", 2024-08-18T17:16:59.809 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:16:59.809 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:16:59.809 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:16:59.809 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:14:28.323120Z", 2024-08-18T17:16:59.809 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:16:59.810 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:16:59.810 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:16:59.810 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:16:59.810 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:16:59.810 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:16:59.810 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:16:59.810 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:16:59.810 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:16:59.810 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:16:59Z" 2024-08-18T17:16:59.810 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:16:59.811 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001414.343331_1533.1 target=1_1724001268.323120_1508.1 2024-08-18T17:17:04.812 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:17:04.812 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:17:04.813 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:17:04.868 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:17:04.868 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:17:04.902 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:17:04.901+0000 7f9f9ae98880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:17:04.903 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:17:04.902+0000 7f9f9ae98880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:17:04.917 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:17:04.918 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:17:04.918 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:17:04.918 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:17:04.918 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:17:04.918 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:17:04.918 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:17:04.918 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:17:04.918 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:17:04.918 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:17:04.918 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:17:04.918 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:17:04.919 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:17:04.919 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:17:04.919 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:17:04.919 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:17:04.919 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:17:04.919 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:17:04.919 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:17:04.919 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:17:04.919 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:17:04.919 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:17:04.919 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:17:04.919 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:17:04.919 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:17:04.920 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:17:04.920 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:17:04.920 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:17:04.920 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:17:04.920 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:17:04.920 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:17:04.920 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:17:04.920 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:17:04.920 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:17:04.920 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:17:04.920 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:17:04.920 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:17:04.920 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:17:04.920 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:17:04.921 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:17:04.921 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:17:04.921 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:17:04.921 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:17:04.921 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:17:04.921 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:17:04.921 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:17:04.921 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:17:04.921 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:17:04.921 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:17:04.921 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001268.323120_1508.1", 2024-08-18T17:17:04.921 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:17:04.921 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:17:04.922 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:17:04.922 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:14:28.323120Z", 2024-08-18T17:17:04.922 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:17:04.922 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:17:04.922 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:17:04.922 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:17:04.922 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:17:04.922 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:17:04.922 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:17:04.922 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:17:04.923 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:17:04.923 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:17:04Z" 2024-08-18T17:17:04.923 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:17:04.925 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001414.343331_1533.1 target=1_1724001268.323120_1508.1 2024-08-18T17:17:09.926 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:17:09.926 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:17:09.927 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:17:09.981 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:17:09.981 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:17:10.016 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:17:10.014+0000 7fe1b5920880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:17:10.016 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:17:10.015+0000 7fe1b5920880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:17:10.030 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:17:10.031 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:17:10.031 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:17:10.031 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:17:10.031 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:17:10.031 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:17:10.031 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:17:10.031 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:17:10.031 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:17:10.031 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:17:10.031 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:17:10.031 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:17:10.031 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:17:10.032 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:17:10.032 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:17:10.032 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:17:10.032 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:17:10.032 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:17:10.032 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:17:10.032 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:17:10.032 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:17:10.032 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:17:10.032 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:17:10.032 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:17:10.032 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:17:10.032 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:17:10.033 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:17:10.033 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:17:10.033 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:17:10.033 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:17:10.033 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:17:10.033 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:17:10.033 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:17:10.033 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:17:10.033 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:17:10.033 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:17:10.033 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:17:10.033 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:17:10.033 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:17:10.033 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:17:10.034 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:17:10.034 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:17:10.034 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:17:10.034 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:17:10.034 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:17:10.034 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:17:10.034 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:17:10.034 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:17:10.034 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:17:10.034 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001268.323120_1508.1", 2024-08-18T17:17:10.034 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:17:10.034 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:17:10.034 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:17:10.035 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:14:28.323120Z", 2024-08-18T17:17:10.035 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:17:10.035 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:17:10.035 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:17:10.035 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:17:10.035 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:17:10.035 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:17:10.035 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:17:10.035 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:17:10.035 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:17:10.035 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:17:10Z" 2024-08-18T17:17:10.035 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:17:10.036 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001414.343331_1533.1 target=1_1724001268.323120_1508.1 2024-08-18T17:17:15.037 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:17:15.038 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:17:15.038 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:17:15.094 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:17:15.094 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:17:15.127 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:17:15.127+0000 7f29b9419880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:17:15.128 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:17:15.127+0000 7f29b9419880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:17:15.143 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:17:15.143 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:17:15.143 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:17:15.144 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:17:15.144 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:17:15.144 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:17:15.144 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:17:15.144 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:17:15.145 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:17:15.145 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:17:15.145 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:17:15.145 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:17:15.145 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:17:15.145 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:17:15.146 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:17:15.146 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:17:15.146 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:17:15.146 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:17:15.146 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:17:15.147 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:17:15.147 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:17:15.147 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:17:15.148 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:17:15.148 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:17:15.148 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:17:15.148 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:17:15.148 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:17:15.148 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:17:15.149 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:17:15.149 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:17:15.149 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:17:15.149 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:17:15.149 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:17:15.150 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:17:15.150 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:17:15.150 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:17:15.150 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:17:15.150 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:17:15.150 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:17:15.151 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:17:15.151 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:17:15.151 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:17:15.151 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:17:15.151 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:17:15.152 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:17:15.152 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:17:15.152 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:17:15.152 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:17:15.152 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:17:15.152 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001268.323120_1508.1", 2024-08-18T17:17:15.153 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:17:15.154 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:17:15.154 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:17:15.154 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:14:28.323120Z", 2024-08-18T17:17:15.154 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:17:15.154 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:17:15.155 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:17:15.155 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:17:15.155 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:17:15.155 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:17:15.155 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:17:15.155 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:17:15.156 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:17:15.156 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:17:15Z" 2024-08-18T17:17:15.156 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:17:15.158 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001414.343331_1533.1 target=1_1724001268.323120_1508.1 2024-08-18T17:17:20.158 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:17:20.159 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:17:20.159 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:17:20.215 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:17:20.215 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:17:20.249 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:17:20.248+0000 7fd702922880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:17:20.250 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:17:20.249+0000 7fd702922880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:17:20.264 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:17:20.265 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:17:20.265 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:17:20.265 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:17:20.265 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:17:20.265 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:17:20.265 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:17:20.265 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:17:20.265 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:17:20.265 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:17:20.265 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:17:20.265 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:17:20.266 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:17:20.266 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:17:20.266 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:17:20.266 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:17:20.266 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:17:20.266 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:17:20.266 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:17:20.266 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:17:20.266 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:17:20.266 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:17:20.266 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:17:20.266 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:17:20.267 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:17:20.267 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:17:20.267 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:17:20.267 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:17:20.267 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:17:20.267 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:17:20.267 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:17:20.267 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:17:20.267 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:17:20.267 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:17:20.267 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:17:20.267 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:17:20.267 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:17:20.268 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:17:20.268 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:17:20.268 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:17:20.268 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:17:20.268 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:17:20.268 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:17:20.268 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:17:20.268 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:17:20.268 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:17:20.268 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:17:20.268 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:17:20.268 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:17:20.269 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001268.323120_1508.1", 2024-08-18T17:17:20.269 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:17:20.269 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:17:20.269 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:17:20.269 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:14:28.323120Z", 2024-08-18T17:17:20.269 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:17:20.269 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:17:20.269 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:17:20.269 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:17:20.269 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:17:20.269 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:17:20.269 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:17:20.269 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:17:20.269 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:17:20.269 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:17:20Z" 2024-08-18T17:17:20.270 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:17:20.271 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001414.343331_1533.1 target=1_1724001268.323120_1508.1 2024-08-18T17:17:25.271 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:17:25.271 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:17:25.272 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:17:25.326 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:17:25.326 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:17:25.360 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:17:25.359+0000 7f4bc6211880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:17:25.360 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:17:25.360+0000 7f4bc6211880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:17:25.375 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:17:25.375 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:17:25.375 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:17:25.376 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:17:25.376 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:17:25.376 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:17:25.376 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:17:25.376 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:17:25.376 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:17:25.376 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:17:25.376 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:17:25.376 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:17:25.376 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:17:25.376 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:17:25.376 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:17:25.376 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:17:25.377 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:17:25.377 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:17:25.377 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:17:25.377 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:17:25.377 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:17:25.377 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:17:25.377 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:17:25.377 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:17:25.377 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:17:25.377 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:17:25.377 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:17:25.377 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:17:25.377 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:17:25.377 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:17:25.378 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:17:25.378 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:17:25.378 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:17:25.378 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:17:25.378 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:17:25.378 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:17:25.378 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:17:25.378 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:17:25.378 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:17:25.378 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:17:25.378 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:17:25.378 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:17:25.379 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:17:25.379 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:17:25.379 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:17:25.379 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:17:25.379 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:17:25.379 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:17:25.379 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:17:25.379 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001268.323120_1508.1", 2024-08-18T17:17:25.379 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:17:25.379 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:17:25.379 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:17:25.379 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:14:28.323120Z", 2024-08-18T17:17:25.379 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:17:25.380 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:17:25.380 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:17:25.380 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:17:25.380 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:17:25.380 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:17:25.380 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:17:25.380 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:17:25.380 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:17:25.380 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:17:25Z" 2024-08-18T17:17:25.380 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:17:25.381 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001414.343331_1533.1 target=1_1724001268.323120_1508.1 2024-08-18T17:17:30.382 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:17:30.383 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:17:30.383 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:17:30.439 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:17:30.439 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:17:30.472 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:17:30.472+0000 7fcd63d36880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:17:30.473 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:17:30.472+0000 7fcd63d36880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:17:30.489 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:17:30.489 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:17:30.489 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:17:30.489 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:17:30.489 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:17:30.489 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:17:30.489 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:17:30.489 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:17:30.489 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:17:30.489 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:17:30.489 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:17:30.490 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:17:30.490 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:17:30.490 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:17:30.490 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:17:30.490 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:17:30.490 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:17:30.490 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:17:30.490 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:17:30.490 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:17:30.490 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:17:30.490 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:17:30.490 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:17:30.490 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:17:30.491 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:17:30.491 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:17:30.491 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:17:30.491 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:17:30.491 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:17:30.491 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:17:30.491 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:17:30.491 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:17:30.491 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:17:30.491 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:17:30.491 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:17:30.491 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:17:30.491 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:17:30.492 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:17:30.492 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:17:30.492 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:17:30.492 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:17:30.492 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:17:30.492 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:17:30.492 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:17:30.492 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:17:30.492 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:17:30.492 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:17:30.492 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:17:30.492 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:17:30.492 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001268.323120_1508.1", 2024-08-18T17:17:30.492 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:17:30.492 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:17:30.493 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:17:30.493 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:14:28.323120Z", 2024-08-18T17:17:30.493 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:17:30.493 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:17:30.493 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:17:30.493 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:17:30.493 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:17:30.493 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:17:30.493 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:17:30.493 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:17:30.493 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:17:30.493 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:17:30Z" 2024-08-18T17:17:30.493 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:17:30.494 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001414.343331_1533.1 target=1_1724001268.323120_1508.1 2024-08-18T17:17:35.495 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:17:35.496 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:17:35.496 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:17:35.550 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:17:35.551 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:17:35.584 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:17:35.583+0000 7f2d1ce13880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:17:35.585 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:17:35.584+0000 7f2d1ce13880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:17:35.600 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:17:35.600 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:17:35.600 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:17:35.600 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:17:35.600 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:17:35.600 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:17:35.600 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:17:35.600 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:17:35.600 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:17:35.601 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:17:35.601 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:17:35.601 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:17:35.601 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:17:35.601 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:17:35.601 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:17:35.601 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:17:35.601 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:17:35.601 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:17:35.601 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:17:35.601 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:17:35.601 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:17:35.601 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:17:35.602 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:17:35.602 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:17:35.602 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:17:35.602 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:17:35.602 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:17:35.602 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:17:35.602 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:17:35.602 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:17:35.602 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:17:35.602 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:17:35.602 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:17:35.602 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:17:35.602 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:17:35.603 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:17:35.603 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:17:35.603 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:17:35.603 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:17:35.603 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:17:35.603 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:17:35.603 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:17:35.603 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:17:35.603 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:17:35.603 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:17:35.604 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:17:35.604 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:17:35.604 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:17:35.604 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:17:35.604 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001268.323120_1508.1", 2024-08-18T17:17:35.604 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:17:35.604 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:17:35.604 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:17:35.604 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:14:28.323120Z", 2024-08-18T17:17:35.604 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:17:35.604 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:17:35.604 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:17:35.605 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:17:35.605 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:17:35.605 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:17:35.605 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:17:35.605 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:17:35.605 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:17:35.605 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:17:35Z" 2024-08-18T17:17:35.605 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:17:35.607 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001414.343331_1533.1 target=1_1724001268.323120_1508.1 2024-08-18T17:17:40.608 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:17:40.608 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:17:40.608 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:17:40.664 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:17:40.665 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:17:40.699 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:17:40.698+0000 7fc230e0e880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:17:40.700 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:17:40.699+0000 7fc230e0e880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:17:40.714 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:17:40.714 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:17:40.714 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:17:40.714 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:17:40.714 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:17:40.714 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:17:40.714 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:17:40.714 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:17:40.715 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:17:40.715 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:17:40.715 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:17:40.715 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:17:40.715 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:17:40.715 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:17:40.715 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:17:40.715 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:17:40.715 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:17:40.715 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:17:40.715 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:17:40.715 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:17:40.715 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:17:40.715 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:17:40.716 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:17:40.716 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:17:40.716 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:17:40.716 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:17:40.716 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:17:40.716 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:17:40.716 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:17:40.716 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:17:40.716 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:17:40.716 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:17:40.716 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:17:40.716 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:17:40.717 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:17:40.717 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:17:40.717 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:17:40.717 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:17:40.717 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:17:40.717 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:17:40.717 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:17:40.717 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:17:40.717 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:17:40.717 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:17:40.717 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:17:40.717 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:17:40.717 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:17:40.718 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:17:40.718 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:17:40.718 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001268.323120_1508.1", 2024-08-18T17:17:40.718 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:17:40.718 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:17:40.718 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:17:40.718 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:14:28.323120Z", 2024-08-18T17:17:40.718 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:17:40.718 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:17:40.718 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:17:40.718 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:17:40.718 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:17:40.719 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:17:40.719 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:17:40.719 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:17:40.719 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:17:40.719 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:17:40Z" 2024-08-18T17:17:40.719 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:17:40.720 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001414.343331_1533.1 target=1_1724001268.323120_1508.1 2024-08-18T17:17:45.721 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:17:45.721 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:17:45.721 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:17:45.776 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:17:45.776 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:17:45.810 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:17:45.808+0000 7fa4c1342880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:17:45.811 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:17:45.809+0000 7fa4c1342880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:17:45.825 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:17:45.826 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:17:45.826 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:17:45.826 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:17:45.826 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:17:45.826 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:17:45.826 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:17:45.826 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:17:45.826 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:17:45.826 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:17:45.827 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:17:45.827 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:17:45.827 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:17:45.827 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:17:45.827 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:17:45.827 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:17:45.827 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:17:45.827 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:17:45.827 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:17:45.827 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:17:45.827 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:17:45.827 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:17:45.828 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:17:45.828 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:17:45.828 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:17:45.828 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:17:45.828 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:17:45.828 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:17:45.828 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:17:45.828 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:17:45.828 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:17:45.828 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:17:45.828 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:17:45.828 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:17:45.828 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:17:45.829 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:17:45.829 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:17:45.829 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:17:45.829 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:17:45.829 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:17:45.829 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:17:45.829 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:17:45.829 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:17:45.829 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:17:45.829 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:17:45.829 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:17:45.829 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:17:45.830 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:17:45.830 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:17:45.830 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001268.323120_1508.1", 2024-08-18T17:17:45.830 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:17:45.830 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:17:45.830 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:17:45.830 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:14:28.323120Z", 2024-08-18T17:17:45.830 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:17:45.830 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:17:45.830 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:17:45.830 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:17:45.830 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:17:45.831 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:17:45.831 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:17:45.831 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:17:45.831 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:17:45.831 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:17:45Z" 2024-08-18T17:17:45.831 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:17:45.832 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001414.343331_1533.1 target=1_1724001268.323120_1508.1 2024-08-18T17:17:50.833 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:17:50.833 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:17:50.834 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:17:50.888 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:17:50.888 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:17:50.922 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:17:50.920+0000 7fda0a20f880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:17:50.923 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:17:50.921+0000 7fda0a20f880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:17:50.938 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:17:50.939 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:17:50.939 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:17:50.939 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:17:50.939 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:17:50.939 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:17:50.939 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:17:50.939 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:17:50.939 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:17:50.939 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:17:50.939 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:17:50.939 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:17:50.940 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:17:50.940 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:17:50.940 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:17:50.940 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:17:50.940 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:17:50.940 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:17:50.940 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:17:50.940 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:17:50.940 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:17:50.940 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:17:50.940 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:17:50.940 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:17:50.940 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:17:50.940 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:17:50.941 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:17:50.941 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:17:50.941 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:17:50.941 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:17:50.941 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:17:50.941 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:17:50.941 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:17:50.941 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:17:50.941 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:17:50.941 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:17:50.941 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:17:50.941 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:17:50.941 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:17:50.942 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:17:50.942 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:17:50.942 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:17:50.942 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:17:50.942 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:17:50.942 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:17:50.942 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:17:50.942 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:17:50.942 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:17:50.942 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:17:50.942 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001268.323120_1508.1", 2024-08-18T17:17:50.942 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:17:50.942 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:17:50.943 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:17:50.943 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:14:28.323120Z", 2024-08-18T17:17:50.943 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:17:50.943 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:17:50.943 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:17:50.943 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:17:50.943 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:17:50.943 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:17:50.943 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:17:50.943 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:17:50.943 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:17:50.943 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:17:50Z" 2024-08-18T17:17:50.943 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:17:50.944 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001414.343331_1533.1 target=1_1724001268.323120_1508.1 2024-08-18T17:17:55.945 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:17:55.945 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:17:55.945 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:17:56.003 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:17:56.003 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:17:56.036 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:17:56.036+0000 7fdc6fe11880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:17:56.037 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:17:56.037+0000 7fdc6fe11880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:17:56.054 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:17:56.054 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:17:56.055 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:17:56.055 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:17:56.055 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:17:56.055 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:17:56.055 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:17:56.055 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:17:56.055 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:17:56.055 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:17:56.055 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:17:56.055 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:17:56.055 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:17:56.056 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:17:56.056 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:17:56.056 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:17:56.056 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:17:56.056 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:17:56.056 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:17:56.056 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:17:56.056 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:17:56.056 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:17:56.056 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:17:56.056 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:17:56.056 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:17:56.057 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:17:56.057 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:17:56.057 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:17:56.057 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:17:56.057 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:17:56.057 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:17:56.057 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:17:56.058 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:17:56.058 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:17:56.058 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:17:56.058 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:17:56.058 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:17:56.058 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:17:56.058 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:17:56.058 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:17:56.059 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:17:56.059 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:17:56.059 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:17:56.059 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:17:56.059 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:17:56.059 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:17:56.059 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:17:56.059 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:17:56.059 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:17:56.059 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001268.323120_1508.1", 2024-08-18T17:17:56.059 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:17:56.060 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:17:56.060 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:17:56.060 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:14:28.323120Z", 2024-08-18T17:17:56.060 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:17:56.060 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:17:56.060 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:17:56.060 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:17:56.060 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:17:56.060 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:17:56.060 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:17:56.060 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:17:56.060 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:17:56.060 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:17:56Z" 2024-08-18T17:17:56.061 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:17:56.061 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001414.343331_1533.1 target=1_1724001268.323120_1508.1 2024-08-18T17:18:01.062 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:18:01.062 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:18:01.063 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:18:01.118 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:18:01.118 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:18:01.151 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:18:01.150+0000 7fc1f3719880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:18:01.151 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:18:01.151+0000 7fc1f3719880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:18:01.166 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:18:01.167 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:18:01.167 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:18:01.167 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:18:01.167 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:18:01.167 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:18:01.167 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:18:01.167 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:01.167 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:18:01.167 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:18:01.167 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:18:01.167 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:18:01.167 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:18:01.167 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:18:01.168 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:18:01.168 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:18:01.168 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:18:01.168 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:18:01.168 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:18:01.168 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:18:01.168 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:01.168 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:18:01.168 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:18:01.168 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:18:01.168 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:18:01.169 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:18:01.169 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:18:01.169 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:18:01.169 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:18:01.169 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:18:01.169 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:18:01.169 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:18:01.169 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:01.169 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:18:01.169 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:18:01.169 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:18:01.169 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:18:01.169 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:18:01.170 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:18:01.170 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:18:01.170 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:18:01.170 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:18:01.170 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:18:01.170 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:18:01.170 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:01.170 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:18:01.170 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:18:01.170 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:18:01.170 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:18:01.170 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001268.323120_1508.1", 2024-08-18T17:18:01.170 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:18:01.171 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:18:01.171 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:18:01.171 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:14:28.323120Z", 2024-08-18T17:18:01.171 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:18:01.171 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:18:01.171 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:18:01.171 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:18:01.171 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:01.171 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:18:01.171 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:18:01.171 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:18:01.171 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:01.171 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:18:01Z" 2024-08-18T17:18:01.171 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:18:01.172 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001414.343331_1533.1 target=1_1724001268.323120_1508.1 2024-08-18T17:18:06.173 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:18:06.173 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:18:06.173 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:18:06.228 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:18:06.228 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:18:06.261 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:18:06.260+0000 7fe2b2136880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:18:06.262 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:18:06.261+0000 7fe2b2136880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:18:06.276 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:18:06.277 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:18:06.277 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:18:06.277 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:18:06.277 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:18:06.277 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:18:06.277 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:18:06.277 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:06.277 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:18:06.277 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:18:06.277 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:18:06.277 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:18:06.278 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:18:06.278 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:18:06.278 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:18:06.278 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:18:06.278 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:18:06.278 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:18:06.278 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:18:06.278 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:18:06.278 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:06.279 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:18:06.279 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:18:06.279 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:18:06.279 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:18:06.279 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:18:06.279 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:18:06.279 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:18:06.279 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:18:06.280 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:18:06.280 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:18:06.280 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:18:06.280 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:06.280 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:18:06.280 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:18:06.280 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:18:06.280 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:18:06.280 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:18:06.280 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:18:06.280 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:18:06.280 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:18:06.281 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:18:06.281 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:18:06.281 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:18:06.281 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:06.281 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:18:06.281 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:18:06.281 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:18:06.281 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:18:06.281 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001268.323120_1508.1", 2024-08-18T17:18:06.281 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:18:06.281 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:18:06.281 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:18:06.281 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:14:28.323120Z", 2024-08-18T17:18:06.282 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:18:06.282 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:18:06.282 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:18:06.282 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:18:06.282 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:06.282 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:18:06.282 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:18:06.282 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:18:06.282 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:06.282 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:18:06Z" 2024-08-18T17:18:06.282 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:18:06.283 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001414.343331_1533.1 target=1_1724001268.323120_1508.1 2024-08-18T17:18:11.284 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:18:11.284 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:18:11.285 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:18:11.339 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:18:11.339 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:18:11.373 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:18:11.372+0000 7f3e83919880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:18:11.373 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:18:11.373+0000 7f3e83919880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:18:11.388 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:18:11.389 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:18:11.389 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:18:11.389 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:18:11.389 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:18:11.389 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:18:11.389 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:18:11.389 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:11.389 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:18:11.389 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:18:11.389 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:18:11.390 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:18:11.390 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:18:11.390 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:18:11.390 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:18:11.390 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:18:11.390 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:18:11.390 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:18:11.390 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:18:11.390 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:18:11.390 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:11.390 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:18:11.391 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:18:11.391 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:18:11.391 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:18:11.391 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:18:11.391 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:18:11.391 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:18:11.391 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:18:11.391 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:18:11.391 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:18:11.392 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:18:11.392 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:11.392 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:18:11.392 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:18:11.392 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:18:11.392 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:18:11.392 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:18:11.392 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:18:11.392 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:18:11.392 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:18:11.392 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:18:11.392 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:18:11.393 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:18:11.393 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:11.393 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:18:11.393 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:18:11.393 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:18:11.393 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:18:11.393 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001268.323120_1508.1", 2024-08-18T17:18:11.393 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:18:11.393 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:18:11.393 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:18:11.394 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:14:28.323120Z", 2024-08-18T17:18:11.394 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:18:11.394 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:18:11.394 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:18:11.394 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:18:11.394 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:11.394 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:18:11.394 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:18:11.394 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:18:11.394 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:11.394 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:18:11Z" 2024-08-18T17:18:11.395 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:18:11.396 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001414.343331_1533.1 target=1_1724001268.323120_1508.1 2024-08-18T17:18:16.397 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:18:16.397 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:18:16.397 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:18:16.452 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:18:16.452 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:18:16.487 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:18:16.486+0000 7f7c74812880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:18:16.487 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:18:16.487+0000 7f7c74812880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:18:16.502 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:18:16.503 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:18:16.503 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:18:16.503 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:18:16.503 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:18:16.503 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:18:16.503 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:18:16.503 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:16.504 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:18:16.504 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:18:16.504 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:18:16.504 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:18:16.504 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:18:16.504 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:18:16.505 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:18:16.505 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:18:16.505 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:18:16.505 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:18:16.505 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:18:16.505 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:18:16.506 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:16.506 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:18:16.506 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:18:16.506 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:18:16.507 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:18:16.507 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:18:16.507 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:18:16.507 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:18:16.507 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:18:16.508 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:18:16.508 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:18:16.508 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:18:16.508 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:16.508 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:18:16.508 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:18:16.509 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:18:16.509 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:18:16.509 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:18:16.509 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:18:16.509 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:18:16.509 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:18:16.509 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:18:16.509 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:18:16.509 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:18:16.509 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:16.509 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:18:16.510 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:18:16.510 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:18:16.510 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:18:16.510 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001268.323120_1508.1", 2024-08-18T17:18:16.510 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:18:16.510 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:18:16.510 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:18:16.510 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:14:28.323120Z", 2024-08-18T17:18:16.510 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:18:16.510 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:18:16.511 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:18:16.511 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:18:16.511 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:16.511 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:18:16.511 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:18:16.511 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:18:16.511 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:16.511 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:18:16Z" 2024-08-18T17:18:16.511 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:18:16.512 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001414.343331_1533.1 target=1_1724001268.323120_1508.1 2024-08-18T17:18:21.513 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:18:21.514 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:18:21.514 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:18:21.569 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:18:21.569 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:18:21.603 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:18:21.602+0000 7fdf9a80f880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:18:21.604 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:18:21.603+0000 7fdf9a80f880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:18:21.619 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:18:21.619 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:18:21.619 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:18:21.619 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:18:21.620 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:18:21.620 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:18:21.620 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:18:21.620 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:21.620 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:18:21.620 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:18:21.620 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:18:21.621 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:18:21.621 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:18:21.621 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:18:21.621 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:18:21.621 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:18:21.621 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:18:21.621 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:18:21.621 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:18:21.622 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:18:21.622 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:21.622 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:18:21.622 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:18:21.622 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:18:21.622 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:18:21.622 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:18:21.622 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:18:21.623 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:18:21.623 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:18:21.623 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:18:21.623 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:18:21.623 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:18:21.623 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:21.623 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:18:21.624 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:18:21.624 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:18:21.624 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:18:21.624 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:18:21.624 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:18:21.624 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:18:21.624 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:18:21.624 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:18:21.624 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:18:21.624 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:18:21.624 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:21.625 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:18:21.625 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:18:21.625 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:18:21.625 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:18:21.625 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001268.323120_1508.1", 2024-08-18T17:18:21.625 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:18:21.625 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:18:21.625 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:18:21.625 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:14:28.323120Z", 2024-08-18T17:18:21.625 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:18:21.625 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:18:21.625 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:18:21.625 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:18:21.625 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:21.626 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:18:21.626 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:18:21.626 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:18:21.626 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:21.626 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:18:21Z" 2024-08-18T17:18:21.626 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:18:21.627 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001414.343331_1533.1 target=1_1724001268.323120_1508.1 2024-08-18T17:18:26.628 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:18:26.628 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:18:26.628 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:18:26.683 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:18:26.683 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:18:26.717 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:18:26.716+0000 7f5a49d23880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:18:26.717 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:18:26.717+0000 7f5a49d23880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:18:26.732 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:18:26.732 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:18:26.732 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:18:26.733 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:18:26.733 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:18:26.733 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:18:26.733 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:18:26.733 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:26.733 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:18:26.733 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:18:26.733 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:18:26.733 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:18:26.733 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:18:26.733 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:18:26.733 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:18:26.733 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:18:26.733 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:18:26.734 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:18:26.734 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:18:26.734 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:18:26.734 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:26.734 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:18:26.734 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:18:26.734 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:18:26.734 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:18:26.734 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:18:26.734 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:18:26.734 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:18:26.734 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:18:26.735 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:18:26.735 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:18:26.735 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:18:26.735 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:26.735 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:18:26.735 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:18:26.735 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:18:26.735 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:18:26.735 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:18:26.735 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:18:26.735 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:18:26.735 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:18:26.735 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:18:26.736 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:18:26.736 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:18:26.736 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:26.736 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:18:26.736 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:18:26.736 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:18:26.736 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:18:26.736 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001268.323120_1508.1", 2024-08-18T17:18:26.736 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:18:26.736 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:18:26.736 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:18:26.736 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:14:28.323120Z", 2024-08-18T17:18:26.736 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:18:26.737 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:18:26.737 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:18:26.737 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:18:26.737 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:26.737 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:18:26.737 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:18:26.737 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:18:26.737 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:26.737 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:18:26Z" 2024-08-18T17:18:26.737 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:18:26.738 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001414.343331_1533.1 target=1_1724001268.323120_1508.1 2024-08-18T17:18:31.740 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:18:31.740 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:18:31.741 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:18:31.796 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:18:31.796 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:18:31.830 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:18:31.829+0000 7f9e4220f880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:18:31.831 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:18:31.830+0000 7f9e4220f880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:18:31.846 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:18:31.846 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:18:31.846 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:18:31.846 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:18:31.846 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:18:31.846 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:18:31.846 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:18:31.846 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:31.846 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:18:31.846 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:18:31.847 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:18:31.847 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:18:31.847 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:18:31.847 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:18:31.847 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:18:31.847 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:18:31.847 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:18:31.847 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:18:31.847 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:18:31.847 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:18:31.847 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:31.847 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:18:31.848 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:18:31.848 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:18:31.848 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:18:31.848 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:18:31.848 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:18:31.848 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:18:31.848 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:18:31.848 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:18:31.848 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:18:31.848 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:18:31.848 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:31.848 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:18:31.849 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:18:31.849 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:18:31.849 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:18:31.849 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:18:31.849 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:18:31.849 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:18:31.849 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:18:31.849 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:18:31.849 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:18:31.849 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:18:31.849 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:31.849 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:18:31.850 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:18:31.850 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:18:31.850 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:18:31.850 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001268.323120_1508.1", 2024-08-18T17:18:31.850 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:18:31.850 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:18:31.850 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:18:31.850 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:14:28.323120Z", 2024-08-18T17:18:31.850 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:18:31.850 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:18:31.850 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:18:31.850 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:18:31.851 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:31.851 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:18:31.851 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:18:31.851 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:18:31.851 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:31.851 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:18:31Z" 2024-08-18T17:18:31.851 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:18:31.852 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001414.343331_1533.1 target=1_1724001268.323120_1508.1 2024-08-18T17:18:36.852 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:18:36.853 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:18:36.853 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:18:36.908 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:18:36.908 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:18:36.941 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:18:36.941+0000 7f1f07a11880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:18:36.942 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:18:36.942+0000 7f1f07a11880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:18:36.957 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:18:36.957 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:18:36.957 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:18:36.957 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:18:36.957 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:18:36.957 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:18:36.957 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:18:36.958 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:36.958 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:18:36.958 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:18:36.958 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:18:36.958 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:18:36.958 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:18:36.958 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:18:36.958 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:18:36.958 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:18:36.958 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:18:36.959 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:18:36.959 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:18:36.959 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:18:36.959 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:36.959 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:18:36.959 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:18:36.959 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:18:36.959 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:18:36.959 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:18:36.959 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:18:36.959 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:18:36.959 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:18:36.960 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:18:36.960 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:18:36.960 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:18:36.960 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:36.960 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:18:36.960 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:18:36.960 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:18:36.960 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:18:36.960 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:18:36.960 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:18:36.960 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:18:36.960 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:18:36.961 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:18:36.961 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:18:36.961 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:18:36.961 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:36.961 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:18:36.961 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:18:36.961 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:18:36.961 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:18:36.961 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001414.169137_1519.1", 2024-08-18T17:18:36.961 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:18:36.962 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:18:36.962 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:18:36.962 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:54.169137Z", 2024-08-18T17:18:36.962 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:18:36.962 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:18:36.962 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:18:36.962 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:18:36.962 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:36.962 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:18:36.962 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:18:36.962 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:18:36.962 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:36.963 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:18:36Z" 2024-08-18T17:18:36.963 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:18:36.964 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001414.343331_1533.1 target=1_1724001414.169137_1519.1 2024-08-18T17:18:41.964 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:18:41.965 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:18:41.965 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:18:42.020 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:18:42.020 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:18:42.057 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:18:42.056+0000 7f94b1ebb880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:18:42.057 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:18:42.057+0000 7f94b1ebb880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:18:42.072 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:18:42.072 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:18:42.073 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:18:42.073 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:18:42.073 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:18:42.073 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:18:42.073 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:18:42.073 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:42.073 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:18:42.073 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:18:42.073 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:18:42.073 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:18:42.073 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:18:42.073 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:18:42.074 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:18:42.074 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:18:42.074 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:18:42.074 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:18:42.074 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:18:42.074 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:18:42.074 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:42.074 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:18:42.074 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:18:42.074 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:18:42.074 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:18:42.075 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:18:42.075 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:18:42.075 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:18:42.075 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:18:42.075 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:18:42.075 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:18:42.075 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:18:42.075 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:42.075 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:18:42.075 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:18:42.075 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:18:42.075 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:18:42.075 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:18:42.075 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:18:42.076 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:18:42.076 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:18:42.076 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:18:42.076 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:18:42.076 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:18:42.076 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:42.076 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:18:42.076 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:18:42.076 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:18:42.076 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:18:42.076 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001414.343331_1533.1", 2024-08-18T17:18:42.076 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:18:42.077 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:18:42.077 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:18:42.077 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:54.343331Z", 2024-08-18T17:18:42.077 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:18:42.077 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:18:42.077 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:18:42.077 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:18:42.077 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:42.077 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:18:42.077 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:18:42.077 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:18:42.077 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:42.077 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:18:42Z" 2024-08-18T17:18:42.077 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:18:42.078 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-08-18T17:18:42.079 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-08-18T17:18:42.079 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:18:42.079 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:18:42.079 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:18:42.136 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:18:42.136 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:18:42.184 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:18:42.184+0000 7f8222336880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:18:42.185 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:18:42.185+0000 7f8222336880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:18:42.206 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T17:18:42.206 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T17:18:42.206 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T17:18:42.207 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T17:18:42.207 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T17:18:42.207 INFO:teuthology.orchestra.run.smithi134.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:18:42.207 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:18:42.207 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:18:42.207 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T17:18:42.207 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:18:42.207 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T17:18:42.207 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:18:42.207 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:18:42.207 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:18:42.207 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:18:42.208 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 1, 2024-08-18T17:18:42.208 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:18:42.208 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:18:42.208 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T17:18:42.208 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:18:42.208 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:18:42.208 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:18:42.208 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T17:18:42.208 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:18:42.208 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:18:42.208 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:18:42.208 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:18:42.208 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:18:42.208 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:18:42.209 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T17:18:42.209 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T17:18:42.209 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:18:42.209 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:18:42.209 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:18:42.209 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T17:18:42.209 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:18:42.209 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:18:42.209 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:18:42.209 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:18:42.209 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:18:42.209 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:18:42.209 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:18:42.209 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:18:42.209 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:18:42.210 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:18:42.210 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:18:42.210 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T17:18:42.210 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:18:42.210 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:18:42.210 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1724001414.343331_1533.1", 2024-08-18T17:18:42.210 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:18:42.210 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:18:42.210 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:18:42.210 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T17:16:54.343331Z", 2024-08-18T17:18:42.210 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:18:42.211 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:18:42.211 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:18:42.211 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:18:42.211 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:18:42.211 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T17:18:42.211 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 1, 2024-08-18T17:18:42.211 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 1 2024-08-18T17:18:42.211 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:18:42.211 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T17:18:42Z" 2024-08-18T17:18:42.211 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T17:18:42.212 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-08-18T17:18:42.212 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:18:42.213 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:18:42.213 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:18:42.260 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:18:42.260 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:18:42.349 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T17:18:42.349 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:18:42.349 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000051594", 2024-08-18T17:18:42.349 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T17:16:54.270479Z" 2024-08-18T17:18:42.350 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:18:42.350 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:18:42.350 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:18:42.350 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:18:42.350 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:18:42.350 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:18:42.350 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000058857", 2024-08-18T17:18:42.350 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T17:16:54.289199Z" 2024-08-18T17:18:42.350 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:18:42.350 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:18:42.350 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000052068", 2024-08-18T17:18:42.350 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T17:05:27.099520Z" 2024-08-18T17:18:42.351 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:18:42.351 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T17:18:42.352 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-08-18T17:18:42.352 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:18:42.352 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:18:42.352 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:18:42.404 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:18:42.404 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:18:42.451 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:18:42.451 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:18:42.451 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:18:42.451 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:18:42.451 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:18:42.452 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:18:42.452 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:42.452 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:18:42.452 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:18:42.452 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:18:42.452 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:18:42.452 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:18:42.452 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000051454", 2024-08-18T17:18:42.452 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:18:42.452 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:18:42.452 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:18:42.452 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.298690Z" 2024-08-18T17:18:42.452 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:18:42.453 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:42.453 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:18:42.453 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:18:42.453 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:18:42.453 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:18:42.453 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:18:42.453 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:18:42.453 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:18:42.453 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:18:42.453 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:18:42.453 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:18:42.453 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:42.453 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:18:42.454 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:18:42.454 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:18:42.454 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:18:42.454 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000058577", 2024-08-18T17:18:42.454 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:18:42.454 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:18:42.454 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:18:42.454 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.298682Z" 2024-08-18T17:18:42.454 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:18:42.454 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:42.454 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:18:42.454 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:18:42.455 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:18:42.455 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:18:42.455 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052068", 2024-08-18T17:18:42.455 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:18:42.455 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:18:42.455 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:18:42.455 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:05:27.099520Z" 2024-08-18T17:18:42.455 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:18:42.455 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:18:42.455 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:18:42.455 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:42.455 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:18:42.455 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:18:42.456 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:18:42.456 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:42.456 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:18:42Z" 2024-08-18T17:18:42.456 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:18:42.457 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000051594 target=00000000000000000000:00000000000000051454, shard=2 master=00000000000000000000:00000000000000058857 target=00000000000000000000:00000000000000058577 2024-08-18T17:18:47.458 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:18:47.458 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:18:47.458 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:18:47.515 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:18:47.516 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:18:47.566 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:18:47.566 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:18:47.566 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:18:47.566 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:18:47.566 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:18:47.566 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:18:47.566 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:47.566 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:18:47.567 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:18:47.567 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:18:47.567 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:18:47.567 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:18:47.567 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000051454", 2024-08-18T17:18:47.567 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:18:47.567 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:18:47.567 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:18:47.567 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.298690Z" 2024-08-18T17:18:47.567 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:18:47.567 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:47.567 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:18:47.567 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:18:47.568 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:18:47.568 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:18:47.568 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:18:47.568 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:18:47.568 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:18:47.568 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:18:47.568 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:18:47.568 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:18:47.568 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:47.568 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:18:47.568 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:18:47.568 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:18:47.568 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:18:47.569 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000058577", 2024-08-18T17:18:47.569 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:18:47.569 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:18:47.569 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:18:47.569 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.298682Z" 2024-08-18T17:18:47.569 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:18:47.569 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:47.569 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:18:47.569 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:18:47.569 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:18:47.569 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:18:47.569 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052068", 2024-08-18T17:18:47.569 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:18:47.570 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:18:47.570 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:18:47.570 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:05:27.099520Z" 2024-08-18T17:18:47.570 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:18:47.570 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:18:47.570 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:18:47.570 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:47.570 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:18:47.570 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:18:47.570 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:18:47.570 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:47.570 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:18:47Z" 2024-08-18T17:18:47.571 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:18:47.571 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000051594 target=00000000000000000000:00000000000000051454, shard=2 master=00000000000000000000:00000000000000058857 target=00000000000000000000:00000000000000058577 2024-08-18T17:18:52.572 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:18:52.573 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:18:52.573 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:18:52.630 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:18:52.630 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:18:52.680 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:18:52.680 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:18:52.680 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:18:52.680 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:18:52.680 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:18:52.680 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:18:52.681 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:52.681 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:18:52.681 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:18:52.681 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:18:52.681 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:18:52.681 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:18:52.681 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000051454", 2024-08-18T17:18:52.681 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:18:52.681 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:18:52.681 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:18:52.681 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.298690Z" 2024-08-18T17:18:52.681 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:18:52.682 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:52.682 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:18:52.682 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:18:52.682 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:18:52.682 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:18:52.682 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:18:52.682 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:18:52.682 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:18:52.682 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:18:52.682 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:18:52.682 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:18:52.682 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:52.682 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:18:52.683 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:18:52.683 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:18:52.683 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:18:52.683 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000058577", 2024-08-18T17:18:52.683 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:18:52.683 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:18:52.683 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:18:52.683 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.298682Z" 2024-08-18T17:18:52.683 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:18:52.683 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:52.683 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:18:52.683 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:18:52.683 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:18:52.684 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:18:52.684 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052068", 2024-08-18T17:18:52.684 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:18:52.684 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:18:52.684 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:18:52.684 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:05:27.099520Z" 2024-08-18T17:18:52.684 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:18:52.684 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:18:52.684 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:18:52.684 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:52.685 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:18:52.685 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:18:52.685 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:18:52.685 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:52.685 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:18:52Z" 2024-08-18T17:18:52.685 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:18:52.686 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000051594 target=00000000000000000000:00000000000000051454, shard=2 master=00000000000000000000:00000000000000058857 target=00000000000000000000:00000000000000058577 2024-08-18T17:18:57.687 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:18:57.687 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:18:57.687 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:18:57.743 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:18:57.744 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:18:57.794 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:18:57.795 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:18:57.795 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:18:57.795 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:18:57.795 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:18:57.795 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:18:57.795 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:57.795 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:18:57.795 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:18:57.796 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:18:57.796 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:18:57.796 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:18:57.796 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000051594", 2024-08-18T17:18:57.796 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:18:57.796 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:18:57.796 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:18:57.796 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:54.270479Z" 2024-08-18T17:18:57.796 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:18:57.797 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:57.797 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:18:57.797 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:18:57.797 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:18:57.797 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:18:57.797 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:18:57.797 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:18:57.797 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:18:57.797 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:18:57.797 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:18:57.798 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:18:57.798 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:57.798 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:18:57.798 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:18:57.798 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:18:57.798 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:18:57.798 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000058857", 2024-08-18T17:18:57.798 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:18:57.798 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:18:57.799 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:18:57.799 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:54.289199Z" 2024-08-18T17:18:57.799 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:18:57.799 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:57.799 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:18:57.799 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:18:57.799 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:18:57.800 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:18:57.800 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052068", 2024-08-18T17:18:57.800 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:18:57.800 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:18:57.800 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:18:57.800 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:05:27.099520Z" 2024-08-18T17:18:57.800 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:18:57.800 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:18:57.800 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:18:57.800 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:57.800 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:18:57.801 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:18:57.801 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:18:57.801 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:57.801 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:18:57Z" 2024-08-18T17:18:57.801 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:18:57.802 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-08-18T17:18:57.817 INFO:tasks.rgw_multisite_tests:test_sync_flow_directional_zonegroup_select: ... FAIL 2024-08-18T17:18:57.818 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:18:57.818 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:18:57.818 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:18:57.871 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:18:57.871 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:18:57.911 INFO:teuthology.orchestra.run.smithi134.stderr:No --period given, using current period=01050e30-179b-43be-bdbb-3bb23cf37ae8 2024-08-18T17:18:57.927 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T17:18:57.927 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:18:57.927 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:18:57.927 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T17:18:57.927 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:18:57.927 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:18:57.927 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:18:57.927 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T17:18:57.927 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:18:57.927 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:18:57.928 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:18:57.928 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:42:47.672461Z" 2024-08-18T17:18:57.928 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:18:57.928 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:18:57.928 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1724001414.343331_1533.1", 2024-08-18T17:18:57.928 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T17:16:54.343331Z" 2024-08-18T17:18:57.928 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:18:57.928 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T17:18:57.929 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-08-18T17:18:57.929 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:18:57.929 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:18:57.930 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:18:57.980 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:18:57.980 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:18:58.010 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:18:58.010+0000 7fe259e2f880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:18:58.011 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:18:58.011+0000 7fe259e2f880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:18:58.027 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:18:58.027 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:18:58.027 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:18:58.027 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:18:58.027 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:18:58.027 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:18:58.027 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:18:58.027 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:58.028 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:18:58.028 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:18:58.028 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:18:58.028 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:18:58.028 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:18:58.028 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:18:58.028 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:18:58.028 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:18:58.028 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:18:58.028 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:18:58.028 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:18:58.028 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:18:58.028 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:58.028 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:18:58.029 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:18:58.029 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:18:58.029 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:18:58.029 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:18:58.029 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:18:58.029 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:18:58.029 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:18:58.029 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:18:58.029 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:18:58.029 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:18:58.029 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:58.029 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:18:58.029 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:18:58.030 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:18:58.030 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:18:58.030 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:18:58.030 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:18:58.030 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:18:58.030 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:18:58.030 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:18:58.030 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:18:58.030 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:18:58.030 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:58.030 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:18:58.030 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:18:58.030 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:18:58.030 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:18:58.031 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001414.343331_1533.1", 2024-08-18T17:18:58.031 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:18:58.031 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:18:58.031 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:18:58.031 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:54.343331Z", 2024-08-18T17:18:58.031 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:18:58.031 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:18:58.031 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:18:58.031 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:18:58.031 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:58.031 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:18:58.031 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:18:58.031 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:18:58.031 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:58.031 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:18:58Z" 2024-08-18T17:18:58.032 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:18:58.032 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-08-18T17:18:58.032 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-08-18T17:18:58.032 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:18:58.032 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:18:58.032 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:18:58.082 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:18:58.083 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:18:58.133 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:18:58.132+0000 7f8d7b898880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:18:58.133 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:18:58.133+0000 7f8d7b898880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:18:58.157 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T17:18:58.157 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T17:18:58.157 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T17:18:58.157 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T17:18:58.157 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T17:18:58.157 INFO:teuthology.orchestra.run.smithi134.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:18:58.157 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:18:58.158 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:18:58.158 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T17:18:58.158 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:18:58.158 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T17:18:58.158 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:18:58.158 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:18:58.158 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:18:58.158 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:18:58.158 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 1, 2024-08-18T17:18:58.158 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:18:58.158 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:18:58.158 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T17:18:58.158 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:18:58.159 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:18:58.159 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:18:58.159 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T17:18:58.159 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:18:58.159 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:18:58.159 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:18:58.159 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:18:58.159 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:18:58.159 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:18:58.159 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T17:18:58.159 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T17:18:58.159 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:18:58.159 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:18:58.159 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:18:58.160 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T17:18:58.160 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:18:58.160 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:18:58.160 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:18:58.160 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:18:58.160 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:18:58.160 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:18:58.160 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:18:58.160 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:18:58.160 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:18:58.160 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:18:58.160 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:18:58.160 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T17:18:58.160 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:18:58.161 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:18:58.161 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1724001414.343331_1533.1", 2024-08-18T17:18:58.161 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:18:58.161 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:18:58.161 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:18:58.161 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T17:16:54.343331Z", 2024-08-18T17:18:58.161 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:18:58.161 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:18:58.161 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:18:58.161 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:18:58.161 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:18:58.161 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T17:18:58.161 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 1, 2024-08-18T17:18:58.161 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 1 2024-08-18T17:18:58.162 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:18:58.162 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T17:18:58Z" 2024-08-18T17:18:58.162 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T17:18:58.163 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-08-18T17:18:58.163 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:18:58.163 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:18:58.163 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:18:58.207 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:18:58.207 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:18:58.272 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-08-18T17:18:58.272 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:18:58.272 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:18:58.272 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:18:58.355 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:18:58.355 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:18:58.406 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-08-18T17:18:58.407 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=kibgmb-91 2024-08-18T17:18:58.436 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=kibgmb-92 2024-08-18T17:18:58.457 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:18:58.457 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:18:58.457 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:18:58.503 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:18:58.503 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:18:58.548 INFO:teuthology.orchestra.run.smithi134.stderr:No --period given, using current period=01050e30-179b-43be-bdbb-3bb23cf37ae8 2024-08-18T17:18:58.563 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T17:18:58.563 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:18:58.564 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:18:58.564 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T17:18:58.564 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:18:58.564 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:18:58.564 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:18:58.564 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T17:18:58.564 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:18:58.564 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:18:58.564 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:18:58.564 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:42:47.672461Z" 2024-08-18T17:18:58.564 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:18:58.564 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:18:58.564 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1724001538.454480_1552.1", 2024-08-18T17:18:58.565 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T17:18:58.454480Z" 2024-08-18T17:18:58.565 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:18:58.565 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T17:18:58.566 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-08-18T17:18:58.566 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:18:58.566 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:18:58.566 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:18:58.619 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:18:58.619 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:18:58.652 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:18:58.651+0000 7f9f5bd42880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:18:58.653 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:18:58.652+0000 7f9f5bd42880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:18:58.667 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:18:58.667 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:18:58.667 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:18:58.667 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:18:58.667 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:18:58.667 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:18:58.667 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:18:58.667 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:58.667 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:18:58.667 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:18:58.667 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:18:58.668 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:18:58.668 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:18:58.668 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:18:58.668 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:18:58.668 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:18:58.668 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:18:58.668 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:18:58.668 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:18:58.668 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:18:58.668 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:58.668 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:18:58.668 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:18:58.668 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:18:58.669 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:18:58.669 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:18:58.669 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:18:58.669 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:18:58.669 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:18:58.669 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:18:58.669 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:18:58.669 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:18:58.669 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:58.669 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:18:58.669 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:18:58.669 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:18:58.669 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:18:58.669 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:18:58.670 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:18:58.670 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:18:58.670 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:18:58.670 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:18:58.670 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:18:58.670 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:18:58.670 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:58.670 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:18:58.670 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:18:58.670 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:18:58.670 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:18:58.670 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001538.419635_1545.1", 2024-08-18T17:18:58.671 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:18:58.671 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:18:58.671 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:18:58.671 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:18:58.419635Z", 2024-08-18T17:18:58.671 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:18:58.671 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:18:58.671 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:18:58.671 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:18:58.671 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:58.671 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:18:58.671 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:18:58.671 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:18:58.671 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:18:58.672 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:18:58Z" 2024-08-18T17:18:58.672 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:18:58.673 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001538.454480_1552.1 target=1_1724001538.419635_1545.1 2024-08-18T17:19:03.674 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:19:03.674 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:19:03.674 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:19:03.729 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:19:03.729 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:19:03.763 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:19:03.762+0000 7f515cb36880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:19:03.763 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:19:03.763+0000 7f515cb36880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:19:03.779 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:19:03.779 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:19:03.779 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:19:03.779 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:19:03.780 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:19:03.780 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:19:03.780 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:19:03.780 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:19:03.780 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:19:03.780 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:19:03.780 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:19:03.781 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:19:03.781 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:19:03.781 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:19:03.781 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:19:03.781 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:19:03.781 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:19:03.781 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:19:03.781 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:19:03.781 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:19:03.781 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:19:03.781 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:19:03.782 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:19:03.782 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:19:03.782 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:19:03.782 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:19:03.782 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:19:03.782 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:19:03.782 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:19:03.782 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:19:03.782 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:19:03.782 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:19:03.782 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:19:03.782 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:19:03.782 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:19:03.782 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:19:03.783 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:19:03.783 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:19:03.783 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:19:03.783 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:19:03.783 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:19:03.783 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:19:03.783 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:19:03.783 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:19:03.783 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:19:03.783 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:19:03.783 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:19:03.783 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:19:03.783 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:19:03.784 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001538.454480_1552.1", 2024-08-18T17:19:03.784 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:19:03.784 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:19:03.784 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:19:03.784 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:18:58.454480Z", 2024-08-18T17:19:03.784 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:19:03.784 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:19:03.784 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:19:03.784 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:19:03.784 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:19:03.784 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:19:03.784 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:19:03.784 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:19:03.784 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:19:03.785 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:19:03Z" 2024-08-18T17:19:03.785 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:19:03.786 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-08-18T17:19:03.786 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-08-18T17:19:03.786 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:19:03.786 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:19:03.786 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:19:03.842 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:19:03.842 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:19:03.892 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:19:03.891+0000 7fee6cc13880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:19:03.893 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:19:03.892+0000 7fee6cc13880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:19:03.910 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T17:19:03.910 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T17:19:03.911 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T17:19:03.911 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T17:19:03.911 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T17:19:03.911 INFO:teuthology.orchestra.run.smithi134.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:19:03.911 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:19:03.911 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:19:03.911 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T17:19:03.911 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:19:03.911 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T17:19:03.911 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:19:03.911 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:19:03.911 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:19:03.912 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:19:03.912 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 1, 2024-08-18T17:19:03.912 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:19:03.912 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:19:03.912 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T17:19:03.912 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:19:03.912 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:19:03.912 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:19:03.912 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T17:19:03.912 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:19:03.912 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:19:03.912 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:19:03.912 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:19:03.912 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:19:03.913 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:19:03.913 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T17:19:03.913 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T17:19:03.913 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:19:03.913 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:19:03.913 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:19:03.913 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T17:19:03.913 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:19:03.913 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:19:03.913 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:19:03.913 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:19:03.913 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:19:03.913 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:19:03.913 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:19:03.914 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:19:03.914 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:19:03.914 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:19:03.914 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:19:03.914 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T17:19:03.914 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:19:03.914 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:19:03.914 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1724001538.419635_1545.1", 2024-08-18T17:19:03.914 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:19:03.914 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:19:03.914 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:19:03.914 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T17:18:58.419635Z", 2024-08-18T17:19:03.914 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:19:03.915 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:19:03.915 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:19:03.915 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:19:03.915 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:19:03.915 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T17:19:03.915 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 1, 2024-08-18T17:19:03.915 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 1 2024-08-18T17:19:03.915 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:19:03.915 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T17:19:03Z" 2024-08-18T17:19:03.915 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T17:19:03.916 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1724001538.454480_1552.1 target=1_1724001538.419635_1545.1 2024-08-18T17:19:08.917 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:19:08.918 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:19:08.918 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:19:08.974 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:19:08.974 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:19:09.011 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:19:09.010+0000 7fbe1aa0e880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:19:09.012 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:19:09.011+0000 7fbe1aa0e880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:19:09.039 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T17:19:09.040 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T17:19:09.040 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T17:19:09.040 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T17:19:09.040 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T17:19:09.040 INFO:teuthology.orchestra.run.smithi134.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:19:09.040 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:19:09.040 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:19:09.040 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T17:19:09.040 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:19:09.041 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T17:19:09.041 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:19:09.041 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:19:09.041 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:19:09.041 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:19:09.041 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 1, 2024-08-18T17:19:09.041 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:19:09.041 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:19:09.041 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T17:19:09.041 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:19:09.042 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:19:09.042 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:19:09.042 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T17:19:09.042 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:19:09.042 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:19:09.042 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:19:09.042 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:19:09.042 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:19:09.042 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:19:09.042 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T17:19:09.043 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T17:19:09.043 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:19:09.043 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:19:09.043 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:19:09.043 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T17:19:09.043 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:19:09.043 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:19:09.044 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:19:09.044 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:19:09.044 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:19:09.044 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:19:09.044 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:19:09.044 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:19:09.045 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:19:09.045 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:19:09.045 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:19:09.045 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T17:19:09.045 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:19:09.045 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:19:09.045 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1724001538.454480_1552.1", 2024-08-18T17:19:09.045 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:19:09.045 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:19:09.046 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:19:09.046 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T17:18:58.454480Z", 2024-08-18T17:19:09.046 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:19:09.046 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:19:09.046 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:19:09.046 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:19:09.046 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:19:09.047 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T17:19:09.047 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 1, 2024-08-18T17:19:09.047 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 1 2024-08-18T17:19:09.047 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:19:09.047 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T17:19:09Z" 2024-08-18T17:19:09.047 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T17:19:09.048 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-08-18T17:19:09.049 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:19:09.049 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:19:09.049 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:19:09.098 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:19:09.098 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:19:09.180 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-08-18T17:19:09.180 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:19:09.180 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:19:09.181 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=test-zone1,test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:19:09.231 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:19:09.231 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:19:09.293 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-08-18T17:19:09.294 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1,test-zone2', '--dest-zones=test-zone1,test-zone2', '--source-bucket=kibgmb-91', '--dest-bucket=kibgmb-91', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:19:09.294 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1,test-zone2', '--dest-zones=test-zone1,test-zone2', '--source-bucket=kibgmb-91', '--dest-bucket=kibgmb-91', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:19:09.294 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=test-zone1,test-zone2 --dest-zones=test-zone1,test-zone2 --source-bucket=kibgmb-91 --dest-bucket=kibgmb-91 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:19:09.347 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:19:09.347 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:19:09.434 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"kibgmb-91","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"kibgmb-91","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-08-18T17:19:09.434 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:19:09.434 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:19:09.434 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:19:09.482 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:19:09.482 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:19:09.535 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"kibgmb-91","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"kibgmb-91","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-08-18T17:19:09.535 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:19:09.535 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:19:09.535 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:19:09.579 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:19:09.579 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:19:09.649 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"kibgmb-91","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"kibgmb-91","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-08-18T17:19:09.649 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:19:09.650 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:19:09.650 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:19:09.736 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:19:09.736 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:19:09.804 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T17:19:09.804 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:19:09.805 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 15, 2024-08-18T17:19:09.805 INFO:teuthology.orchestra.run.smithi134.stdout: "predecessor_uuid": "64309209-7567-46e5-8881-65783d84cf3c", 2024-08-18T17:19:09.805 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": [ 2024-08-18T17:19:09.805 INFO:teuthology.orchestra.run.smithi134.stdout: "", 2024-08-18T17:19:09.805 INFO:teuthology.orchestra.run.smithi134.stdout: "", 2024-08-18T17:19:09.805 INFO:teuthology.orchestra.run.smithi134.stdout: "", 2024-08-18T17:19:09.805 INFO:teuthology.orchestra.run.smithi134.stdout: "1_1723995848.930494_187.1" 2024-08-18T17:19:09.805 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:19:09.805 INFO:teuthology.orchestra.run.smithi134.stdout: "period_map": { 2024-08-18T17:19:09.805 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:19:09.805 INFO:teuthology.orchestra.run.smithi134.stdout: "zonegroups": [ 2024-08-18T17:19:09.806 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:19:09.806 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "test-zonegroup", 2024-08-18T17:19:09.806 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "test-zonegroup", 2024-08-18T17:19:09.806 INFO:teuthology.orchestra.run.smithi134.stdout: "api_name": "", 2024-08-18T17:19:09.806 INFO:teuthology.orchestra.run.smithi134.stdout: "is_master": true, 2024-08-18T17:19:09.806 INFO:teuthology.orchestra.run.smithi134.stdout: "endpoints": [ 2024-08-18T17:19:09.806 INFO:teuthology.orchestra.run.smithi134.stdout: "http://smithi134.front.sepia.ceph.com:8000" 2024-08-18T17:19:09.806 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:19:09.806 INFO:teuthology.orchestra.run.smithi134.stdout: "hostnames": [], 2024-08-18T17:19:09.806 INFO:teuthology.orchestra.run.smithi134.stdout: "hostnames_s3website": [], 2024-08-18T17:19:09.806 INFO:teuthology.orchestra.run.smithi134.stdout: "master_zone": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T17:19:09.807 INFO:teuthology.orchestra.run.smithi134.stdout: "zones": [ 2024-08-18T17:19:09.807 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:19:09.807 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T17:19:09.807 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "test-zone3", 2024-08-18T17:19:09.807 INFO:teuthology.orchestra.run.smithi134.stdout: "endpoints": [ 2024-08-18T17:19:09.807 INFO:teuthology.orchestra.run.smithi134.stdout: "http://smithi134.front.sepia.ceph.com:8001" 2024-08-18T17:19:09.807 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:19:09.807 INFO:teuthology.orchestra.run.smithi134.stdout: "log_meta": false, 2024-08-18T17:19:09.807 INFO:teuthology.orchestra.run.smithi134.stdout: "log_data": true, 2024-08-18T17:19:09.807 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_index_max_shards": 11, 2024-08-18T17:19:09.807 INFO:teuthology.orchestra.run.smithi134.stdout: "read_only": false, 2024-08-18T17:19:09.807 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_type": "", 2024-08-18T17:19:09.807 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from_all": true, 2024-08-18T17:19:09.808 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from": [], 2024-08-18T17:19:09.808 INFO:teuthology.orchestra.run.smithi134.stdout: "redirect_zone": "", 2024-08-18T17:19:09.808 INFO:teuthology.orchestra.run.smithi134.stdout: "supported_features": [ 2024-08-18T17:19:09.808 INFO:teuthology.orchestra.run.smithi134.stdout: "compress-encrypted", 2024-08-18T17:19:09.808 INFO:teuthology.orchestra.run.smithi134.stdout: "resharding" 2024-08-18T17:19:09.808 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:19:09.808 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:19:09.808 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:19:09.808 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "10090ccf-aad6-416d-9775-df8b7b27345a", 2024-08-18T17:19:09.808 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "test-zone2", 2024-08-18T17:19:09.808 INFO:teuthology.orchestra.run.smithi134.stdout: "endpoints": [ 2024-08-18T17:19:09.808 INFO:teuthology.orchestra.run.smithi134.stdout: "http://smithi193.front.sepia.ceph.com:8000" 2024-08-18T17:19:09.808 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:19:09.809 INFO:teuthology.orchestra.run.smithi134.stdout: "log_meta": false, 2024-08-18T17:19:09.809 INFO:teuthology.orchestra.run.smithi134.stdout: "log_data": true, 2024-08-18T17:19:09.809 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_index_max_shards": 11, 2024-08-18T17:19:09.809 INFO:teuthology.orchestra.run.smithi134.stdout: "read_only": false, 2024-08-18T17:19:09.809 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_type": "", 2024-08-18T17:19:09.809 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from_all": true, 2024-08-18T17:19:09.809 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from": [], 2024-08-18T17:19:09.809 INFO:teuthology.orchestra.run.smithi134.stdout: "redirect_zone": "", 2024-08-18T17:19:09.809 INFO:teuthology.orchestra.run.smithi134.stdout: "supported_features": [ 2024-08-18T17:19:09.809 INFO:teuthology.orchestra.run.smithi134.stdout: "compress-encrypted", 2024-08-18T17:19:09.809 INFO:teuthology.orchestra.run.smithi134.stdout: "resharding" 2024-08-18T17:19:09.809 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:19:09.809 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:19:09.809 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:19:09.810 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T17:19:09.810 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "test-zone1", 2024-08-18T17:19:09.810 INFO:teuthology.orchestra.run.smithi134.stdout: "endpoints": [ 2024-08-18T17:19:09.810 INFO:teuthology.orchestra.run.smithi134.stdout: "http://smithi134.front.sepia.ceph.com:8000" 2024-08-18T17:19:09.810 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:19:09.810 INFO:teuthology.orchestra.run.smithi134.stdout: "log_meta": false, 2024-08-18T17:19:09.810 INFO:teuthology.orchestra.run.smithi134.stdout: "log_data": true, 2024-08-18T17:19:09.810 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_index_max_shards": 11, 2024-08-18T17:19:09.810 INFO:teuthology.orchestra.run.smithi134.stdout: "read_only": false, 2024-08-18T17:19:09.810 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_type": "", 2024-08-18T17:19:09.810 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from_all": true, 2024-08-18T17:19:09.810 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from": [], 2024-08-18T17:19:09.811 INFO:teuthology.orchestra.run.smithi134.stdout: "redirect_zone": "", 2024-08-18T17:19:09.811 INFO:teuthology.orchestra.run.smithi134.stdout: "supported_features": [ 2024-08-18T17:19:09.811 INFO:teuthology.orchestra.run.smithi134.stdout: "compress-encrypted", 2024-08-18T17:19:09.811 INFO:teuthology.orchestra.run.smithi134.stdout: "resharding" 2024-08-18T17:19:09.811 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:19:09.811 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:19:09.811 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:19:09.811 INFO:teuthology.orchestra.run.smithi134.stdout: "placement_targets": [ 2024-08-18T17:19:09.811 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:19:09.811 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "default-placement", 2024-08-18T17:19:09.812 INFO:teuthology.orchestra.run.smithi134.stdout: "tags": [], 2024-08-18T17:19:09.812 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_classes": [ 2024-08-18T17:19:09.812 INFO:teuthology.orchestra.run.smithi134.stdout: "STANDARD" 2024-08-18T17:19:09.812 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:19:09.812 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:19:09.812 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:19:09.812 INFO:teuthology.orchestra.run.smithi134.stdout: "default_placement": "default-placement", 2024-08-18T17:19:09.812 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_id": "2827c6f0-bf0d-4836-944c-9e7fcf22457b", 2024-08-18T17:19:09.812 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_policy": { 2024-08-18T17:19:09.812 INFO:teuthology.orchestra.run.smithi134.stdout: "groups": [ 2024-08-18T17:19:09.812 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:19:09.812 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "sync-group", 2024-08-18T17:19:09.812 INFO:teuthology.orchestra.run.smithi134.stdout: "data_flow": { 2024-08-18T17:19:09.813 INFO:teuthology.orchestra.run.smithi134.stdout: "symmetrical": [ 2024-08-18T17:19:09.813 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:19:09.813 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "sync-flow1", 2024-08-18T17:19:09.813 INFO:teuthology.orchestra.run.smithi134.stdout: "zones": [ 2024-08-18T17:19:09.819 INFO:teuthology.orchestra.run.smithi134.stdout: "10090ccf-aad6-416d-9775-df8b7b27345a", 2024-08-18T17:19:09.819 INFO:teuthology.orchestra.run.smithi134.stdout: "96bf8502-2f39-4d53-8196-3378c28da3b4" 2024-08-18T17:19:09.819 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:19:09.819 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:19:09.819 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:19:09.819 INFO:teuthology.orchestra.run.smithi134.stdout: "directional": [ 2024-08-18T17:19:09.819 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:19:09.820 INFO:teuthology.orchestra.run.smithi134.stdout: "source_zone": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T17:19:09.820 INFO:teuthology.orchestra.run.smithi134.stdout: "dest_zone": "10090ccf-aad6-416d-9775-df8b7b27345a" 2024-08-18T17:19:09.820 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:19:09.820 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:19:09.820 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:19:09.820 INFO:teuthology.orchestra.run.smithi134.stdout: "pipes": [ 2024-08-18T17:19:09.820 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:19:09.820 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "sync-pipe", 2024-08-18T17:19:09.820 INFO:teuthology.orchestra.run.smithi134.stdout: "source": { 2024-08-18T17:19:09.820 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket": "kibgmb-91", 2024-08-18T17:19:09.820 INFO:teuthology.orchestra.run.smithi134.stdout: "zones": [ 2024-08-18T17:19:09.820 INFO:teuthology.orchestra.run.smithi134.stdout: "10090ccf-aad6-416d-9775-df8b7b27345a", 2024-08-18T17:19:09.821 INFO:teuthology.orchestra.run.smithi134.stdout: "96bf8502-2f39-4d53-8196-3378c28da3b4" 2024-08-18T17:19:09.821 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:19:09.821 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:19:09.821 INFO:teuthology.orchestra.run.smithi134.stdout: "dest": { 2024-08-18T17:19:09.821 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket": "kibgmb-91", 2024-08-18T17:19:09.821 INFO:teuthology.orchestra.run.smithi134.stdout: "zones": [ 2024-08-18T17:19:09.821 INFO:teuthology.orchestra.run.smithi134.stdout: "10090ccf-aad6-416d-9775-df8b7b27345a", 2024-08-18T17:19:09.821 INFO:teuthology.orchestra.run.smithi134.stdout: "96bf8502-2f39-4d53-8196-3378c28da3b4" 2024-08-18T17:19:09.821 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:19:09.821 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:19:09.821 INFO:teuthology.orchestra.run.smithi134.stdout: "params": { 2024-08-18T17:19:09.821 INFO:teuthology.orchestra.run.smithi134.stdout: "source": { 2024-08-18T17:19:09.821 INFO:teuthology.orchestra.run.smithi134.stdout: "filter": { 2024-08-18T17:19:09.821 INFO:teuthology.orchestra.run.smithi134.stdout: "tags": [] 2024-08-18T17:19:09.822 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:19:09.822 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:19:09.822 INFO:teuthology.orchestra.run.smithi134.stdout: "dest": {}, 2024-08-18T17:19:09.822 INFO:teuthology.orchestra.run.smithi134.stdout: "priority": 0, 2024-08-18T17:19:09.822 INFO:teuthology.orchestra.run.smithi134.stdout: "mode": "system", 2024-08-18T17:19:09.822 INFO:teuthology.orchestra.run.smithi134.stdout: "user": "" 2024-08-18T17:19:09.822 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:19:09.822 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:19:09.822 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:19:09.822 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "enabled" 2024-08-18T17:19:09.822 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:19:09.822 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:19:09.822 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:19:09.823 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled_features": [ 2024-08-18T17:19:09.823 INFO:teuthology.orchestra.run.smithi134.stdout: "resharding" 2024-08-18T17:19:09.823 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:19:09.823 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:19:09.823 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:19:09.823 INFO:teuthology.orchestra.run.smithi134.stdout: "short_zone_ids": [ 2024-08-18T17:19:09.823 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:19:09.823 INFO:teuthology.orchestra.run.smithi134.stdout: "key": "018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T17:19:09.823 INFO:teuthology.orchestra.run.smithi134.stdout: "val": 462036791 2024-08-18T17:19:09.824 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:19:09.824 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:19:09.824 INFO:teuthology.orchestra.run.smithi134.stdout: "key": "10090ccf-aad6-416d-9775-df8b7b27345a", 2024-08-18T17:19:09.824 INFO:teuthology.orchestra.run.smithi134.stdout: "val": 874289624 2024-08-18T17:19:09.824 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:19:09.824 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:19:09.825 INFO:teuthology.orchestra.run.smithi134.stdout: "key": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T17:19:09.825 INFO:teuthology.orchestra.run.smithi134.stdout: "val": 1721399886 2024-08-18T17:19:09.825 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:19:09.825 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:19:09.825 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:19:09.825 INFO:teuthology.orchestra.run.smithi134.stdout: "master_zonegroup": "test-zonegroup", 2024-08-18T17:19:09.825 INFO:teuthology.orchestra.run.smithi134.stdout: "master_zone": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T17:19:09.825 INFO:teuthology.orchestra.run.smithi134.stdout: "period_config": { 2024-08-18T17:19:09.825 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_quota": { 2024-08-18T17:19:09.825 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false, 2024-08-18T17:19:09.825 INFO:teuthology.orchestra.run.smithi134.stdout: "check_on_raw": false, 2024-08-18T17:19:09.825 INFO:teuthology.orchestra.run.smithi134.stdout: "max_size": -1, 2024-08-18T17:19:09.825 INFO:teuthology.orchestra.run.smithi134.stdout: "max_size_kb": 0, 2024-08-18T17:19:09.826 INFO:teuthology.orchestra.run.smithi134.stdout: "max_objects": -1 2024-08-18T17:19:09.826 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:19:09.826 INFO:teuthology.orchestra.run.smithi134.stdout: "user_quota": { 2024-08-18T17:19:09.826 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false, 2024-08-18T17:19:09.826 INFO:teuthology.orchestra.run.smithi134.stdout: "check_on_raw": false, 2024-08-18T17:19:09.826 INFO:teuthology.orchestra.run.smithi134.stdout: "max_size": -1, 2024-08-18T17:19:09.826 INFO:teuthology.orchestra.run.smithi134.stdout: "max_size_kb": 0, 2024-08-18T17:19:09.826 INFO:teuthology.orchestra.run.smithi134.stdout: "max_objects": -1 2024-08-18T17:19:09.826 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:19:09.826 INFO:teuthology.orchestra.run.smithi134.stdout: "user_ratelimit": { 2024-08-18T17:19:09.826 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_ops": 0, 2024-08-18T17:19:09.826 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_ops": 0, 2024-08-18T17:19:09.827 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_bytes": 0, 2024-08-18T17:19:09.827 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_bytes": 0, 2024-08-18T17:19:09.827 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false 2024-08-18T17:19:09.827 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:19:09.827 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_ratelimit": { 2024-08-18T17:19:09.827 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_ops": 0, 2024-08-18T17:19:09.827 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_ops": 0, 2024-08-18T17:19:09.827 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_bytes": 0, 2024-08-18T17:19:09.827 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_bytes": 0, 2024-08-18T17:19:09.827 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false 2024-08-18T17:19:09.827 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:19:09.827 INFO:teuthology.orchestra.run.smithi134.stdout: "anonymous_ratelimit": { 2024-08-18T17:19:09.827 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_ops": 0, 2024-08-18T17:19:09.828 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_ops": 0, 2024-08-18T17:19:09.828 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_bytes": 0, 2024-08-18T17:19:09.828 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_bytes": 0, 2024-08-18T17:19:09.828 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false 2024-08-18T17:19:09.828 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:19:09.828 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:19:09.828 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_id": "2827c6f0-bf0d-4836-944c-9e7fcf22457b", 2024-08-18T17:19:09.828 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_name": "test-realm", 2024-08-18T17:19:09.828 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:19:09.828 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T17:19:09.830 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'info', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:19:09.830 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'info', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:19:09.830 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync info --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:19:09.876 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:19:09.876 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:19:09.997 INFO:teuthology.orchestra.run.smithi134.stdout:{"sources":[{"id":"sync-pipe","source":{"zone":"test-zone2","bucket":"kibgmb-91"},"dest":{"zone":"test-zone1","bucket":"kibgmb-91"},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"dests":[{"id":"sync-pipe","source":{"zone":"test-zone1","bucket":"kibgmb-91"},"dest":{"zone":"test-zone2","bucket":"kibgmb-91"},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"hints":{"sources":[],"dests":[]},"resolved-hints-1":{"sources":[],"dests":[]},"resolved-hints":{"sources":[],"dests":[]}} 2024-08-18T17:19:10.187 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:19:10.187 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:19:10.187 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:19:10.253 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:19:10.253 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:19:10.335 INFO:teuthology.orchestra.run.smithi134.stderr:No --period given, using current period=01050e30-179b-43be-bdbb-3bb23cf37ae8 2024-08-18T17:19:10.394 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T17:19:10.395 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:19:10.395 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:19:10.395 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T17:19:10.395 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:19:10.395 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:19:10.395 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:19:10.395 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T17:19:10.395 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:19:10.395 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:19:10.395 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:19:10.395 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:42:47.672461Z" 2024-08-18T17:19:10.395 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:19:10.395 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:19:10.396 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1724001538.454480_1552.1", 2024-08-18T17:19:10.396 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T17:18:58.454480Z" 2024-08-18T17:19:10.396 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:19:10.396 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T17:19:10.397 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-08-18T17:19:10.397 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:19:10.397 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:19:10.397 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:19:10.448 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:19:10.449 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:19:10.484 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:19:10.483+0000 7f32a8919880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:19:10.485 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:19:10.484+0000 7f32a8919880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:19:10.498 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:19:10.498 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:19:10.498 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:19:10.499 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:19:10.499 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:19:10.499 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:19:10.499 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:19:10.499 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:19:10.499 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:19:10.499 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:19:10.499 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:19:10.499 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:19:10.499 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:19:10.499 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:19:10.499 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:19:10.499 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:19:10.500 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:19:10.500 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:19:10.500 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:19:10.500 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:19:10.500 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:19:10.500 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:19:10.500 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:19:10.500 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:19:10.500 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:19:10.500 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:19:10.500 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:19:10.500 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:19:10.500 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:19:10.500 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:19:10.501 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:19:10.501 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:19:10.501 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:19:10.501 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:19:10.501 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:19:10.501 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:19:10.501 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:19:10.501 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:19:10.501 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:19:10.501 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:19:10.501 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:19:10.501 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:19:10.501 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:19:10.501 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:19:10.502 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:19:10.502 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:19:10.502 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:19:10.502 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:19:10.502 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:19:10.502 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001538.454480_1552.1", 2024-08-18T17:19:10.502 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:19:10.502 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:19:10.502 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:19:10.502 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:18:58.454480Z", 2024-08-18T17:19:10.502 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:19:10.502 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:19:10.502 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:19:10.503 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:19:10.503 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:19:10.503 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:19:10.503 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:19:10.503 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:19:10.503 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:19:10.503 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:19:10Z" 2024-08-18T17:19:10.503 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:19:10.504 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-08-18T17:19:10.504 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-08-18T17:19:10.504 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:19:10.504 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:19:10.505 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:19:10.569 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:19:10.570 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:19:10.628 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:19:10.627+0000 7fb007b24880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:19:10.629 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:19:10.629+0000 7fb007b24880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:19:10.680 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T17:19:10.680 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T17:19:10.680 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T17:19:10.681 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T17:19:10.681 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T17:19:10.681 INFO:teuthology.orchestra.run.smithi134.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:19:10.681 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:19:10.681 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:19:10.681 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T17:19:10.681 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:19:10.681 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T17:19:10.681 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:19:10.681 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:19:10.681 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:19:10.681 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:19:10.682 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 1, 2024-08-18T17:19:10.682 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:19:10.682 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:19:10.682 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T17:19:10.682 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:19:10.682 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:19:10.682 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:19:10.682 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T17:19:10.682 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:19:10.682 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:19:10.682 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:19:10.683 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:19:10.683 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:19:10.683 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:19:10.683 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T17:19:10.683 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T17:19:10.683 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:19:10.683 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:19:10.683 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:19:10.683 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T17:19:10.683 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:19:10.683 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:19:10.683 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:19:10.684 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:19:10.684 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:19:10.684 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:19:10.684 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:19:10.684 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:19:10.684 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:19:10.684 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:19:10.684 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:19:10.684 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T17:19:10.684 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:19:10.684 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:19:10.684 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1724001538.454480_1552.1", 2024-08-18T17:19:10.685 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:19:10.685 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:19:10.685 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:19:10.685 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T17:18:58.454480Z", 2024-08-18T17:19:10.685 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:19:10.685 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:19:10.685 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:19:10.685 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:19:10.685 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:19:10.685 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T17:19:10.685 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 1, 2024-08-18T17:19:10.685 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 1 2024-08-18T17:19:10.685 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:19:10.686 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T17:19:10Z" 2024-08-18T17:19:10.686 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T17:19:10.687 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-08-18T17:19:10.687 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:19:10.687 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:19:10.687 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:19:10.738 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:19:10.739 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:19:10.866 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T17:19:10.866 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:19:10.866 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000051734", 2024-08-18T17:19:10.866 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T17:19:10.076905Z" 2024-08-18T17:19:10.866 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:19:10.866 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:19:10.866 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:19:10.866 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:19:10.867 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:19:10.867 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:19:10.867 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000059137", 2024-08-18T17:19:10.867 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T17:19:10.125128Z" 2024-08-18T17:19:10.867 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:19:10.867 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:19:10.867 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:19:10.867 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:19:10.867 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:19:10.867 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T17:19:10.868 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-08-18T17:19:10.868 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:19:10.869 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:19:10.869 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:19:10.918 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:19:10.919 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:19:10.964 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:19:10.965 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:19:10.965 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:19:10.965 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:19:10.965 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:19:10.965 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:19:10.965 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:19:10.966 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:19:10.966 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:19:10.966 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:19:10.966 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:19:10.966 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:19:10.966 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000051594", 2024-08-18T17:19:10.967 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:19:10.967 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:19:10.967 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:19:10.967 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:54.270479Z" 2024-08-18T17:19:10.967 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:19:10.967 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:19:10.968 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:19:10.968 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:19:10.968 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:19:10.968 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:19:10.968 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:19:10.968 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:19:10.968 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:19:10.969 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:19:10.969 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:19:10.969 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:19:10.969 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:19:10.969 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:19:10.969 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:19:10.969 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:19:10.970 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:19:10.970 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000058857", 2024-08-18T17:19:10.970 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:19:10.970 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:19:10.970 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:19:10.970 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:54.289199Z" 2024-08-18T17:19:10.971 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:19:10.971 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:19:10.971 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:19:10.971 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:19:10.971 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:19:10.971 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:19:10.971 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052068", 2024-08-18T17:19:10.972 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:19:10.972 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:19:10.972 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:19:10.972 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:05:27.099520Z" 2024-08-18T17:19:10.972 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:19:10.972 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:19:10.972 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:19:10.972 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:19:10.973 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:19:10.973 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:19:10.973 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:19:10.973 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:19:10.973 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:19:10Z" 2024-08-18T17:19:10.973 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:19:10.975 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000051734 target=00000000000000000000:00000000000000051594, shard=2 master=00000000000000000000:00000000000000059137 target=00000000000000000000:00000000000000058857, shard=3 master=00000000000000000000:00000000000000052210 target=00000000000000000000:00000000000000052068 2024-08-18T17:19:15.976 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:19:15.977 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:19:15.977 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:19:16.034 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:19:16.034 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:19:16.085 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:19:16.085 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:19:16.085 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:19:16.085 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:19:16.085 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:19:16.085 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:19:16.085 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:19:16.085 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:19:16.085 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:19:16.085 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:19:16.086 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:19:16.086 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:19:16.086 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000051594", 2024-08-18T17:19:16.086 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:19:16.086 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:19:16.086 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:19:16.086 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:54.270479Z" 2024-08-18T17:19:16.086 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:19:16.086 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:19:16.086 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:19:16.086 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:19:16.086 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:19:16.086 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:19:16.087 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:19:16.087 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:19:16.087 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:19:16.087 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:19:16.087 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:19:16.087 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:19:16.087 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:19:16.087 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:19:16.087 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:19:16.088 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:19:16.088 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:19:16.088 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000058857", 2024-08-18T17:19:16.088 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:19:16.088 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:19:16.088 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:19:16.088 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:54.289199Z" 2024-08-18T17:19:16.088 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:19:16.088 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:19:16.088 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:19:16.088 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:19:16.088 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:19:16.089 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:19:16.089 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052068", 2024-08-18T17:19:16.089 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:19:16.089 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:19:16.089 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:19:16.089 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:05:27.099520Z" 2024-08-18T17:19:16.089 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:19:16.089 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:19:16.089 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:19:16.089 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:19:16.089 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:19:16.089 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:19:16.089 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:19:16.089 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:19:16.090 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:19:16Z" 2024-08-18T17:19:16.090 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:19:16.091 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000051734 target=00000000000000000000:00000000000000051594, shard=2 master=00000000000000000000:00000000000000059137 target=00000000000000000000:00000000000000058857, shard=3 master=00000000000000000000:00000000000000052210 target=00000000000000000000:00000000000000052068 2024-08-18T17:19:21.092 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:19:21.092 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:19:21.093 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:19:21.150 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:19:21.150 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:19:21.200 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:19:21.200 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:19:21.200 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:19:21.200 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:19:21.201 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:19:21.201 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:19:21.201 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:19:21.201 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:19:21.201 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:19:21.201 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:19:21.201 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:19:21.201 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:19:21.201 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000051594", 2024-08-18T17:19:21.201 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:19:21.201 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:19:21.201 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:19:21.201 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:54.270479Z" 2024-08-18T17:19:21.202 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:19:21.202 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:19:21.202 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:19:21.202 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:19:21.202 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:19:21.202 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:19:21.202 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:19:21.202 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:19:21.202 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:19:21.202 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:19:21.202 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:19:21.202 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:19:21.202 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:19:21.202 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:19:21.203 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:19:21.203 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:19:21.203 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:19:21.203 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000058857", 2024-08-18T17:19:21.203 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:19:21.203 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:19:21.203 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:19:21.203 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:54.289199Z" 2024-08-18T17:19:21.203 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:19:21.203 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:19:21.203 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:19:21.204 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:19:21.204 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:19:21.204 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:19:21.204 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052068", 2024-08-18T17:19:21.204 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:19:21.204 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:19:21.204 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:19:21.204 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:05:27.099520Z" 2024-08-18T17:19:21.204 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:19:21.204 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:19:21.204 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:19:21.204 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:19:21.204 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:19:21.204 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:19:21.204 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:19:21.205 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:19:21.205 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:19:21Z" 2024-08-18T17:19:21.205 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:19:21.206 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000051734 target=00000000000000000000:00000000000000051594, shard=2 master=00000000000000000000:00000000000000059137 target=00000000000000000000:00000000000000058857, shard=3 master=00000000000000000000:00000000000000052210 target=00000000000000000000:00000000000000052068 2024-08-18T17:19:26.207 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:19:26.207 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:19:26.208 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:19:26.264 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:19:26.264 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:19:26.314 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:19:26.314 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:19:26.314 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:19:26.314 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:19:26.314 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:19:26.314 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:19:26.314 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:19:26.314 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:19:26.314 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:19:26.314 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:19:26.315 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:19:26.315 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:19:26.315 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000051594", 2024-08-18T17:19:26.315 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:19:26.315 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:19:26.315 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:19:26.315 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:54.270479Z" 2024-08-18T17:19:26.315 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:19:26.315 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:19:26.316 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:19:26.316 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:19:26.316 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:19:26.316 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:19:26.316 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:19:26.316 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:19:26.316 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:19:26.316 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:19:26.316 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:19:26.316 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:19:26.316 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:19:26.317 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:19:26.317 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:19:26.317 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:19:26.317 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:19:26.317 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000058857", 2024-08-18T17:19:26.317 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:19:26.317 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:19:26.317 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:19:26.317 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:54.289199Z" 2024-08-18T17:19:26.317 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:19:26.318 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:19:26.318 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:19:26.318 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:19:26.318 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:19:26.318 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:19:26.318 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052068", 2024-08-18T17:19:26.319 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:19:26.319 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:19:26.319 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:19:26.319 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:05:27.099520Z" 2024-08-18T17:19:26.319 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:19:26.319 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:19:26.319 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:19:26.319 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:19:26.319 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:19:26.319 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:19:26.319 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:19:26.319 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:19:26.319 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:19:26Z" 2024-08-18T17:19:26.319 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:19:26.320 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000051734 target=00000000000000000000:00000000000000051594, shard=2 master=00000000000000000000:00000000000000059137 target=00000000000000000000:00000000000000058857, shard=3 master=00000000000000000000:00000000000000052210 target=00000000000000000000:00000000000000052068 2024-08-18T17:19:31.322 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:19:31.322 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:19:31.322 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:19:31.379 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:19:31.379 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:19:31.430 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:19:31.430 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:19:31.430 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:19:31.430 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:19:31.430 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:19:31.430 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:19:31.430 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:19:31.430 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:19:31.430 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:19:31.431 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:19:31.431 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:19:31.431 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:19:31.431 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000051594", 2024-08-18T17:19:31.431 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:19:31.431 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:19:31.431 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:19:31.431 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:54.270479Z" 2024-08-18T17:19:31.431 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:19:31.431 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:19:31.431 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:19:31.431 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:19:31.431 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:19:31.432 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:19:31.432 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:19:31.432 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:19:31.432 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:19:31.432 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:19:31.432 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:19:31.432 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:19:31.432 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:19:31.432 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:19:31.432 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:19:31.432 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:19:31.432 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:19:31.433 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000058857", 2024-08-18T17:19:31.433 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:19:31.433 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:19:31.433 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:19:31.433 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:54.289199Z" 2024-08-18T17:19:31.433 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:19:31.433 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:19:31.433 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:19:31.433 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:19:31.433 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:19:31.433 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:19:31.433 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052068", 2024-08-18T17:19:31.433 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:19:31.433 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:19:31.434 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:19:31.434 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:05:27.099520Z" 2024-08-18T17:19:31.434 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:19:31.435 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:19:31.435 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:19:31.435 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:19:31.435 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:19:31.435 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:19:31.436 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:19:31.436 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:19:31.436 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:19:31Z" 2024-08-18T17:19:31.436 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:19:31.437 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000051734 target=00000000000000000000:00000000000000051594, shard=2 master=00000000000000000000:00000000000000059137 target=00000000000000000000:00000000000000058857, shard=3 master=00000000000000000000:00000000000000052210 target=00000000000000000000:00000000000000052068 2024-08-18T17:19:36.438 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:19:36.438 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:19:36.439 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:19:36.495 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:19:36.495 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:19:36.546 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:19:36.546 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:19:36.546 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:19:36.547 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:19:36.547 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:19:36.547 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:19:36.547 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:19:36.547 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:19:36.547 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:19:36.547 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:19:36.547 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:19:36.547 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:19:36.547 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000051594", 2024-08-18T17:19:36.547 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:19:36.547 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:19:36.548 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:19:36.548 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:54.270479Z" 2024-08-18T17:19:36.548 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:19:36.548 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:19:36.548 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:19:36.548 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:19:36.548 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:19:36.548 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:19:36.548 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:19:36.548 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:19:36.548 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:19:36.548 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:19:36.548 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:19:36.549 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:19:36.549 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:19:36.549 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:19:36.549 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:19:36.549 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:19:36.549 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:19:36.549 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000058857", 2024-08-18T17:19:36.549 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:19:36.549 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:19:36.549 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:19:36.549 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:54.289199Z" 2024-08-18T17:19:36.549 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:19:36.549 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:19:36.550 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:19:36.550 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:19:36.550 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:19:36.550 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:19:36.550 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052068", 2024-08-18T17:19:36.550 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:19:36.550 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:19:36.550 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:19:36.550 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:05:27.099520Z" 2024-08-18T17:19:36.550 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:19:36.550 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:19:36.551 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:19:36.551 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:19:36.551 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:19:36.551 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:19:36.551 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:19:36.551 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:19:36.551 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:19:36Z" 2024-08-18T17:19:36.551 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:19:36.552 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000051734 target=00000000000000000000:00000000000000051594, shard=2 master=00000000000000000000:00000000000000059137 target=00000000000000000000:00000000000000058857, shard=3 master=00000000000000000000:00000000000000052210 target=00000000000000000000:00000000000000052068 2024-08-18T17:19:41.553 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:19:41.553 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:19:41.554 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:19:41.611 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:19:41.611 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:19:41.661 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:19:41.661 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:19:41.661 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:19:41.662 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:19:41.662 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:19:41.662 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:19:41.662 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:19:41.662 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:19:41.662 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:19:41.662 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:19:41.662 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:19:41.662 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:19:41.662 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000051594", 2024-08-18T17:19:41.662 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:19:41.662 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:19:41.663 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:19:41.663 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:54.270479Z" 2024-08-18T17:19:41.663 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:19:41.663 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:19:41.663 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:19:41.663 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:19:41.663 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:19:41.663 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:19:41.663 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:19:41.663 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:19:41.663 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:19:41.663 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:19:41.663 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:19:41.663 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:19:41.664 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:19:41.664 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:19:41.664 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:19:41.664 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:19:41.664 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:19:41.664 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000058857", 2024-08-18T17:19:41.664 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:19:41.664 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:19:41.664 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:19:41.664 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:54.289199Z" 2024-08-18T17:19:41.664 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:19:41.664 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:19:41.664 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:19:41.665 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:19:41.665 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:19:41.665 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:19:41.665 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052068", 2024-08-18T17:19:41.665 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:19:41.665 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:19:41.665 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:19:41.665 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:05:27.099520Z" 2024-08-18T17:19:41.665 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:19:41.665 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:19:41.665 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:19:41.665 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:19:41.665 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:19:41.666 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:19:41.666 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:19:41.666 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:19:41.666 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:19:41Z" 2024-08-18T17:19:41.666 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:19:41.667 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000051734 target=00000000000000000000:00000000000000051594, shard=2 master=00000000000000000000:00000000000000059137 target=00000000000000000000:00000000000000058857, shard=3 master=00000000000000000000:00000000000000052210 target=00000000000000000000:00000000000000052068 2024-08-18T17:19:46.667 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:19:46.668 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:19:46.668 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:19:46.726 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:19:46.726 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:19:46.776 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:19:46.776 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:19:46.776 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:19:46.777 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:19:46.777 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:19:46.777 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:19:46.777 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:19:46.777 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:19:46.777 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:19:46.778 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:19:46.778 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:19:46.778 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:19:46.778 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000051594", 2024-08-18T17:19:46.778 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:19:46.778 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:19:46.778 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:19:46.778 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:54.270479Z" 2024-08-18T17:19:46.779 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:19:46.779 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:19:46.779 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:19:46.779 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:19:46.779 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:19:46.779 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:19:46.779 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:19:46.779 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:19:46.779 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:19:46.779 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:19:46.779 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:19:46.779 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:19:46.780 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:19:46.780 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:19:46.780 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:19:46.780 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:19:46.780 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:19:46.780 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000058857", 2024-08-18T17:19:46.780 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:19:46.780 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:19:46.780 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:19:46.780 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:54.289199Z" 2024-08-18T17:19:46.780 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:19:46.780 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:19:46.780 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:19:46.780 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:19:46.781 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:19:46.781 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:19:46.781 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052068", 2024-08-18T17:19:46.781 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:19:46.781 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:19:46.781 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:19:46.781 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:05:27.099520Z" 2024-08-18T17:19:46.781 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:19:46.781 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:19:46.781 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:19:46.781 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:19:46.781 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:19:46.781 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:19:46.782 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:19:46.782 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:19:46.782 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:19:46Z" 2024-08-18T17:19:46.782 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:19:46.785 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000051734 target=00000000000000000000:00000000000000051594, shard=2 master=00000000000000000000:00000000000000059137 target=00000000000000000000:00000000000000058857, shard=3 master=00000000000000000000:00000000000000052210 target=00000000000000000000:00000000000000052068 2024-08-18T17:19:51.786 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:19:51.786 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:19:51.787 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:19:51.847 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:19:51.847 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:19:51.897 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:19:51.897 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:19:51.897 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:19:51.898 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:19:51.898 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:19:51.898 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:19:51.898 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:19:51.898 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:19:51.898 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:19:51.898 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:19:51.898 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:19:51.898 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:19:51.898 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000051594", 2024-08-18T17:19:51.898 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:19:51.898 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:19:51.898 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:19:51.899 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:54.270479Z" 2024-08-18T17:19:51.899 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:19:51.899 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:19:51.899 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:19:51.899 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:19:51.899 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:19:51.899 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:19:51.899 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:19:51.899 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:19:51.899 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:19:51.899 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:19:51.899 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:19:51.899 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:19:51.899 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:19:51.900 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:19:51.900 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:19:51.900 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:19:51.900 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:19:51.900 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000058857", 2024-08-18T17:19:51.900 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:19:51.900 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:19:51.900 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:19:51.900 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:54.289199Z" 2024-08-18T17:19:51.900 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:19:51.900 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:19:51.900 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:19:51.900 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:19:51.901 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:19:51.901 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:19:51.901 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052068", 2024-08-18T17:19:51.901 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:19:51.901 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:19:51.901 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:19:51.901 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:05:27.099520Z" 2024-08-18T17:19:51.901 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:19:51.901 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:19:51.901 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:19:51.901 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:19:51.901 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:19:51.901 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:19:51.902 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:19:51.902 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:19:51.902 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:19:51Z" 2024-08-18T17:19:51.902 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:19:51.903 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000051734 target=00000000000000000000:00000000000000051594, shard=2 master=00000000000000000000:00000000000000059137 target=00000000000000000000:00000000000000058857, shard=3 master=00000000000000000000:00000000000000052210 target=00000000000000000000:00000000000000052068 2024-08-18T17:19:56.904 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:19:56.904 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:19:56.905 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:19:56.963 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:19:56.963 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:19:57.014 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:19:57.014 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:19:57.014 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:19:57.015 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:19:57.015 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:19:57.015 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:19:57.015 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:19:57.015 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:19:57.015 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:19:57.015 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:19:57.015 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:19:57.015 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:19:57.015 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000051594", 2024-08-18T17:19:57.015 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:19:57.016 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:19:57.016 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:19:57.016 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:54.270479Z" 2024-08-18T17:19:57.016 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:19:57.016 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:19:57.016 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:19:57.016 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:19:57.016 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:19:57.016 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:19:57.016 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:19:57.016 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:19:57.016 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:19:57.016 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:19:57.017 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:19:57.017 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:19:57.017 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:19:57.017 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:19:57.017 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:19:57.017 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:19:57.017 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:19:57.017 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000058857", 2024-08-18T17:19:57.017 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:19:57.017 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:19:57.017 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:19:57.017 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:54.289199Z" 2024-08-18T17:19:57.018 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:19:57.018 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:19:57.018 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:19:57.018 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:19:57.018 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:19:57.018 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:19:57.018 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052068", 2024-08-18T17:19:57.018 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:19:57.018 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:19:57.018 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:19:57.018 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:05:27.099520Z" 2024-08-18T17:19:57.018 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:19:57.019 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:19:57.019 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:19:57.019 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:19:57.019 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:19:57.019 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:19:57.019 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:19:57.019 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:19:57.019 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:19:57Z" 2024-08-18T17:19:57.019 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:19:57.020 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000051734 target=00000000000000000000:00000000000000051594, shard=2 master=00000000000000000000:00000000000000059137 target=00000000000000000000:00000000000000058857, shard=3 master=00000000000000000000:00000000000000052210 target=00000000000000000000:00000000000000052068 2024-08-18T17:20:02.022 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:20:02.022 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:20:02.022 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:20:02.079 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:20:02.079 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:20:02.134 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:20:02.134 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:20:02.135 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:20:02.135 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:20:02.135 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:20:02.135 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:20:02.135 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:20:02.135 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:20:02.135 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:20:02.135 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:20:02.135 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:20:02.135 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:20:02.135 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000051594", 2024-08-18T17:20:02.136 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:20:02.136 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:20:02.136 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:20:02.136 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:54.270479Z" 2024-08-18T17:20:02.136 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:20:02.136 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:20:02.136 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:20:02.136 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:20:02.136 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:20:02.136 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:20:02.136 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:20:02.136 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:20:02.136 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:20:02.137 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:20:02.137 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:20:02.137 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:20:02.137 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:20:02.137 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:20:02.137 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:20:02.137 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:20:02.137 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:20:02.137 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000058857", 2024-08-18T17:20:02.137 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:20:02.137 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:20:02.137 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:20:02.137 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:54.289199Z" 2024-08-18T17:20:02.138 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:20:02.138 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:20:02.138 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:20:02.138 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:20:02.138 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:20:02.138 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:20:02.138 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052068", 2024-08-18T17:20:02.138 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:20:02.138 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:20:02.138 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:20:02.138 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:05:27.099520Z" 2024-08-18T17:20:02.138 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:20:02.139 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:20:02.139 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:20:02.139 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:20:02.139 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:20:02.139 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:20:02.139 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:20:02.139 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:20:02.139 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:20:02Z" 2024-08-18T17:20:02.139 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:20:02.140 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000051734 target=00000000000000000000:00000000000000051594, shard=2 master=00000000000000000000:00000000000000059137 target=00000000000000000000:00000000000000058857, shard=3 master=00000000000000000000:00000000000000052210 target=00000000000000000000:00000000000000052068 2024-08-18T17:20:07.142 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:20:07.142 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:20:07.142 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:20:07.199 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:20:07.199 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:20:07.251 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:20:07.252 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:20:07.252 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:20:07.252 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:20:07.252 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:20:07.252 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:20:07.252 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:20:07.252 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:20:07.252 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:20:07.252 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:20:07.252 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:20:07.253 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:20:07.253 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000051594", 2024-08-18T17:20:07.253 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:20:07.253 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:20:07.253 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:20:07.253 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:54.270479Z" 2024-08-18T17:20:07.253 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:20:07.253 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:20:07.253 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:20:07.253 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:20:07.253 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:20:07.253 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:20:07.254 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:20:07.254 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:20:07.254 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:20:07.254 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:20:07.254 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:20:07.254 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:20:07.254 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:20:07.254 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:20:07.254 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:20:07.254 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:20:07.254 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:20:07.255 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000058857", 2024-08-18T17:20:07.255 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:20:07.255 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:20:07.255 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:20:07.255 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:54.289199Z" 2024-08-18T17:20:07.255 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:20:07.255 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:20:07.255 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:20:07.255 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:20:07.255 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:20:07.255 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:20:07.255 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052068", 2024-08-18T17:20:07.256 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:20:07.256 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:20:07.256 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:20:07.256 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:05:27.099520Z" 2024-08-18T17:20:07.256 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:20:07.256 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:20:07.256 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:20:07.256 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:20:07.256 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:20:07.256 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:20:07.256 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:20:07.256 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:20:07.256 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:20:07Z" 2024-08-18T17:20:07.257 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:20:07.258 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000051734 target=00000000000000000000:00000000000000051594, shard=2 master=00000000000000000000:00000000000000059137 target=00000000000000000000:00000000000000058857, shard=3 master=00000000000000000000:00000000000000052210 target=00000000000000000000:00000000000000052068 2024-08-18T17:20:12.259 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:20:12.259 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:20:12.259 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:20:12.316 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:20:12.316 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:20:12.367 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:20:12.367 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:20:12.367 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:20:12.367 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:20:12.367 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:20:12.367 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:20:12.367 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:20:12.368 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:20:12.368 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:20:12.368 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:20:12.368 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:20:12.368 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:20:12.368 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000051594", 2024-08-18T17:20:12.368 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:20:12.368 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:20:12.368 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:20:12.368 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:54.270479Z" 2024-08-18T17:20:12.368 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:20:12.368 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:20:12.369 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:20:12.369 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:20:12.369 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:20:12.369 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:20:12.369 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:20:12.369 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:20:12.369 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:20:12.369 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:20:12.369 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:20:12.369 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:20:12.369 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:20:12.369 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:20:12.369 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:20:12.370 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:20:12.370 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:20:12.370 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000058857", 2024-08-18T17:20:12.370 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:20:12.370 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:20:12.370 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:20:12.370 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:54.289199Z" 2024-08-18T17:20:12.370 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:20:12.370 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:20:12.370 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:20:12.370 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:20:12.370 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:20:12.371 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:20:12.371 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052068", 2024-08-18T17:20:12.371 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:20:12.371 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:20:12.371 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:20:12.371 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:05:27.099520Z" 2024-08-18T17:20:12.371 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:20:12.371 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:20:12.371 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:20:12.371 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:20:12.371 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:20:12.371 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:20:12.371 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:20:12.372 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:20:12.372 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:20:12Z" 2024-08-18T17:20:12.372 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:20:12.373 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000051734 target=00000000000000000000:00000000000000051594, shard=2 master=00000000000000000000:00000000000000059137 target=00000000000000000000:00000000000000058857, shard=3 master=00000000000000000000:00000000000000052210 target=00000000000000000000:00000000000000052068 2024-08-18T17:20:17.374 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:20:17.374 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:20:17.374 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:20:17.431 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:20:17.431 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:20:17.483 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:20:17.483 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:20:17.483 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:20:17.483 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:20:17.483 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:20:17.484 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:20:17.484 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:20:17.484 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:20:17.484 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:20:17.484 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:20:17.484 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:20:17.484 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:20:17.484 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000051594", 2024-08-18T17:20:17.484 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:20:17.484 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:20:17.484 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:20:17.484 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:54.270479Z" 2024-08-18T17:20:17.484 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:20:17.484 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:20:17.485 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:20:17.485 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:20:17.485 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:20:17.485 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:20:17.485 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:20:17.485 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:20:17.485 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:20:17.485 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:20:17.485 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:20:17.485 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:20:17.485 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:20:17.485 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:20:17.485 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:20:17.486 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:20:17.486 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:20:17.486 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000058857", 2024-08-18T17:20:17.486 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:20:17.486 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:20:17.486 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:20:17.486 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:54.289199Z" 2024-08-18T17:20:17.486 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:20:17.486 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:20:17.486 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:20:17.486 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:20:17.486 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:20:17.487 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:20:17.487 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052068", 2024-08-18T17:20:17.487 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:20:17.487 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:20:17.487 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:20:17.487 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:05:27.099520Z" 2024-08-18T17:20:17.487 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:20:17.487 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:20:17.487 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:20:17.487 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:20:17.487 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:20:17.487 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:20:17.487 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:20:17.487 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:20:17.488 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:20:17Z" 2024-08-18T17:20:17.488 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:20:17.489 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000051734 target=00000000000000000000:00000000000000051594, shard=2 master=00000000000000000000:00000000000000059137 target=00000000000000000000:00000000000000058857, shard=3 master=00000000000000000000:00000000000000052210 target=00000000000000000000:00000000000000052068 2024-08-18T17:20:22.489 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:20:22.489 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:20:22.490 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:20:22.547 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:20:22.547 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:20:22.598 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:20:22.598 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:20:22.598 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:20:22.599 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:20:22.599 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:20:22.599 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:20:22.599 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:20:22.599 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:20:22.599 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:20:22.599 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:20:22.599 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:20:22.599 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:20:22.599 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000051594", 2024-08-18T17:20:22.599 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:20:22.599 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:20:22.599 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:20:22.600 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:54.270479Z" 2024-08-18T17:20:22.600 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:20:22.600 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:20:22.600 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:20:22.600 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:20:22.600 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:20:22.600 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:20:22.600 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:20:22.600 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:20:22.600 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:20:22.600 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:20:22.600 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:20:22.600 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:20:22.600 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:20:22.601 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:20:22.601 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:20:22.601 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:20:22.601 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:20:22.601 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000058857", 2024-08-18T17:20:22.601 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:20:22.601 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:20:22.601 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:20:22.601 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:54.289199Z" 2024-08-18T17:20:22.601 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:20:22.601 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:20:22.601 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:20:22.602 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:20:22.602 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:20:22.602 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:20:22.602 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052068", 2024-08-18T17:20:22.602 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:20:22.602 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:20:22.602 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:20:22.602 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:05:27.099520Z" 2024-08-18T17:20:22.602 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:20:22.602 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:20:22.602 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:20:22.602 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:20:22.603 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:20:22.603 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:20:22.603 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:20:22.603 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:20:22.603 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:20:22Z" 2024-08-18T17:20:22.603 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:20:22.604 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000051734 target=00000000000000000000:00000000000000051594, shard=2 master=00000000000000000000:00000000000000059137 target=00000000000000000000:00000000000000058857, shard=3 master=00000000000000000000:00000000000000052210 target=00000000000000000000:00000000000000052068 2024-08-18T17:20:27.605 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:20:27.605 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:20:27.605 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:20:27.661 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:20:27.661 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:20:27.711 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:20:27.711 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:20:27.711 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:20:27.712 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:20:27.712 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:20:27.712 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:20:27.712 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:20:27.712 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:20:27.712 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:20:27.712 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:20:27.712 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:20:27.712 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:20:27.712 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000051594", 2024-08-18T17:20:27.712 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:20:27.712 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:20:27.712 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:20:27.713 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:54.270479Z" 2024-08-18T17:20:27.713 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:20:27.713 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:20:27.713 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:20:27.713 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:20:27.713 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:20:27.713 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:20:27.713 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:20:27.713 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:20:27.713 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:20:27.713 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:20:27.713 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:20:27.713 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:20:27.714 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:20:27.714 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:20:27.714 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:20:27.714 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:20:27.714 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:20:27.714 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000058857", 2024-08-18T17:20:27.714 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:20:27.714 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:20:27.714 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:20:27.714 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:54.289199Z" 2024-08-18T17:20:27.714 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:20:27.714 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:20:27.714 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:20:27.715 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:20:27.715 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:20:27.715 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:20:27.715 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052068", 2024-08-18T17:20:27.715 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:20:27.715 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:20:27.715 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:20:27.715 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:05:27.099520Z" 2024-08-18T17:20:27.715 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:20:27.715 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:20:27.715 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:20:27.715 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:20:27.715 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:20:27.715 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:20:27.716 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:20:27.716 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:20:27.716 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:20:27Z" 2024-08-18T17:20:27.716 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:20:27.717 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000051734 target=00000000000000000000:00000000000000051594, shard=2 master=00000000000000000000:00000000000000059137 target=00000000000000000000:00000000000000058857, shard=3 master=00000000000000000000:00000000000000052210 target=00000000000000000000:00000000000000052068 2024-08-18T17:20:32.718 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:20:32.718 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:20:32.718 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:20:32.775 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:20:32.776 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:20:32.826 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:20:32.827 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:20:32.827 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:20:32.827 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:20:32.827 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:20:32.827 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:20:32.828 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:20:32.828 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:20:32.828 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:20:32.828 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:20:32.828 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:20:32.828 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:20:32.829 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000051594", 2024-08-18T17:20:32.829 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:20:32.829 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:20:32.829 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:20:32.829 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:54.270479Z" 2024-08-18T17:20:32.830 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:20:32.830 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:20:32.830 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:20:32.830 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:20:32.830 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:20:32.831 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:20:32.831 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:20:32.831 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:20:32.831 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:20:32.831 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:20:32.831 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:20:32.832 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:20:32.832 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:20:32.832 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:20:32.832 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:20:32.832 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:20:32.833 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:20:32.833 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000058857", 2024-08-18T17:20:32.833 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:20:32.834 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:20:32.834 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:20:32.834 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:54.289199Z" 2024-08-18T17:20:32.835 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:20:32.835 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:20:32.835 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:20:32.835 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:20:32.835 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:20:32.835 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:20:32.835 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052068", 2024-08-18T17:20:32.835 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:20:32.835 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:20:32.835 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:20:32.835 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:05:27.099520Z" 2024-08-18T17:20:32.835 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:20:32.836 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:20:32.836 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:20:32.836 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:20:32.836 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:20:32.836 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:20:32.836 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:20:32.836 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:20:32.836 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:20:32Z" 2024-08-18T17:20:32.836 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:20:32.838 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000051734 target=00000000000000000000:00000000000000051594, shard=2 master=00000000000000000000:00000000000000059137 target=00000000000000000000:00000000000000058857, shard=3 master=00000000000000000000:00000000000000052210 target=00000000000000000000:00000000000000052068 2024-08-18T17:20:37.839 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:20:37.839 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:20:37.840 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:20:37.896 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:20:37.896 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:20:37.947 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:20:37.947 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:20:37.947 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:20:37.947 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:20:37.947 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:20:37.948 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:20:37.948 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:20:37.948 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:20:37.948 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:20:37.948 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:20:37.948 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:20:37.948 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:20:37.948 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000051594", 2024-08-18T17:20:37.948 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:20:37.948 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:20:37.948 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:20:37.948 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:54.270479Z" 2024-08-18T17:20:37.948 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:20:37.948 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:20:37.949 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:20:37.949 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:20:37.949 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:20:37.949 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:20:37.949 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:20:37.949 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:20:37.949 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:20:37.949 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:20:37.949 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:20:37.949 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:20:37.949 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:20:37.949 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:20:37.949 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:20:37.950 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:20:37.950 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:20:37.950 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000058857", 2024-08-18T17:20:37.950 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:20:37.950 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:20:37.950 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:20:37.950 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:54.289199Z" 2024-08-18T17:20:37.950 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:20:37.950 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:20:37.950 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:20:37.950 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:20:37.950 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:20:37.950 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:20:37.951 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052068", 2024-08-18T17:20:37.951 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:20:37.951 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:20:37.951 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:20:37.951 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:05:27.099520Z" 2024-08-18T17:20:37.951 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:20:37.951 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:20:37.951 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:20:37.951 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:20:37.951 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:20:37.951 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:20:37.951 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:20:37.951 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:20:37.951 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:20:37Z" 2024-08-18T17:20:37.952 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:20:37.952 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000051734 target=00000000000000000000:00000000000000051594, shard=2 master=00000000000000000000:00000000000000059137 target=00000000000000000000:00000000000000058857, shard=3 master=00000000000000000000:00000000000000052210 target=00000000000000000000:00000000000000052068 2024-08-18T17:20:42.954 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:20:42.954 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:20:42.954 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:20:43.011 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:20:43.011 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:20:43.063 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:20:43.063 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:20:43.063 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:20:43.063 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:20:43.063 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:20:43.063 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:20:43.064 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:20:43.064 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:20:43.064 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:20:43.064 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:20:43.064 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:20:43.064 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:20:43.064 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000051594", 2024-08-18T17:20:43.064 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:20:43.064 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:20:43.064 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:20:43.064 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:54.270479Z" 2024-08-18T17:20:43.064 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:20:43.065 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:20:43.065 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:20:43.065 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:20:43.065 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:20:43.065 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:20:43.065 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:20:43.065 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:20:43.065 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:20:43.065 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:20:43.065 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:20:43.065 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:20:43.065 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:20:43.065 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:20:43.066 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:20:43.066 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:20:43.066 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:20:43.066 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000058857", 2024-08-18T17:20:43.066 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:20:43.066 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:20:43.066 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:20:43.066 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:54.289199Z" 2024-08-18T17:20:43.066 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:20:43.066 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:20:43.066 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:20:43.066 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:20:43.067 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:20:43.067 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:20:43.067 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052068", 2024-08-18T17:20:43.067 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:20:43.067 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:20:43.067 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:20:43.067 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:05:27.099520Z" 2024-08-18T17:20:43.067 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:20:43.067 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:20:43.067 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:20:43.067 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:20:43.067 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:20:43.067 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:20:43.068 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:20:43.068 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:20:43.068 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:20:43Z" 2024-08-18T17:20:43.068 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:20:43.069 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000051734 target=00000000000000000000:00000000000000051594, shard=2 master=00000000000000000000:00000000000000059137 target=00000000000000000000:00000000000000058857, shard=3 master=00000000000000000000:00000000000000052210 target=00000000000000000000:00000000000000052068 2024-08-18T17:20:48.070 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:20:48.070 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:20:48.070 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:20:48.128 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:20:48.129 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:20:48.179 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:20:48.179 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:20:48.180 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:20:48.180 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:20:48.180 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:20:48.180 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:20:48.180 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:20:48.180 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:20:48.180 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:20:48.180 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:20:48.180 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:20:48.180 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:20:48.180 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000051594", 2024-08-18T17:20:48.180 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:20:48.180 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:20:48.181 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:20:48.181 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:54.270479Z" 2024-08-18T17:20:48.181 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:20:48.181 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:20:48.181 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:20:48.181 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:20:48.181 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:20:48.181 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:20:48.181 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:20:48.181 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:20:48.181 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:20:48.181 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:20:48.182 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:20:48.182 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:20:48.182 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:20:48.182 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:20:48.182 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:20:48.182 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:20:48.182 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:20:48.182 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000058857", 2024-08-18T17:20:48.182 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:20:48.182 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:20:48.182 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:20:48.183 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:54.289199Z" 2024-08-18T17:20:48.183 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:20:48.183 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:20:48.183 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:20:48.183 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:20:48.183 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:20:48.183 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:20:48.183 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052068", 2024-08-18T17:20:48.183 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:20:48.183 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:20:48.183 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:20:48.183 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:05:27.099520Z" 2024-08-18T17:20:48.183 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:20:48.184 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:20:48.184 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:20:48.184 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:20:48.184 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:20:48.184 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:20:48.184 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:20:48.184 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:20:48.184 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:20:48Z" 2024-08-18T17:20:48.184 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:20:48.185 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000051734 target=00000000000000000000:00000000000000051594, shard=2 master=00000000000000000000:00000000000000059137 target=00000000000000000000:00000000000000058857, shard=3 master=00000000000000000000:00000000000000052210 target=00000000000000000000:00000000000000052068 2024-08-18T17:20:53.187 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:20:53.187 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:20:53.187 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:20:53.244 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:20:53.245 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:20:53.295 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:20:53.295 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:20:53.295 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:20:53.295 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:20:53.295 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:20:53.295 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:20:53.296 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:20:53.296 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:20:53.296 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:20:53.296 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:20:53.296 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:20:53.296 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:20:53.296 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000051594", 2024-08-18T17:20:53.296 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:20:53.296 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:20:53.296 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:20:53.296 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:54.270479Z" 2024-08-18T17:20:53.296 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:20:53.296 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:20:53.297 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:20:53.297 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:20:53.297 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:20:53.297 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:20:53.297 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:20:53.297 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:20:53.297 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:20:53.297 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:20:53.297 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:20:53.297 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:20:53.297 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:20:53.297 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:20:53.297 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:20:53.298 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:20:53.298 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:20:53.298 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000058857", 2024-08-18T17:20:53.298 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:20:53.298 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:20:53.298 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:20:53.298 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:54.289199Z" 2024-08-18T17:20:53.298 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:20:53.298 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:20:53.298 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:20:53.298 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:20:53.298 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:20:53.298 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:20:53.299 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052068", 2024-08-18T17:20:53.299 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:20:53.299 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:20:53.299 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:20:53.299 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:05:27.099520Z" 2024-08-18T17:20:53.299 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:20:53.299 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:20:53.299 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:20:53.299 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:20:53.300 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:20:53.300 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:20:53.300 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:20:53.300 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:20:53.300 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:20:53Z" 2024-08-18T17:20:53.300 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:20:53.301 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000051734 target=00000000000000000000:00000000000000051594, shard=2 master=00000000000000000000:00000000000000059137 target=00000000000000000000:00000000000000058857, shard=3 master=00000000000000000000:00000000000000052210 target=00000000000000000000:00000000000000052068 2024-08-18T17:20:58.302 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:20:58.302 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:20:58.303 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:20:58.360 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:20:58.361 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:20:58.412 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:20:58.412 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:20:58.412 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:20:58.412 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:20:58.413 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:20:58.413 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:20:58.413 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:20:58.413 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:20:58.413 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:20:58.413 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:20:58.413 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:20:58.413 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:20:58.413 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000051594", 2024-08-18T17:20:58.413 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:20:58.413 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:20:58.413 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:20:58.414 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:54.270479Z" 2024-08-18T17:20:58.414 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:20:58.414 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:20:58.414 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:20:58.414 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:20:58.414 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:20:58.414 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:20:58.414 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:20:58.414 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:20:58.414 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:20:58.415 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:20:58.415 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:20:58.415 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:20:58.415 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:20:58.415 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:20:58.415 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:20:58.415 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:20:58.415 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:20:58.415 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000058857", 2024-08-18T17:20:58.415 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:20:58.415 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:20:58.415 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:20:58.415 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:54.289199Z" 2024-08-18T17:20:58.416 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:20:58.416 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:20:58.416 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:20:58.416 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:20:58.416 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:20:58.416 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:20:58.416 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052068", 2024-08-18T17:20:58.416 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:20:58.416 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:20:58.416 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:20:58.416 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:05:27.099520Z" 2024-08-18T17:20:58.416 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:20:58.416 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:20:58.417 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:20:58.417 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:20:58.417 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:20:58.417 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:20:58.417 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:20:58.417 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:20:58.417 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:20:58Z" 2024-08-18T17:20:58.417 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:20:58.418 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000051734 target=00000000000000000000:00000000000000051594, shard=2 master=00000000000000000000:00000000000000059137 target=00000000000000000000:00000000000000058857, shard=3 master=00000000000000000000:00000000000000052210 target=00000000000000000000:00000000000000052068 2024-08-18T17:21:03.419 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:21:03.420 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:21:03.420 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:21:03.479 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:21:03.479 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:21:03.529 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:21:03.529 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:21:03.529 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:21:03.529 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:21:03.529 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:21:03.529 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:21:03.530 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:21:03.530 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:21:03.530 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:21:03.530 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:21:03.530 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:21:03.530 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:21:03.531 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000051594", 2024-08-18T17:21:03.531 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:21:03.531 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:21:03.531 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:21:03.531 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:54.270479Z" 2024-08-18T17:21:03.531 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:21:03.531 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:21:03.531 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:21:03.532 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:21:03.532 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:21:03.532 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:21:03.532 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:21:03.532 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:21:03.532 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:21:03.532 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:21:03.533 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:21:03.533 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:21:03.533 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:21:03.533 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:21:03.533 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:21:03.533 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:21:03.533 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:21:03.533 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000058857", 2024-08-18T17:21:03.533 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:21:03.534 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:21:03.534 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:21:03.534 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:54.289199Z" 2024-08-18T17:21:03.534 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:21:03.534 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:21:03.534 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:21:03.535 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:21:03.535 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:21:03.535 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:21:03.535 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052068", 2024-08-18T17:21:03.535 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:21:03.535 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:21:03.535 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:21:03.535 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:05:27.099520Z" 2024-08-18T17:21:03.535 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:21:03.535 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:21:03.535 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:21:03.535 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:21:03.535 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:21:03.536 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:21:03.536 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:21:03.536 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:21:03.536 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:21:03Z" 2024-08-18T17:21:03.536 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:21:03.537 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000051734 target=00000000000000000000:00000000000000051594, shard=2 master=00000000000000000000:00000000000000059137 target=00000000000000000000:00000000000000058857, shard=3 master=00000000000000000000:00000000000000052210 target=00000000000000000000:00000000000000052068 2024-08-18T17:21:08.538 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:21:08.538 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:21:08.538 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:21:08.595 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:21:08.595 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:21:08.647 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:21:08.647 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:21:08.647 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:21:08.647 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:21:08.647 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:21:08.647 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:21:08.647 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:21:08.648 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:21:08.648 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:21:08.648 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:21:08.648 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:21:08.648 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:21:08.648 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000051594", 2024-08-18T17:21:08.648 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:21:08.648 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:21:08.648 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:21:08.648 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:54.270479Z" 2024-08-18T17:21:08.648 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:21:08.648 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:21:08.648 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:21:08.649 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:21:08.649 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:21:08.649 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:21:08.649 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:21:08.649 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:21:08.649 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:21:08.649 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:21:08.649 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:21:08.649 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:21:08.649 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:21:08.649 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:21:08.649 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:21:08.649 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:21:08.650 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:21:08.650 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000058857", 2024-08-18T17:21:08.650 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:21:08.650 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:21:08.650 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:21:08.650 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:54.289199Z" 2024-08-18T17:21:08.650 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:21:08.650 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:21:08.650 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:21:08.650 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:21:08.650 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:21:08.650 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:21:08.651 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052068", 2024-08-18T17:21:08.651 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:21:08.651 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:21:08.651 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:21:08.651 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:05:27.099520Z" 2024-08-18T17:21:08.651 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:21:08.651 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:21:08.651 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:21:08.651 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:21:08.651 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:21:08.651 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:21:08.651 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:21:08.651 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:21:08.651 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:21:08Z" 2024-08-18T17:21:08.652 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:21:08.653 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000051734 target=00000000000000000000:00000000000000051594, shard=2 master=00000000000000000000:00000000000000059137 target=00000000000000000000:00000000000000058857, shard=3 master=00000000000000000000:00000000000000052210 target=00000000000000000000:00000000000000052068 2024-08-18T17:21:13.653 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:21:13.653 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:21:13.654 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:21:13.710 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:21:13.710 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:21:13.760 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:21:13.760 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:21:13.760 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:21:13.760 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:21:13.760 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:21:13.761 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:21:13.761 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:21:13.761 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:21:13.761 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:21:13.761 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:21:13.761 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:21:13.761 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:21:13.761 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000051734", 2024-08-18T17:21:13.761 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:21:13.761 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:21:13.761 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:21:13.761 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.076905Z" 2024-08-18T17:21:13.761 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:21:13.762 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:21:13.762 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:21:13.762 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:21:13.762 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:21:13.762 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:21:13.762 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:21:13.762 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:21:13.762 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:21:13.762 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:21:13.762 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:21:13.762 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:21:13.762 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:21:13.763 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:21:13.763 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:21:13.763 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:21:13.763 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:21:13.763 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000059137", 2024-08-18T17:21:13.763 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:21:13.763 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:21:13.763 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:21:13.763 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.125128Z" 2024-08-18T17:21:13.763 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:21:13.763 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:21:13.763 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:21:13.764 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:21:13.764 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:21:13.764 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:21:13.764 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:21:13.764 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:21:13.764 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:21:13.764 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:21:13.764 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:21:13.764 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:21:13.764 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:21:13.764 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:21:13.764 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:21:13.764 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:21:13.765 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:21:13.765 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:21:13.765 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:21:13.765 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:21:13Z" 2024-08-18T17:21:13.765 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:21:13.766 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-08-18T17:21:13.789 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:21:13.789 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:21:13.790 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-group --pipe-id sync-pipe --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:21:13.848 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:21:13.848 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:21:13.921 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[],"status":"enabled"}]} 2024-08-18T17:21:13.922 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1,test-zone2', '--dest-zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:21:13.922 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1,test-zone2', '--dest-zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:21:13.922 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=test-zone1,test-zone2 --dest-zones=test-zone1,test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:21:13.972 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:21:13.973 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:21:14.041 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-08-18T17:21:14.041 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:21:14.042 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:21:14.042 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:21:14.089 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:21:14.089 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:21:14.149 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-08-18T17:21:14.149 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:21:14.150 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:21:14.150 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:21:14.196 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:21:14.196 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:21:14.249 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T17:21:14.249 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:21:14.249 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 16, 2024-08-18T17:21:14.249 INFO:teuthology.orchestra.run.smithi134.stdout: "predecessor_uuid": "64309209-7567-46e5-8881-65783d84cf3c", 2024-08-18T17:21:14.249 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": [ 2024-08-18T17:21:14.249 INFO:teuthology.orchestra.run.smithi134.stdout: "", 2024-08-18T17:21:14.249 INFO:teuthology.orchestra.run.smithi134.stdout: "", 2024-08-18T17:21:14.249 INFO:teuthology.orchestra.run.smithi134.stdout: "", 2024-08-18T17:21:14.249 INFO:teuthology.orchestra.run.smithi134.stdout: "1_1723995848.930494_187.1" 2024-08-18T17:21:14.250 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:21:14.250 INFO:teuthology.orchestra.run.smithi134.stdout: "period_map": { 2024-08-18T17:21:14.250 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:21:14.250 INFO:teuthology.orchestra.run.smithi134.stdout: "zonegroups": [ 2024-08-18T17:21:14.250 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:21:14.250 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "test-zonegroup", 2024-08-18T17:21:14.250 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "test-zonegroup", 2024-08-18T17:21:14.250 INFO:teuthology.orchestra.run.smithi134.stdout: "api_name": "", 2024-08-18T17:21:14.250 INFO:teuthology.orchestra.run.smithi134.stdout: "is_master": true, 2024-08-18T17:21:14.250 INFO:teuthology.orchestra.run.smithi134.stdout: "endpoints": [ 2024-08-18T17:21:14.250 INFO:teuthology.orchestra.run.smithi134.stdout: "http://smithi134.front.sepia.ceph.com:8000" 2024-08-18T17:21:14.250 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:21:14.250 INFO:teuthology.orchestra.run.smithi134.stdout: "hostnames": [], 2024-08-18T17:21:14.251 INFO:teuthology.orchestra.run.smithi134.stdout: "hostnames_s3website": [], 2024-08-18T17:21:14.251 INFO:teuthology.orchestra.run.smithi134.stdout: "master_zone": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T17:21:14.251 INFO:teuthology.orchestra.run.smithi134.stdout: "zones": [ 2024-08-18T17:21:14.251 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:21:14.251 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T17:21:14.251 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "test-zone3", 2024-08-18T17:21:14.251 INFO:teuthology.orchestra.run.smithi134.stdout: "endpoints": [ 2024-08-18T17:21:14.251 INFO:teuthology.orchestra.run.smithi134.stdout: "http://smithi134.front.sepia.ceph.com:8001" 2024-08-18T17:21:14.251 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:21:14.251 INFO:teuthology.orchestra.run.smithi134.stdout: "log_meta": false, 2024-08-18T17:21:14.251 INFO:teuthology.orchestra.run.smithi134.stdout: "log_data": true, 2024-08-18T17:21:14.251 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_index_max_shards": 11, 2024-08-18T17:21:14.251 INFO:teuthology.orchestra.run.smithi134.stdout: "read_only": false, 2024-08-18T17:21:14.252 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_type": "", 2024-08-18T17:21:14.252 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from_all": true, 2024-08-18T17:21:14.252 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from": [], 2024-08-18T17:21:14.252 INFO:teuthology.orchestra.run.smithi134.stdout: "redirect_zone": "", 2024-08-18T17:21:14.252 INFO:teuthology.orchestra.run.smithi134.stdout: "supported_features": [ 2024-08-18T17:21:14.252 INFO:teuthology.orchestra.run.smithi134.stdout: "compress-encrypted", 2024-08-18T17:21:14.252 INFO:teuthology.orchestra.run.smithi134.stdout: "resharding" 2024-08-18T17:21:14.252 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:21:14.252 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:21:14.252 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:21:14.252 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "10090ccf-aad6-416d-9775-df8b7b27345a", 2024-08-18T17:21:14.252 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "test-zone2", 2024-08-18T17:21:14.252 INFO:teuthology.orchestra.run.smithi134.stdout: "endpoints": [ 2024-08-18T17:21:14.253 INFO:teuthology.orchestra.run.smithi134.stdout: "http://smithi193.front.sepia.ceph.com:8000" 2024-08-18T17:21:14.253 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:21:14.253 INFO:teuthology.orchestra.run.smithi134.stdout: "log_meta": false, 2024-08-18T17:21:14.253 INFO:teuthology.orchestra.run.smithi134.stdout: "log_data": true, 2024-08-18T17:21:14.253 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_index_max_shards": 11, 2024-08-18T17:21:14.253 INFO:teuthology.orchestra.run.smithi134.stdout: "read_only": false, 2024-08-18T17:21:14.253 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_type": "", 2024-08-18T17:21:14.253 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from_all": true, 2024-08-18T17:21:14.253 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from": [], 2024-08-18T17:21:14.253 INFO:teuthology.orchestra.run.smithi134.stdout: "redirect_zone": "", 2024-08-18T17:21:14.254 INFO:teuthology.orchestra.run.smithi134.stdout: "supported_features": [ 2024-08-18T17:21:14.254 INFO:teuthology.orchestra.run.smithi134.stdout: "compress-encrypted", 2024-08-18T17:21:14.254 INFO:teuthology.orchestra.run.smithi134.stdout: "resharding" 2024-08-18T17:21:14.254 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:21:14.254 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:21:14.254 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:21:14.254 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T17:21:14.254 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "test-zone1", 2024-08-18T17:21:14.254 INFO:teuthology.orchestra.run.smithi134.stdout: "endpoints": [ 2024-08-18T17:21:14.254 INFO:teuthology.orchestra.run.smithi134.stdout: "http://smithi134.front.sepia.ceph.com:8000" 2024-08-18T17:21:14.254 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:21:14.255 INFO:teuthology.orchestra.run.smithi134.stdout: "log_meta": false, 2024-08-18T17:21:14.255 INFO:teuthology.orchestra.run.smithi134.stdout: "log_data": true, 2024-08-18T17:21:14.255 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_index_max_shards": 11, 2024-08-18T17:21:14.255 INFO:teuthology.orchestra.run.smithi134.stdout: "read_only": false, 2024-08-18T17:21:14.255 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_type": "", 2024-08-18T17:21:14.255 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from_all": true, 2024-08-18T17:21:14.255 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from": [], 2024-08-18T17:21:14.255 INFO:teuthology.orchestra.run.smithi134.stdout: "redirect_zone": "", 2024-08-18T17:21:14.255 INFO:teuthology.orchestra.run.smithi134.stdout: "supported_features": [ 2024-08-18T17:21:14.255 INFO:teuthology.orchestra.run.smithi134.stdout: "compress-encrypted", 2024-08-18T17:21:14.255 INFO:teuthology.orchestra.run.smithi134.stdout: "resharding" 2024-08-18T17:21:14.255 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:21:14.255 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:21:14.256 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:21:14.256 INFO:teuthology.orchestra.run.smithi134.stdout: "placement_targets": [ 2024-08-18T17:21:14.256 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:21:14.256 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "default-placement", 2024-08-18T17:21:14.256 INFO:teuthology.orchestra.run.smithi134.stdout: "tags": [], 2024-08-18T17:21:14.256 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_classes": [ 2024-08-18T17:21:14.256 INFO:teuthology.orchestra.run.smithi134.stdout: "STANDARD" 2024-08-18T17:21:14.256 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:21:14.256 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:21:14.256 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:21:14.256 INFO:teuthology.orchestra.run.smithi134.stdout: "default_placement": "default-placement", 2024-08-18T17:21:14.256 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_id": "2827c6f0-bf0d-4836-944c-9e7fcf22457b", 2024-08-18T17:21:14.256 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_policy": { 2024-08-18T17:21:14.256 INFO:teuthology.orchestra.run.smithi134.stdout: "groups": [ 2024-08-18T17:21:14.257 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:21:14.257 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "sync-group", 2024-08-18T17:21:14.257 INFO:teuthology.orchestra.run.smithi134.stdout: "data_flow": { 2024-08-18T17:21:14.257 INFO:teuthology.orchestra.run.smithi134.stdout: "symmetrical": [ 2024-08-18T17:21:14.257 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:21:14.257 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "sync-flow1", 2024-08-18T17:21:14.257 INFO:teuthology.orchestra.run.smithi134.stdout: "zones": [ 2024-08-18T17:21:14.272 INFO:teuthology.orchestra.run.smithi134.stdout: "10090ccf-aad6-416d-9775-df8b7b27345a", 2024-08-18T17:21:14.272 INFO:teuthology.orchestra.run.smithi134.stdout: "96bf8502-2f39-4d53-8196-3378c28da3b4" 2024-08-18T17:21:14.272 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:21:14.273 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:21:14.273 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:21:14.273 INFO:teuthology.orchestra.run.smithi134.stdout: "directional": [ 2024-08-18T17:21:14.273 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:21:14.273 INFO:teuthology.orchestra.run.smithi134.stdout: "source_zone": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T17:21:14.273 INFO:teuthology.orchestra.run.smithi134.stdout: "dest_zone": "10090ccf-aad6-416d-9775-df8b7b27345a" 2024-08-18T17:21:14.273 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:21:14.273 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:21:14.273 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:21:14.273 INFO:teuthology.orchestra.run.smithi134.stdout: "pipes": [ 2024-08-18T17:21:14.273 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:21:14.273 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "sync-pipe", 2024-08-18T17:21:14.274 INFO:teuthology.orchestra.run.smithi134.stdout: "source": { 2024-08-18T17:21:14.274 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket": "*", 2024-08-18T17:21:14.274 INFO:teuthology.orchestra.run.smithi134.stdout: "zones": [ 2024-08-18T17:21:14.274 INFO:teuthology.orchestra.run.smithi134.stdout: "10090ccf-aad6-416d-9775-df8b7b27345a", 2024-08-18T17:21:14.274 INFO:teuthology.orchestra.run.smithi134.stdout: "96bf8502-2f39-4d53-8196-3378c28da3b4" 2024-08-18T17:21:14.274 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:21:14.274 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:21:14.274 INFO:teuthology.orchestra.run.smithi134.stdout: "dest": { 2024-08-18T17:21:14.274 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket": "*", 2024-08-18T17:21:14.274 INFO:teuthology.orchestra.run.smithi134.stdout: "zones": [ 2024-08-18T17:21:14.274 INFO:teuthology.orchestra.run.smithi134.stdout: "10090ccf-aad6-416d-9775-df8b7b27345a", 2024-08-18T17:21:14.274 INFO:teuthology.orchestra.run.smithi134.stdout: "96bf8502-2f39-4d53-8196-3378c28da3b4" 2024-08-18T17:21:14.275 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:21:14.275 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:21:14.275 INFO:teuthology.orchestra.run.smithi134.stdout: "params": { 2024-08-18T17:21:14.275 INFO:teuthology.orchestra.run.smithi134.stdout: "source": { 2024-08-18T17:21:14.275 INFO:teuthology.orchestra.run.smithi134.stdout: "filter": { 2024-08-18T17:21:14.275 INFO:teuthology.orchestra.run.smithi134.stdout: "tags": [] 2024-08-18T17:21:14.275 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:21:14.275 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:21:14.275 INFO:teuthology.orchestra.run.smithi134.stdout: "dest": {}, 2024-08-18T17:21:14.275 INFO:teuthology.orchestra.run.smithi134.stdout: "priority": 0, 2024-08-18T17:21:14.275 INFO:teuthology.orchestra.run.smithi134.stdout: "mode": "system", 2024-08-18T17:21:14.275 INFO:teuthology.orchestra.run.smithi134.stdout: "user": "" 2024-08-18T17:21:14.275 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:21:14.276 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:21:14.276 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:21:14.276 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "allowed" 2024-08-18T17:21:14.276 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:21:14.276 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:21:14.276 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:21:14.276 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled_features": [ 2024-08-18T17:21:14.276 INFO:teuthology.orchestra.run.smithi134.stdout: "resharding" 2024-08-18T17:21:14.276 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:21:14.276 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:21:14.276 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:21:14.276 INFO:teuthology.orchestra.run.smithi134.stdout: "short_zone_ids": [ 2024-08-18T17:21:14.276 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:21:14.276 INFO:teuthology.orchestra.run.smithi134.stdout: "key": "018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T17:21:14.277 INFO:teuthology.orchestra.run.smithi134.stdout: "val": 462036791 2024-08-18T17:21:14.277 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:21:14.277 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:21:14.277 INFO:teuthology.orchestra.run.smithi134.stdout: "key": "10090ccf-aad6-416d-9775-df8b7b27345a", 2024-08-18T17:21:14.277 INFO:teuthology.orchestra.run.smithi134.stdout: "val": 874289624 2024-08-18T17:21:14.277 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:21:14.277 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:21:14.277 INFO:teuthology.orchestra.run.smithi134.stdout: "key": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T17:21:14.277 INFO:teuthology.orchestra.run.smithi134.stdout: "val": 1721399886 2024-08-18T17:21:14.277 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:21:14.277 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:21:14.277 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:21:14.277 INFO:teuthology.orchestra.run.smithi134.stdout: "master_zonegroup": "test-zonegroup", 2024-08-18T17:21:14.277 INFO:teuthology.orchestra.run.smithi134.stdout: "master_zone": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T17:21:14.278 INFO:teuthology.orchestra.run.smithi134.stdout: "period_config": { 2024-08-18T17:21:14.278 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_quota": { 2024-08-18T17:21:14.278 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false, 2024-08-18T17:21:14.278 INFO:teuthology.orchestra.run.smithi134.stdout: "check_on_raw": false, 2024-08-18T17:21:14.278 INFO:teuthology.orchestra.run.smithi134.stdout: "max_size": -1, 2024-08-18T17:21:14.278 INFO:teuthology.orchestra.run.smithi134.stdout: "max_size_kb": 0, 2024-08-18T17:21:14.278 INFO:teuthology.orchestra.run.smithi134.stdout: "max_objects": -1 2024-08-18T17:21:14.278 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:21:14.278 INFO:teuthology.orchestra.run.smithi134.stdout: "user_quota": { 2024-08-18T17:21:14.278 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false, 2024-08-18T17:21:14.278 INFO:teuthology.orchestra.run.smithi134.stdout: "check_on_raw": false, 2024-08-18T17:21:14.278 INFO:teuthology.orchestra.run.smithi134.stdout: "max_size": -1, 2024-08-18T17:21:14.278 INFO:teuthology.orchestra.run.smithi134.stdout: "max_size_kb": 0, 2024-08-18T17:21:14.278 INFO:teuthology.orchestra.run.smithi134.stdout: "max_objects": -1 2024-08-18T17:21:14.279 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:21:14.279 INFO:teuthology.orchestra.run.smithi134.stdout: "user_ratelimit": { 2024-08-18T17:21:14.279 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_ops": 0, 2024-08-18T17:21:14.279 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_ops": 0, 2024-08-18T17:21:14.279 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_bytes": 0, 2024-08-18T17:21:14.279 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_bytes": 0, 2024-08-18T17:21:14.279 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false 2024-08-18T17:21:14.279 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:21:14.279 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_ratelimit": { 2024-08-18T17:21:14.279 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_ops": 0, 2024-08-18T17:21:14.279 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_ops": 0, 2024-08-18T17:21:14.279 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_bytes": 0, 2024-08-18T17:21:14.279 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_bytes": 0, 2024-08-18T17:21:14.279 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false 2024-08-18T17:21:14.280 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:21:14.280 INFO:teuthology.orchestra.run.smithi134.stdout: "anonymous_ratelimit": { 2024-08-18T17:21:14.280 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_ops": 0, 2024-08-18T17:21:14.280 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_ops": 0, 2024-08-18T17:21:14.280 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_bytes": 0, 2024-08-18T17:21:14.280 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_bytes": 0, 2024-08-18T17:21:14.280 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false 2024-08-18T17:21:14.280 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:21:14.280 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:21:14.280 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_id": "2827c6f0-bf0d-4836-944c-9e7fcf22457b", 2024-08-18T17:21:14.280 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_name": "test-realm", 2024-08-18T17:21:14.280 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:21:14.280 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T17:21:14.281 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:21:14.282 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:21:14.282 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:21:14.329 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:21:14.329 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:21:14.424 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-08-18T17:21:14.424 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'kibgmb-91', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:21:14.425 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'kibgmb-91', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:21:14.425 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-bucket --status allowed --bucket kibgmb-91 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:21:14.483 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:21:14.483 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:21:14.620 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"allowed"}]} 2024-08-18T17:21:14.620 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--bucket', 'kibgmb-91', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:21:14.620 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--bucket', 'kibgmb-91', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:21:14.620 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flowA --flow-type symmetrical --zones=test-zone1,test-zone2 --bucket kibgmb-91 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:21:14.690 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:21:14.690 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:21:14.815 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["test-zone2","test-zone1"]}]},"pipes":[],"status":"allowed"}]} 2024-08-18T17:21:14.815 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1,test-zone2', '--dest-zones=test-zone1,test-zone2', '--bucket=kibgmb-91', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:21:14.815 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1,test-zone2', '--dest-zones=test-zone1,test-zone2', '--bucket=kibgmb-91', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:21:14.815 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe --source-zones=test-zone1,test-zone2 --dest-zones=test-zone1,test-zone2 --bucket=kibgmb-91 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:21:14.874 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:21:14.874 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:21:14.973 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["test-zone2","test-zone1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2024-08-18T17:21:14.973 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'kibgmb-91', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:21:14.973 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'kibgmb-91', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:21:14.974 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-bucket --status enabled --bucket kibgmb-91 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:21:15.027 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:21:15.027 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:21:15.143 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["test-zone2","test-zone1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2024-08-18T17:21:15.143 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--bucket', 'kibgmb-91', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:21:15.143 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--bucket', 'kibgmb-91', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:21:15.144 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --bucket kibgmb-91 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:21:15.194 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:21:15.194 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:21:15.331 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["test-zone2","test-zone1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2024-08-18T17:21:15.365 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:21:15.365 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:21:15.366 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:21:15.415 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:21:15.415 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:21:15.479 INFO:teuthology.orchestra.run.smithi134.stderr:No --period given, using current period=01050e30-179b-43be-bdbb-3bb23cf37ae8 2024-08-18T17:21:15.506 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T17:21:15.507 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:21:15.507 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:21:15.507 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T17:21:15.507 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:21:15.507 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:21:15.507 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:21:15.507 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T17:21:15.507 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:21:15.507 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:21:15.507 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:21:15.508 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:42:47.672461Z" 2024-08-18T17:21:15.508 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:21:15.508 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:21:15.508 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1724001675.093496_1592.1", 2024-08-18T17:21:15.508 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T17:21:15.093496Z" 2024-08-18T17:21:15.508 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:21:15.508 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T17:21:15.509 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-08-18T17:21:15.509 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:21:15.509 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:21:15.509 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:21:15.561 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:21:15.561 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:21:15.594 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:21:15.593+0000 7fe20a610880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:21:15.595 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:21:15.594+0000 7fe20a610880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:21:15.609 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:21:15.610 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:21:15.610 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:21:15.610 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:21:15.610 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:21:15.610 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:21:15.610 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:21:15.611 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:21:15.611 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:21:15.611 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:21:15.611 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:21:15.611 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:21:15.611 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:21:15.611 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:21:15.611 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:21:15.611 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:21:15.611 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:21:15.611 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:21:15.611 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:21:15.612 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:21:15.612 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:21:15.612 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:21:15.612 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:21:15.612 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:21:15.612 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:21:15.612 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:21:15.612 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:21:15.612 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:21:15.612 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:21:15.612 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:21:15.613 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:21:15.613 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:21:15.613 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:21:15.613 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:21:15.613 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:21:15.613 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:21:15.613 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:21:15.613 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:21:15.613 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:21:15.613 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:21:15.613 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:21:15.613 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:21:15.613 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:21:15.614 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:21:15.614 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:21:15.614 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:21:15.614 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:21:15.614 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:21:15.614 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:21:15.614 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001538.454480_1552.1", 2024-08-18T17:21:15.614 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:21:15.614 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:21:15.614 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:21:15.614 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:18:58.454480Z", 2024-08-18T17:21:15.614 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:21:15.614 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:21:15.615 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:21:15.615 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:21:15.615 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:21:15.615 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:21:15.615 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:21:15.615 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:21:15.615 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:21:15.615 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:21:15Z" 2024-08-18T17:21:15.615 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:21:15.616 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001675.093496_1592.1 target=1_1724001538.454480_1552.1 2024-08-18T17:21:20.617 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:21:20.617 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:21:20.618 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:21:20.673 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:21:20.674 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:21:20.707 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:21:20.707+0000 7f46b2338880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:21:20.708 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:21:20.708+0000 7f46b2338880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:21:20.723 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:21:20.723 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:21:20.723 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:21:20.723 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:21:20.723 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:21:20.723 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:21:20.723 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:21:20.723 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:21:20.723 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:21:20.723 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:21:20.724 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:21:20.724 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:21:20.724 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:21:20.724 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:21:20.724 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:21:20.724 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:21:20.724 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:21:20.724 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:21:20.724 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:21:20.725 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:21:20.725 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:21:20.725 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:21:20.725 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:21:20.725 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:21:20.725 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:21:20.725 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:21:20.725 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:21:20.726 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:21:20.726 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:21:20.726 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:21:20.726 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:21:20.726 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:21:20.726 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:21:20.726 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:21:20.726 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:21:20.726 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:21:20.726 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:21:20.726 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:21:20.726 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:21:20.727 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:21:20.727 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:21:20.727 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:21:20.727 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:21:20.727 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:21:20.727 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:21:20.727 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:21:20.727 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:21:20.727 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:21:20.727 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:21:20.727 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001538.454480_1552.1", 2024-08-18T17:21:20.727 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:21:20.727 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:21:20.728 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:21:20.728 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:18:58.454480Z", 2024-08-18T17:21:20.728 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:21:20.728 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:21:20.728 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:21:20.728 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:21:20.728 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:21:20.728 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:21:20.728 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:21:20.728 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:21:20.728 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:21:20.728 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:21:20Z" 2024-08-18T17:21:20.728 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:21:20.730 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001675.093496_1592.1 target=1_1724001538.454480_1552.1 2024-08-18T17:21:25.731 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:21:25.731 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:21:25.732 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:21:25.787 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:21:25.787 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:21:25.821 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:21:25.821+0000 7f0564522880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:21:25.822 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:21:25.822+0000 7f0564522880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:21:25.837 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:21:25.837 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:21:25.837 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:21:25.837 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:21:25.837 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:21:25.837 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:21:25.837 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:21:25.837 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:21:25.838 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:21:25.838 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:21:25.838 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:21:25.838 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:21:25.838 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:21:25.838 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:21:25.838 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:21:25.838 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:21:25.838 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:21:25.838 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:21:25.838 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:21:25.838 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:21:25.842 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:21:25.842 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:21:25.842 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:21:25.842 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:21:25.842 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:21:25.842 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:21:25.842 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:21:25.842 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:21:25.842 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:21:25.842 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:21:25.842 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:21:25.843 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:21:25.843 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:21:25.843 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:21:25.843 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:21:25.843 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:21:25.843 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:21:25.843 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:21:25.843 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:21:25.843 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:21:25.843 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:21:25.843 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:21:25.843 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:21:25.843 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:21:25.843 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:21:25.844 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:21:25.844 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:21:25.844 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:21:25.844 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:21:25.844 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001538.454480_1552.1", 2024-08-18T17:21:25.844 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:21:25.844 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:21:25.844 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:21:25.844 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:18:58.454480Z", 2024-08-18T17:21:25.844 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:21:25.844 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:21:25.845 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:21:25.845 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:21:25.845 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:21:25.845 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:21:25.845 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:21:25.845 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:21:25.846 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:21:25.846 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:21:25Z" 2024-08-18T17:21:25.846 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:21:25.847 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001675.093496_1592.1 target=1_1724001538.454480_1552.1 2024-08-18T17:21:30.848 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:21:30.848 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:21:30.848 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:21:30.904 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:21:30.904 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:21:30.938 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:21:30.937+0000 7f4d35d26880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:21:30.938 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:21:30.938+0000 7f4d35d26880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:21:30.953 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:21:30.953 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:21:30.953 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:21:30.953 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:21:30.953 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:21:30.954 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:21:30.954 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:21:30.954 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:21:30.954 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:21:30.954 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:21:30.954 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:21:30.954 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:21:30.954 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:21:30.954 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:21:30.954 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:21:30.954 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:21:30.954 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:21:30.955 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:21:30.955 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:21:30.955 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:21:30.955 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:21:30.955 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:21:30.955 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:21:30.955 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:21:30.955 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:21:30.955 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:21:30.955 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:21:30.955 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:21:30.955 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:21:30.955 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:21:30.956 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:21:30.956 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:21:30.956 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:21:30.956 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:21:30.956 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:21:30.956 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:21:30.956 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:21:30.956 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:21:30.956 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:21:30.956 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:21:30.956 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:21:30.956 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:21:30.956 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:21:30.956 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:21:30.957 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:21:30.957 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:21:30.957 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:21:30.957 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:21:30.957 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:21:30.957 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001538.454480_1552.1", 2024-08-18T17:21:30.958 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:21:30.958 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:21:30.958 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:21:30.958 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:18:58.454480Z", 2024-08-18T17:21:30.958 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:21:30.958 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:21:30.958 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:21:30.958 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:21:30.962 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:21:30.962 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:21:30.962 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:21:30.962 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:21:30.962 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:21:30.962 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:21:30Z" 2024-08-18T17:21:30.962 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:21:30.963 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001675.093496_1592.1 target=1_1724001538.454480_1552.1 2024-08-18T17:21:35.964 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:21:35.964 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:21:35.965 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:21:36.020 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:21:36.020 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:21:36.054 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:21:36.053+0000 7f97baa35880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:21:36.054 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:21:36.054+0000 7f97baa35880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:21:36.069 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:21:36.069 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:21:36.069 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:21:36.069 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:21:36.069 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:21:36.069 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:21:36.070 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:21:36.070 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:21:36.070 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:21:36.070 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:21:36.070 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:21:36.070 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:21:36.070 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:21:36.070 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:21:36.070 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:21:36.070 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:21:36.070 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:21:36.070 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:21:36.070 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:21:36.071 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:21:36.071 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:21:36.071 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:21:36.071 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:21:36.071 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:21:36.071 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:21:36.071 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:21:36.071 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:21:36.071 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:21:36.071 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:21:36.071 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:21:36.071 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:21:36.071 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:21:36.071 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:21:36.072 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:21:36.072 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:21:36.072 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:21:36.072 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:21:36.072 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:21:36.072 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:21:36.072 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:21:36.072 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:21:36.072 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:21:36.072 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:21:36.072 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:21:36.072 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:21:36.072 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:21:36.072 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:21:36.073 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:21:36.073 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:21:36.073 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001538.454480_1552.1", 2024-08-18T17:21:36.073 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:21:36.073 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:21:36.073 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:21:36.073 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:18:58.454480Z", 2024-08-18T17:21:36.073 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:21:36.073 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:21:36.073 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:21:36.073 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:21:36.073 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:21:36.073 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:21:36.073 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:21:36.074 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:21:36.074 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:21:36.074 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:21:36Z" 2024-08-18T17:21:36.074 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:21:36.075 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001675.093496_1592.1 target=1_1724001538.454480_1552.1 2024-08-18T17:21:41.076 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:21:41.076 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:21:41.077 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:21:41.132 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:21:41.132 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:21:41.166 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:21:41.165+0000 7f9bdae15880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:21:41.166 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:21:41.165+0000 7f9bdae15880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:21:41.181 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:21:41.181 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:21:41.181 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:21:41.181 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:21:41.181 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:21:41.181 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:21:41.181 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:21:41.181 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:21:41.181 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:21:41.181 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:21:41.181 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:21:41.182 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:21:41.182 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:21:41.182 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:21:41.182 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:21:41.182 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:21:41.182 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:21:41.182 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:21:41.182 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:21:41.182 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:21:41.182 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:21:41.182 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:21:41.182 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:21:41.182 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:21:41.183 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:21:41.183 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:21:41.183 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:21:41.183 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:21:41.183 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:21:41.183 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:21:41.183 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:21:41.183 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:21:41.183 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:21:41.183 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:21:41.183 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:21:41.183 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:21:41.183 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:21:41.183 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:21:41.184 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:21:41.184 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:21:41.184 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:21:41.184 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:21:41.184 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:21:41.184 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:21:41.184 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:21:41.184 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:21:41.184 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:21:41.184 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:21:41.184 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:21:41.184 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001538.454480_1552.1", 2024-08-18T17:21:41.184 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:21:41.184 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:21:41.185 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:21:41.185 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:18:58.454480Z", 2024-08-18T17:21:41.185 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:21:41.185 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:21:41.185 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:21:41.185 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:21:41.185 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:21:41.185 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:21:41.185 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:21:41.185 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:21:41.185 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:21:41.185 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:21:41Z" 2024-08-18T17:21:41.185 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:21:41.186 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001675.093496_1592.1 target=1_1724001538.454480_1552.1 2024-08-18T17:21:46.188 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:21:46.188 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:21:46.189 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:21:46.243 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:21:46.243 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:21:46.276 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:21:46.275+0000 7ff79472d880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:21:46.277 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:21:46.276+0000 7ff79472d880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:21:46.291 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:21:46.292 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:21:46.292 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:21:46.292 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:21:46.292 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:21:46.292 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:21:46.292 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:21:46.292 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:21:46.292 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:21:46.292 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:21:46.292 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:21:46.292 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:21:46.293 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:21:46.293 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:21:46.293 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:21:46.293 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:21:46.293 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:21:46.293 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:21:46.293 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:21:46.293 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:21:46.293 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:21:46.293 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:21:46.293 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:21:46.293 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:21:46.293 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:21:46.294 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:21:46.294 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:21:46.294 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:21:46.294 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:21:46.294 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:21:46.294 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:21:46.294 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:21:46.294 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:21:46.294 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:21:46.294 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:21:46.295 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:21:46.295 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:21:46.295 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:21:46.295 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:21:46.295 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:21:46.295 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:21:46.295 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:21:46.295 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:21:46.295 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:21:46.295 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:21:46.295 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:21:46.295 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:21:46.296 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:21:46.296 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:21:46.296 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001538.454480_1552.1", 2024-08-18T17:21:46.296 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:21:46.296 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:21:46.296 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:21:46.296 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:18:58.454480Z", 2024-08-18T17:21:46.296 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:21:46.296 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:21:46.296 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:21:46.296 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:21:46.296 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:21:46.297 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:21:46.297 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:21:46.297 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:21:46.297 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:21:46.297 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:21:46Z" 2024-08-18T17:21:46.297 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:21:46.298 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001675.093496_1592.1 target=1_1724001538.454480_1552.1 2024-08-18T17:21:51.299 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:21:51.299 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:21:51.300 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:21:51.357 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:21:51.357 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:21:51.393 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:21:51.392+0000 7f4326b26880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:21:51.394 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:21:51.393+0000 7f4326b26880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:21:51.409 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:21:51.409 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:21:51.409 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:21:51.409 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:21:51.409 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:21:51.409 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:21:51.409 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:21:51.409 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:21:51.410 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:21:51.410 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:21:51.410 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:21:51.410 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:21:51.410 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:21:51.410 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:21:51.410 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:21:51.410 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:21:51.410 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:21:51.410 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:21:51.410 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:21:51.410 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:21:51.411 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:21:51.411 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:21:51.411 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:21:51.411 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:21:51.411 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:21:51.411 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:21:51.411 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:21:51.411 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:21:51.411 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:21:51.411 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:21:51.411 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:21:51.411 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:21:51.411 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:21:51.412 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:21:51.412 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:21:51.412 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:21:51.412 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:21:51.412 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:21:51.412 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:21:51.412 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:21:51.412 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:21:51.412 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:21:51.412 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:21:51.412 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:21:51.413 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:21:51.413 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:21:51.413 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:21:51.413 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:21:51.413 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:21:51.413 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001538.454480_1552.1", 2024-08-18T17:21:51.413 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:21:51.413 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:21:51.413 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:21:51.413 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:18:58.454480Z", 2024-08-18T17:21:51.413 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:21:51.413 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:21:51.414 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:21:51.414 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:21:51.414 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:21:51.414 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:21:51.414 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:21:51.414 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:21:51.414 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:21:51.414 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:21:51Z" 2024-08-18T17:21:51.414 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:21:51.415 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001675.093496_1592.1 target=1_1724001538.454480_1552.1 2024-08-18T17:21:56.416 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:21:56.416 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:21:56.417 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:21:56.472 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:21:56.472 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:21:56.506 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:21:56.505+0000 7f2962634880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:21:56.506 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:21:56.506+0000 7f2962634880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:21:56.521 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:21:56.521 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:21:56.521 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:21:56.521 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:21:56.521 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:21:56.521 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:21:56.522 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:21:56.522 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:21:56.522 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:21:56.522 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:21:56.522 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:21:56.522 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:21:56.522 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:21:56.522 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:21:56.522 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:21:56.522 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:21:56.522 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:21:56.522 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:21:56.523 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:21:56.523 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:21:56.523 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:21:56.523 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:21:56.523 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:21:56.523 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:21:56.523 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:21:56.523 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:21:56.523 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:21:56.523 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:21:56.523 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:21:56.523 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:21:56.524 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:21:56.524 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:21:56.524 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:21:56.524 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:21:56.524 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:21:56.524 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:21:56.524 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:21:56.524 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:21:56.524 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:21:56.524 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:21:56.525 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:21:56.525 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:21:56.525 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:21:56.525 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:21:56.525 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:21:56.525 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:21:56.525 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:21:56.525 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:21:56.525 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:21:56.525 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001538.454480_1552.1", 2024-08-18T17:21:56.525 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:21:56.525 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:21:56.526 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:21:56.526 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:18:58.454480Z", 2024-08-18T17:21:56.526 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:21:56.526 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:21:56.526 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:21:56.526 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:21:56.526 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:21:56.526 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:21:56.526 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:21:56.526 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:21:56.527 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:21:56.527 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:21:56Z" 2024-08-18T17:21:56.527 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:21:56.528 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001675.093496_1592.1 target=1_1724001538.454480_1552.1 2024-08-18T17:22:01.530 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:22:01.530 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:22:01.530 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:22:01.585 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:22:01.585 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:22:01.619 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:22:01.617+0000 7faa45c15880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:22:01.619 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:22:01.618+0000 7faa45c15880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:22:01.634 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:22:01.635 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:22:01.635 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:22:01.635 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:22:01.635 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:22:01.635 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:22:01.635 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:22:01.635 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:22:01.635 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:22:01.635 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:22:01.635 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:22:01.635 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:22:01.635 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:22:01.635 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:22:01.636 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:22:01.636 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:22:01.636 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:22:01.636 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:22:01.636 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:22:01.636 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:22:01.636 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:22:01.636 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:22:01.636 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:22:01.636 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:22:01.636 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:22:01.636 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:22:01.636 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:22:01.637 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:22:01.637 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:22:01.637 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:22:01.637 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:22:01.637 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:22:01.637 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:22:01.637 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:22:01.637 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:22:01.637 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:22:01.637 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:22:01.637 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:22:01.637 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:22:01.637 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:22:01.638 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:22:01.638 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:22:01.638 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:22:01.638 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:22:01.638 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:22:01.638 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:22:01.638 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:22:01.638 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:22:01.638 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:22:01.638 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001538.454480_1552.1", 2024-08-18T17:22:01.638 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:22:01.638 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:22:01.638 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:22:01.639 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:18:58.454480Z", 2024-08-18T17:22:01.639 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:22:01.639 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:22:01.639 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:22:01.639 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:22:01.639 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:22:01.639 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:22:01.639 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:22:01.639 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:22:01.639 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:22:01.639 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:22:01Z" 2024-08-18T17:22:01.639 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:22:01.640 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001675.093496_1592.1 target=1_1724001538.454480_1552.1 2024-08-18T17:22:06.642 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:22:06.642 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:22:06.642 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:22:06.698 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:22:06.698 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:22:06.732 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:22:06.731+0000 7ffa14738880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:22:06.733 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:22:06.732+0000 7ffa14738880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:22:06.748 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:22:06.748 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:22:06.748 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:22:06.748 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:22:06.748 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:22:06.748 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:22:06.748 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:22:06.748 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:22:06.749 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:22:06.749 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:22:06.749 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:22:06.749 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:22:06.749 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:22:06.749 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:22:06.749 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:22:06.749 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:22:06.749 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:22:06.749 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:22:06.749 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:22:06.749 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:22:06.749 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:22:06.750 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:22:06.750 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:22:06.750 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:22:06.750 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:22:06.750 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:22:06.750 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:22:06.750 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:22:06.750 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:22:06.750 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:22:06.750 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:22:06.750 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:22:06.750 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:22:06.751 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:22:06.751 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:22:06.751 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:22:06.751 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:22:06.751 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:22:06.751 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:22:06.751 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:22:06.751 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:22:06.751 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:22:06.751 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:22:06.751 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:22:06.751 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:22:06.752 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:22:06.752 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:22:06.752 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:22:06.752 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:22:06.752 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001538.454480_1552.1", 2024-08-18T17:22:06.753 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:22:06.753 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:22:06.753 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:22:06.753 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:18:58.454480Z", 2024-08-18T17:22:06.753 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:22:06.753 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:22:06.753 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:22:06.753 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:22:06.753 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:22:06.753 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:22:06.753 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:22:06.753 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:22:06.753 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:22:06.753 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:22:06Z" 2024-08-18T17:22:06.754 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:22:06.755 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001675.093496_1592.1 target=1_1724001538.454480_1552.1 2024-08-18T17:22:11.756 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:22:11.756 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:22:11.757 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:22:11.812 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:22:11.812 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:22:11.846 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:22:11.845+0000 7f7ca7a10880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:22:11.847 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:22:11.846+0000 7f7ca7a10880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:22:11.861 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:22:11.862 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:22:11.862 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:22:11.862 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:22:11.862 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:22:11.862 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:22:11.862 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:22:11.862 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:22:11.862 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:22:11.863 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:22:11.863 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:22:11.863 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:22:11.863 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:22:11.863 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:22:11.863 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:22:11.863 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:22:11.863 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:22:11.863 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:22:11.863 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:22:11.863 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:22:11.863 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:22:11.863 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:22:11.863 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:22:11.864 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:22:11.864 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:22:11.864 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:22:11.864 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:22:11.864 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:22:11.864 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:22:11.864 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:22:11.864 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:22:11.864 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:22:11.864 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:22:11.864 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:22:11.864 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:22:11.865 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:22:11.865 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:22:11.865 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:22:11.865 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:22:11.865 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:22:11.865 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:22:11.865 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:22:11.865 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:22:11.865 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:22:11.865 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:22:11.865 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:22:11.865 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:22:11.865 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:22:11.866 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:22:11.866 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001538.454480_1552.1", 2024-08-18T17:22:11.866 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:22:11.866 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:22:11.866 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:22:11.866 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:18:58.454480Z", 2024-08-18T17:22:11.866 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:22:11.866 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:22:11.866 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:22:11.866 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:22:11.866 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:22:11.867 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:22:11.867 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:22:11.867 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:22:11.867 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:22:11.867 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:22:11Z" 2024-08-18T17:22:11.867 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:22:11.868 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001675.093496_1592.1 target=1_1724001538.454480_1552.1 2024-08-18T17:22:16.869 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:22:16.869 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:22:16.869 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:22:16.925 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:22:16.925 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:22:16.958 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:22:16.958+0000 7fb715c0f880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:22:16.959 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:22:16.959+0000 7fb715c0f880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:22:16.974 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:22:16.974 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:22:16.974 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:22:16.974 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:22:16.974 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:22:16.975 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:22:16.975 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:22:16.975 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:22:16.975 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:22:16.975 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:22:16.975 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:22:16.975 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:22:16.975 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:22:16.975 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:22:16.975 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:22:16.975 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:22:16.975 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:22:16.975 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:22:16.976 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:22:16.976 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:22:16.976 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:22:16.976 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:22:16.976 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:22:16.976 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:22:16.976 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:22:16.976 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:22:16.976 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:22:16.976 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:22:16.976 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:22:16.976 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:22:16.977 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:22:16.977 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:22:16.977 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:22:16.977 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:22:16.977 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:22:16.977 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:22:16.977 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:22:16.977 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:22:16.977 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:22:16.977 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:22:16.977 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:22:16.977 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:22:16.977 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:22:16.977 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:22:16.978 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:22:16.978 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:22:16.978 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:22:16.978 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:22:16.978 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:22:16.978 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001538.454480_1552.1", 2024-08-18T17:22:16.978 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:22:16.978 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:22:16.978 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:22:16.978 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:18:58.454480Z", 2024-08-18T17:22:16.978 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:22:16.978 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:22:16.978 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:22:16.979 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:22:16.979 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:22:16.979 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:22:16.979 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:22:16.979 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:22:16.979 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:22:16.979 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:22:16Z" 2024-08-18T17:22:16.979 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:22:16.980 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001675.093496_1592.1 target=1_1724001538.454480_1552.1 2024-08-18T17:22:21.981 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:22:21.981 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:22:21.982 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:22:22.037 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:22:22.037 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:22:22.070 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:22:22.069+0000 7ff79e210880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:22:22.071 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:22:22.070+0000 7ff79e210880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:22:22.086 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:22:22.086 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:22:22.086 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:22:22.086 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:22:22.086 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:22:22.087 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:22:22.087 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:22:22.087 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:22:22.087 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:22:22.087 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:22:22.087 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:22:22.087 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:22:22.087 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:22:22.087 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:22:22.087 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:22:22.087 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:22:22.087 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:22:22.088 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:22:22.088 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:22:22.088 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:22:22.088 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:22:22.088 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:22:22.088 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:22:22.088 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:22:22.088 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:22:22.088 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:22:22.088 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:22:22.088 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:22:22.088 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:22:22.088 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:22:22.089 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:22:22.089 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:22:22.089 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:22:22.089 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:22:22.089 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:22:22.089 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:22:22.089 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:22:22.089 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:22:22.089 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:22:22.089 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:22:22.089 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:22:22.089 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:22:22.089 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:22:22.090 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:22:22.090 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:22:22.090 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:22:22.090 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:22:22.090 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:22:22.090 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:22:22.090 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001538.454480_1552.1", 2024-08-18T17:22:22.090 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:22:22.090 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:22:22.090 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:22:22.090 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:18:58.454480Z", 2024-08-18T17:22:22.090 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:22:22.090 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:22:22.091 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:22:22.091 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:22:22.091 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:22:22.091 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:22:22.091 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:22:22.091 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:22:22.091 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:22:22.091 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:22:22Z" 2024-08-18T17:22:22.091 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:22:22.092 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001675.093496_1592.1 target=1_1724001538.454480_1552.1 2024-08-18T17:22:27.093 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:22:27.093 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:22:27.093 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:22:27.148 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:22:27.148 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:22:27.181 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:22:27.180+0000 7f1101d36880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:22:27.182 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:22:27.181+0000 7f1101d36880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:22:27.197 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:22:27.197 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:22:27.197 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:22:27.197 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:22:27.197 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:22:27.197 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:22:27.197 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:22:27.198 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:22:27.198 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:22:27.198 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:22:27.198 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:22:27.198 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:22:27.198 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:22:27.198 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:22:27.198 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:22:27.198 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:22:27.198 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:22:27.198 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:22:27.198 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:22:27.198 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:22:27.199 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:22:27.199 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:22:27.199 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:22:27.199 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:22:27.199 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:22:27.199 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:22:27.199 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:22:27.199 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:22:27.199 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:22:27.199 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:22:27.199 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:22:27.200 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:22:27.200 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:22:27.200 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:22:27.200 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:22:27.200 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:22:27.200 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:22:27.200 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:22:27.200 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:22:27.200 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:22:27.200 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:22:27.200 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:22:27.200 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:22:27.200 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:22:27.200 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:22:27.201 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:22:27.201 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:22:27.201 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:22:27.201 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:22:27.201 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001538.454480_1552.1", 2024-08-18T17:22:27.201 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:22:27.201 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:22:27.201 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:22:27.201 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:18:58.454480Z", 2024-08-18T17:22:27.201 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:22:27.201 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:22:27.201 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:22:27.202 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:22:27.202 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:22:27.202 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:22:27.202 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:22:27.202 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:22:27.202 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:22:27.202 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:22:27Z" 2024-08-18T17:22:27.202 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:22:27.203 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001675.093496_1592.1 target=1_1724001538.454480_1552.1 2024-08-18T17:22:32.205 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:22:32.205 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:22:32.205 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:22:32.260 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:22:32.261 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:22:32.294 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:22:32.293+0000 7f9c60926880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:22:32.295 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:22:32.294+0000 7f9c60926880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:22:32.310 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:22:32.310 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:22:32.310 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:22:32.311 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:22:32.311 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:22:32.311 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:22:32.311 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:22:32.311 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:22:32.311 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:22:32.311 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:22:32.311 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:22:32.311 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:22:32.311 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:22:32.311 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:22:32.311 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:22:32.312 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:22:32.312 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:22:32.312 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:22:32.312 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:22:32.312 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:22:32.312 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:22:32.312 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:22:32.312 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:22:32.312 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:22:32.312 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:22:32.313 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:22:32.313 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:22:32.313 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:22:32.313 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:22:32.313 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:22:32.313 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:22:32.313 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:22:32.313 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:22:32.313 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:22:32.313 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:22:32.313 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:22:32.313 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:22:32.314 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:22:32.314 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:22:32.314 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:22:32.314 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:22:32.314 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:22:32.314 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:22:32.314 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:22:32.314 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:22:32.314 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:22:32.314 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:22:32.315 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:22:32.315 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:22:32.315 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001538.454480_1552.1", 2024-08-18T17:22:32.315 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:22:32.315 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:22:32.315 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:22:32.315 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:18:58.454480Z", 2024-08-18T17:22:32.315 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:22:32.315 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:22:32.315 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:22:32.315 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:22:32.315 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:22:32.316 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:22:32.316 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:22:32.316 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:22:32.316 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:22:32.316 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:22:32Z" 2024-08-18T17:22:32.316 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:22:32.317 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001675.093496_1592.1 target=1_1724001538.454480_1552.1 2024-08-18T17:22:37.318 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:22:37.319 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:22:37.319 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:22:37.374 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:22:37.374 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:22:37.409 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:22:37.408+0000 7eff6b630880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:22:37.409 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:22:37.409+0000 7eff6b630880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:22:37.424 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:22:37.424 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:22:37.424 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:22:37.424 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:22:37.425 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:22:37.425 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:22:37.425 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:22:37.425 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:22:37.425 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:22:37.425 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:22:37.425 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:22:37.425 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:22:37.425 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:22:37.425 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:22:37.425 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:22:37.425 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:22:37.425 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:22:37.426 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:22:37.426 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:22:37.426 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:22:37.426 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:22:37.426 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:22:37.426 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:22:37.426 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:22:37.426 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:22:37.426 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:22:37.426 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:22:37.426 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:22:37.426 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:22:37.427 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:22:37.427 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:22:37.427 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:22:37.427 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:22:37.427 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:22:37.427 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:22:37.427 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:22:37.427 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:22:37.427 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:22:37.427 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:22:37.427 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:22:37.428 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:22:37.428 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:22:37.428 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:22:37.428 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:22:37.428 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:22:37.428 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:22:37.428 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:22:37.428 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:22:37.428 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:22:37.428 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001538.454480_1552.1", 2024-08-18T17:22:37.429 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:22:37.429 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:22:37.429 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:22:37.429 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:18:58.454480Z", 2024-08-18T17:22:37.429 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:22:37.429 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:22:37.429 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:22:37.429 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:22:37.429 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:22:37.429 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:22:37.429 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:22:37.429 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:22:37.429 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:22:37.430 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:22:37Z" 2024-08-18T17:22:37.430 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:22:37.431 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001675.093496_1592.1 target=1_1724001538.454480_1552.1 2024-08-18T17:22:42.431 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:22:42.431 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:22:42.431 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:22:42.487 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:22:42.487 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:22:42.522 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:22:42.521+0000 7fe90800f880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:22:42.522 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:22:42.522+0000 7fe90800f880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:22:42.537 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:22:42.537 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:22:42.537 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:22:42.537 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:22:42.537 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:22:42.538 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:22:42.538 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:22:42.538 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:22:42.538 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:22:42.538 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:22:42.538 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:22:42.538 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:22:42.538 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:22:42.538 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:22:42.538 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:22:42.538 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:22:42.538 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:22:42.539 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:22:42.539 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:22:42.539 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:22:42.539 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:22:42.539 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:22:42.539 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:22:42.539 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:22:42.539 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:22:42.539 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:22:42.539 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:22:42.539 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:22:42.539 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:22:42.539 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:22:42.540 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:22:42.540 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:22:42.540 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:22:42.540 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:22:42.540 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:22:42.540 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:22:42.540 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:22:42.540 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:22:42.540 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:22:42.540 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:22:42.540 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:22:42.540 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:22:42.540 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:22:42.541 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:22:42.541 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:22:42.541 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:22:42.541 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:22:42.541 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:22:42.541 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:22:42.541 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001538.454480_1552.1", 2024-08-18T17:22:42.541 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:22:42.541 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:22:42.541 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:22:42.541 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:18:58.454480Z", 2024-08-18T17:22:42.541 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:22:42.541 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:22:42.542 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:22:42.542 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:22:42.542 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:22:42.542 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:22:42.542 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:22:42.542 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:22:42.542 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:22:42.542 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:22:42Z" 2024-08-18T17:22:42.542 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:22:42.543 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001675.093496_1592.1 target=1_1724001538.454480_1552.1 2024-08-18T17:22:47.544 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:22:47.544 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:22:47.544 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:22:47.601 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:22:47.601 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:22:47.635 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:22:47.634+0000 7f8bf3612880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:22:47.636 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:22:47.635+0000 7f8bf3612880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:22:47.651 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:22:47.651 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:22:47.651 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:22:47.651 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:22:47.651 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:22:47.651 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:22:47.651 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:22:47.651 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:22:47.651 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:22:47.651 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:22:47.652 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:22:47.652 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:22:47.652 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:22:47.652 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:22:47.652 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:22:47.652 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:22:47.652 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:22:47.652 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:22:47.652 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:22:47.652 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:22:47.652 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:22:47.652 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:22:47.652 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:22:47.653 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:22:47.653 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:22:47.653 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:22:47.653 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:22:47.653 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:22:47.653 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:22:47.653 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:22:47.653 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:22:47.653 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:22:47.653 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:22:47.653 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:22:47.653 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:22:47.653 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:22:47.653 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:22:47.654 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:22:47.654 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:22:47.654 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:22:47.654 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:22:47.654 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:22:47.654 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:22:47.654 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:22:47.654 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:22:47.654 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:22:47.654 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:22:47.655 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:22:47.655 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:22:47.655 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001675.093496_1592.1", 2024-08-18T17:22:47.655 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:22:47.655 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:22:47.655 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:22:47.655 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:21:15.093496Z", 2024-08-18T17:22:47.655 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:22:47.655 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:22:47.655 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:22:47.655 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:22:47.655 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:22:47.655 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:22:47.656 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:22:47.656 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:22:47.656 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:22:47.656 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:22:47Z" 2024-08-18T17:22:47.656 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:22:47.657 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-08-18T17:22:47.657 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-08-18T17:22:47.657 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:22:47.657 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:22:47.658 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:22:47.712 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:22:47.712 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:22:47.753 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:22:47.752+0000 7f835bb22880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:22:47.753 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:22:47.753+0000 7f835bb22880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:22:47.776 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T17:22:47.776 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T17:22:47.776 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T17:22:47.776 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T17:22:47.776 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T17:22:47.776 INFO:teuthology.orchestra.run.smithi134.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:22:47.777 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:22:47.777 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:22:47.777 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T17:22:47.777 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:22:47.777 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T17:22:47.777 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:22:47.777 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:22:47.777 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:22:47.777 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:22:47.777 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 1, 2024-08-18T17:22:47.777 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:22:47.777 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:22:47.777 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T17:22:47.777 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:22:47.778 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:22:47.778 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:22:47.778 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T17:22:47.778 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:22:47.778 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:22:47.778 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:22:47.778 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:22:47.778 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:22:47.778 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:22:47.778 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T17:22:47.778 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T17:22:47.778 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:22:47.779 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:22:47.779 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:22:47.779 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T17:22:47.779 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:22:47.779 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:22:47.779 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:22:47.779 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:22:47.779 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:22:47.779 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:22:47.779 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:22:47.779 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:22:47.779 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:22:47.779 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:22:47.779 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:22:47.780 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T17:22:47.780 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:22:47.780 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:22:47.780 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1724001538.454480_1552.1", 2024-08-18T17:22:47.780 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:22:47.780 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:22:47.780 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:22:47.780 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T17:18:58.454480Z", 2024-08-18T17:22:47.780 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:22:47.780 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:22:47.780 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:22:47.780 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:22:47.780 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:22:47.781 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T17:22:47.781 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 1, 2024-08-18T17:22:47.781 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 1 2024-08-18T17:22:47.781 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:22:47.781 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T17:22:47Z" 2024-08-18T17:22:47.781 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T17:22:47.782 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1724001675.093496_1592.1 target=1_1724001538.454480_1552.1 2024-08-18T17:22:52.783 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:22:52.783 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:22:52.783 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:22:52.838 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:22:52.838 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:22:52.873 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:22:52.872+0000 7fe34f411880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:22:52.874 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:22:52.873+0000 7fe34f411880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:22:52.888 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T17:22:52.888 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T17:22:52.888 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T17:22:52.888 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T17:22:52.888 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T17:22:52.889 INFO:teuthology.orchestra.run.smithi134.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:22:52.889 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:22:52.889 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:22:52.889 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T17:22:52.889 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:22:52.889 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T17:22:52.889 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:22:52.889 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:22:52.889 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:22:52.889 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:22:52.889 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 1, 2024-08-18T17:22:52.889 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:22:52.889 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:22:52.890 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T17:22:52.890 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:22:52.890 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:22:52.890 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:22:52.890 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T17:22:52.890 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:22:52.890 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:22:52.890 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:22:52.890 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:22:52.890 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:22:52.890 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:22:52.890 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T17:22:52.890 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T17:22:52.891 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:22:52.891 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:22:52.891 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:22:52.891 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T17:22:52.891 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:22:52.891 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:22:52.891 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:22:52.891 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:22:52.891 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:22:52.891 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:22:52.891 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:22:52.891 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:22:52.892 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:22:52.892 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:22:52.892 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:22:52.892 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T17:22:52.892 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:22:52.892 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:22:52.892 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1724001538.454480_1552.1", 2024-08-18T17:22:52.892 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:22:52.892 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:22:52.892 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:22:52.892 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T17:18:58.454480Z", 2024-08-18T17:22:52.892 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:22:52.892 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:22:52.892 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:22:52.893 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:22:52.893 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:22:52.893 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T17:22:52.893 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 1, 2024-08-18T17:22:52.893 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 1 2024-08-18T17:22:52.893 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:22:52.893 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T17:22:52Z" 2024-08-18T17:22:52.893 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T17:22:52.894 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1724001675.093496_1592.1 target=1_1724001538.454480_1552.1 2024-08-18T17:22:57.895 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:22:57.895 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:22:57.895 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:22:57.950 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:22:57.950 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:22:57.988 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:22:57.987+0000 7ffaf5410880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:22:57.988 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:22:57.988+0000 7ffaf5410880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:22:58.005 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T17:22:58.005 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T17:22:58.006 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T17:22:58.006 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T17:22:58.006 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T17:22:58.006 INFO:teuthology.orchestra.run.smithi134.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:22:58.006 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:22:58.006 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:22:58.006 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T17:22:58.006 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:22:58.006 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T17:22:58.006 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:22:58.006 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:22:58.007 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:22:58.007 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:22:58.007 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 1, 2024-08-18T17:22:58.007 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:22:58.007 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:22:58.007 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T17:22:58.007 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:22:58.007 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:22:58.007 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:22:58.007 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T17:22:58.007 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:22:58.007 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:22:58.007 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:22:58.008 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:22:58.008 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:22:58.008 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:22:58.008 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T17:22:58.008 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T17:22:58.008 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:22:58.008 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:22:58.008 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:22:58.008 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T17:22:58.008 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:22:58.008 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:22:58.008 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:22:58.008 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:22:58.009 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:22:58.009 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:22:58.009 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:22:58.009 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:22:58.009 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:22:58.009 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:22:58.009 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:22:58.009 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T17:22:58.009 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:22:58.009 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:22:58.009 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1724001538.454480_1552.1", 2024-08-18T17:22:58.009 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:22:58.009 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:22:58.010 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:22:58.010 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T17:18:58.454480Z", 2024-08-18T17:22:58.010 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:22:58.010 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:22:58.010 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:22:58.010 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:22:58.010 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:22:58.010 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T17:22:58.010 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 1, 2024-08-18T17:22:58.010 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 1 2024-08-18T17:22:58.010 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:22:58.010 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T17:22:57Z" 2024-08-18T17:22:58.011 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T17:22:58.012 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1724001675.093496_1592.1 target=1_1724001538.454480_1552.1 2024-08-18T17:23:03.013 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:23:03.013 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:23:03.013 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:23:03.069 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:23:03.069 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:23:03.112 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:23:03.111+0000 7f58b0b38880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:23:03.113 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:23:03.112+0000 7f58b0b38880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:23:03.130 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T17:23:03.130 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T17:23:03.130 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T17:23:03.130 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T17:23:03.130 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T17:23:03.130 INFO:teuthology.orchestra.run.smithi134.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:23:03.130 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:23:03.130 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:23:03.131 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T17:23:03.131 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:23:03.131 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T17:23:03.131 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:23:03.131 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:23:03.131 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:23:03.131 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:23:03.131 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 1, 2024-08-18T17:23:03.131 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:23:03.131 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:23:03.131 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T17:23:03.131 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:23:03.131 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:23:03.132 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:23:03.132 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T17:23:03.132 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:23:03.132 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:23:03.132 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:23:03.132 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:23:03.132 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:23:03.132 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:23:03.132 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T17:23:03.132 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T17:23:03.132 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:23:03.132 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:23:03.132 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:23:03.132 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T17:23:03.133 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:23:03.133 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:23:03.133 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:23:03.133 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:23:03.133 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:23:03.133 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:23:03.133 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:23:03.133 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:23:03.133 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:23:03.133 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:23:03.133 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:23:03.133 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T17:23:03.133 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:23:03.133 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:23:03.134 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1724001538.454480_1552.1", 2024-08-18T17:23:03.134 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:23:03.134 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:23:03.134 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:23:03.134 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T17:18:58.454480Z", 2024-08-18T17:23:03.134 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:23:03.134 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:23:03.134 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:23:03.134 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:23:03.134 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:23:03.134 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T17:23:03.134 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 1, 2024-08-18T17:23:03.134 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 1 2024-08-18T17:23:03.135 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:23:03.135 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T17:23:03Z" 2024-08-18T17:23:03.135 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T17:23:03.136 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1724001675.093496_1592.1 target=1_1724001538.454480_1552.1 2024-08-18T17:23:08.136 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:23:08.137 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:23:08.137 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:23:08.192 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:23:08.192 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:23:08.229 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:23:08.228+0000 7f4c2d035880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:23:08.230 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:23:08.229+0000 7f4c2d035880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:23:08.250 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T17:23:08.250 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T17:23:08.250 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T17:23:08.250 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T17:23:08.250 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T17:23:08.250 INFO:teuthology.orchestra.run.smithi134.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:23:08.250 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:23:08.250 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:23:08.251 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T17:23:08.251 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:23:08.251 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T17:23:08.251 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:23:08.251 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:23:08.251 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:23:08.251 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:23:08.251 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 1, 2024-08-18T17:23:08.251 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:23:08.251 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:23:08.251 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T17:23:08.251 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:23:08.251 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:23:08.251 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:23:08.252 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T17:23:08.252 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:23:08.252 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:23:08.252 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:23:08.252 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:23:08.252 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:23:08.252 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:23:08.252 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T17:23:08.252 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T17:23:08.252 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:23:08.252 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:23:08.252 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:23:08.252 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T17:23:08.253 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:23:08.253 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:23:08.253 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:23:08.253 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:23:08.253 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:23:08.253 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:23:08.253 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:23:08.253 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:23:08.253 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:23:08.254 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:23:08.254 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:23:08.254 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T17:23:08.254 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:23:08.254 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:23:08.254 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1724001538.454480_1552.1", 2024-08-18T17:23:08.254 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:23:08.254 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:23:08.254 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:23:08.254 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T17:18:58.454480Z", 2024-08-18T17:23:08.255 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:23:08.255 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:23:08.255 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:23:08.255 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:23:08.255 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:23:08.255 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T17:23:08.255 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 1, 2024-08-18T17:23:08.255 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 1 2024-08-18T17:23:08.255 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:23:08.255 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T17:23:08Z" 2024-08-18T17:23:08.255 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T17:23:08.256 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1724001675.093496_1592.1 target=1_1724001538.454480_1552.1 2024-08-18T17:23:13.257 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:23:13.258 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:23:13.258 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:23:13.314 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:23:13.314 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:23:13.369 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:23:13.368+0000 7fabc9a0f880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:23:13.369 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:23:13.369+0000 7fabc9a0f880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:23:13.394 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T17:23:13.394 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T17:23:13.394 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T17:23:13.394 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T17:23:13.394 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T17:23:13.394 INFO:teuthology.orchestra.run.smithi134.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:23:13.394 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:23:13.394 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:23:13.395 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T17:23:13.395 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:23:13.395 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T17:23:13.395 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:23:13.395 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:23:13.395 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:23:13.395 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:23:13.395 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 1, 2024-08-18T17:23:13.395 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:23:13.395 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:23:13.395 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T17:23:13.395 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:23:13.395 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:23:13.395 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:23:13.396 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T17:23:13.396 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:23:13.396 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:23:13.396 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:23:13.396 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:23:13.396 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:23:13.396 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:23:13.396 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T17:23:13.396 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T17:23:13.396 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:23:13.396 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:23:13.396 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:23:13.396 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T17:23:13.397 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:23:13.397 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:23:13.397 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:23:13.397 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:23:13.397 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:23:13.397 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:23:13.397 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:23:13.397 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:23:13.397 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:23:13.397 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:23:13.397 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:23:13.397 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T17:23:13.397 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:23:13.397 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:23:13.398 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1724001538.454480_1552.1", 2024-08-18T17:23:13.398 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:23:13.398 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:23:13.398 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:23:13.398 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T17:18:58.454480Z", 2024-08-18T17:23:13.398 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:23:13.398 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:23:13.398 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:23:13.398 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:23:13.398 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:23:13.398 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T17:23:13.398 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 1, 2024-08-18T17:23:13.398 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 1 2024-08-18T17:23:13.398 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:23:13.399 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T17:23:13Z" 2024-08-18T17:23:13.399 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T17:23:13.400 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1724001675.093496_1592.1 target=1_1724001538.454480_1552.1 2024-08-18T17:23:18.401 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:23:18.401 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:23:18.401 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:23:18.457 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:23:18.458 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:23:18.495 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:23:18.494+0000 7f60b8e15880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:23:18.495 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:23:18.495+0000 7f60b8e15880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:23:18.513 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T17:23:18.513 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T17:23:18.514 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T17:23:18.514 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T17:23:18.514 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T17:23:18.514 INFO:teuthology.orchestra.run.smithi134.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:23:18.514 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:23:18.514 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:23:18.514 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T17:23:18.514 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:23:18.514 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T17:23:18.514 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:23:18.514 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:23:18.514 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:23:18.515 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:23:18.515 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 1, 2024-08-18T17:23:18.515 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:23:18.515 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:23:18.515 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T17:23:18.515 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:23:18.515 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:23:18.515 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:23:18.515 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T17:23:18.515 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:23:18.515 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:23:18.515 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:23:18.515 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:23:18.515 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:23:18.516 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:23:18.516 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T17:23:18.516 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T17:23:18.516 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:23:18.516 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:23:18.516 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:23:18.516 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T17:23:18.516 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:23:18.516 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:23:18.516 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:23:18.516 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:23:18.516 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:23:18.517 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:23:18.517 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:23:18.517 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:23:18.517 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:23:18.517 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:23:18.517 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:23:18.517 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T17:23:18.517 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:23:18.517 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:23:18.517 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1724001675.093496_1592.1", 2024-08-18T17:23:18.517 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:23:18.517 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:23:18.517 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:23:18.517 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T17:21:15.093496Z", 2024-08-18T17:23:18.518 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:23:18.518 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:23:18.518 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:23:18.518 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:23:18.518 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:23:18.518 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T17:23:18.518 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 1, 2024-08-18T17:23:18.518 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 1 2024-08-18T17:23:18.518 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:23:18.518 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T17:23:18Z" 2024-08-18T17:23:18.518 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T17:23:18.519 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-08-18T17:23:18.520 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:23:18.520 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:23:18.520 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:23:18.567 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:23:18.567 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:23:18.655 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T17:23:18.655 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:23:18.656 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000051734", 2024-08-18T17:23:18.656 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T17:19:10.076905Z" 2024-08-18T17:23:18.656 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:23:18.656 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:23:18.656 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:23:18.656 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:23:18.656 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:23:18.656 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:23:18.656 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000059417", 2024-08-18T17:23:18.656 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T17:21:15.361961Z" 2024-08-18T17:23:18.656 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:23:18.656 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:23:18.657 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:23:18.657 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:23:18.657 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:23:18.657 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T17:23:18.658 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-08-18T17:23:18.658 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:23:18.658 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:23:18.658 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:23:18.714 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:23:18.715 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:23:18.766 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:23:18.766 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:23:18.766 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:23:18.766 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:23:18.766 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:23:18.766 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:23:18.766 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:23:18.766 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:23:18.766 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:23:18.766 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:23:18.766 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:23:18.767 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:23:18.767 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000051734", 2024-08-18T17:23:18.767 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:23:18.767 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:23:18.767 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:23:18.767 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.076905Z" 2024-08-18T17:23:18.767 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:23:18.767 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:23:18.767 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:23:18.767 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:23:18.767 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:23:18.767 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:23:18.767 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:23:18.767 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:23:18.768 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:23:18.768 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:23:18.768 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:23:18.768 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:23:18.768 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:23:18.768 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:23:18.768 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:23:18.768 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:23:18.768 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:23:18.768 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000059417", 2024-08-18T17:23:18.768 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:23:18.768 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:23:18.768 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:23:18.769 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:21:15.361961Z" 2024-08-18T17:23:18.769 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:23:18.769 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:23:18.769 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:23:18.769 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:23:18.769 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:23:18.769 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:23:18.769 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:23:18.769 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:23:18.769 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:23:18.769 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:23:18.769 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:23:18.769 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:23:18.770 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:23:18.770 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:23:18.770 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:23:18.770 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:23:18.770 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:23:18.770 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:23:18.770 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:23:18.770 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:23:18Z" 2024-08-18T17:23:18.770 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:23:18.771 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-08-18T17:23:18.785 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'kibgmb-91', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:23:18.785 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'kibgmb-91', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:23:18.785 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-bucket --bucket kibgmb-91 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:23:18.833 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:23:18.833 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:23:18.910 INFO:teuthology.orchestra.run.smithi134.stdout:{"sync_policy":{"groups":[]}} 2024-08-18T17:23:18.911 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:23:18.911 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:23:18.911 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:23:18.956 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:23:18.956 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:23:19.026 INFO:teuthology.orchestra.run.smithi134.stdout:{"sync_policy":{"groups":[]}} 2024-08-18T17:23:19.027 INFO:tasks.rgw_multisite_tests:test_sync_single_bucket: ... ok 2024-08-18T17:23:19.027 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:23:19.027 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:23:19.027 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:23:19.073 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:23:19.074 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:23:19.113 INFO:teuthology.orchestra.run.smithi134.stderr:No --period given, using current period=01050e30-179b-43be-bdbb-3bb23cf37ae8 2024-08-18T17:23:19.131 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T17:23:19.131 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:23:19.132 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:23:19.132 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T17:23:19.132 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:23:19.132 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:23:19.132 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:23:19.132 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T17:23:19.132 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:23:19.132 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:23:19.133 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:23:19.133 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:42:47.672461Z" 2024-08-18T17:23:19.133 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:23:19.133 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:23:19.133 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1724001798.886581_1599.1", 2024-08-18T17:23:19.133 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T17:23:18.886581Z" 2024-08-18T17:23:19.133 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:23:19.134 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T17:23:19.135 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-08-18T17:23:19.135 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:23:19.135 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:23:19.136 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:23:19.187 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:23:19.187 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:23:19.219 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:23:19.218+0000 7fb0eecbb880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:23:19.219 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:23:19.219+0000 7fb0eecbb880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:23:19.234 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:23:19.234 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:23:19.234 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:23:19.234 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:23:19.234 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:23:19.234 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:23:19.234 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:23:19.234 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:23:19.235 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:23:19.235 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:23:19.235 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:23:19.235 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:23:19.235 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:23:19.235 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:23:19.235 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:23:19.235 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:23:19.235 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:23:19.235 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:23:19.235 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:23:19.236 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:23:19.236 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:23:19.236 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:23:19.236 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:23:19.236 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:23:19.236 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:23:19.236 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:23:19.236 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:23:19.236 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:23:19.236 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:23:19.236 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:23:19.236 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:23:19.237 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:23:19.237 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:23:19.237 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:23:19.237 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:23:19.237 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:23:19.237 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:23:19.237 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:23:19.237 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:23:19.238 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:23:19.238 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:23:19.238 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:23:19.238 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:23:19.238 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:23:19.238 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:23:19.238 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:23:19.238 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:23:19.238 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:23:19.238 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:23:19.238 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001675.093496_1592.1", 2024-08-18T17:23:19.239 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:23:19.239 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:23:19.239 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:23:19.239 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:21:15.093496Z", 2024-08-18T17:23:19.239 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:23:19.239 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:23:19.239 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:23:19.239 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:23:19.240 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:23:19.240 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:23:19.240 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:23:19.240 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:23:19.240 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:23:19.240 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:23:19Z" 2024-08-18T17:23:19.240 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:23:19.241 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001798.886581_1599.1 target=1_1724001675.093496_1592.1 2024-08-18T17:23:24.242 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:23:24.243 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:23:24.243 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:23:24.298 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:23:24.298 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:23:24.332 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:23:24.331+0000 7f3f2e662880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:23:24.333 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:23:24.332+0000 7f3f2e662880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:23:24.348 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:23:24.348 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:23:24.348 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:23:24.348 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:23:24.348 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:23:24.348 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:23:24.348 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:23:24.349 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:23:24.349 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:23:24.349 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:23:24.349 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:23:24.349 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:23:24.349 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:23:24.349 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:23:24.349 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:23:24.349 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:23:24.349 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:23:24.349 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:23:24.349 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:23:24.349 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:23:24.349 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:23:24.350 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:23:24.350 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:23:24.350 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:23:24.350 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:23:24.350 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:23:24.350 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:23:24.350 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:23:24.350 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:23:24.350 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:23:24.350 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:23:24.350 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:23:24.350 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:23:24.350 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:23:24.351 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:23:24.351 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:23:24.351 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:23:24.351 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:23:24.351 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:23:24.351 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:23:24.351 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:23:24.351 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:23:24.351 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:23:24.351 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:23:24.351 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:23:24.351 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:23:24.351 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:23:24.352 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:23:24.352 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:23:24.352 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001675.093496_1592.1", 2024-08-18T17:23:24.352 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:23:24.352 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:23:24.352 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:23:24.352 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:21:15.093496Z", 2024-08-18T17:23:24.352 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:23:24.352 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:23:24.352 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:23:24.352 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:23:24.352 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:23:24.352 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:23:24.352 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:23:24.353 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:23:24.353 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:23:24.353 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:23:24Z" 2024-08-18T17:23:24.353 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:23:24.354 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001798.886581_1599.1 target=1_1724001675.093496_1592.1 2024-08-18T17:23:29.355 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:23:29.355 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:23:29.355 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:23:29.410 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:23:29.410 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:23:29.443 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:23:29.442+0000 7f7c43b20880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:23:29.444 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:23:29.443+0000 7f7c43b20880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:23:29.458 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:23:29.459 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:23:29.459 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:23:29.459 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:23:29.459 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:23:29.459 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:23:29.459 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:23:29.459 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:23:29.459 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:23:29.459 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:23:29.459 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:23:29.460 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:23:29.460 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:23:29.460 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:23:29.460 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:23:29.460 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:23:29.460 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:23:29.460 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:23:29.460 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:23:29.460 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:23:29.460 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:23:29.460 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:23:29.460 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:23:29.460 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:23:29.460 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:23:29.461 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:23:29.461 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:23:29.461 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:23:29.461 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:23:29.461 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:23:29.461 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:23:29.461 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:23:29.461 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:23:29.461 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:23:29.461 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:23:29.461 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:23:29.461 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:23:29.462 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:23:29.462 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:23:29.462 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:23:29.462 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:23:29.462 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:23:29.462 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:23:29.462 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:23:29.462 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:23:29.462 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:23:29.462 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:23:29.463 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:23:29.463 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:23:29.463 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001798.886581_1599.1", 2024-08-18T17:23:29.463 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:23:29.463 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:23:29.463 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:23:29.463 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:23:18.886581Z", 2024-08-18T17:23:29.463 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:23:29.463 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:23:29.463 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:23:29.463 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:23:29.463 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:23:29.463 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:23:29.464 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:23:29.464 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:23:29.464 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:23:29.464 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:23:29Z" 2024-08-18T17:23:29.464 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:23:29.465 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-08-18T17:23:29.465 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-08-18T17:23:29.465 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:23:29.465 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:23:29.465 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:23:29.520 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:23:29.520 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:23:29.567 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:23:29.567+0000 7f512f232880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:23:29.568 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:23:29.568+0000 7f512f232880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:23:29.587 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T17:23:29.587 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T17:23:29.587 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T17:23:29.587 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T17:23:29.587 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T17:23:29.587 INFO:teuthology.orchestra.run.smithi134.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:23:29.588 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:23:29.588 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:23:29.588 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T17:23:29.588 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:23:29.588 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T17:23:29.588 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:23:29.588 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:23:29.588 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:23:29.588 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:23:29.588 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 1, 2024-08-18T17:23:29.588 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:23:29.589 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:23:29.589 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T17:23:29.589 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:23:29.589 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:23:29.589 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:23:29.589 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T17:23:29.589 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:23:29.589 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:23:29.589 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:23:29.589 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:23:29.589 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:23:29.589 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:23:29.589 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T17:23:29.590 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T17:23:29.590 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:23:29.590 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:23:29.590 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:23:29.590 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T17:23:29.590 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:23:29.590 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:23:29.590 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:23:29.590 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:23:29.590 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:23:29.590 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:23:29.590 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:23:29.591 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:23:29.591 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:23:29.591 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:23:29.591 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:23:29.591 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T17:23:29.591 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:23:29.591 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:23:29.591 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1724001675.093496_1592.1", 2024-08-18T17:23:29.591 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:23:29.591 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:23:29.591 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:23:29.591 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T17:21:15.093496Z", 2024-08-18T17:23:29.591 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:23:29.591 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:23:29.592 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:23:29.592 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:23:29.592 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:23:29.592 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T17:23:29.592 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 1, 2024-08-18T17:23:29.592 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 1 2024-08-18T17:23:29.593 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:23:29.593 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T17:23:29Z" 2024-08-18T17:23:29.593 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T17:23:29.594 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1724001798.886581_1599.1 target=1_1724001675.093496_1592.1 2024-08-18T17:23:34.595 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:23:34.596 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:23:34.596 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:23:34.650 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:23:34.651 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:23:34.687 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:23:34.686+0000 7f3be9c0f880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:23:34.688 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:23:34.687+0000 7f3be9c0f880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:23:34.703 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T17:23:34.703 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T17:23:34.703 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T17:23:34.703 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T17:23:34.703 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T17:23:34.704 INFO:teuthology.orchestra.run.smithi134.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:23:34.704 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:23:34.704 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:23:34.704 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T17:23:34.704 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:23:34.704 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T17:23:34.704 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:23:34.704 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:23:34.704 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:23:34.704 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:23:34.704 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 1, 2024-08-18T17:23:34.704 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:23:34.704 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:23:34.705 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T17:23:34.705 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:23:34.705 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:23:34.705 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:23:34.705 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T17:23:34.705 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:23:34.705 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:23:34.705 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:23:34.705 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:23:34.705 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:23:34.705 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:23:34.705 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T17:23:34.706 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T17:23:34.706 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:23:34.706 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:23:34.706 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:23:34.706 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T17:23:34.706 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:23:34.706 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:23:34.706 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:23:34.706 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:23:34.706 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:23:34.706 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:23:34.706 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:23:34.706 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:23:34.707 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:23:34.707 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:23:34.707 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:23:34.707 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T17:23:34.707 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:23:34.707 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:23:34.707 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1724001675.093496_1592.1", 2024-08-18T17:23:34.707 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:23:34.707 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:23:34.707 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:23:34.707 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T17:21:15.093496Z", 2024-08-18T17:23:34.707 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:23:34.707 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:23:34.708 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:23:34.708 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:23:34.708 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:23:34.708 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T17:23:34.708 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 1, 2024-08-18T17:23:34.708 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 1 2024-08-18T17:23:34.708 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:23:34.708 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T17:23:34Z" 2024-08-18T17:23:34.708 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T17:23:34.709 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1724001798.886581_1599.1 target=1_1724001675.093496_1592.1 2024-08-18T17:23:39.710 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:23:39.710 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:23:39.710 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:23:39.765 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:23:39.766 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:23:39.822 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:23:39.821+0000 7fab9d32a880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:23:39.822 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:23:39.822+0000 7fab9d32a880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:23:39.841 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T17:23:39.841 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T17:23:39.841 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T17:23:39.842 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T17:23:39.842 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T17:23:39.842 INFO:teuthology.orchestra.run.smithi134.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:23:39.842 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:23:39.842 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:23:39.842 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T17:23:39.842 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:23:39.842 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T17:23:39.842 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:23:39.842 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:23:39.842 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:23:39.842 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:23:39.843 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 1, 2024-08-18T17:23:39.843 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:23:39.843 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:23:39.843 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T17:23:39.843 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:23:39.843 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:23:39.843 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:23:39.843 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T17:23:39.843 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:23:39.843 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:23:39.843 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:23:39.843 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:23:39.843 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:23:39.843 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:23:39.843 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T17:23:39.844 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T17:23:39.844 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:23:39.844 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:23:39.844 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:23:39.844 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T17:23:39.844 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:23:39.844 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:23:39.844 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:23:39.844 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:23:39.844 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:23:39.844 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:23:39.844 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:23:39.844 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:23:39.844 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:23:39.844 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:23:39.844 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:23:39.845 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T17:23:39.845 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:23:39.845 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:23:39.845 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1724001798.886581_1599.1", 2024-08-18T17:23:39.845 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:23:39.845 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:23:39.845 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:23:39.845 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T17:23:18.886581Z", 2024-08-18T17:23:39.845 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:23:39.845 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:23:39.845 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:23:39.845 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:23:39.845 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:23:39.845 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T17:23:39.845 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 1, 2024-08-18T17:23:39.846 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 1 2024-08-18T17:23:39.846 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:23:39.846 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T17:23:39Z" 2024-08-18T17:23:39.846 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T17:23:39.847 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-08-18T17:23:39.847 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:23:39.847 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:23:39.847 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:23:39.891 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:23:39.892 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:23:39.941 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[]} 2024-08-18T17:23:39.942 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=kibgmb-93 2024-08-18T17:23:39.961 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=kibgmb-94 2024-08-18T17:23:39.980 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:23:39.980 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:23:39.980 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:23:40.026 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:23:40.026 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:23:40.071 INFO:teuthology.orchestra.run.smithi134.stderr:No --period given, using current period=01050e30-179b-43be-bdbb-3bb23cf37ae8 2024-08-18T17:23:40.093 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T17:23:40.093 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:23:40.093 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:23:40.093 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T17:23:40.093 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:23:40.093 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:23:40.093 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:23:40.093 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T17:23:40.093 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:23:40.093 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:23:40.093 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:23:40.094 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:42:47.672461Z" 2024-08-18T17:23:40.094 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:23:40.094 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:23:40.094 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1724001819.978375_1607.1", 2024-08-18T17:23:40.094 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T17:23:39.978375Z" 2024-08-18T17:23:40.094 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:23:40.094 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T17:23:40.095 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-08-18T17:23:40.095 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:23:40.095 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:23:40.096 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:23:40.153 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:23:40.153 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:23:40.192 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:23:40.191+0000 7fed0e010880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:23:40.193 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:23:40.192+0000 7fed0e010880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:23:40.211 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:23:40.212 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:23:40.212 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:23:40.212 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:23:40.212 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:23:40.212 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:23:40.212 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:23:40.212 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:23:40.213 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:23:40.213 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:23:40.213 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:23:40.213 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:23:40.213 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:23:40.213 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:23:40.213 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:23:40.213 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:23:40.213 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:23:40.213 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:23:40.213 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:23:40.214 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:23:40.214 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:23:40.214 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:23:40.214 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:23:40.214 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:23:40.214 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:23:40.214 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:23:40.215 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:23:40.215 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:23:40.215 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:23:40.215 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:23:40.215 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:23:40.215 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:23:40.215 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:23:40.215 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:23:40.215 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:23:40.215 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:23:40.215 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:23:40.215 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:23:40.215 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:23:40.216 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:23:40.216 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:23:40.216 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:23:40.216 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:23:40.216 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:23:40.216 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:23:40.216 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:23:40.216 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:23:40.216 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:23:40.216 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:23:40.217 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001798.886581_1599.1", 2024-08-18T17:23:40.217 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:23:40.217 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:23:40.217 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:23:40.217 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:23:18.886581Z", 2024-08-18T17:23:40.217 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:23:40.217 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:23:40.217 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:23:40.217 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:23:40.217 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:23:40.218 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:23:40.218 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:23:40.218 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:23:40.218 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:23:40.218 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:23:40Z" 2024-08-18T17:23:40.218 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:23:40.220 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001819.978375_1607.1 target=1_1724001798.886581_1599.1 2024-08-18T17:23:45.221 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:23:45.221 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:23:45.221 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:23:45.278 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:23:45.278 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:23:45.311 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:23:45.311+0000 7f8bae019880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:23:45.312 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:23:45.312+0000 7f8bae019880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:23:45.327 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:23:45.327 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:23:45.327 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:23:45.327 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:23:45.328 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:23:45.328 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:23:45.328 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:23:45.328 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:23:45.328 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:23:45.328 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:23:45.328 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:23:45.328 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:23:45.328 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:23:45.328 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:23:45.328 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:23:45.328 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:23:45.329 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:23:45.329 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:23:45.329 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:23:45.329 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:23:45.329 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:23:45.329 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:23:45.329 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:23:45.329 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:23:45.329 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:23:45.329 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:23:45.329 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:23:45.329 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:23:45.329 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:23:45.330 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:23:45.330 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:23:45.330 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:23:45.330 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:23:45.330 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:23:45.330 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:23:45.330 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:23:45.330 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:23:45.330 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:23:45.330 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:23:45.330 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:23:45.330 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:23:45.331 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:23:45.331 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:23:45.331 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:23:45.331 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:23:45.331 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:23:45.331 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:23:45.331 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:23:45.331 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:23:45.331 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001819.978375_1607.1", 2024-08-18T17:23:45.331 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:23:45.331 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:23:45.331 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:23:45.331 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:23:39.978375Z", 2024-08-18T17:23:45.332 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:23:45.332 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:23:45.332 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:23:45.332 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:23:45.332 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:23:45.332 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:23:45.332 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:23:45.332 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:23:45.332 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:23:45.332 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:23:45Z" 2024-08-18T17:23:45.332 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:23:45.333 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-08-18T17:23:45.333 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-08-18T17:23:45.334 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:23:45.334 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:23:45.334 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:23:45.390 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:23:45.390 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:23:45.436 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:23:45.435+0000 7f861e817880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:23:45.437 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:23:45.436+0000 7f861e817880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:23:45.455 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T17:23:45.455 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T17:23:45.456 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T17:23:45.456 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T17:23:45.456 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T17:23:45.456 INFO:teuthology.orchestra.run.smithi134.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:23:45.456 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:23:45.456 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:23:45.456 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T17:23:45.456 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:23:45.456 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T17:23:45.456 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:23:45.456 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:23:45.457 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:23:45.457 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:23:45.457 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 1, 2024-08-18T17:23:45.457 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:23:45.457 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:23:45.457 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T17:23:45.457 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:23:45.457 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:23:45.457 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:23:45.457 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T17:23:45.457 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:23:45.457 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:23:45.457 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:23:45.458 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:23:45.458 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:23:45.458 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:23:45.458 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T17:23:45.458 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T17:23:45.458 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:23:45.458 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:23:45.458 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:23:45.458 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T17:23:45.458 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:23:45.458 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:23:45.458 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:23:45.459 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:23:45.459 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:23:45.459 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:23:45.459 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:23:45.459 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:23:45.459 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:23:45.459 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:23:45.459 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:23:45.459 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T17:23:45.460 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:23:45.460 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:23:45.460 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1724001819.978375_1607.1", 2024-08-18T17:23:45.460 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:23:45.460 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:23:45.460 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:23:45.460 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T17:23:39.978375Z", 2024-08-18T17:23:45.460 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:23:45.461 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:23:45.461 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:23:45.461 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:23:45.461 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:23:45.461 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T17:23:45.461 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 1, 2024-08-18T17:23:45.461 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 1 2024-08-18T17:23:45.461 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:23:45.461 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T17:23:45Z" 2024-08-18T17:23:45.461 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T17:23:45.463 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-08-18T17:23:45.463 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:23:45.463 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:23:45.463 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:23:45.511 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:23:45.511 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:23:45.586 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2024-08-18T17:23:45.586 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:23:45.586 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:23:45.586 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=test-zone1,test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:23:45.637 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:23:45.637 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:23:45.703 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}]},"pipes":[],"status":"allowed"}]} 2024-08-18T17:23:45.703 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--source-bucket=kibgmb-93', '--dest-bucket=kibgmb-94', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:23:45.703 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--source-bucket=kibgmb-93', '--dest-bucket=kibgmb-94', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:23:45.704 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=test-zone1 --dest-zones=test-zone2 --source-bucket=kibgmb-93 --dest-bucket=kibgmb-94 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:23:45.756 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:23:45.756 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:23:45.808 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"kibgmb-93","zones":["test-zone1"]},"dest":{"bucket":"kibgmb-94","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-08-18T17:23:45.808 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:23:45.808 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:23:45.808 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:23:45.855 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:23:45.856 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:23:45.942 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"kibgmb-93","zones":["test-zone1"]},"dest":{"bucket":"kibgmb-94","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-08-18T17:23:45.943 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:23:45.943 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:23:45.943 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:23:45.989 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:23:45.989 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:23:46.055 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T17:23:46.056 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:23:46.056 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 17, 2024-08-18T17:23:46.056 INFO:teuthology.orchestra.run.smithi134.stdout: "predecessor_uuid": "64309209-7567-46e5-8881-65783d84cf3c", 2024-08-18T17:23:46.056 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": [ 2024-08-18T17:23:46.056 INFO:teuthology.orchestra.run.smithi134.stdout: "", 2024-08-18T17:23:46.056 INFO:teuthology.orchestra.run.smithi134.stdout: "", 2024-08-18T17:23:46.056 INFO:teuthology.orchestra.run.smithi134.stdout: "", 2024-08-18T17:23:46.056 INFO:teuthology.orchestra.run.smithi134.stdout: "1_1723995848.930494_187.1" 2024-08-18T17:23:46.056 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:23:46.056 INFO:teuthology.orchestra.run.smithi134.stdout: "period_map": { 2024-08-18T17:23:46.056 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:23:46.056 INFO:teuthology.orchestra.run.smithi134.stdout: "zonegroups": [ 2024-08-18T17:23:46.056 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:23:46.057 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "test-zonegroup", 2024-08-18T17:23:46.057 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "test-zonegroup", 2024-08-18T17:23:46.057 INFO:teuthology.orchestra.run.smithi134.stdout: "api_name": "", 2024-08-18T17:23:46.057 INFO:teuthology.orchestra.run.smithi134.stdout: "is_master": true, 2024-08-18T17:23:46.057 INFO:teuthology.orchestra.run.smithi134.stdout: "endpoints": [ 2024-08-18T17:23:46.057 INFO:teuthology.orchestra.run.smithi134.stdout: "http://smithi134.front.sepia.ceph.com:8000" 2024-08-18T17:23:46.057 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:23:46.057 INFO:teuthology.orchestra.run.smithi134.stdout: "hostnames": [], 2024-08-18T17:23:46.057 INFO:teuthology.orchestra.run.smithi134.stdout: "hostnames_s3website": [], 2024-08-18T17:23:46.057 INFO:teuthology.orchestra.run.smithi134.stdout: "master_zone": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T17:23:46.057 INFO:teuthology.orchestra.run.smithi134.stdout: "zones": [ 2024-08-18T17:23:46.057 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:23:46.057 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T17:23:46.058 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "test-zone3", 2024-08-18T17:23:46.058 INFO:teuthology.orchestra.run.smithi134.stdout: "endpoints": [ 2024-08-18T17:23:46.058 INFO:teuthology.orchestra.run.smithi134.stdout: "http://smithi134.front.sepia.ceph.com:8001" 2024-08-18T17:23:46.058 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:23:46.058 INFO:teuthology.orchestra.run.smithi134.stdout: "log_meta": false, 2024-08-18T17:23:46.058 INFO:teuthology.orchestra.run.smithi134.stdout: "log_data": true, 2024-08-18T17:23:46.058 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_index_max_shards": 11, 2024-08-18T17:23:46.058 INFO:teuthology.orchestra.run.smithi134.stdout: "read_only": false, 2024-08-18T17:23:46.058 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_type": "", 2024-08-18T17:23:46.058 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from_all": true, 2024-08-18T17:23:46.058 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from": [], 2024-08-18T17:23:46.059 INFO:teuthology.orchestra.run.smithi134.stdout: "redirect_zone": "", 2024-08-18T17:23:46.059 INFO:teuthology.orchestra.run.smithi134.stdout: "supported_features": [ 2024-08-18T17:23:46.059 INFO:teuthology.orchestra.run.smithi134.stdout: "compress-encrypted", 2024-08-18T17:23:46.059 INFO:teuthology.orchestra.run.smithi134.stdout: "resharding" 2024-08-18T17:23:46.059 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:23:46.059 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:23:46.059 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:23:46.059 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "10090ccf-aad6-416d-9775-df8b7b27345a", 2024-08-18T17:23:46.060 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "test-zone2", 2024-08-18T17:23:46.060 INFO:teuthology.orchestra.run.smithi134.stdout: "endpoints": [ 2024-08-18T17:23:46.060 INFO:teuthology.orchestra.run.smithi134.stdout: "http://smithi193.front.sepia.ceph.com:8000" 2024-08-18T17:23:46.060 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:23:46.061 INFO:teuthology.orchestra.run.smithi134.stdout: "log_meta": false, 2024-08-18T17:23:46.061 INFO:teuthology.orchestra.run.smithi134.stdout: "log_data": true, 2024-08-18T17:23:46.061 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_index_max_shards": 11, 2024-08-18T17:23:46.061 INFO:teuthology.orchestra.run.smithi134.stdout: "read_only": false, 2024-08-18T17:23:46.061 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_type": "", 2024-08-18T17:23:46.061 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from_all": true, 2024-08-18T17:23:46.061 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from": [], 2024-08-18T17:23:46.061 INFO:teuthology.orchestra.run.smithi134.stdout: "redirect_zone": "", 2024-08-18T17:23:46.061 INFO:teuthology.orchestra.run.smithi134.stdout: "supported_features": [ 2024-08-18T17:23:46.061 INFO:teuthology.orchestra.run.smithi134.stdout: "compress-encrypted", 2024-08-18T17:23:46.061 INFO:teuthology.orchestra.run.smithi134.stdout: "resharding" 2024-08-18T17:23:46.061 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:23:46.061 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:23:46.061 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:23:46.062 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T17:23:46.062 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "test-zone1", 2024-08-18T17:23:46.062 INFO:teuthology.orchestra.run.smithi134.stdout: "endpoints": [ 2024-08-18T17:23:46.062 INFO:teuthology.orchestra.run.smithi134.stdout: "http://smithi134.front.sepia.ceph.com:8000" 2024-08-18T17:23:46.062 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:23:46.062 INFO:teuthology.orchestra.run.smithi134.stdout: "log_meta": false, 2024-08-18T17:23:46.063 INFO:teuthology.orchestra.run.smithi134.stdout: "log_data": true, 2024-08-18T17:23:46.063 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_index_max_shards": 11, 2024-08-18T17:23:46.063 INFO:teuthology.orchestra.run.smithi134.stdout: "read_only": false, 2024-08-18T17:23:46.063 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_type": "", 2024-08-18T17:23:46.063 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from_all": true, 2024-08-18T17:23:46.063 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from": [], 2024-08-18T17:23:46.063 INFO:teuthology.orchestra.run.smithi134.stdout: "redirect_zone": "", 2024-08-18T17:23:46.063 INFO:teuthology.orchestra.run.smithi134.stdout: "supported_features": [ 2024-08-18T17:23:46.064 INFO:teuthology.orchestra.run.smithi134.stdout: "compress-encrypted", 2024-08-18T17:23:46.064 INFO:teuthology.orchestra.run.smithi134.stdout: "resharding" 2024-08-18T17:23:46.064 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:23:46.064 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:23:46.064 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:23:46.064 INFO:teuthology.orchestra.run.smithi134.stdout: "placement_targets": [ 2024-08-18T17:23:46.064 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:23:46.064 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "default-placement", 2024-08-18T17:23:46.064 INFO:teuthology.orchestra.run.smithi134.stdout: "tags": [], 2024-08-18T17:23:46.064 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_classes": [ 2024-08-18T17:23:46.064 INFO:teuthology.orchestra.run.smithi134.stdout: "STANDARD" 2024-08-18T17:23:46.064 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:23:46.065 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:23:46.065 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:23:46.065 INFO:teuthology.orchestra.run.smithi134.stdout: "default_placement": "default-placement", 2024-08-18T17:23:46.065 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_id": "2827c6f0-bf0d-4836-944c-9e7fcf22457b", 2024-08-18T17:23:46.065 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_policy": { 2024-08-18T17:23:46.065 INFO:teuthology.orchestra.run.smithi134.stdout: "groups": [ 2024-08-18T17:23:46.065 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:23:46.065 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "sync-group", 2024-08-18T17:23:46.065 INFO:teuthology.orchestra.run.smithi134.stdout: "data_flow": { 2024-08-18T17:23:46.065 INFO:teuthology.orchestra.run.smithi134.stdout: "symmetrical": [ 2024-08-18T17:23:46.065 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:23:46.065 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "sync-flow1", 2024-08-18T17:23:46.065 INFO:teuthology.orchestra.run.smithi134.stdout: "zones": [ 2024-08-18T17:23:46.069 INFO:teuthology.orchestra.run.smithi134.stdout: "10090ccf-aad6-416d-9775-df8b7b27345a", 2024-08-18T17:23:46.070 INFO:teuthology.orchestra.run.smithi134.stdout: "96bf8502-2f39-4d53-8196-3378c28da3b4" 2024-08-18T17:23:46.070 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:23:46.070 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:23:46.070 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:23:46.070 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:23:46.070 INFO:teuthology.orchestra.run.smithi134.stdout: "pipes": [ 2024-08-18T17:23:46.070 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:23:46.070 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "sync-pipe", 2024-08-18T17:23:46.070 INFO:teuthology.orchestra.run.smithi134.stdout: "source": { 2024-08-18T17:23:46.070 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket": "kibgmb-93", 2024-08-18T17:23:46.070 INFO:teuthology.orchestra.run.smithi134.stdout: "zones": [ 2024-08-18T17:23:46.071 INFO:teuthology.orchestra.run.smithi134.stdout: "96bf8502-2f39-4d53-8196-3378c28da3b4" 2024-08-18T17:23:46.071 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:23:46.071 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:23:46.071 INFO:teuthology.orchestra.run.smithi134.stdout: "dest": { 2024-08-18T17:23:46.071 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket": "kibgmb-94", 2024-08-18T17:23:46.071 INFO:teuthology.orchestra.run.smithi134.stdout: "zones": [ 2024-08-18T17:23:46.071 INFO:teuthology.orchestra.run.smithi134.stdout: "10090ccf-aad6-416d-9775-df8b7b27345a" 2024-08-18T17:23:46.071 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:23:46.071 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:23:46.071 INFO:teuthology.orchestra.run.smithi134.stdout: "params": { 2024-08-18T17:23:46.071 INFO:teuthology.orchestra.run.smithi134.stdout: "source": { 2024-08-18T17:23:46.071 INFO:teuthology.orchestra.run.smithi134.stdout: "filter": { 2024-08-18T17:23:46.071 INFO:teuthology.orchestra.run.smithi134.stdout: "tags": [] 2024-08-18T17:23:46.072 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:23:46.072 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:23:46.072 INFO:teuthology.orchestra.run.smithi134.stdout: "dest": {}, 2024-08-18T17:23:46.072 INFO:teuthology.orchestra.run.smithi134.stdout: "priority": 0, 2024-08-18T17:23:46.072 INFO:teuthology.orchestra.run.smithi134.stdout: "mode": "system", 2024-08-18T17:23:46.072 INFO:teuthology.orchestra.run.smithi134.stdout: "user": "" 2024-08-18T17:23:46.072 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:23:46.072 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:23:46.072 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:23:46.073 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "enabled" 2024-08-18T17:23:46.073 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:23:46.073 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:23:46.073 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:23:46.073 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled_features": [ 2024-08-18T17:23:46.073 INFO:teuthology.orchestra.run.smithi134.stdout: "resharding" 2024-08-18T17:23:46.073 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:23:46.074 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:23:46.074 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:23:46.074 INFO:teuthology.orchestra.run.smithi134.stdout: "short_zone_ids": [ 2024-08-18T17:23:46.074 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:23:46.074 INFO:teuthology.orchestra.run.smithi134.stdout: "key": "018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T17:23:46.074 INFO:teuthology.orchestra.run.smithi134.stdout: "val": 462036791 2024-08-18T17:23:46.074 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:23:46.074 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:23:46.074 INFO:teuthology.orchestra.run.smithi134.stdout: "key": "10090ccf-aad6-416d-9775-df8b7b27345a", 2024-08-18T17:23:46.074 INFO:teuthology.orchestra.run.smithi134.stdout: "val": 874289624 2024-08-18T17:23:46.075 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:23:46.075 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:23:46.075 INFO:teuthology.orchestra.run.smithi134.stdout: "key": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T17:23:46.075 INFO:teuthology.orchestra.run.smithi134.stdout: "val": 1721399886 2024-08-18T17:23:46.075 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:23:46.075 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:23:46.076 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:23:46.076 INFO:teuthology.orchestra.run.smithi134.stdout: "master_zonegroup": "test-zonegroup", 2024-08-18T17:23:46.076 INFO:teuthology.orchestra.run.smithi134.stdout: "master_zone": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T17:23:46.076 INFO:teuthology.orchestra.run.smithi134.stdout: "period_config": { 2024-08-18T17:23:46.076 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_quota": { 2024-08-18T17:23:46.077 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false, 2024-08-18T17:23:46.077 INFO:teuthology.orchestra.run.smithi134.stdout: "check_on_raw": false, 2024-08-18T17:23:46.077 INFO:teuthology.orchestra.run.smithi134.stdout: "max_size": -1, 2024-08-18T17:23:46.077 INFO:teuthology.orchestra.run.smithi134.stdout: "max_size_kb": 0, 2024-08-18T17:23:46.077 INFO:teuthology.orchestra.run.smithi134.stdout: "max_objects": -1 2024-08-18T17:23:46.078 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:23:46.078 INFO:teuthology.orchestra.run.smithi134.stdout: "user_quota": { 2024-08-18T17:23:46.078 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false, 2024-08-18T17:23:46.078 INFO:teuthology.orchestra.run.smithi134.stdout: "check_on_raw": false, 2024-08-18T17:23:46.078 INFO:teuthology.orchestra.run.smithi134.stdout: "max_size": -1, 2024-08-18T17:23:46.079 INFO:teuthology.orchestra.run.smithi134.stdout: "max_size_kb": 0, 2024-08-18T17:23:46.079 INFO:teuthology.orchestra.run.smithi134.stdout: "max_objects": -1 2024-08-18T17:23:46.079 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:23:46.079 INFO:teuthology.orchestra.run.smithi134.stdout: "user_ratelimit": { 2024-08-18T17:23:46.079 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_ops": 0, 2024-08-18T17:23:46.079 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_ops": 0, 2024-08-18T17:23:46.080 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_bytes": 0, 2024-08-18T17:23:46.080 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_bytes": 0, 2024-08-18T17:23:46.080 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false 2024-08-18T17:23:46.080 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:23:46.080 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_ratelimit": { 2024-08-18T17:23:46.081 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_ops": 0, 2024-08-18T17:23:46.081 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_ops": 0, 2024-08-18T17:23:46.081 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_bytes": 0, 2024-08-18T17:23:46.081 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_bytes": 0, 2024-08-18T17:23:46.081 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false 2024-08-18T17:23:46.081 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:23:46.081 INFO:teuthology.orchestra.run.smithi134.stdout: "anonymous_ratelimit": { 2024-08-18T17:23:46.082 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_ops": 0, 2024-08-18T17:23:46.082 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_ops": 0, 2024-08-18T17:23:46.082 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_bytes": 0, 2024-08-18T17:23:46.082 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_bytes": 0, 2024-08-18T17:23:46.082 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false 2024-08-18T17:23:46.082 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:23:46.082 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:23:46.082 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_id": "2827c6f0-bf0d-4836-944c-9e7fcf22457b", 2024-08-18T17:23:46.082 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_name": "test-realm", 2024-08-18T17:23:46.082 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:23:46.082 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T17:23:46.084 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:23:46.084 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:23:46.084 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:23:46.133 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:23:46.133 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:23:46.244 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"kibgmb-93","zones":["test-zone1"]},"dest":{"bucket":"kibgmb-94","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-08-18T17:23:46.487 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:23:46.487 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:23:46.487 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:23:46.554 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:23:46.554 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:23:46.642 INFO:teuthology.orchestra.run.smithi134.stderr:No --period given, using current period=01050e30-179b-43be-bdbb-3bb23cf37ae8 2024-08-18T17:23:46.678 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T17:23:46.678 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:23:46.678 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:23:46.678 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T17:23:46.678 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:23:46.679 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:23:46.679 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:23:46.679 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T17:23:46.679 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:23:46.679 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:23:46.679 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:23:46.679 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:42:47.672461Z" 2024-08-18T17:23:46.679 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:23:46.679 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:23:46.679 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1724001819.978375_1607.1", 2024-08-18T17:23:46.679 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T17:23:39.978375Z" 2024-08-18T17:23:46.680 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:23:46.680 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T17:23:46.681 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-08-18T17:23:46.681 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:23:46.681 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:23:46.681 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:23:46.729 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:23:46.729 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:23:46.759 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:23:46.758+0000 7fbd5ba0e880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:23:46.760 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:23:46.759+0000 7fbd5ba0e880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:23:46.774 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:23:46.774 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:23:46.774 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:23:46.774 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:23:46.774 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:23:46.774 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:23:46.774 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:23:46.774 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:23:46.774 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:23:46.775 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:23:46.775 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:23:46.775 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:23:46.775 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:23:46.775 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:23:46.775 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:23:46.775 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:23:46.775 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:23:46.775 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:23:46.775 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:23:46.775 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:23:46.775 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:23:46.775 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:23:46.775 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:23:46.776 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:23:46.776 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:23:46.776 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:23:46.776 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:23:46.776 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:23:46.776 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:23:46.776 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:23:46.776 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:23:46.776 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:23:46.776 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:23:46.776 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:23:46.776 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:23:46.776 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:23:46.776 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:23:46.777 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:23:46.777 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:23:46.777 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:23:46.777 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:23:46.777 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:23:46.777 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:23:46.777 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:23:46.777 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:23:46.777 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:23:46.777 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:23:46.777 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:23:46.777 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:23:46.777 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001819.978375_1607.1", 2024-08-18T17:23:46.778 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:23:46.778 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:23:46.778 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:23:46.778 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:23:39.978375Z", 2024-08-18T17:23:46.778 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:23:46.778 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:23:46.778 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:23:46.778 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:23:46.778 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:23:46.778 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:23:46.778 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:23:46.778 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:23:46.778 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:23:46.779 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:23:46Z" 2024-08-18T17:23:46.779 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:23:46.780 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-08-18T17:23:46.780 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-08-18T17:23:46.781 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:23:46.781 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:23:46.781 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:23:46.845 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:23:46.845 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:23:46.924 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:23:46.923+0000 7fa2ae0bb880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:23:46.925 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:23:46.924+0000 7fa2ae0bb880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:23:46.961 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T17:23:46.961 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T17:23:46.961 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T17:23:46.961 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T17:23:46.962 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T17:23:46.962 INFO:teuthology.orchestra.run.smithi134.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:23:46.962 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:23:46.962 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:23:46.962 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T17:23:46.962 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:23:46.962 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T17:23:46.962 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:23:46.962 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:23:46.962 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:23:46.962 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:23:46.963 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 1, 2024-08-18T17:23:46.963 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:23:46.963 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:23:46.963 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T17:23:46.963 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:23:46.963 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:23:46.963 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:23:46.963 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T17:23:46.963 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:23:46.963 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:23:46.963 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:23:46.963 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:23:46.964 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:23:46.964 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:23:46.964 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T17:23:46.964 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T17:23:46.964 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:23:46.964 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:23:46.964 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:23:46.964 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T17:23:46.964 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:23:46.964 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:23:46.964 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:23:46.964 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:23:46.965 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:23:46.965 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:23:46.965 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:23:46.965 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:23:46.965 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:23:46.965 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:23:46.965 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:23:46.965 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T17:23:46.965 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:23:46.965 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:23:46.965 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1724001819.978375_1607.1", 2024-08-18T17:23:46.965 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:23:46.965 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:23:46.966 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:23:46.966 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T17:23:39.978375Z", 2024-08-18T17:23:46.966 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:23:46.966 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:23:46.966 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:23:46.966 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:23:46.966 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:23:46.966 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T17:23:46.966 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 1, 2024-08-18T17:23:46.966 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 1 2024-08-18T17:23:46.966 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:23:46.966 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T17:23:46Z" 2024-08-18T17:23:46.967 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T17:23:46.968 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-08-18T17:23:46.968 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:23:46.968 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:23:46.968 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:23:47.024 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:23:47.024 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:23:47.150 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T17:23:47.150 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:23:47.151 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000051874", 2024-08-18T17:23:47.151 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T17:23:46.463578Z" 2024-08-18T17:23:47.151 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:23:47.151 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:23:47.151 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:23:47.151 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:23:47.151 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:23:47.151 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:23:47.151 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000059557", 2024-08-18T17:23:47.151 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T17:23:46.436203Z" 2024-08-18T17:23:47.151 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:23:47.151 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:23:47.152 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:23:47.152 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:23:47.152 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:23:47.152 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T17:23:47.153 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-08-18T17:23:47.153 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:23:47.153 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:23:47.153 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:23:47.202 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:23:47.203 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:23:47.249 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:23:47.249 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:23:47.249 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:23:47.249 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:23:47.249 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:23:47.249 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:23:47.249 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:23:47.249 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:23:47.249 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:23:47.249 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:23:47.249 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:23:47.250 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:23:47.250 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000051734", 2024-08-18T17:23:47.250 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:23:47.250 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:23:47.250 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:23:47.250 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.076905Z" 2024-08-18T17:23:47.250 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:23:47.250 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:23:47.250 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:23:47.250 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:23:47.250 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:23:47.250 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:23:47.251 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:23:47.251 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:23:47.251 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:23:47.251 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:23:47.251 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:23:47.251 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:23:47.251 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:23:47.251 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:23:47.251 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:23:47.251 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:23:47.251 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:23:47.251 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000059417", 2024-08-18T17:23:47.251 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:23:47.251 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:23:47.252 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:23:47.252 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:21:15.361961Z" 2024-08-18T17:23:47.252 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:23:47.252 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:23:47.252 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:23:47.252 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:23:47.252 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:23:47.252 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:23:47.252 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:23:47.252 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:23:47.252 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:23:47.252 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:23:47.252 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:23:47.252 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:23:47.253 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:23:47.253 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:23:47.253 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:23:47.253 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:23:47.253 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:23:47.253 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:23:47.253 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:23:47.253 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:23:47Z" 2024-08-18T17:23:47.253 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:23:47.254 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000051874 target=00000000000000000000:00000000000000051734, shard=2 master=00000000000000000000:00000000000000059557 target=00000000000000000000:00000000000000059417 2024-08-18T17:23:52.255 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:23:52.256 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:23:52.256 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:23:52.313 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:23:52.313 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:23:52.363 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:23:52.363 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:23:52.364 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:23:52.364 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:23:52.364 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:23:52.364 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:23:52.364 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:23:52.364 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:23:52.364 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:23:52.364 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:23:52.364 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:23:52.364 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:23:52.365 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000051734", 2024-08-18T17:23:52.365 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:23:52.365 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:23:52.365 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:23:52.365 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.076905Z" 2024-08-18T17:23:52.365 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:23:52.365 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:23:52.365 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:23:52.365 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:23:52.365 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:23:52.365 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:23:52.365 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:23:52.366 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:23:52.366 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:23:52.366 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:23:52.366 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:23:52.366 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:23:52.366 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:23:52.366 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:23:52.366 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:23:52.366 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:23:52.366 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:23:52.366 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000059417", 2024-08-18T17:23:52.367 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:23:52.367 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:23:52.367 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:23:52.367 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:21:15.361961Z" 2024-08-18T17:23:52.367 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:23:52.368 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:23:52.368 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:23:52.368 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:23:52.368 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:23:52.368 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:23:52.368 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:23:52.368 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:23:52.368 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:23:52.368 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:23:52.368 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:23:52.369 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:23:52.369 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:23:52.369 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:23:52.369 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:23:52.369 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:23:52.369 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:23:52.369 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:23:52.369 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:23:52.369 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:23:52Z" 2024-08-18T17:23:52.369 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:23:52.371 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000051874 target=00000000000000000000:00000000000000051734, shard=2 master=00000000000000000000:00000000000000059557 target=00000000000000000000:00000000000000059417 2024-08-18T17:23:57.372 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:23:57.373 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:23:57.373 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:23:57.430 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:23:57.430 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:23:57.481 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:23:57.482 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:23:57.482 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:23:57.482 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:23:57.482 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:23:57.482 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:23:57.482 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:23:57.482 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:23:57.482 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:23:57.482 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:23:57.482 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:23:57.482 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:23:57.483 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000051734", 2024-08-18T17:23:57.483 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:23:57.483 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:23:57.483 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:23:57.483 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.076905Z" 2024-08-18T17:23:57.483 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:23:57.483 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:23:57.483 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:23:57.483 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:23:57.483 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:23:57.483 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:23:57.483 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:23:57.484 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:23:57.484 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:23:57.484 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:23:57.484 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:23:57.484 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:23:57.484 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:23:57.484 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:23:57.484 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:23:57.484 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:23:57.484 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:23:57.484 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000059417", 2024-08-18T17:23:57.484 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:23:57.484 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:23:57.484 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:23:57.485 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:21:15.361961Z" 2024-08-18T17:23:57.485 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:23:57.485 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:23:57.485 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:23:57.485 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:23:57.485 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:23:57.485 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:23:57.485 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:23:57.485 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:23:57.486 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:23:57.486 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:23:57.486 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:23:57.486 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:23:57.486 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:23:57.486 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:23:57.486 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:23:57.486 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:23:57.486 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:23:57.486 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:23:57.486 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:23:57.486 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:23:57Z" 2024-08-18T17:23:57.486 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:23:57.487 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000051874 target=00000000000000000000:00000000000000051734, shard=2 master=00000000000000000000:00000000000000059557 target=00000000000000000000:00000000000000059417 2024-08-18T17:24:02.489 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:24:02.489 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:24:02.489 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:24:02.547 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:24:02.547 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:24:02.601 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:24:02.601 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:24:02.601 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:24:02.601 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:24:02.601 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:24:02.601 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:24:02.602 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:24:02.602 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:24:02.602 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:24:02.602 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:24:02.602 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:24:02.602 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:24:02.602 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000051734", 2024-08-18T17:24:02.602 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:24:02.602 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:24:02.602 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:24:02.602 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.076905Z" 2024-08-18T17:24:02.602 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:24:02.603 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:24:02.603 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:24:02.603 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:24:02.603 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:24:02.603 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:24:02.603 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:24:02.603 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:24:02.603 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:24:02.603 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:24:02.603 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:24:02.603 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:24:02.603 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:24:02.603 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:24:02.604 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:24:02.604 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:24:02.604 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:24:02.604 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000059417", 2024-08-18T17:24:02.604 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:24:02.604 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:24:02.604 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:24:02.604 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:21:15.361961Z" 2024-08-18T17:24:02.604 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:24:02.604 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:24:02.604 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:24:02.604 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:24:02.604 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:24:02.604 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:24:02.605 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:24:02.605 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:24:02.605 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:24:02.605 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:24:02.605 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:24:02.605 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:24:02.605 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:24:02.605 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:24:02.605 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:24:02.605 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:24:02.605 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:24:02.605 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:24:02.605 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:24:02.606 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:24:02Z" 2024-08-18T17:24:02.606 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:24:02.607 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000051874 target=00000000000000000000:00000000000000051734, shard=2 master=00000000000000000000:00000000000000059557 target=00000000000000000000:00000000000000059417 2024-08-18T17:24:07.607 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:24:07.608 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:24:07.608 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:24:07.665 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:24:07.665 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:24:07.716 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:24:07.716 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:24:07.716 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:24:07.716 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:24:07.716 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:24:07.716 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:24:07.716 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:24:07.716 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:24:07.716 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:24:07.716 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:24:07.717 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:24:07.717 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:24:07.717 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000051734", 2024-08-18T17:24:07.717 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:24:07.717 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:24:07.717 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:24:07.717 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.076905Z" 2024-08-18T17:24:07.717 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:24:07.717 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:24:07.717 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:24:07.717 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:24:07.717 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:24:07.717 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:24:07.718 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:24:07.718 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:24:07.718 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:24:07.718 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:24:07.718 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:24:07.718 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:24:07.718 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:24:07.718 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:24:07.718 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:24:07.718 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:24:07.718 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:24:07.718 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000059417", 2024-08-18T17:24:07.719 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:24:07.719 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:24:07.719 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:24:07.719 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:21:15.361961Z" 2024-08-18T17:24:07.719 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:24:07.719 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:24:07.719 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:24:07.719 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:24:07.719 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:24:07.719 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:24:07.719 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:24:07.719 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:24:07.719 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:24:07.720 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:24:07.720 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:24:07.720 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:24:07.720 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:24:07.720 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:24:07.720 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:24:07.720 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:24:07.720 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:24:07.720 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:24:07.720 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:24:07.720 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:24:07Z" 2024-08-18T17:24:07.720 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:24:07.721 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000051874 target=00000000000000000000:00000000000000051734, shard=2 master=00000000000000000000:00000000000000059557 target=00000000000000000000:00000000000000059417 2024-08-18T17:24:12.726 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:24:12.727 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:24:12.727 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:24:12.784 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:24:12.784 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:24:12.834 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:24:12.834 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:24:12.835 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:24:12.835 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:24:12.835 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:24:12.835 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:24:12.835 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:24:12.835 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:24:12.835 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:24:12.835 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:24:12.836 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:24:12.836 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:24:12.836 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000051734", 2024-08-18T17:24:12.836 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:24:12.836 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:24:12.836 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:24:12.836 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.076905Z" 2024-08-18T17:24:12.836 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:24:12.836 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:24:12.837 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:24:12.837 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:24:12.837 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:24:12.837 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:24:12.837 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:24:12.837 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:24:12.837 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:24:12.837 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:24:12.837 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:24:12.837 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:24:12.837 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:24:12.838 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:24:12.838 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:24:12.838 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:24:12.838 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:24:12.838 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000059417", 2024-08-18T17:24:12.838 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:24:12.838 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:24:12.838 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:24:12.838 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:21:15.361961Z" 2024-08-18T17:24:12.838 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:24:12.839 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:24:12.839 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:24:12.839 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:24:12.839 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:24:12.839 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:24:12.839 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:24:12.840 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:24:12.840 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:24:12.840 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:24:12.840 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:24:12.840 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:24:12.841 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:24:12.841 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:24:12.841 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:24:12.841 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:24:12.842 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:24:12.842 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:24:12.842 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:24:12.843 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:24:12Z" 2024-08-18T17:24:12.843 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:24:12.845 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000051874 target=00000000000000000000:00000000000000051734, shard=2 master=00000000000000000000:00000000000000059557 target=00000000000000000000:00000000000000059417 2024-08-18T17:24:17.846 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:24:17.847 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:24:17.847 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:24:17.906 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:24:17.907 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:24:17.957 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:24:17.957 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:24:17.958 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:24:17.958 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:24:17.958 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:24:17.958 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:24:17.958 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:24:17.958 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:24:17.958 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:24:17.958 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:24:17.958 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:24:17.958 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:24:17.958 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000051734", 2024-08-18T17:24:17.958 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:24:17.959 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:24:17.959 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:24:17.959 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.076905Z" 2024-08-18T17:24:17.959 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:24:17.959 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:24:17.959 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:24:17.959 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:24:17.959 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:24:17.959 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:24:17.959 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:24:17.959 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:24:17.959 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:24:17.959 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:24:17.959 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:24:17.960 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:24:17.960 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:24:17.960 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:24:17.960 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:24:17.960 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:24:17.960 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:24:17.960 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000059417", 2024-08-18T17:24:17.960 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:24:17.960 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:24:17.960 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:24:17.960 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:21:15.361961Z" 2024-08-18T17:24:17.960 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:24:17.960 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:24:17.961 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:24:17.961 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:24:17.961 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:24:17.961 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:24:17.961 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:24:17.961 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:24:17.961 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:24:17.961 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:24:17.962 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:24:17.962 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:24:17.962 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:24:17.962 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:24:17.962 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:24:17.962 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:24:17.962 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:24:17.962 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:24:17.962 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:24:17.963 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:24:17Z" 2024-08-18T17:24:17.963 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:24:17.964 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000051874 target=00000000000000000000:00000000000000051734, shard=2 master=00000000000000000000:00000000000000059557 target=00000000000000000000:00000000000000059417 2024-08-18T17:24:22.965 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:24:22.965 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:24:22.965 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:24:23.022 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:24:23.022 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:24:23.073 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:24:23.073 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:24:23.073 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:24:23.073 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:24:23.073 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:24:23.073 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:24:23.073 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:24:23.073 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:24:23.073 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:24:23.073 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:24:23.074 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:24:23.074 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:24:23.074 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000051734", 2024-08-18T17:24:23.074 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:24:23.074 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:24:23.074 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:24:23.074 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.076905Z" 2024-08-18T17:24:23.074 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:24:23.074 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:24:23.074 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:24:23.074 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:24:23.074 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:24:23.074 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:24:23.075 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:24:23.075 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:24:23.075 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:24:23.075 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:24:23.075 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:24:23.075 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:24:23.075 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:24:23.075 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:24:23.075 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:24:23.075 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:24:23.075 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:24:23.075 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000059417", 2024-08-18T17:24:23.075 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:24:23.076 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:24:23.076 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:24:23.076 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:21:15.361961Z" 2024-08-18T17:24:23.076 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:24:23.076 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:24:23.076 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:24:23.076 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:24:23.076 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:24:23.076 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:24:23.076 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:24:23.076 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:24:23.076 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:24:23.076 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:24:23.076 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:24:23.077 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:24:23.077 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:24:23.077 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:24:23.077 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:24:23.077 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:24:23.077 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:24:23.077 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:24:23.077 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:24:23.077 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:24:23Z" 2024-08-18T17:24:23.077 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:24:23.079 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000051874 target=00000000000000000000:00000000000000051734, shard=2 master=00000000000000000000:00000000000000059557 target=00000000000000000000:00000000000000059417 2024-08-18T17:24:28.080 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:24:28.080 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:24:28.080 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:24:28.139 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:24:28.139 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:24:28.189 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:24:28.189 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:24:28.189 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:24:28.189 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:24:28.190 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:24:28.190 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:24:28.190 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:24:28.190 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:24:28.190 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:24:28.190 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:24:28.190 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:24:28.190 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:24:28.190 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000051734", 2024-08-18T17:24:28.190 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:24:28.190 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:24:28.190 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:24:28.191 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.076905Z" 2024-08-18T17:24:28.191 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:24:28.191 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:24:28.191 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:24:28.191 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:24:28.191 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:24:28.191 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:24:28.191 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:24:28.191 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:24:28.191 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:24:28.191 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:24:28.191 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:24:28.191 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:24:28.192 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:24:28.192 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:24:28.192 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:24:28.192 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:24:28.192 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:24:28.192 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000059417", 2024-08-18T17:24:28.192 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:24:28.192 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:24:28.192 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:24:28.192 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:21:15.361961Z" 2024-08-18T17:24:28.192 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:24:28.192 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:24:28.192 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:24:28.193 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:24:28.193 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:24:28.193 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:24:28.193 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:24:28.193 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:24:28.193 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:24:28.193 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:24:28.193 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:24:28.193 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:24:28.193 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:24:28.193 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:24:28.193 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:24:28.193 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:24:28.194 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:24:28.194 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:24:28.194 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:24:28.194 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:24:28Z" 2024-08-18T17:24:28.194 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:24:28.195 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000051874 target=00000000000000000000:00000000000000051734, shard=2 master=00000000000000000000:00000000000000059557 target=00000000000000000000:00000000000000059417 2024-08-18T17:24:33.195 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:24:33.196 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:24:33.196 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:24:33.255 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:24:33.255 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:24:33.305 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:24:33.305 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:24:33.306 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:24:33.306 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:24:33.306 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:24:33.306 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:24:33.306 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:24:33.306 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:24:33.306 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:24:33.306 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:24:33.306 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:24:33.306 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:24:33.306 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000051734", 2024-08-18T17:24:33.307 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:24:33.307 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:24:33.307 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:24:33.307 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.076905Z" 2024-08-18T17:24:33.307 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:24:33.307 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:24:33.307 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:24:33.307 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:24:33.307 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:24:33.307 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:24:33.307 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:24:33.307 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:24:33.308 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:24:33.308 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:24:33.308 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:24:33.308 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:24:33.308 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:24:33.308 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:24:33.308 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:24:33.308 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:24:33.308 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:24:33.308 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000059417", 2024-08-18T17:24:33.308 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:24:33.308 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:24:33.309 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:24:33.309 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:21:15.361961Z" 2024-08-18T17:24:33.309 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:24:33.309 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:24:33.309 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:24:33.309 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:24:33.309 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:24:33.309 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:24:33.309 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:24:33.309 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:24:33.309 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:24:33.310 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:24:33.310 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:24:33.310 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:24:33.310 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:24:33.310 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:24:33.310 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:24:33.310 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:24:33.310 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:24:33.310 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:24:33.310 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:24:33.310 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:24:33Z" 2024-08-18T17:24:33.311 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:24:33.312 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000051874 target=00000000000000000000:00000000000000051734, shard=2 master=00000000000000000000:00000000000000059557 target=00000000000000000000:00000000000000059417 2024-08-18T17:24:38.313 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:24:38.313 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:24:38.313 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:24:38.371 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:24:38.371 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:24:38.421 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:24:38.421 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:24:38.422 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:24:38.422 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:24:38.422 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:24:38.422 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:24:38.422 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:24:38.422 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:24:38.422 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:24:38.422 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:24:38.422 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:24:38.422 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:24:38.422 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000051734", 2024-08-18T17:24:38.422 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:24:38.422 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:24:38.423 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:24:38.423 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.076905Z" 2024-08-18T17:24:38.423 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:24:38.423 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:24:38.423 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:24:38.423 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:24:38.423 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:24:38.423 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:24:38.423 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:24:38.423 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:24:38.423 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:24:38.423 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:24:38.424 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:24:38.424 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:24:38.424 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:24:38.424 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:24:38.424 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:24:38.424 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:24:38.424 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:24:38.424 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000059417", 2024-08-18T17:24:38.424 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:24:38.424 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:24:38.424 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:24:38.424 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:21:15.361961Z" 2024-08-18T17:24:38.424 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:24:38.425 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:24:38.425 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:24:38.425 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:24:38.425 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:24:38.425 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:24:38.425 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:24:38.425 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:24:38.425 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:24:38.425 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:24:38.425 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:24:38.425 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:24:38.425 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:24:38.425 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:24:38.426 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:24:38.426 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:24:38.426 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:24:38.426 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:24:38.426 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:24:38.426 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:24:38Z" 2024-08-18T17:24:38.426 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:24:38.427 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000051874 target=00000000000000000000:00000000000000051734, shard=2 master=00000000000000000000:00000000000000059557 target=00000000000000000000:00000000000000059417 2024-08-18T17:24:43.428 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:24:43.429 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:24:43.429 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:24:43.487 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:24:43.487 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:24:43.537 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:24:43.537 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:24:43.537 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:24:43.538 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:24:43.538 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:24:43.538 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:24:43.538 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:24:43.538 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:24:43.538 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:24:43.538 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:24:43.538 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:24:43.538 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:24:43.538 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000051734", 2024-08-18T17:24:43.539 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:24:43.539 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:24:43.539 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:24:43.539 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.076905Z" 2024-08-18T17:24:43.539 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:24:43.539 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:24:43.539 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:24:43.539 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:24:43.539 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:24:43.539 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:24:43.539 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:24:43.539 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:24:43.540 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:24:43.540 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:24:43.540 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:24:43.540 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:24:43.540 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:24:43.540 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:24:43.540 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:24:43.540 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:24:43.540 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:24:43.540 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000059417", 2024-08-18T17:24:43.540 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:24:43.540 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:24:43.540 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:24:43.541 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:21:15.361961Z" 2024-08-18T17:24:43.541 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:24:43.541 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:24:43.541 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:24:43.541 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:24:43.541 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:24:43.541 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:24:43.541 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:24:43.541 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:24:43.541 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:24:43.541 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:24:43.541 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:24:43.541 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:24:43.542 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:24:43.542 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:24:43.542 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:24:43.542 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:24:43.542 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:24:43.542 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:24:43.542 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:24:43.542 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:24:43Z" 2024-08-18T17:24:43.542 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:24:43.543 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000051874 target=00000000000000000000:00000000000000051734, shard=2 master=00000000000000000000:00000000000000059557 target=00000000000000000000:00000000000000059417 2024-08-18T17:24:48.544 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:24:48.545 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:24:48.545 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:24:48.602 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:24:48.602 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:24:48.652 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:24:48.652 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:24:48.653 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:24:48.653 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:24:48.653 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:24:48.653 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:24:48.653 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:24:48.653 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:24:48.653 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:24:48.653 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:24:48.653 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:24:48.653 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:24:48.653 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000051734", 2024-08-18T17:24:48.653 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:24:48.653 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:24:48.654 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:24:48.654 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.076905Z" 2024-08-18T17:24:48.654 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:24:48.654 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:24:48.654 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:24:48.654 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:24:48.654 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:24:48.654 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:24:48.654 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:24:48.654 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:24:48.654 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:24:48.654 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:24:48.655 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:24:48.655 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:24:48.655 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:24:48.655 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:24:48.655 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:24:48.655 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:24:48.655 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:24:48.655 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000059417", 2024-08-18T17:24:48.655 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:24:48.655 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:24:48.655 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:24:48.655 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:21:15.361961Z" 2024-08-18T17:24:48.655 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:24:48.655 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:24:48.656 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:24:48.656 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:24:48.656 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:24:48.656 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:24:48.656 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:24:48.656 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:24:48.656 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:24:48.656 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:24:48.656 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:24:48.656 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:24:48.656 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:24:48.656 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:24:48.656 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:24:48.656 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:24:48.657 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:24:48.657 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:24:48.657 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:24:48.657 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:24:48Z" 2024-08-18T17:24:48.657 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:24:48.658 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000051874 target=00000000000000000000:00000000000000051734, shard=2 master=00000000000000000000:00000000000000059557 target=00000000000000000000:00000000000000059417 2024-08-18T17:24:53.658 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:24:53.659 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:24:53.659 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:24:53.717 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:24:53.717 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:24:53.767 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:24:53.767 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:24:53.767 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:24:53.767 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:24:53.768 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:24:53.768 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:24:53.768 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:24:53.768 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:24:53.768 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:24:53.768 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:24:53.768 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:24:53.768 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:24:53.768 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000051734", 2024-08-18T17:24:53.768 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:24:53.768 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:24:53.768 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:24:53.769 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.076905Z" 2024-08-18T17:24:53.769 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:24:53.769 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:24:53.769 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:24:53.769 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:24:53.769 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:24:53.769 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:24:53.769 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:24:53.769 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:24:53.769 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:24:53.769 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:24:53.769 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:24:53.769 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:24:53.770 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:24:53.770 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:24:53.770 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:24:53.770 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:24:53.770 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:24:53.770 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000059417", 2024-08-18T17:24:53.770 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:24:53.770 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:24:53.770 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:24:53.770 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:21:15.361961Z" 2024-08-18T17:24:53.770 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:24:53.770 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:24:53.770 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:24:53.771 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:24:53.771 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:24:53.771 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:24:53.771 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:24:53.771 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:24:53.771 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:24:53.771 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:24:53.771 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:24:53.771 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:24:53.771 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:24:53.771 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:24:53.771 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:24:53.771 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:24:53.772 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:24:53.772 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:24:53.772 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:24:53.772 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:24:53Z" 2024-08-18T17:24:53.772 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:24:53.773 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000051874 target=00000000000000000000:00000000000000051734, shard=2 master=00000000000000000000:00000000000000059557 target=00000000000000000000:00000000000000059417 2024-08-18T17:24:58.774 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:24:58.774 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:24:58.774 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:24:58.831 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:24:58.832 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:24:58.882 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:24:58.882 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:24:58.882 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:24:58.883 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:24:58.883 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:24:58.883 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:24:58.883 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:24:58.883 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:24:58.883 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:24:58.883 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:24:58.883 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:24:58.883 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:24:58.883 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000051734", 2024-08-18T17:24:58.883 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:24:58.883 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:24:58.883 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:24:58.884 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.076905Z" 2024-08-18T17:24:58.884 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:24:58.884 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:24:58.884 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:24:58.884 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:24:58.884 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:24:58.884 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:24:58.884 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:24:58.884 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:24:58.884 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:24:58.884 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:24:58.884 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:24:58.884 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:24:58.885 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:24:58.885 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:24:58.885 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:24:58.885 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:24:58.885 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:24:58.885 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000059417", 2024-08-18T17:24:58.885 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:24:58.885 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:24:58.885 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:24:58.885 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:21:15.361961Z" 2024-08-18T17:24:58.885 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:24:58.885 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:24:58.885 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:24:58.886 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:24:58.886 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:24:58.886 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:24:58.886 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:24:58.886 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:24:58.886 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:24:58.886 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:24:58.886 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:24:58.886 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:24:58.886 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:24:58.886 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:24:58.886 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:24:58.886 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:24:58.887 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:24:58.887 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:24:58.887 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:24:58.887 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:24:58Z" 2024-08-18T17:24:58.887 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:24:58.888 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000051874 target=00000000000000000000:00000000000000051734, shard=2 master=00000000000000000000:00000000000000059557 target=00000000000000000000:00000000000000059417 2024-08-18T17:25:03.889 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:25:03.889 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:25:03.889 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:25:03.947 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:25:03.947 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:25:03.998 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:25:03.998 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:25:03.998 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:25:03.998 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:25:03.998 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:25:03.998 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:25:03.998 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:25:03.998 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:25:03.998 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:25:03.998 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:25:03.999 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:25:03.999 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:25:03.999 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000051734", 2024-08-18T17:25:03.999 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:25:03.999 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:25:03.999 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:25:03.999 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.076905Z" 2024-08-18T17:25:03.999 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:25:03.999 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:25:03.999 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:25:03.999 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:25:03.999 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:25:03.999 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:25:04.000 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:25:04.000 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:25:04.000 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:25:04.000 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:25:04.000 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:25:04.000 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:25:04.000 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:25:04.000 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:25:04.000 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:25:04.000 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:25:04.000 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:25:04.000 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000059417", 2024-08-18T17:25:04.000 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:25:04.001 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:25:04.001 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:25:04.001 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:21:15.361961Z" 2024-08-18T17:25:04.001 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:25:04.001 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:25:04.001 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:25:04.001 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:25:04.001 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:25:04.001 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:25:04.001 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:25:04.001 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:25:04.001 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:25:04.001 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:25:04.002 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:25:04.002 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:25:04.002 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:25:04.002 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:25:04.002 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:25:04.002 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:25:04.002 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:25:04.002 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:25:04.002 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:25:04.002 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:25:03Z" 2024-08-18T17:25:04.002 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:25:04.003 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000051874 target=00000000000000000000:00000000000000051734, shard=2 master=00000000000000000000:00000000000000059557 target=00000000000000000000:00000000000000059417 2024-08-18T17:25:09.004 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:25:09.004 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:25:09.005 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:25:09.061 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:25:09.062 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:25:09.112 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:25:09.112 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:25:09.112 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:25:09.112 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:25:09.112 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:25:09.112 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:25:09.113 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:25:09.113 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:25:09.113 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:25:09.113 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:25:09.113 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:25:09.113 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:25:09.113 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000051734", 2024-08-18T17:25:09.113 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:25:09.113 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:25:09.113 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:25:09.113 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.076905Z" 2024-08-18T17:25:09.113 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:25:09.113 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:25:09.114 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:25:09.114 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:25:09.114 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:25:09.114 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:25:09.114 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:25:09.114 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:25:09.114 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:25:09.114 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:25:09.114 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:25:09.114 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:25:09.114 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:25:09.114 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:25:09.114 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:25:09.115 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:25:09.115 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:25:09.115 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000059417", 2024-08-18T17:25:09.115 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:25:09.115 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:25:09.115 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:25:09.115 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:21:15.361961Z" 2024-08-18T17:25:09.115 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:25:09.115 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:25:09.115 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:25:09.115 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:25:09.115 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:25:09.115 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:25:09.116 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:25:09.116 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:25:09.116 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:25:09.116 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:25:09.116 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:25:09.116 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:25:09.116 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:25:09.116 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:25:09.116 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:25:09.116 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:25:09.116 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:25:09.116 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:25:09.116 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:25:09.117 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:25:09Z" 2024-08-18T17:25:09.117 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:25:09.118 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000051874 target=00000000000000000000:00000000000000051734, shard=2 master=00000000000000000000:00000000000000059557 target=00000000000000000000:00000000000000059417 2024-08-18T17:25:14.119 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:25:14.119 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:25:14.119 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:25:14.176 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:25:14.176 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:25:14.228 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:25:14.228 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:25:14.229 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:25:14.229 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:25:14.229 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:25:14.229 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:25:14.229 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:25:14.229 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:25:14.229 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:25:14.229 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:25:14.229 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:25:14.229 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:25:14.229 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000051734", 2024-08-18T17:25:14.229 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:25:14.230 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:25:14.230 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:25:14.230 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.076905Z" 2024-08-18T17:25:14.230 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:25:14.230 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:25:14.230 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:25:14.230 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:25:14.230 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:25:14.230 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:25:14.230 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:25:14.230 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:25:14.230 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:25:14.231 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:25:14.231 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:25:14.231 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:25:14.231 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:25:14.231 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:25:14.231 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:25:14.231 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:25:14.231 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:25:14.231 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000059417", 2024-08-18T17:25:14.231 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:25:14.231 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:25:14.231 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:25:14.232 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:21:15.361961Z" 2024-08-18T17:25:14.232 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:25:14.232 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:25:14.232 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:25:14.232 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:25:14.232 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:25:14.232 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:25:14.232 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:25:14.232 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:25:14.232 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:25:14.232 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:25:14.233 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:25:14.233 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:25:14.233 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:25:14.233 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:25:14.233 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:25:14.233 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:25:14.233 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:25:14.233 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:25:14.233 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:25:14.233 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:25:14Z" 2024-08-18T17:25:14.233 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:25:14.234 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000051874 target=00000000000000000000:00000000000000051734, shard=2 master=00000000000000000000:00000000000000059557 target=00000000000000000000:00000000000000059417 2024-08-18T17:25:19.235 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:25:19.236 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:25:19.236 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:25:19.292 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:25:19.293 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:25:19.343 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:25:19.344 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:25:19.344 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:25:19.344 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:25:19.344 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:25:19.344 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:25:19.344 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:25:19.344 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:25:19.344 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:25:19.344 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:25:19.344 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:25:19.344 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:25:19.345 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000051874", 2024-08-18T17:25:19.345 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:25:19.345 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:25:19.345 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:25:19.345 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:23:46.463578Z" 2024-08-18T17:25:19.345 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:25:19.345 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:25:19.345 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:25:19.345 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:25:19.345 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:25:19.345 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:25:19.345 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:25:19.345 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:25:19.346 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:25:19.346 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:25:19.346 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:25:19.346 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:25:19.346 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:25:19.346 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:25:19.346 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:25:19.346 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:25:19.346 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:25:19.346 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000059557", 2024-08-18T17:25:19.346 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:25:19.346 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:25:19.346 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:25:19.347 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:23:46.436203Z" 2024-08-18T17:25:19.347 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:25:19.347 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:25:19.347 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:25:19.347 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:25:19.347 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:25:19.347 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:25:19.347 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:25:19.347 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:25:19.347 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:25:19.347 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:25:19.347 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:25:19.347 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:25:19.347 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:25:19.348 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:25:19.348 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:25:19.348 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:25:19.348 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:25:19.348 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:25:19.348 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:25:19.348 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:25:19Z" 2024-08-18T17:25:19.348 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:25:19.349 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-08-18T17:25:19.410 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:25:19.410 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:25:19.410 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-group --pipe-id sync-pipe --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:25:19.466 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:25:19.466 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:25:19.529 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}]},"pipes":[],"status":"enabled"}]} 2024-08-18T17:25:19.529 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1,test-zone2', '--dest-zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:25:19.529 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1,test-zone2', '--dest-zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:25:19.530 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=test-zone1,test-zone2 --dest-zones=test-zone1,test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:25:19.581 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:25:19.581 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:25:19.660 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-08-18T17:25:19.660 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:25:19.660 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:25:19.660 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:25:19.707 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:25:19.707 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:25:19.768 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-08-18T17:25:19.768 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:25:19.769 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:25:19.769 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:25:19.815 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:25:19.815 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:25:19.866 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T17:25:19.866 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:25:19.866 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 18, 2024-08-18T17:25:19.866 INFO:teuthology.orchestra.run.smithi134.stdout: "predecessor_uuid": "64309209-7567-46e5-8881-65783d84cf3c", 2024-08-18T17:25:19.866 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": [ 2024-08-18T17:25:19.866 INFO:teuthology.orchestra.run.smithi134.stdout: "", 2024-08-18T17:25:19.866 INFO:teuthology.orchestra.run.smithi134.stdout: "", 2024-08-18T17:25:19.866 INFO:teuthology.orchestra.run.smithi134.stdout: "", 2024-08-18T17:25:19.867 INFO:teuthology.orchestra.run.smithi134.stdout: "1_1723995848.930494_187.1" 2024-08-18T17:25:19.867 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:25:19.867 INFO:teuthology.orchestra.run.smithi134.stdout: "period_map": { 2024-08-18T17:25:19.867 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:25:19.867 INFO:teuthology.orchestra.run.smithi134.stdout: "zonegroups": [ 2024-08-18T17:25:19.867 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:25:19.867 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "test-zonegroup", 2024-08-18T17:25:19.867 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "test-zonegroup", 2024-08-18T17:25:19.867 INFO:teuthology.orchestra.run.smithi134.stdout: "api_name": "", 2024-08-18T17:25:19.867 INFO:teuthology.orchestra.run.smithi134.stdout: "is_master": true, 2024-08-18T17:25:19.867 INFO:teuthology.orchestra.run.smithi134.stdout: "endpoints": [ 2024-08-18T17:25:19.867 INFO:teuthology.orchestra.run.smithi134.stdout: "http://smithi134.front.sepia.ceph.com:8000" 2024-08-18T17:25:19.867 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:25:19.868 INFO:teuthology.orchestra.run.smithi134.stdout: "hostnames": [], 2024-08-18T17:25:19.868 INFO:teuthology.orchestra.run.smithi134.stdout: "hostnames_s3website": [], 2024-08-18T17:25:19.868 INFO:teuthology.orchestra.run.smithi134.stdout: "master_zone": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T17:25:19.868 INFO:teuthology.orchestra.run.smithi134.stdout: "zones": [ 2024-08-18T17:25:19.868 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:25:19.868 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T17:25:19.868 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "test-zone3", 2024-08-18T17:25:19.868 INFO:teuthology.orchestra.run.smithi134.stdout: "endpoints": [ 2024-08-18T17:25:19.868 INFO:teuthology.orchestra.run.smithi134.stdout: "http://smithi134.front.sepia.ceph.com:8001" 2024-08-18T17:25:19.868 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:25:19.868 INFO:teuthology.orchestra.run.smithi134.stdout: "log_meta": false, 2024-08-18T17:25:19.868 INFO:teuthology.orchestra.run.smithi134.stdout: "log_data": true, 2024-08-18T17:25:19.868 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_index_max_shards": 11, 2024-08-18T17:25:19.868 INFO:teuthology.orchestra.run.smithi134.stdout: "read_only": false, 2024-08-18T17:25:19.868 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_type": "", 2024-08-18T17:25:19.869 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from_all": true, 2024-08-18T17:25:19.869 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from": [], 2024-08-18T17:25:19.869 INFO:teuthology.orchestra.run.smithi134.stdout: "redirect_zone": "", 2024-08-18T17:25:19.869 INFO:teuthology.orchestra.run.smithi134.stdout: "supported_features": [ 2024-08-18T17:25:19.869 INFO:teuthology.orchestra.run.smithi134.stdout: "compress-encrypted", 2024-08-18T17:25:19.869 INFO:teuthology.orchestra.run.smithi134.stdout: "resharding" 2024-08-18T17:25:19.869 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:25:19.869 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:25:19.869 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:25:19.869 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "10090ccf-aad6-416d-9775-df8b7b27345a", 2024-08-18T17:25:19.869 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "test-zone2", 2024-08-18T17:25:19.869 INFO:teuthology.orchestra.run.smithi134.stdout: "endpoints": [ 2024-08-18T17:25:19.870 INFO:teuthology.orchestra.run.smithi134.stdout: "http://smithi193.front.sepia.ceph.com:8000" 2024-08-18T17:25:19.870 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:25:19.870 INFO:teuthology.orchestra.run.smithi134.stdout: "log_meta": false, 2024-08-18T17:25:19.870 INFO:teuthology.orchestra.run.smithi134.stdout: "log_data": true, 2024-08-18T17:25:19.870 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_index_max_shards": 11, 2024-08-18T17:25:19.870 INFO:teuthology.orchestra.run.smithi134.stdout: "read_only": false, 2024-08-18T17:25:19.870 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_type": "", 2024-08-18T17:25:19.870 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from_all": true, 2024-08-18T17:25:19.870 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from": [], 2024-08-18T17:25:19.870 INFO:teuthology.orchestra.run.smithi134.stdout: "redirect_zone": "", 2024-08-18T17:25:19.870 INFO:teuthology.orchestra.run.smithi134.stdout: "supported_features": [ 2024-08-18T17:25:19.870 INFO:teuthology.orchestra.run.smithi134.stdout: "compress-encrypted", 2024-08-18T17:25:19.870 INFO:teuthology.orchestra.run.smithi134.stdout: "resharding" 2024-08-18T17:25:19.871 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:25:19.871 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:25:19.871 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:25:19.871 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T17:25:19.871 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "test-zone1", 2024-08-18T17:25:19.871 INFO:teuthology.orchestra.run.smithi134.stdout: "endpoints": [ 2024-08-18T17:25:19.871 INFO:teuthology.orchestra.run.smithi134.stdout: "http://smithi134.front.sepia.ceph.com:8000" 2024-08-18T17:25:19.871 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:25:19.871 INFO:teuthology.orchestra.run.smithi134.stdout: "log_meta": false, 2024-08-18T17:25:19.871 INFO:teuthology.orchestra.run.smithi134.stdout: "log_data": true, 2024-08-18T17:25:19.871 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_index_max_shards": 11, 2024-08-18T17:25:19.871 INFO:teuthology.orchestra.run.smithi134.stdout: "read_only": false, 2024-08-18T17:25:19.871 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_type": "", 2024-08-18T17:25:19.871 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from_all": true, 2024-08-18T17:25:19.872 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from": [], 2024-08-18T17:25:19.872 INFO:teuthology.orchestra.run.smithi134.stdout: "redirect_zone": "", 2024-08-18T17:25:19.872 INFO:teuthology.orchestra.run.smithi134.stdout: "supported_features": [ 2024-08-18T17:25:19.872 INFO:teuthology.orchestra.run.smithi134.stdout: "compress-encrypted", 2024-08-18T17:25:19.872 INFO:teuthology.orchestra.run.smithi134.stdout: "resharding" 2024-08-18T17:25:19.872 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:25:19.872 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:25:19.872 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:25:19.872 INFO:teuthology.orchestra.run.smithi134.stdout: "placement_targets": [ 2024-08-18T17:25:19.872 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:25:19.872 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "default-placement", 2024-08-18T17:25:19.872 INFO:teuthology.orchestra.run.smithi134.stdout: "tags": [], 2024-08-18T17:25:19.873 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_classes": [ 2024-08-18T17:25:19.873 INFO:teuthology.orchestra.run.smithi134.stdout: "STANDARD" 2024-08-18T17:25:19.873 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:25:19.873 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:25:19.873 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:25:19.873 INFO:teuthology.orchestra.run.smithi134.stdout: "default_placement": "default-placement", 2024-08-18T17:25:19.873 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_id": "2827c6f0-bf0d-4836-944c-9e7fcf22457b", 2024-08-18T17:25:19.873 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_policy": { 2024-08-18T17:25:19.873 INFO:teuthology.orchestra.run.smithi134.stdout: "groups": [ 2024-08-18T17:25:19.873 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:25:19.873 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "sync-group", 2024-08-18T17:25:19.873 INFO:teuthology.orchestra.run.smithi134.stdout: "data_flow": { 2024-08-18T17:25:19.873 INFO:teuthology.orchestra.run.smithi134.stdout: "symmetrical": [ 2024-08-18T17:25:19.873 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:25:19.874 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "sync-flow1", 2024-08-18T17:25:19.874 INFO:teuthology.orchestra.run.smithi134.stdout: "zones": [ 2024-08-18T17:25:19.899 INFO:teuthology.orchestra.run.smithi134.stdout: "10090ccf-aad6-416d-9775-df8b7b27345a", 2024-08-18T17:25:19.899 INFO:teuthology.orchestra.run.smithi134.stdout: "96bf8502-2f39-4d53-8196-3378c28da3b4" 2024-08-18T17:25:19.899 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:25:19.899 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:25:19.899 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:25:19.899 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:25:19.899 INFO:teuthology.orchestra.run.smithi134.stdout: "pipes": [ 2024-08-18T17:25:19.899 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:25:19.899 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "sync-pipe", 2024-08-18T17:25:19.899 INFO:teuthology.orchestra.run.smithi134.stdout: "source": { 2024-08-18T17:25:19.899 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket": "*", 2024-08-18T17:25:19.900 INFO:teuthology.orchestra.run.smithi134.stdout: "zones": [ 2024-08-18T17:25:19.900 INFO:teuthology.orchestra.run.smithi134.stdout: "10090ccf-aad6-416d-9775-df8b7b27345a", 2024-08-18T17:25:19.900 INFO:teuthology.orchestra.run.smithi134.stdout: "96bf8502-2f39-4d53-8196-3378c28da3b4" 2024-08-18T17:25:19.900 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:25:19.900 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:25:19.900 INFO:teuthology.orchestra.run.smithi134.stdout: "dest": { 2024-08-18T17:25:19.900 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket": "*", 2024-08-18T17:25:19.900 INFO:teuthology.orchestra.run.smithi134.stdout: "zones": [ 2024-08-18T17:25:19.900 INFO:teuthology.orchestra.run.smithi134.stdout: "10090ccf-aad6-416d-9775-df8b7b27345a", 2024-08-18T17:25:19.900 INFO:teuthology.orchestra.run.smithi134.stdout: "96bf8502-2f39-4d53-8196-3378c28da3b4" 2024-08-18T17:25:19.900 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:25:19.901 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:25:19.901 INFO:teuthology.orchestra.run.smithi134.stdout: "params": { 2024-08-18T17:25:19.901 INFO:teuthology.orchestra.run.smithi134.stdout: "source": { 2024-08-18T17:25:19.901 INFO:teuthology.orchestra.run.smithi134.stdout: "filter": { 2024-08-18T17:25:19.901 INFO:teuthology.orchestra.run.smithi134.stdout: "tags": [] 2024-08-18T17:25:19.901 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:25:19.901 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:25:19.901 INFO:teuthology.orchestra.run.smithi134.stdout: "dest": {}, 2024-08-18T17:25:19.901 INFO:teuthology.orchestra.run.smithi134.stdout: "priority": 0, 2024-08-18T17:25:19.901 INFO:teuthology.orchestra.run.smithi134.stdout: "mode": "system", 2024-08-18T17:25:19.901 INFO:teuthology.orchestra.run.smithi134.stdout: "user": "" 2024-08-18T17:25:19.901 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:25:19.902 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:25:19.902 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:25:19.902 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "allowed" 2024-08-18T17:25:19.902 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:25:19.902 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:25:19.902 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:25:19.902 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled_features": [ 2024-08-18T17:25:19.902 INFO:teuthology.orchestra.run.smithi134.stdout: "resharding" 2024-08-18T17:25:19.902 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:25:19.902 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:25:19.902 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:25:19.902 INFO:teuthology.orchestra.run.smithi134.stdout: "short_zone_ids": [ 2024-08-18T17:25:19.902 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:25:19.903 INFO:teuthology.orchestra.run.smithi134.stdout: "key": "018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T17:25:19.903 INFO:teuthology.orchestra.run.smithi134.stdout: "val": 462036791 2024-08-18T17:25:19.903 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:25:19.903 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:25:19.903 INFO:teuthology.orchestra.run.smithi134.stdout: "key": "10090ccf-aad6-416d-9775-df8b7b27345a", 2024-08-18T17:25:19.903 INFO:teuthology.orchestra.run.smithi134.stdout: "val": 874289624 2024-08-18T17:25:19.903 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:25:19.903 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:25:19.903 INFO:teuthology.orchestra.run.smithi134.stdout: "key": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T17:25:19.903 INFO:teuthology.orchestra.run.smithi134.stdout: "val": 1721399886 2024-08-18T17:25:19.903 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:25:19.903 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:25:19.903 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:25:19.904 INFO:teuthology.orchestra.run.smithi134.stdout: "master_zonegroup": "test-zonegroup", 2024-08-18T17:25:19.904 INFO:teuthology.orchestra.run.smithi134.stdout: "master_zone": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T17:25:19.904 INFO:teuthology.orchestra.run.smithi134.stdout: "period_config": { 2024-08-18T17:25:19.904 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_quota": { 2024-08-18T17:25:19.904 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false, 2024-08-18T17:25:19.904 INFO:teuthology.orchestra.run.smithi134.stdout: "check_on_raw": false, 2024-08-18T17:25:19.904 INFO:teuthology.orchestra.run.smithi134.stdout: "max_size": -1, 2024-08-18T17:25:19.904 INFO:teuthology.orchestra.run.smithi134.stdout: "max_size_kb": 0, 2024-08-18T17:25:19.904 INFO:teuthology.orchestra.run.smithi134.stdout: "max_objects": -1 2024-08-18T17:25:19.904 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:25:19.904 INFO:teuthology.orchestra.run.smithi134.stdout: "user_quota": { 2024-08-18T17:25:19.904 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false, 2024-08-18T17:25:19.905 INFO:teuthology.orchestra.run.smithi134.stdout: "check_on_raw": false, 2024-08-18T17:25:19.905 INFO:teuthology.orchestra.run.smithi134.stdout: "max_size": -1, 2024-08-18T17:25:19.905 INFO:teuthology.orchestra.run.smithi134.stdout: "max_size_kb": 0, 2024-08-18T17:25:19.905 INFO:teuthology.orchestra.run.smithi134.stdout: "max_objects": -1 2024-08-18T17:25:19.905 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:25:19.905 INFO:teuthology.orchestra.run.smithi134.stdout: "user_ratelimit": { 2024-08-18T17:25:19.906 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_ops": 0, 2024-08-18T17:25:19.906 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_ops": 0, 2024-08-18T17:25:19.906 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_bytes": 0, 2024-08-18T17:25:19.906 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_bytes": 0, 2024-08-18T17:25:19.906 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false 2024-08-18T17:25:19.906 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:25:19.906 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_ratelimit": { 2024-08-18T17:25:19.906 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_ops": 0, 2024-08-18T17:25:19.906 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_ops": 0, 2024-08-18T17:25:19.906 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_bytes": 0, 2024-08-18T17:25:19.906 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_bytes": 0, 2024-08-18T17:25:19.906 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false 2024-08-18T17:25:19.906 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:25:19.907 INFO:teuthology.orchestra.run.smithi134.stdout: "anonymous_ratelimit": { 2024-08-18T17:25:19.907 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_ops": 0, 2024-08-18T17:25:19.907 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_ops": 0, 2024-08-18T17:25:19.907 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_bytes": 0, 2024-08-18T17:25:19.907 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_bytes": 0, 2024-08-18T17:25:19.907 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false 2024-08-18T17:25:19.907 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:25:19.907 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:25:19.907 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_id": "2827c6f0-bf0d-4836-944c-9e7fcf22457b", 2024-08-18T17:25:19.907 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_name": "test-realm", 2024-08-18T17:25:19.907 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:25:19.907 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T17:25:19.909 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:25:19.909 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:25:19.909 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:25:19.957 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:25:19.957 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:25:20.033 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-08-18T17:25:20.033 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'kibgmb-93', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:25:20.033 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'kibgmb-93', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:25:20.033 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-bucket --status allowed --bucket kibgmb-93 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:25:20.099 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:25:20.099 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:25:20.248 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"allowed"}]} 2024-08-18T17:25:20.249 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--bucket', 'kibgmb-93', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:25:20.249 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--bucket', 'kibgmb-93', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:25:20.249 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flowA --flow-type symmetrical --zones=test-zone1,test-zone2 --bucket kibgmb-93 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:25:20.320 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:25:20.320 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:25:20.457 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["test-zone2","test-zone1"]}]},"pipes":[],"status":"allowed"}]} 2024-08-18T17:25:20.457 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipeA', '--source-zones=test-zone1,test-zone2', '--dest-zones=test-zone1,test-zone2', '--bucket=kibgmb-93', '--source-bucket=*', '--dest-bucket=kibgmb-94', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:25:20.458 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipeA', '--source-zones=test-zone1,test-zone2', '--dest-zones=test-zone1,test-zone2', '--bucket=kibgmb-93', '--source-bucket=*', '--dest-bucket=kibgmb-94', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:25:20.458 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipeA --source-zones=test-zone1,test-zone2 --dest-zones=test-zone1,test-zone2 --bucket=kibgmb-93 '--source-bucket=*' --dest-bucket=kibgmb-94 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:25:20.518 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:25:20.518 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:25:20.651 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["test-zone2","test-zone1"]}]},"pipes":[{"id":"sync-pipeA","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"kibgmb-94","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2024-08-18T17:25:20.651 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'kibgmb-93', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:25:20.651 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'kibgmb-93', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:25:20.652 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-bucket --status enabled --bucket kibgmb-93 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:25:20.708 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:25:20.708 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:25:20.835 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["test-zone2","test-zone1"]}]},"pipes":[{"id":"sync-pipeA","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"kibgmb-94","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2024-08-18T17:25:20.836 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--bucket', 'kibgmb-93', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:25:20.836 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--bucket', 'kibgmb-93', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:25:20.836 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --bucket kibgmb-93 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:25:20.887 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:25:20.888 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:25:20.971 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["test-zone2","test-zone1"]}]},"pipes":[{"id":"sync-pipeA","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"kibgmb-94","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2024-08-18T17:25:20.972 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:25:20.972 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:25:20.972 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:25:21.023 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:25:21.023 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:25:21.089 INFO:teuthology.orchestra.run.smithi134.stderr:No --period given, using current period=01050e30-179b-43be-bdbb-3bb23cf37ae8 2024-08-18T17:25:21.122 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T17:25:21.123 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:25:21.123 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:25:21.123 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T17:25:21.123 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:25:21.123 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:25:21.123 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:25:21.123 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T17:25:21.123 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:25:21.123 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:25:21.123 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:25:21.123 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:42:47.672461Z" 2024-08-18T17:25:21.124 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:25:21.124 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:25:21.124 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1724001920.785288_1647.1", 2024-08-18T17:25:21.124 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T17:25:20.785288Z" 2024-08-18T17:25:21.124 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:25:21.124 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T17:25:21.125 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-08-18T17:25:21.125 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:25:21.125 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:25:21.126 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:25:21.176 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:25:21.176 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:25:21.208 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:25:21.207+0000 7fcbf9886880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:25:21.208 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:25:21.208+0000 7fcbf9886880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:25:21.222 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:25:21.223 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:25:21.223 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:25:21.223 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:25:21.223 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:25:21.223 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:25:21.223 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:25:21.223 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:25:21.223 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:25:21.223 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:25:21.223 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:25:21.223 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:25:21.224 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:25:21.224 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:25:21.224 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:25:21.224 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:25:21.224 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:25:21.224 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:25:21.224 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:25:21.224 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:25:21.224 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:25:21.224 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:25:21.224 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:25:21.224 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:25:21.224 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:25:21.224 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:25:21.225 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:25:21.225 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:25:21.225 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:25:21.225 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:25:21.225 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:25:21.225 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:25:21.225 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:25:21.225 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:25:21.225 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:25:21.225 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:25:21.225 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:25:21.225 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:25:21.225 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:25:21.226 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:25:21.226 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:25:21.226 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:25:21.226 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:25:21.226 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:25:21.226 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:25:21.226 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:25:21.226 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:25:21.226 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:25:21.226 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:25:21.226 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001819.978375_1607.1", 2024-08-18T17:25:21.226 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:25:21.226 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:25:21.227 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:25:21.227 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:23:39.978375Z", 2024-08-18T17:25:21.227 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:25:21.227 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:25:21.227 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:25:21.227 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:25:21.227 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:25:21.227 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:25:21.227 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:25:21.227 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:25:21.227 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:25:21.227 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:25:21Z" 2024-08-18T17:25:21.227 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:25:21.228 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001920.785288_1647.1 target=1_1724001819.978375_1607.1 2024-08-18T17:25:26.229 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:25:26.229 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:25:26.229 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:25:26.284 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:25:26.284 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:25:26.318 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:25:26.317+0000 7f3b0aa3f880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:25:26.318 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:25:26.318+0000 7f3b0aa3f880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:25:26.333 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:25:26.334 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:25:26.334 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:25:26.334 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:25:26.334 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:25:26.334 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:25:26.334 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:25:26.334 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:25:26.334 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:25:26.334 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:25:26.334 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:25:26.334 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:25:26.335 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:25:26.335 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:25:26.335 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:25:26.335 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:25:26.335 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:25:26.335 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:25:26.335 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:25:26.335 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:25:26.335 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:25:26.335 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:25:26.335 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:25:26.336 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:25:26.336 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:25:26.336 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:25:26.336 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:25:26.336 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:25:26.336 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:25:26.336 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:25:26.336 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:25:26.336 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:25:26.336 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:25:26.336 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:25:26.336 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:25:26.336 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:25:26.336 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:25:26.337 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:25:26.337 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:25:26.337 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:25:26.337 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:25:26.337 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:25:26.337 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:25:26.337 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:25:26.337 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:25:26.337 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:25:26.337 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:25:26.337 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:25:26.337 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:25:26.337 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001819.978375_1607.1", 2024-08-18T17:25:26.338 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:25:26.338 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:25:26.338 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:25:26.338 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:23:39.978375Z", 2024-08-18T17:25:26.338 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:25:26.338 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:25:26.338 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:25:26.338 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:25:26.338 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:25:26.338 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:25:26.338 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:25:26.338 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:25:26.339 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:25:26.339 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:25:26Z" 2024-08-18T17:25:26.339 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:25:26.340 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001920.785288_1647.1 target=1_1724001819.978375_1607.1 2024-08-18T17:25:31.341 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:25:31.342 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:25:31.342 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:25:31.399 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:25:31.399 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:25:31.434 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:25:31.433+0000 7f51d0426880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:25:31.435 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:25:31.434+0000 7f51d0426880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:25:31.449 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:25:31.450 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:25:31.450 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:25:31.450 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:25:31.450 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:25:31.450 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:25:31.450 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:25:31.450 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:25:31.450 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:25:31.450 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:25:31.451 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:25:31.451 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:25:31.451 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:25:31.451 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:25:31.451 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:25:31.451 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:25:31.451 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:25:31.451 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:25:31.451 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:25:31.451 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:25:31.451 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:25:31.451 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:25:31.451 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:25:31.452 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:25:31.452 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:25:31.452 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:25:31.452 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:25:31.452 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:25:31.452 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:25:31.452 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:25:31.452 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:25:31.452 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:25:31.452 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:25:31.452 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:25:31.452 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:25:31.453 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:25:31.453 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:25:31.453 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:25:31.453 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:25:31.453 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:25:31.453 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:25:31.453 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:25:31.453 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:25:31.454 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:25:31.454 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:25:31.454 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:25:31.454 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:25:31.454 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:25:31.454 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:25:31.454 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001819.978375_1607.1", 2024-08-18T17:25:31.454 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:25:31.455 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:25:31.455 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:25:31.455 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:23:39.978375Z", 2024-08-18T17:25:31.455 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:25:31.455 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:25:31.455 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:25:31.455 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:25:31.455 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:25:31.455 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:25:31.455 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:25:31.455 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:25:31.455 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:25:31.455 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:25:31Z" 2024-08-18T17:25:31.456 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:25:31.457 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001920.785288_1647.1 target=1_1724001819.978375_1607.1 2024-08-18T17:25:36.458 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:25:36.458 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:25:36.459 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:25:36.514 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:25:36.514 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:25:36.548 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:25:36.547+0000 7f9267f40880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:25:36.548 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:25:36.548+0000 7f9267f40880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:25:36.564 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:25:36.564 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:25:36.564 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:25:36.564 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:25:36.564 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:25:36.564 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:25:36.564 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:25:36.564 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:25:36.564 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:25:36.564 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:25:36.565 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:25:36.565 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:25:36.565 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:25:36.565 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:25:36.565 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:25:36.565 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:25:36.565 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:25:36.565 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:25:36.565 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:25:36.565 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:25:36.565 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:25:36.566 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:25:36.566 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:25:36.566 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:25:36.566 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:25:36.566 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:25:36.566 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:25:36.566 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:25:36.566 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:25:36.566 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:25:36.566 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:25:36.566 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:25:36.566 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:25:36.567 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:25:36.567 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:25:36.567 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:25:36.567 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:25:36.567 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:25:36.567 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:25:36.567 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:25:36.567 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:25:36.567 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:25:36.567 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:25:36.568 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:25:36.568 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:25:36.568 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:25:36.568 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:25:36.568 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:25:36.568 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:25:36.568 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001819.978375_1607.1", 2024-08-18T17:25:36.568 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:25:36.568 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:25:36.568 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:25:36.568 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:23:39.978375Z", 2024-08-18T17:25:36.568 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:25:36.569 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:25:36.569 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:25:36.569 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:25:36.569 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:25:36.569 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:25:36.569 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:25:36.569 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:25:36.569 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:25:36.569 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:25:36Z" 2024-08-18T17:25:36.569 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:25:36.570 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001920.785288_1647.1 target=1_1724001819.978375_1607.1 2024-08-18T17:25:41.571 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:25:41.572 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:25:41.572 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:25:41.627 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:25:41.627 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:25:41.661 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:25:41.661+0000 7fb30c122880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:25:41.662 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:25:41.662+0000 7fb30c122880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:25:41.677 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:25:41.677 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:25:41.677 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:25:41.677 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:25:41.677 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:25:41.678 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:25:41.678 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:25:41.678 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:25:41.678 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:25:41.678 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:25:41.678 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:25:41.678 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:25:41.678 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:25:41.678 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:25:41.678 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:25:41.678 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:25:41.679 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:25:41.679 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:25:41.679 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:25:41.679 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:25:41.679 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:25:41.679 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:25:41.679 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:25:41.679 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:25:41.679 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:25:41.679 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:25:41.679 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:25:41.679 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:25:41.680 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:25:41.680 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:25:41.680 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:25:41.680 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:25:41.680 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:25:41.680 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:25:41.680 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:25:41.680 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:25:41.680 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:25:41.680 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:25:41.680 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:25:41.680 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:25:41.681 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:25:41.681 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:25:41.681 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:25:41.681 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:25:41.681 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:25:41.681 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:25:41.681 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:25:41.681 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:25:41.682 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:25:41.682 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001819.978375_1607.1", 2024-08-18T17:25:41.682 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:25:41.682 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:25:41.682 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:25:41.682 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:23:39.978375Z", 2024-08-18T17:25:41.682 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:25:41.682 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:25:41.682 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:25:41.682 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:25:41.682 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:25:41.682 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:25:41.683 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:25:41.683 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:25:41.683 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:25:41.683 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:25:41Z" 2024-08-18T17:25:41.683 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:25:41.684 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001920.785288_1647.1 target=1_1724001819.978375_1607.1 2024-08-18T17:25:46.685 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:25:46.685 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:25:46.685 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:25:46.740 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:25:46.740 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:25:46.774 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:25:46.773+0000 7fdef4a62880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:25:46.774 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:25:46.774+0000 7fdef4a62880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:25:46.789 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:25:46.789 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:25:46.789 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:25:46.790 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:25:46.790 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:25:46.790 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:25:46.790 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:25:46.790 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:25:46.790 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:25:46.790 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:25:46.790 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:25:46.790 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:25:46.790 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:25:46.790 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:25:46.791 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:25:46.791 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:25:46.791 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:25:46.791 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:25:46.791 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:25:46.791 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:25:46.791 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:25:46.791 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:25:46.791 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:25:46.791 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:25:46.791 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:25:46.791 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:25:46.792 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:25:46.792 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:25:46.792 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:25:46.792 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:25:46.792 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:25:46.792 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:25:46.792 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:25:46.792 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:25:46.792 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:25:46.792 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:25:46.792 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:25:46.793 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:25:46.793 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:25:46.793 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:25:46.793 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:25:46.793 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:25:46.793 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:25:46.794 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:25:46.794 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:25:46.794 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:25:46.794 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:25:46.794 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:25:46.794 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:25:46.794 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001819.978375_1607.1", 2024-08-18T17:25:46.794 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:25:46.794 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:25:46.795 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:25:46.795 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:23:39.978375Z", 2024-08-18T17:25:46.795 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:25:46.795 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:25:46.795 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:25:46.795 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:25:46.796 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:25:46.796 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:25:46.796 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:25:46.796 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:25:46.796 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:25:46.796 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:25:46Z" 2024-08-18T17:25:46.796 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:25:46.798 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001920.785288_1647.1 target=1_1724001819.978375_1607.1 2024-08-18T17:25:51.798 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:25:51.799 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:25:51.799 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:25:51.853 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:25:51.854 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:25:51.888 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:25:51.887+0000 7f043ff32880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:25:51.889 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:25:51.888+0000 7f043ff32880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:25:51.903 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:25:51.903 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:25:51.903 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:25:51.904 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:25:51.904 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:25:51.904 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:25:51.904 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:25:51.904 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:25:51.904 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:25:51.904 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:25:51.904 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:25:51.904 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:25:51.904 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:25:51.904 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:25:51.904 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:25:51.904 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:25:51.904 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:25:51.905 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:25:51.905 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:25:51.905 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:25:51.905 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:25:51.905 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:25:51.905 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:25:51.905 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:25:51.905 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:25:51.905 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:25:51.905 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:25:51.905 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:25:51.905 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:25:51.905 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:25:51.906 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:25:51.906 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:25:51.906 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:25:51.906 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:25:51.906 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:25:51.906 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:25:51.906 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:25:51.906 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:25:51.906 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:25:51.906 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:25:51.906 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:25:51.907 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:25:51.907 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:25:51.907 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:25:51.907 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:25:51.907 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:25:51.907 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:25:51.907 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:25:51.907 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:25:51.908 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001819.978375_1607.1", 2024-08-18T17:25:51.908 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:25:51.908 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:25:51.908 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:25:51.908 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:23:39.978375Z", 2024-08-18T17:25:51.908 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:25:51.908 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:25:51.908 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:25:51.908 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:25:51.908 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:25:51.909 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:25:51.909 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:25:51.909 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:25:51.909 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:25:51.909 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:25:51Z" 2024-08-18T17:25:51.909 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:25:51.910 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724001920.785288_1647.1 target=1_1724001819.978375_1607.1 2024-08-18T17:25:56.911 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:25:56.911 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:25:56.911 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:25:56.966 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:25:56.966 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:25:57.000 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:25:56.999+0000 7f191cd36880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:25:57.000 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:25:57.000+0000 7f191cd36880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:25:57.016 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:25:57.017 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:25:57.017 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:25:57.017 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:25:57.017 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:25:57.017 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:25:57.017 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:25:57.017 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:25:57.017 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:25:57.017 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:25:57.018 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:25:57.018 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:25:57.018 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:25:57.018 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:25:57.018 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:25:57.018 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:25:57.018 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:25:57.018 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:25:57.018 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:25:57.018 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:25:57.018 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:25:57.018 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:25:57.019 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:25:57.019 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:25:57.019 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:25:57.019 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:25:57.019 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:25:57.019 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:25:57.019 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:25:57.019 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:25:57.019 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:25:57.019 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:25:57.019 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:25:57.019 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:25:57.020 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:25:57.020 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:25:57.020 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:25:57.020 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:25:57.020 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:25:57.020 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:25:57.020 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:25:57.020 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:25:57.020 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:25:57.020 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:25:57.020 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:25:57.020 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:25:57.020 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:25:57.021 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:25:57.021 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:25:57.021 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001920.785288_1647.1", 2024-08-18T17:25:57.021 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:25:57.021 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:25:57.021 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:25:57.021 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:25:20.785288Z", 2024-08-18T17:25:57.021 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:25:57.021 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:25:57.021 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:25:57.021 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:25:57.021 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:25:57.022 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:25:57.022 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:25:57.022 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:25:57.022 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:25:57.022 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:25:57Z" 2024-08-18T17:25:57.022 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:25:57.023 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-08-18T17:25:57.023 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-08-18T17:25:57.023 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:25:57.023 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:25:57.024 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:25:57.078 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:25:57.078 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:25:57.122 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:25:57.121+0000 7f1db4922880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:25:57.123 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:25:57.122+0000 7f1db4922880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:25:57.145 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T17:25:57.145 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T17:25:57.145 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T17:25:57.145 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T17:25:57.145 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T17:25:57.145 INFO:teuthology.orchestra.run.smithi134.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:25:57.145 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:25:57.145 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:25:57.145 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T17:25:57.146 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:25:57.146 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T17:25:57.146 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:25:57.146 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:25:57.146 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:25:57.146 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:25:57.146 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 1, 2024-08-18T17:25:57.146 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:25:57.146 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:25:57.147 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T17:25:57.147 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:25:57.147 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:25:57.147 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:25:57.147 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T17:25:57.147 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:25:57.147 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:25:57.147 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:25:57.147 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:25:57.147 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:25:57.147 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:25:57.147 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T17:25:57.147 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T17:25:57.148 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:25:57.148 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:25:57.148 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:25:57.148 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T17:25:57.148 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:25:57.148 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:25:57.148 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:25:57.148 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:25:57.148 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:25:57.149 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:25:57.149 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:25:57.149 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:25:57.149 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:25:57.149 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:25:57.149 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:25:57.149 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T17:25:57.149 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:25:57.149 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:25:57.149 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1724001920.785288_1647.1", 2024-08-18T17:25:57.149 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:25:57.149 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:25:57.149 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:25:57.150 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T17:25:20.785288Z", 2024-08-18T17:25:57.150 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:25:57.150 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:25:57.150 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:25:57.150 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:25:57.150 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:25:57.150 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T17:25:57.150 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 1, 2024-08-18T17:25:57.150 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 1 2024-08-18T17:25:57.151 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:25:57.151 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T17:25:57Z" 2024-08-18T17:25:57.151 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T17:25:57.152 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-08-18T17:25:57.215 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:25:57.215 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:25:57.215 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:25:57.261 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:25:57.262 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:25:57.295 INFO:teuthology.orchestra.run.smithi134.stderr:No --period given, using current period=01050e30-179b-43be-bdbb-3bb23cf37ae8 2024-08-18T17:25:57.310 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T17:25:57.310 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:25:57.310 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:25:57.310 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T17:25:57.310 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:25:57.311 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:25:57.311 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:25:57.311 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T17:25:57.311 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:25:57.311 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:25:57.311 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:25:57.311 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:42:47.672461Z" 2024-08-18T17:25:57.311 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:25:57.311 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:25:57.311 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1724001920.785288_1647.1", 2024-08-18T17:25:57.311 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T17:25:20.785288Z" 2024-08-18T17:25:57.311 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:25:57.312 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T17:25:57.312 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-08-18T17:25:57.313 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:25:57.313 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:25:57.313 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:25:57.366 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:25:57.367 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:25:57.399 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:25:57.398+0000 7f0293b26880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:25:57.400 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:25:57.399+0000 7f0293b26880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:25:57.414 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:25:57.414 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:25:57.414 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:25:57.414 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:25:57.415 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:25:57.415 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:25:57.415 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:25:57.415 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:25:57.415 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:25:57.415 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:25:57.415 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:25:57.415 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:25:57.415 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:25:57.415 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:25:57.415 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:25:57.415 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:25:57.415 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:25:57.416 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:25:57.416 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:25:57.416 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:25:57.416 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:25:57.416 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:25:57.416 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:25:57.416 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:25:57.416 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:25:57.416 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:25:57.416 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:25:57.416 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:25:57.416 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:25:57.416 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:25:57.417 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:25:57.417 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:25:57.417 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:25:57.417 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:25:57.417 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:25:57.417 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:25:57.417 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:25:57.417 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:25:57.417 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:25:57.417 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:25:57.417 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:25:57.417 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:25:57.417 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:25:57.418 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:25:57.418 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:25:57.418 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:25:57.418 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:25:57.418 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:25:57.418 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:25:57.418 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001920.785288_1647.1", 2024-08-18T17:25:57.418 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:25:57.418 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:25:57.418 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:25:57.418 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:25:20.785288Z", 2024-08-18T17:25:57.418 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:25:57.419 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:25:57.419 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:25:57.419 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:25:57.419 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:25:57.419 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:25:57.419 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:25:57.419 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:25:57.419 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:25:57.419 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:25:57Z" 2024-08-18T17:25:57.419 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:25:57.420 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-08-18T17:25:57.420 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-08-18T17:25:57.420 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:25:57.421 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:25:57.421 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:25:57.468 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:25:57.469 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:25:57.502 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:25:57.502+0000 7fcf43126880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:25:57.503 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:25:57.502+0000 7fcf43126880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:25:57.526 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T17:25:57.526 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T17:25:57.526 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T17:25:57.526 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T17:25:57.527 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T17:25:57.527 INFO:teuthology.orchestra.run.smithi134.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:25:57.527 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:25:57.527 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:25:57.527 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T17:25:57.527 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:25:57.527 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T17:25:57.527 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:25:57.527 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:25:57.527 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:25:57.527 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:25:57.527 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 1, 2024-08-18T17:25:57.527 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:25:57.527 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:25:57.528 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T17:25:57.528 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:25:57.528 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:25:57.528 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:25:57.528 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T17:25:57.528 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:25:57.528 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:25:57.528 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:25:57.528 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:25:57.528 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:25:57.528 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:25:57.528 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T17:25:57.528 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T17:25:57.528 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:25:57.529 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:25:57.529 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:25:57.529 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T17:25:57.529 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:25:57.529 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:25:57.529 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:25:57.529 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:25:57.529 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:25:57.529 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:25:57.529 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:25:57.529 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:25:57.529 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:25:57.529 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:25:57.529 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:25:57.530 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T17:25:57.530 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:25:57.530 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:25:57.530 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1724001920.785288_1647.1", 2024-08-18T17:25:57.530 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:25:57.530 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:25:57.530 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:25:57.530 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T17:25:20.785288Z", 2024-08-18T17:25:57.531 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:25:57.531 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:25:57.531 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:25:57.531 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:25:57.531 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:25:57.531 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T17:25:57.531 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 1, 2024-08-18T17:25:57.531 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 1 2024-08-18T17:25:57.531 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:25:57.531 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T17:25:57Z" 2024-08-18T17:25:57.532 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T17:25:57.532 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-08-18T17:25:57.533 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:25:57.533 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:25:57.533 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:25:57.578 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:25:57.579 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:25:57.649 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T17:25:57.649 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:25:57.649 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000052014", 2024-08-18T17:25:57.649 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T17:25:57.178435Z" 2024-08-18T17:25:57.650 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:25:57.650 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:25:57.650 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:25:57.651 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:25:57.651 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:25:57.651 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:25:57.651 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000059697", 2024-08-18T17:25:57.651 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T17:25:57.169082Z" 2024-08-18T17:25:57.651 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:25:57.652 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:25:57.652 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:25:57.652 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:25:57.652 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:25:57.652 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T17:25:57.654 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-08-18T17:25:57.654 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:25:57.655 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:25:57.655 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:25:57.707 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:25:57.707 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:25:57.754 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:25:57.755 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:25:57.755 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:25:57.755 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:25:57.755 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:25:57.755 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:25:57.755 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:25:57.755 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:25:57.756 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:25:57.756 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:25:57.756 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:25:57.756 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:25:57.756 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000051874", 2024-08-18T17:25:57.757 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:25:57.757 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:25:57.757 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:25:57.757 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:23:46.463578Z" 2024-08-18T17:25:57.757 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:25:57.757 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:25:57.758 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:25:57.758 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:25:57.758 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:25:57.758 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:25:57.758 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:25:57.758 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:25:57.758 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:25:57.758 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:25:57.759 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:25:57.759 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:25:57.759 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:25:57.759 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:25:57.759 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:25:57.759 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:25:57.759 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:25:57.759 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000059557", 2024-08-18T17:25:57.759 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:25:57.759 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:25:57.760 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:25:57.760 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:23:46.436203Z" 2024-08-18T17:25:57.760 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:25:57.760 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:25:57.760 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:25:57.760 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:25:57.760 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:25:57.761 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:25:57.761 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:25:57.761 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:25:57.761 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:25:57.761 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:25:57.761 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:25:57.761 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:25:57.762 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:25:57.762 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:25:57.762 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:25:57.762 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:25:57.762 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:25:57.762 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:25:57.762 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:25:57.763 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:25:57Z" 2024-08-18T17:25:57.763 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:25:57.764 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000052014 target=00000000000000000000:00000000000000051874, shard=2 master=00000000000000000000:00000000000000059697 target=00000000000000000000:00000000000000059557 2024-08-18T17:26:02.766 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:26:02.766 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:26:02.766 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:26:02.824 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:26:02.824 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:26:02.874 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:26:02.875 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:26:02.875 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:26:02.875 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:26:02.875 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:26:02.875 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:26:02.875 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:26:02.875 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:26:02.875 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:26:02.875 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:26:02.876 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:26:02.876 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:26:02.876 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000051874", 2024-08-18T17:26:02.876 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:26:02.876 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:26:02.876 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:26:02.876 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:23:46.463578Z" 2024-08-18T17:26:02.876 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:26:02.876 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:26:02.876 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:26:02.876 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:26:02.876 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:26:02.876 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:26:02.877 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:26:02.877 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:26:02.877 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:26:02.877 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:26:02.877 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:26:02.877 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:26:02.877 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:26:02.877 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:26:02.877 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:26:02.877 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:26:02.878 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:26:02.878 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000059557", 2024-08-18T17:26:02.878 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:26:02.878 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:26:02.878 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:26:02.878 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:23:46.436203Z" 2024-08-18T17:26:02.878 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:26:02.878 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:26:02.878 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:26:02.878 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:26:02.879 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:26:02.879 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:26:02.879 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:26:02.879 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:26:02.879 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:26:02.879 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:26:02.879 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:26:02.879 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:26:02.879 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:26:02.879 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:26:02.879 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:26:02.879 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:26:02.879 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:26:02.879 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:26:02.879 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:26:02.880 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:26:02Z" 2024-08-18T17:26:02.880 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:26:02.881 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000052014 target=00000000000000000000:00000000000000051874, shard=2 master=00000000000000000000:00000000000000059697 target=00000000000000000000:00000000000000059557 2024-08-18T17:26:07.881 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:26:07.881 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:26:07.882 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:26:07.940 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:26:07.941 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:26:07.992 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:26:07.992 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:26:07.992 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:26:07.992 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:26:07.992 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:26:07.992 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:26:07.993 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:26:07.993 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:26:07.993 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:26:07.993 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:26:07.993 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:26:07.993 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:26:07.993 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000051874", 2024-08-18T17:26:07.993 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:26:07.993 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:26:07.993 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:26:07.993 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:23:46.463578Z" 2024-08-18T17:26:07.993 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:26:07.993 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:26:07.994 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:26:07.994 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:26:07.994 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:26:07.994 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:26:07.994 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:26:07.994 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:26:07.994 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:26:07.994 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:26:07.994 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:26:07.994 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:26:07.994 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:26:07.994 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:26:07.994 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:26:07.995 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:26:07.995 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:26:07.995 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000059557", 2024-08-18T17:26:07.995 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:26:07.995 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:26:07.995 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:26:07.995 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:23:46.436203Z" 2024-08-18T17:26:07.995 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:26:07.995 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:26:07.995 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:26:07.995 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:26:07.995 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:26:07.995 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:26:07.996 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:26:07.996 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:26:07.996 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:26:07.996 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:26:07.996 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:26:07.996 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:26:07.996 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:26:07.996 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:26:07.996 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:26:07.996 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:26:07.996 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:26:07.996 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:26:07.996 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:26:07.997 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:26:07Z" 2024-08-18T17:26:07.997 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:26:07.998 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000052014 target=00000000000000000000:00000000000000051874, shard=2 master=00000000000000000000:00000000000000059697 target=00000000000000000000:00000000000000059557 2024-08-18T17:26:12.999 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:26:12.999 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:26:12.999 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:26:13.056 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:26:13.057 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:26:13.109 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:26:13.109 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:26:13.109 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:26:13.109 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:26:13.109 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:26:13.109 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:26:13.110 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:26:13.110 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:26:13.110 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:26:13.110 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:26:13.110 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:26:13.110 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:26:13.110 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000051874", 2024-08-18T17:26:13.110 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:26:13.110 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:26:13.110 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:26:13.110 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:23:46.463578Z" 2024-08-18T17:26:13.110 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:26:13.111 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:26:13.111 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:26:13.111 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:26:13.111 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:26:13.111 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:26:13.111 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:26:13.111 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:26:13.111 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:26:13.111 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:26:13.111 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:26:13.111 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:26:13.111 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:26:13.112 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:26:13.112 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:26:13.112 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:26:13.112 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:26:13.112 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000059557", 2024-08-18T17:26:13.112 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:26:13.112 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:26:13.112 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:26:13.112 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:23:46.436203Z" 2024-08-18T17:26:13.112 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:26:13.112 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:26:13.112 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:26:13.112 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:26:13.113 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:26:13.113 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:26:13.113 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:26:13.113 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:26:13.113 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:26:13.113 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:26:13.113 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:26:13.113 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:26:13.113 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:26:13.113 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:26:13.113 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:26:13.113 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:26:13.113 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:26:13.114 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:26:13.114 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:26:13.114 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:26:13Z" 2024-08-18T17:26:13.114 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:26:13.115 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000052014 target=00000000000000000000:00000000000000051874, shard=2 master=00000000000000000000:00000000000000059697 target=00000000000000000000:00000000000000059557 2024-08-18T17:26:18.116 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:26:18.117 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:26:18.117 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:26:18.175 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:26:18.175 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:26:18.226 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:26:18.227 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:26:18.227 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:26:18.227 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:26:18.227 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:26:18.227 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:26:18.227 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:26:18.227 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:26:18.227 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:26:18.227 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:26:18.227 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:26:18.228 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:26:18.228 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000051874", 2024-08-18T17:26:18.228 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:26:18.228 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:26:18.228 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:26:18.228 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:23:46.463578Z" 2024-08-18T17:26:18.228 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:26:18.228 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:26:18.228 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:26:18.228 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:26:18.229 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:26:18.229 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:26:18.229 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:26:18.229 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:26:18.229 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:26:18.229 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:26:18.229 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:26:18.229 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:26:18.229 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:26:18.229 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:26:18.229 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:26:18.230 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:26:18.230 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:26:18.230 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000059557", 2024-08-18T17:26:18.230 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:26:18.230 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:26:18.230 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:26:18.230 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:23:46.436203Z" 2024-08-18T17:26:18.231 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:26:18.231 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:26:18.231 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:26:18.231 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:26:18.231 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:26:18.231 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:26:18.231 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:26:18.231 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:26:18.231 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:26:18.231 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:26:18.232 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:26:18.232 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:26:18.232 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:26:18.232 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:26:18.232 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:26:18.232 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:26:18.232 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:26:18.232 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:26:18.232 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:26:18.232 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:26:18Z" 2024-08-18T17:26:18.232 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:26:18.234 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000052014 target=00000000000000000000:00000000000000051874, shard=2 master=00000000000000000000:00000000000000059697 target=00000000000000000000:00000000000000059557 2024-08-18T17:26:23.234 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:26:23.235 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:26:23.235 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:26:23.292 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:26:23.292 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:26:23.343 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:26:23.343 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:26:23.343 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:26:23.344 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:26:23.344 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:26:23.344 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:26:23.344 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:26:23.344 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:26:23.344 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:26:23.344 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:26:23.344 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:26:23.344 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:26:23.344 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000051874", 2024-08-18T17:26:23.344 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:26:23.344 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:26:23.345 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:26:23.345 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:23:46.463578Z" 2024-08-18T17:26:23.345 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:26:23.345 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:26:23.345 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:26:23.345 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:26:23.345 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:26:23.345 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:26:23.345 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:26:23.345 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:26:23.345 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:26:23.345 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:26:23.345 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:26:23.346 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:26:23.346 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:26:23.346 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:26:23.346 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:26:23.346 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:26:23.346 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:26:23.346 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000059557", 2024-08-18T17:26:23.346 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:26:23.346 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:26:23.346 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:26:23.346 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:23:46.436203Z" 2024-08-18T17:26:23.346 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:26:23.347 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:26:23.347 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:26:23.347 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:26:23.347 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:26:23.347 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:26:23.347 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:26:23.347 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:26:23.347 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:26:23.347 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:26:23.347 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:26:23.347 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:26:23.347 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:26:23.347 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:26:23.348 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:26:23.348 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:26:23.348 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:26:23.348 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:26:23.348 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:26:23.348 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:26:23Z" 2024-08-18T17:26:23.348 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:26:23.349 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000052014 target=00000000000000000000:00000000000000051874, shard=2 master=00000000000000000000:00000000000000059697 target=00000000000000000000:00000000000000059557 2024-08-18T17:26:28.349 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:26:28.350 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:26:28.350 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:26:28.407 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:26:28.408 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:26:28.458 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:26:28.458 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:26:28.458 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:26:28.458 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:26:28.458 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:26:28.458 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:26:28.458 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:26:28.458 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:26:28.458 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:26:28.459 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:26:28.459 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:26:28.459 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:26:28.459 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000051874", 2024-08-18T17:26:28.459 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:26:28.459 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:26:28.459 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:26:28.459 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:23:46.463578Z" 2024-08-18T17:26:28.459 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:26:28.459 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:26:28.459 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:26:28.460 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:26:28.460 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:26:28.460 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:26:28.460 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:26:28.460 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:26:28.460 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:26:28.460 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:26:28.460 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:26:28.460 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:26:28.460 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:26:28.460 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:26:28.460 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:26:28.461 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:26:28.461 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:26:28.461 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000059557", 2024-08-18T17:26:28.461 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:26:28.461 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:26:28.461 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:26:28.461 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:23:46.436203Z" 2024-08-18T17:26:28.461 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:26:28.462 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:26:28.462 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:26:28.462 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:26:28.462 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:26:28.462 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:26:28.462 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:26:28.462 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:26:28.462 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:26:28.462 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:26:28.462 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:26:28.463 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:26:28.463 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:26:28.463 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:26:28.463 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:26:28.463 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:26:28.463 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:26:28.463 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:26:28.463 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:26:28.463 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:26:28Z" 2024-08-18T17:26:28.463 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:26:28.465 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000052014 target=00000000000000000000:00000000000000051874, shard=2 master=00000000000000000000:00000000000000059697 target=00000000000000000000:00000000000000059557 2024-08-18T17:26:33.465 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:26:33.466 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:26:33.466 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:26:33.523 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:26:33.523 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:26:33.575 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:26:33.575 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:26:33.575 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:26:33.575 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:26:33.575 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:26:33.576 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:26:33.576 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:26:33.576 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:26:33.576 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:26:33.576 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:26:33.576 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:26:33.576 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:26:33.576 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000051874", 2024-08-18T17:26:33.576 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:26:33.576 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:26:33.577 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:26:33.577 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:23:46.463578Z" 2024-08-18T17:26:33.577 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:26:33.577 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:26:33.577 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:26:33.577 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:26:33.578 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:26:33.578 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:26:33.578 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:26:33.578 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:26:33.578 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:26:33.578 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:26:33.578 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:26:33.578 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:26:33.578 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:26:33.578 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:26:33.578 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:26:33.579 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:26:33.579 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:26:33.579 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000059557", 2024-08-18T17:26:33.579 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:26:33.579 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:26:33.579 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:26:33.579 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:23:46.436203Z" 2024-08-18T17:26:33.579 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:26:33.579 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:26:33.580 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:26:33.580 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:26:33.580 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:26:33.580 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:26:33.580 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:26:33.580 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:26:33.580 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:26:33.580 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:26:33.580 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:26:33.580 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:26:33.580 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:26:33.581 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:26:33.581 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:26:33.581 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:26:33.581 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:26:33.581 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:26:33.581 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:26:33.581 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:26:33Z" 2024-08-18T17:26:33.581 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:26:33.583 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000052014 target=00000000000000000000:00000000000000051874, shard=2 master=00000000000000000000:00000000000000059697 target=00000000000000000000:00000000000000059557 2024-08-18T17:26:38.584 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:26:38.585 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:26:38.585 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:26:38.641 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:26:38.641 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:26:38.695 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:26:38.695 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:26:38.695 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:26:38.695 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:26:38.695 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:26:38.695 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:26:38.695 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:26:38.695 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:26:38.696 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:26:38.696 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:26:38.696 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:26:38.696 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:26:38.696 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000051874", 2024-08-18T17:26:38.696 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:26:38.696 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:26:38.696 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:26:38.696 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:23:46.463578Z" 2024-08-18T17:26:38.696 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:26:38.696 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:26:38.696 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:26:38.697 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:26:38.697 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:26:38.697 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:26:38.697 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:26:38.697 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:26:38.697 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:26:38.697 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:26:38.697 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:26:38.697 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:26:38.697 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:26:38.697 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:26:38.697 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:26:38.698 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:26:38.698 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:26:38.698 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000059557", 2024-08-18T17:26:38.698 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:26:38.698 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:26:38.698 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:26:38.698 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:23:46.436203Z" 2024-08-18T17:26:38.698 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:26:38.698 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:26:38.698 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:26:38.698 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:26:38.698 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:26:38.698 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:26:38.699 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:26:38.699 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:26:38.699 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:26:38.699 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:26:38.699 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:26:38.699 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:26:38.699 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:26:38.699 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:26:38.699 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:26:38.700 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:26:38.700 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:26:38.700 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:26:38.700 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:26:38.700 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:26:38Z" 2024-08-18T17:26:38.700 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:26:38.701 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000052014 target=00000000000000000000:00000000000000051874, shard=2 master=00000000000000000000:00000000000000059697 target=00000000000000000000:00000000000000059557 2024-08-18T17:26:43.702 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:26:43.702 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:26:43.703 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:26:43.760 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:26:43.760 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:26:43.809 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:26:43.810 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:26:43.810 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:26:43.810 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:26:43.810 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:26:43.810 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:26:43.810 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:26:43.810 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:26:43.810 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:26:43.810 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:26:43.810 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:26:43.811 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:26:43.811 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000051874", 2024-08-18T17:26:43.811 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:26:43.811 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:26:43.811 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:26:43.811 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:23:46.463578Z" 2024-08-18T17:26:43.811 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:26:43.811 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:26:43.811 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:26:43.811 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:26:43.811 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:26:43.811 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:26:43.812 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:26:43.812 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:26:43.812 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:26:43.812 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:26:43.812 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:26:43.812 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:26:43.812 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:26:43.812 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:26:43.812 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:26:43.812 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:26:43.812 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:26:43.812 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000059557", 2024-08-18T17:26:43.813 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:26:43.813 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:26:43.813 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:26:43.813 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:23:46.436203Z" 2024-08-18T17:26:43.813 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:26:43.813 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:26:43.813 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:26:43.813 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:26:43.813 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:26:43.814 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:26:43.814 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:26:43.814 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:26:43.814 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:26:43.814 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:26:43.814 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:26:43.814 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:26:43.814 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:26:43.814 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:26:43.814 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:26:43.814 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:26:43.814 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:26:43.815 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:26:43.815 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:26:43.815 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:26:43Z" 2024-08-18T17:26:43.815 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:26:43.816 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000052014 target=00000000000000000000:00000000000000051874, shard=2 master=00000000000000000000:00000000000000059697 target=00000000000000000000:00000000000000059557 2024-08-18T17:26:48.817 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:26:48.817 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:26:48.817 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:26:48.874 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:26:48.874 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:26:48.925 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:26:48.925 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:26:48.926 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:26:48.926 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:26:48.926 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:26:48.926 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:26:48.926 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:26:48.926 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:26:48.926 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:26:48.926 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:26:48.926 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:26:48.926 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:26:48.927 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000051874", 2024-08-18T17:26:48.927 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:26:48.927 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:26:48.927 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:26:48.927 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:23:46.463578Z" 2024-08-18T17:26:48.927 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:26:48.927 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:26:48.927 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:26:48.927 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:26:48.927 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:26:48.927 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:26:48.927 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:26:48.928 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:26:48.928 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:26:48.928 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:26:48.928 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:26:48.928 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:26:48.928 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:26:48.928 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:26:48.928 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:26:48.928 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:26:48.928 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:26:48.928 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000059557", 2024-08-18T17:26:48.928 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:26:48.928 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:26:48.928 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:26:48.929 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:23:46.436203Z" 2024-08-18T17:26:48.929 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:26:48.929 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:26:48.929 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:26:48.929 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:26:48.929 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:26:48.929 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:26:48.929 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:26:48.930 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:26:48.930 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:26:48.930 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:26:48.930 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:26:48.930 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:26:48.930 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:26:48.930 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:26:48.930 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:26:48.930 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:26:48.931 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:26:48.931 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:26:48.931 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:26:48.931 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:26:48Z" 2024-08-18T17:26:48.931 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:26:48.932 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000052014 target=00000000000000000000:00000000000000051874, shard=2 master=00000000000000000000:00000000000000059697 target=00000000000000000000:00000000000000059557 2024-08-18T17:26:53.933 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:26:53.933 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:26:53.934 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:26:53.990 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:26:53.990 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:26:54.040 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:26:54.040 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:26:54.040 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:26:54.041 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:26:54.041 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:26:54.041 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:26:54.041 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:26:54.041 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:26:54.041 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:26:54.041 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:26:54.041 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:26:54.041 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:26:54.041 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000051874", 2024-08-18T17:26:54.041 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:26:54.041 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:26:54.041 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:26:54.042 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:23:46.463578Z" 2024-08-18T17:26:54.042 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:26:54.042 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:26:54.042 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:26:54.042 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:26:54.042 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:26:54.042 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:26:54.042 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:26:54.042 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:26:54.042 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:26:54.043 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:26:54.043 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:26:54.043 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:26:54.043 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:26:54.043 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:26:54.043 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:26:54.043 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:26:54.043 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:26:54.043 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000059557", 2024-08-18T17:26:54.043 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:26:54.043 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:26:54.043 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:26:54.043 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:23:46.436203Z" 2024-08-18T17:26:54.044 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:26:54.044 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:26:54.044 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:26:54.044 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:26:54.044 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:26:54.044 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:26:54.044 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:26:54.044 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:26:54.044 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:26:54.044 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:26:54.044 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:26:54.044 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:26:54.044 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:26:54.044 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:26:54.045 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:26:54.045 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:26:54.045 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:26:54.045 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:26:54.045 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:26:54.045 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:26:54Z" 2024-08-18T17:26:54.045 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:26:54.046 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000052014 target=00000000000000000000:00000000000000051874, shard=2 master=00000000000000000000:00000000000000059697 target=00000000000000000000:00000000000000059557 2024-08-18T17:26:59.047 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:26:59.047 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:26:59.047 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:26:59.104 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:26:59.104 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:26:59.154 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:26:59.155 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:26:59.155 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:26:59.155 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:26:59.155 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:26:59.155 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:26:59.155 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:26:59.155 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:26:59.155 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:26:59.155 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:26:59.155 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:26:59.155 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:26:59.155 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000051874", 2024-08-18T17:26:59.155 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:26:59.156 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:26:59.156 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:26:59.156 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:23:46.463578Z" 2024-08-18T17:26:59.156 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:26:59.156 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:26:59.156 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:26:59.156 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:26:59.156 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:26:59.156 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:26:59.156 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:26:59.156 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:26:59.156 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:26:59.156 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:26:59.157 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:26:59.157 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:26:59.157 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:26:59.157 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:26:59.157 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:26:59.157 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:26:59.157 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:26:59.157 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000059557", 2024-08-18T17:26:59.157 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:26:59.157 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:26:59.157 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:26:59.157 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:23:46.436203Z" 2024-08-18T17:26:59.157 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:26:59.157 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:26:59.158 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:26:59.158 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:26:59.158 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:26:59.158 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:26:59.158 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:26:59.158 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:26:59.158 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:26:59.158 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:26:59.158 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:26:59.158 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:26:59.158 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:26:59.158 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:26:59.158 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:26:59.159 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:26:59.159 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:26:59.159 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:26:59.159 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:26:59.159 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:26:59Z" 2024-08-18T17:26:59.159 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:26:59.160 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000052014 target=00000000000000000000:00000000000000051874, shard=2 master=00000000000000000000:00000000000000059697 target=00000000000000000000:00000000000000059557 2024-08-18T17:27:04.161 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:27:04.161 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:27:04.161 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:27:04.219 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:27:04.219 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:27:04.269 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:27:04.270 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:27:04.270 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:27:04.270 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:27:04.270 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:27:04.270 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:27:04.270 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:04.270 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:27:04.270 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:27:04.270 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:27:04.270 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:27:04.271 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:27:04.271 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000051874", 2024-08-18T17:27:04.271 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:27:04.271 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:27:04.271 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:27:04.271 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:23:46.463578Z" 2024-08-18T17:27:04.271 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:27:04.271 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:04.271 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:27:04.271 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:27:04.271 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:27:04.271 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:27:04.272 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:27:04.272 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:27:04.272 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:27:04.272 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:27:04.272 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:27:04.272 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:27:04.272 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:04.272 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:27:04.272 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:27:04.272 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:27:04.272 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:27:04.272 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000059557", 2024-08-18T17:27:04.272 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:27:04.273 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:27:04.273 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:27:04.273 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:23:46.436203Z" 2024-08-18T17:27:04.273 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:27:04.273 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:04.273 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:27:04.273 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:27:04.273 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:27:04.273 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:27:04.273 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:27:04.273 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:27:04.273 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:27:04.273 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:27:04.274 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:27:04.274 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:27:04.274 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:27:04.274 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:27:04.274 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:04.274 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:27:04.274 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:27:04.274 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:27:04.274 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:04.274 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:27:04Z" 2024-08-18T17:27:04.274 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:27:04.275 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000052014 target=00000000000000000000:00000000000000051874, shard=2 master=00000000000000000000:00000000000000059697 target=00000000000000000000:00000000000000059557 2024-08-18T17:27:09.276 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:27:09.276 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:27:09.277 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:27:09.333 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:27:09.333 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:27:09.384 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:27:09.384 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:27:09.384 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:27:09.384 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:27:09.384 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:27:09.385 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:27:09.385 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:09.385 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:27:09.385 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:27:09.385 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:27:09.385 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:27:09.385 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:27:09.385 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000051874", 2024-08-18T17:27:09.385 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:27:09.385 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:27:09.385 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:27:09.385 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:23:46.463578Z" 2024-08-18T17:27:09.385 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:27:09.386 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:09.386 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:27:09.386 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:27:09.386 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:27:09.386 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:27:09.386 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:27:09.386 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:27:09.386 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:27:09.386 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:27:09.386 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:27:09.386 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:27:09.386 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:09.386 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:27:09.387 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:27:09.387 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:27:09.387 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:27:09.387 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000059557", 2024-08-18T17:27:09.387 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:27:09.387 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:27:09.387 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:27:09.387 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:23:46.436203Z" 2024-08-18T17:27:09.387 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:27:09.387 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:09.387 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:27:09.387 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:27:09.388 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:27:09.388 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:27:09.388 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:27:09.388 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:27:09.388 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:27:09.388 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:27:09.388 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:27:09.388 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:27:09.388 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:27:09.388 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:27:09.388 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:09.388 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:27:09.388 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:27:09.388 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:27:09.389 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:09.389 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:27:09Z" 2024-08-18T17:27:09.389 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:27:09.390 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000052014 target=00000000000000000000:00000000000000051874, shard=2 master=00000000000000000000:00000000000000059697 target=00000000000000000000:00000000000000059557 2024-08-18T17:27:14.391 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:27:14.391 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:27:14.392 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:27:14.449 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:27:14.449 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:27:14.499 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:27:14.499 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:27:14.499 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:27:14.499 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:27:14.500 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:27:14.500 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:27:14.500 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:14.500 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:27:14.500 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:27:14.500 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:27:14.500 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:27:14.500 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:27:14.500 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000051874", 2024-08-18T17:27:14.500 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:27:14.500 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:27:14.500 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:27:14.501 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:23:46.463578Z" 2024-08-18T17:27:14.501 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:27:14.501 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:14.501 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:27:14.501 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:27:14.501 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:27:14.501 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:27:14.501 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:27:14.501 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:27:14.501 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:27:14.501 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:27:14.501 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:27:14.501 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:27:14.501 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:14.502 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:27:14.502 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:27:14.502 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:27:14.502 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:27:14.502 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000059557", 2024-08-18T17:27:14.502 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:27:14.502 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:27:14.502 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:27:14.502 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:23:46.436203Z" 2024-08-18T17:27:14.502 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:27:14.502 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:14.502 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:27:14.502 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:27:14.503 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:27:14.503 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:27:14.503 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:27:14.503 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:27:14.503 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:27:14.503 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:27:14.503 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:27:14.503 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:27:14.503 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:27:14.503 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:27:14.503 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:14.503 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:27:14.503 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:27:14.503 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:27:14.504 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:14.504 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:27:14Z" 2024-08-18T17:27:14.504 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:27:14.505 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000052014 target=00000000000000000000:00000000000000051874, shard=2 master=00000000000000000000:00000000000000059697 target=00000000000000000000:00000000000000059557 2024-08-18T17:27:19.506 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:27:19.506 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:27:19.507 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:27:19.564 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:27:19.565 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:27:19.614 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:27:19.615 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:27:19.615 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:27:19.615 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:27:19.615 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:27:19.615 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:27:19.615 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:19.615 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:27:19.615 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:27:19.615 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:27:19.616 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:27:19.616 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:27:19.616 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000051874", 2024-08-18T17:27:19.616 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:27:19.616 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:27:19.616 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:27:19.616 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:23:46.463578Z" 2024-08-18T17:27:19.616 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:27:19.616 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:19.616 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:27:19.616 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:27:19.616 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:27:19.616 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:27:19.617 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:27:19.617 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:27:19.617 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:27:19.617 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:27:19.617 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:27:19.617 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:27:19.617 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:19.617 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:27:19.617 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:27:19.617 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:27:19.617 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:27:19.617 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000059557", 2024-08-18T17:27:19.617 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:27:19.618 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:27:19.618 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:27:19.618 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:23:46.436203Z" 2024-08-18T17:27:19.618 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:27:19.618 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:19.618 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:27:19.618 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:27:19.618 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:27:19.618 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:27:19.618 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:27:19.618 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:27:19.618 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:27:19.619 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:27:19.619 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:27:19.619 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:27:19.619 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:27:19.619 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:27:19.619 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:19.619 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:27:19.619 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:27:19.619 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:27:19.619 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:19.619 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:27:19Z" 2024-08-18T17:27:19.619 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:27:19.620 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000052014 target=00000000000000000000:00000000000000051874, shard=2 master=00000000000000000000:00000000000000059697 target=00000000000000000000:00000000000000059557 2024-08-18T17:27:24.621 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:27:24.622 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:27:24.622 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:27:24.679 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:27:24.679 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:27:24.731 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:27:24.731 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:27:24.731 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:27:24.731 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:27:24.731 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:27:24.731 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:27:24.731 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:24.731 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:27:24.732 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:27:24.732 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:27:24.732 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:27:24.732 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:27:24.732 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052014", 2024-08-18T17:27:24.732 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:27:24.732 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:27:24.732 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:27:24.732 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:25:57.178435Z" 2024-08-18T17:27:24.732 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:27:24.732 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:24.732 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:27:24.732 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:27:24.733 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:27:24.733 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:27:24.733 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:27:24.733 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:27:24.733 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:27:24.733 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:27:24.733 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:27:24.733 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:27:24.733 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:24.733 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:27:24.733 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:27:24.733 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:27:24.733 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:27:24.734 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000059697", 2024-08-18T17:27:24.734 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:27:24.734 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:27:24.734 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:27:24.734 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:25:57.169082Z" 2024-08-18T17:27:24.734 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:27:24.734 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:24.734 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:27:24.734 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:27:24.734 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:27:24.734 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:27:24.735 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:27:24.735 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:27:24.735 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:27:24.735 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:27:24.735 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:27:24.735 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:27:24.735 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:27:24.735 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:27:24.735 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:24.735 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:27:24.735 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:27:24.735 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:27:24.735 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:24.735 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:27:24Z" 2024-08-18T17:27:24.736 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:27:24.736 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-08-18T17:27:24.751 INFO:tasks.rgw_multisite_tests:test_sync_different_buckets: ... FAIL 2024-08-18T17:27:24.751 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:27:24.752 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:27:24.752 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:27:24.807 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:27:24.807 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:27:24.845 INFO:teuthology.orchestra.run.smithi134.stderr:No --period given, using current period=01050e30-179b-43be-bdbb-3bb23cf37ae8 2024-08-18T17:27:24.865 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T17:27:24.865 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:27:24.865 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:27:24.865 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T17:27:24.865 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:27:24.865 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:27:24.865 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:27:24.866 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T17:27:24.866 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:27:24.866 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:27:24.866 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:27:24.866 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:42:47.672461Z" 2024-08-18T17:27:24.866 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:27:24.866 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:27:24.867 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1724001920.785288_1647.1", 2024-08-18T17:27:24.867 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T17:25:20.785288Z" 2024-08-18T17:27:24.867 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:27:24.867 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T17:27:24.868 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-08-18T17:27:24.868 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:27:24.868 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:27:24.868 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:27:24.917 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:27:24.918 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:27:24.947 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:27:24.946+0000 7f36fc126880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:27:24.948 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:27:24.947+0000 7f36fc126880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:27:24.961 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:27:24.961 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:27:24.961 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:27:24.961 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:27:24.962 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:27:24.962 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:27:24.962 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:27:24.962 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:24.962 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:27:24.962 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:27:24.962 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:27:24.962 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:27:24.962 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:27:24.962 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:27:24.962 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:27:24.962 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:27:24.963 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:27:24.963 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:27:24.963 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:27:24.963 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:27:24.963 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:24.963 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:27:24.963 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:27:24.963 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:27:24.963 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:27:24.963 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:27:24.963 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:27:24.963 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:27:24.963 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:27:24.963 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:27:24.964 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:27:24.964 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:27:24.964 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:24.964 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:27:24.964 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:27:24.964 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:27:24.964 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:27:24.964 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:27:24.964 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:27:24.964 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:27:24.964 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:27:24.964 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:27:24.964 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:27:24.964 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:27:24.965 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:24.965 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:27:24.965 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:27:24.965 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:27:24.965 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:27:24.965 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724001920.785288_1647.1", 2024-08-18T17:27:24.965 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:27:24.965 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:27:24.965 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:27:24.965 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:25:20.785288Z", 2024-08-18T17:27:24.965 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:27:24.965 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:27:24.965 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:27:24.966 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:27:24.966 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:24.966 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:27:24.966 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:27:24.966 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:27:24.966 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:24.966 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:27:24Z" 2024-08-18T17:27:24.966 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:27:24.967 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-08-18T17:27:24.968 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-08-18T17:27:24.968 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:27:24.968 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:27:24.968 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:27:25.017 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:27:25.017 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:27:25.070 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:27:25.069+0000 7f4036812880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:27:25.071 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:27:25.070+0000 7f4036812880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:27:25.093 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T17:27:25.093 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T17:27:25.093 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T17:27:25.093 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T17:27:25.093 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T17:27:25.093 INFO:teuthology.orchestra.run.smithi134.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:27:25.093 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:27:25.093 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:27:25.093 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T17:27:25.093 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:27:25.093 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T17:27:25.094 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:27:25.094 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:27:25.094 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:27:25.094 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:27:25.094 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 1, 2024-08-18T17:27:25.094 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:27:25.094 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:27:25.094 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T17:27:25.094 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:27:25.094 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:27:25.094 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:27:25.094 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T17:27:25.095 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:27:25.095 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:27:25.095 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:27:25.095 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:27:25.095 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:27:25.095 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:27:25.095 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T17:27:25.095 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T17:27:25.095 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:27:25.095 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:27:25.095 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:27:25.095 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T17:27:25.095 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:27:25.096 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:27:25.096 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:27:25.096 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:27:25.096 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:27:25.096 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:27:25.096 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:27:25.096 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:27:25.096 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:27:25.096 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:27:25.096 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:27:25.097 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T17:27:25.097 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:27:25.097 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:27:25.097 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1724001920.785288_1647.1", 2024-08-18T17:27:25.097 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:27:25.097 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:27:25.097 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:27:25.097 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T17:25:20.785288Z", 2024-08-18T17:27:25.097 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:27:25.097 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:27:25.097 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:27:25.097 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:27:25.097 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:27:25.098 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T17:27:25.098 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 1, 2024-08-18T17:27:25.098 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 1 2024-08-18T17:27:25.098 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:27:25.098 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T17:27:25Z" 2024-08-18T17:27:25.098 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T17:27:25.099 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-08-18T17:27:25.100 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:27:25.100 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:27:25.100 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:27:25.144 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:27:25.144 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:27:25.221 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-08-18T17:27:25.222 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=kibgmb-95 2024-08-18T17:27:25.248 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=kibgmb-96 2024-08-18T17:27:25.279 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:27:25.280 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:27:25.280 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:27:25.329 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:27:25.329 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:27:25.375 INFO:teuthology.orchestra.run.smithi134.stderr:No --period given, using current period=01050e30-179b-43be-bdbb-3bb23cf37ae8 2024-08-18T17:27:25.400 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T17:27:25.400 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:27:25.400 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:27:25.400 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T17:27:25.401 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:27:25.401 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:27:25.401 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:27:25.401 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T17:27:25.401 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:27:25.402 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:27:25.402 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:27:25.402 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:42:47.672461Z" 2024-08-18T17:27:25.402 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:27:25.402 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:27:25.402 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1724002045.275297_1660.1", 2024-08-18T17:27:25.403 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T17:27:25.275297Z" 2024-08-18T17:27:25.403 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:27:25.403 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T17:27:25.405 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-08-18T17:27:25.405 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:27:25.405 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:27:25.406 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:27:25.454 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:27:25.455 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:27:25.484 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:27:25.484+0000 7f799a946880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:27:25.485 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:27:25.485+0000 7f799a946880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:27:25.499 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:27:25.499 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:27:25.499 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:27:25.499 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:27:25.499 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:27:25.500 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:27:25.500 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:27:25.500 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:25.500 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:27:25.500 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:27:25.500 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:27:25.500 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:27:25.500 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:27:25.500 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:27:25.500 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:27:25.500 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:27:25.500 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:27:25.501 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:27:25.501 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:27:25.501 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:27:25.501 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:25.501 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:27:25.501 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:27:25.501 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:27:25.501 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:27:25.501 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:27:25.501 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:27:25.501 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:27:25.502 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:27:25.502 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:27:25.502 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:27:25.502 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:27:25.502 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:25.502 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:27:25.502 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:27:25.502 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:27:25.502 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:27:25.502 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:27:25.503 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:27:25.503 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:27:25.503 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:27:25.503 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:27:25.503 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:27:25.504 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:27:25.504 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:25.504 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:27:25.504 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:27:25.504 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:27:25.504 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:27:25.504 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724002045.233556_1653.1", 2024-08-18T17:27:25.504 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:27:25.505 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:27:25.505 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:27:25.505 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:27:25.233556Z", 2024-08-18T17:27:25.505 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:27:25.505 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:27:25.505 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:27:25.505 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:27:25.505 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:25.505 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:27:25.505 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:27:25.505 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:27:25.506 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:25.506 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:27:25Z" 2024-08-18T17:27:25.506 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:27:25.507 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724002045.275297_1660.1 target=1_1724002045.233556_1653.1 2024-08-18T17:27:30.508 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:27:30.508 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:27:30.508 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:27:30.564 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:27:30.564 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:27:30.598 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:27:30.597+0000 7ff74ad2a880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:27:30.599 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:27:30.598+0000 7ff74ad2a880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:27:30.613 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:27:30.614 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:27:30.614 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:27:30.614 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:27:30.614 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:27:30.614 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:27:30.614 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:27:30.614 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:30.614 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:27:30.614 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:27:30.614 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:27:30.614 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:27:30.614 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:27:30.615 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:27:30.615 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:27:30.615 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:27:30.615 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:27:30.615 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:27:30.615 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:27:30.615 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:27:30.615 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:30.615 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:27:30.615 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:27:30.615 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:27:30.615 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:27:30.616 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:27:30.616 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:27:30.616 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:27:30.616 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:27:30.616 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:27:30.616 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:27:30.616 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:27:30.616 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:30.616 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:27:30.616 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:27:30.616 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:27:30.616 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:27:30.616 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:27:30.617 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:27:30.617 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:27:30.617 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:27:30.617 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:27:30.617 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:27:30.617 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:27:30.617 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:30.617 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:27:30.617 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:27:30.617 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:27:30.617 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:27:30.617 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724002045.233556_1653.1", 2024-08-18T17:27:30.617 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:27:30.618 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:27:30.618 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:27:30.618 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:27:25.233556Z", 2024-08-18T17:27:30.618 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:27:30.618 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:27:30.618 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:27:30.618 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:27:30.618 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:30.618 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:27:30.618 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:27:30.618 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:27:30.618 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:30.618 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:27:30Z" 2024-08-18T17:27:30.619 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:27:30.619 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724002045.275297_1660.1 target=1_1724002045.233556_1653.1 2024-08-18T17:27:35.620 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:27:35.621 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:27:35.621 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:27:35.676 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:27:35.677 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:27:35.713 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:27:35.711+0000 7fe045015880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:27:35.713 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:27:35.712+0000 7fe045015880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:27:35.728 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:27:35.728 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:27:35.728 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:27:35.728 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:27:35.729 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:27:35.729 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:27:35.729 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:27:35.729 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:35.729 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:27:35.729 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:27:35.730 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:27:35.730 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:27:35.730 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:27:35.730 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:27:35.730 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:27:35.731 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:27:35.731 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:27:35.731 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:27:35.731 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:27:35.731 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:27:35.731 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:35.732 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:27:35.732 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:27:35.732 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:27:35.732 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:27:35.732 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:27:35.732 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:27:35.733 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:27:35.733 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:27:35.733 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:27:35.733 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:27:35.733 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:27:35.734 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:35.734 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:27:35.734 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:27:35.734 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:27:35.734 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:27:35.735 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:27:35.735 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:27:35.735 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:27:35.735 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:27:35.735 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:27:35.736 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:27:35.736 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:27:35.736 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:35.736 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:27:35.736 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:27:35.737 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:27:35.737 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:27:35.737 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724002045.275297_1660.1", 2024-08-18T17:27:35.737 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:27:35.737 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:27:35.737 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:27:35.738 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:27:25.275297Z", 2024-08-18T17:27:35.738 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:27:35.738 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:27:35.738 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:27:35.738 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:27:35.739 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:35.739 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:27:35.739 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:27:35.739 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:27:35.739 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:35.740 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:27:35Z" 2024-08-18T17:27:35.740 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:27:35.741 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-08-18T17:27:35.742 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-08-18T17:27:35.742 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:27:35.742 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:27:35.743 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:27:35.800 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:27:35.801 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:27:35.852 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:27:35.851+0000 7f888e530880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:27:35.853 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:27:35.853+0000 7f888e530880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:27:35.877 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T17:27:35.877 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T17:27:35.877 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T17:27:35.877 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T17:27:35.877 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T17:27:35.878 INFO:teuthology.orchestra.run.smithi134.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:27:35.878 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:27:35.878 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:27:35.878 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T17:27:35.878 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:27:35.878 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T17:27:35.878 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:27:35.878 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:27:35.878 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:27:35.878 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:27:35.878 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 1, 2024-08-18T17:27:35.878 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:27:35.879 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:27:35.879 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T17:27:35.879 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:27:35.879 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:27:35.879 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:27:35.879 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T17:27:35.879 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:27:35.879 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:27:35.879 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:27:35.879 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:27:35.879 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:27:35.879 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:27:35.880 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T17:27:35.880 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T17:27:35.880 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:27:35.880 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:27:35.880 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:27:35.880 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T17:27:35.880 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:27:35.880 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:27:35.880 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:27:35.880 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:27:35.880 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:27:35.880 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:27:35.881 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:27:35.881 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:27:35.881 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:27:35.881 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:27:35.881 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:27:35.881 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T17:27:35.881 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:27:35.881 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:27:35.881 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1724002045.275297_1660.1", 2024-08-18T17:27:35.881 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:27:35.881 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:27:35.881 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:27:35.881 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T17:27:25.275297Z", 2024-08-18T17:27:35.882 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:27:35.882 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:27:35.882 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:27:35.882 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:27:35.882 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:27:35.882 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T17:27:35.882 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 1, 2024-08-18T17:27:35.882 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 1 2024-08-18T17:27:35.882 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:27:35.882 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T17:27:35Z" 2024-08-18T17:27:35.882 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T17:27:35.883 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-08-18T17:27:35.883 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:27:35.883 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:27:35.884 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:27:35.931 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:27:35.932 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:27:35.983 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-08-18T17:27:35.984 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'directional', '--source-zone=test-zone1', '--dest-zone=test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:27:35.984 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'directional', '--source-zone=test-zone1', '--dest-zone=test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:27:35.984 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow --flow-type directional --source-zone=test-zone1 --dest-zone=test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:27:36.035 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:27:36.035 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:27:36.108 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-08-18T17:27:36.108 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-kibgmb-95', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--source-bucket=kibgmb-95', '--dest-bucket=kibgmb-96', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:27:36.108 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-kibgmb-95', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--source-bucket=kibgmb-95', '--dest-bucket=kibgmb-96', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:27:36.109 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe-kibgmb-95 --source-zones=test-zone1 --dest-zones=test-zone2 --source-bucket=kibgmb-95 --dest-bucket=kibgmb-96 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:27:36.160 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:27:36.160 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:27:36.240 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-kibgmb-95","source":{"bucket":"kibgmb-95","zones":["test-zone1"]},"dest":{"bucket":"kibgmb-96","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-08-18T17:27:36.240 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-kibgmb-96', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--source-bucket=kibgmb-96', '--dest-bucket=kibgmb-96', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:27:36.240 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-kibgmb-96', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--source-bucket=kibgmb-96', '--dest-bucket=kibgmb-96', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:27:36.241 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe-kibgmb-96 --source-zones=test-zone1 --dest-zones=test-zone2 --source-bucket=kibgmb-96 --dest-bucket=kibgmb-96 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:27:36.292 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:27:36.292 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:27:36.364 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-kibgmb-95","source":{"bucket":"kibgmb-95","zones":["test-zone1"]},"dest":{"bucket":"kibgmb-96","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-kibgmb-96","source":{"bucket":"kibgmb-96","zones":["test-zone1"]},"dest":{"bucket":"kibgmb-96","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-08-18T17:27:36.365 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:27:36.365 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:27:36.365 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:27:36.412 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:27:36.412 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:27:36.465 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-kibgmb-95","source":{"bucket":"kibgmb-95","zones":["test-zone1"]},"dest":{"bucket":"kibgmb-96","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-kibgmb-96","source":{"bucket":"kibgmb-96","zones":["test-zone1"]},"dest":{"bucket":"kibgmb-96","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-08-18T17:27:36.465 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:27:36.465 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:27:36.466 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:27:36.511 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:27:36.512 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:27:36.583 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T17:27:36.583 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:27:36.583 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 19, 2024-08-18T17:27:36.583 INFO:teuthology.orchestra.run.smithi134.stdout: "predecessor_uuid": "64309209-7567-46e5-8881-65783d84cf3c", 2024-08-18T17:27:36.583 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": [ 2024-08-18T17:27:36.583 INFO:teuthology.orchestra.run.smithi134.stdout: "", 2024-08-18T17:27:36.583 INFO:teuthology.orchestra.run.smithi134.stdout: "", 2024-08-18T17:27:36.583 INFO:teuthology.orchestra.run.smithi134.stdout: "", 2024-08-18T17:27:36.583 INFO:teuthology.orchestra.run.smithi134.stdout: "1_1723995848.930494_187.1" 2024-08-18T17:27:36.584 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:27:36.584 INFO:teuthology.orchestra.run.smithi134.stdout: "period_map": { 2024-08-18T17:27:36.584 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:27:36.584 INFO:teuthology.orchestra.run.smithi134.stdout: "zonegroups": [ 2024-08-18T17:27:36.584 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:27:36.584 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "test-zonegroup", 2024-08-18T17:27:36.584 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "test-zonegroup", 2024-08-18T17:27:36.584 INFO:teuthology.orchestra.run.smithi134.stdout: "api_name": "", 2024-08-18T17:27:36.584 INFO:teuthology.orchestra.run.smithi134.stdout: "is_master": true, 2024-08-18T17:27:36.584 INFO:teuthology.orchestra.run.smithi134.stdout: "endpoints": [ 2024-08-18T17:27:36.584 INFO:teuthology.orchestra.run.smithi134.stdout: "http://smithi134.front.sepia.ceph.com:8000" 2024-08-18T17:27:36.584 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:27:36.584 INFO:teuthology.orchestra.run.smithi134.stdout: "hostnames": [], 2024-08-18T17:27:36.585 INFO:teuthology.orchestra.run.smithi134.stdout: "hostnames_s3website": [], 2024-08-18T17:27:36.585 INFO:teuthology.orchestra.run.smithi134.stdout: "master_zone": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T17:27:36.585 INFO:teuthology.orchestra.run.smithi134.stdout: "zones": [ 2024-08-18T17:27:36.585 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:27:36.585 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T17:27:36.585 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "test-zone3", 2024-08-18T17:27:36.585 INFO:teuthology.orchestra.run.smithi134.stdout: "endpoints": [ 2024-08-18T17:27:36.585 INFO:teuthology.orchestra.run.smithi134.stdout: "http://smithi134.front.sepia.ceph.com:8001" 2024-08-18T17:27:36.585 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:27:36.585 INFO:teuthology.orchestra.run.smithi134.stdout: "log_meta": false, 2024-08-18T17:27:36.585 INFO:teuthology.orchestra.run.smithi134.stdout: "log_data": true, 2024-08-18T17:27:36.585 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_index_max_shards": 11, 2024-08-18T17:27:36.585 INFO:teuthology.orchestra.run.smithi134.stdout: "read_only": false, 2024-08-18T17:27:36.586 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_type": "", 2024-08-18T17:27:36.586 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from_all": true, 2024-08-18T17:27:36.586 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from": [], 2024-08-18T17:27:36.586 INFO:teuthology.orchestra.run.smithi134.stdout: "redirect_zone": "", 2024-08-18T17:27:36.586 INFO:teuthology.orchestra.run.smithi134.stdout: "supported_features": [ 2024-08-18T17:27:36.586 INFO:teuthology.orchestra.run.smithi134.stdout: "compress-encrypted", 2024-08-18T17:27:36.586 INFO:teuthology.orchestra.run.smithi134.stdout: "resharding" 2024-08-18T17:27:36.586 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:27:36.586 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:27:36.586 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:27:36.586 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "10090ccf-aad6-416d-9775-df8b7b27345a", 2024-08-18T17:27:36.586 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "test-zone2", 2024-08-18T17:27:36.586 INFO:teuthology.orchestra.run.smithi134.stdout: "endpoints": [ 2024-08-18T17:27:36.586 INFO:teuthology.orchestra.run.smithi134.stdout: "http://smithi193.front.sepia.ceph.com:8000" 2024-08-18T17:27:36.587 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:27:36.587 INFO:teuthology.orchestra.run.smithi134.stdout: "log_meta": false, 2024-08-18T17:27:36.587 INFO:teuthology.orchestra.run.smithi134.stdout: "log_data": true, 2024-08-18T17:27:36.587 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_index_max_shards": 11, 2024-08-18T17:27:36.587 INFO:teuthology.orchestra.run.smithi134.stdout: "read_only": false, 2024-08-18T17:27:36.587 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_type": "", 2024-08-18T17:27:36.587 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from_all": true, 2024-08-18T17:27:36.587 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from": [], 2024-08-18T17:27:36.587 INFO:teuthology.orchestra.run.smithi134.stdout: "redirect_zone": "", 2024-08-18T17:27:36.587 INFO:teuthology.orchestra.run.smithi134.stdout: "supported_features": [ 2024-08-18T17:27:36.587 INFO:teuthology.orchestra.run.smithi134.stdout: "compress-encrypted", 2024-08-18T17:27:36.587 INFO:teuthology.orchestra.run.smithi134.stdout: "resharding" 2024-08-18T17:27:36.588 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:27:36.588 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:27:36.588 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:27:36.588 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T17:27:36.588 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "test-zone1", 2024-08-18T17:27:36.588 INFO:teuthology.orchestra.run.smithi134.stdout: "endpoints": [ 2024-08-18T17:27:36.588 INFO:teuthology.orchestra.run.smithi134.stdout: "http://smithi134.front.sepia.ceph.com:8000" 2024-08-18T17:27:36.588 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:27:36.588 INFO:teuthology.orchestra.run.smithi134.stdout: "log_meta": false, 2024-08-18T17:27:36.588 INFO:teuthology.orchestra.run.smithi134.stdout: "log_data": true, 2024-08-18T17:27:36.588 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_index_max_shards": 11, 2024-08-18T17:27:36.588 INFO:teuthology.orchestra.run.smithi134.stdout: "read_only": false, 2024-08-18T17:27:36.588 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_type": "", 2024-08-18T17:27:36.588 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from_all": true, 2024-08-18T17:27:36.589 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from": [], 2024-08-18T17:27:36.589 INFO:teuthology.orchestra.run.smithi134.stdout: "redirect_zone": "", 2024-08-18T17:27:36.589 INFO:teuthology.orchestra.run.smithi134.stdout: "supported_features": [ 2024-08-18T17:27:36.589 INFO:teuthology.orchestra.run.smithi134.stdout: "compress-encrypted", 2024-08-18T17:27:36.589 INFO:teuthology.orchestra.run.smithi134.stdout: "resharding" 2024-08-18T17:27:36.589 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:27:36.589 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:27:36.589 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:27:36.589 INFO:teuthology.orchestra.run.smithi134.stdout: "placement_targets": [ 2024-08-18T17:27:36.589 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:27:36.589 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "default-placement", 2024-08-18T17:27:36.589 INFO:teuthology.orchestra.run.smithi134.stdout: "tags": [], 2024-08-18T17:27:36.589 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_classes": [ 2024-08-18T17:27:36.589 INFO:teuthology.orchestra.run.smithi134.stdout: "STANDARD" 2024-08-18T17:27:36.590 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:27:36.590 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:27:36.590 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:27:36.590 INFO:teuthology.orchestra.run.smithi134.stdout: "default_placement": "default-placement", 2024-08-18T17:27:36.590 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_id": "2827c6f0-bf0d-4836-944c-9e7fcf22457b", 2024-08-18T17:27:36.590 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_policy": { 2024-08-18T17:27:36.590 INFO:teuthology.orchestra.run.smithi134.stdout: "groups": [ 2024-08-18T17:27:36.590 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:27:36.590 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "sync-group", 2024-08-18T17:27:36.590 INFO:teuthology.orchestra.run.smithi134.stdout: "data_flow": { 2024-08-18T17:27:36.590 INFO:teuthology.orchestra.run.smithi134.stdout: "symmetrical": [ 2024-08-18T17:27:36.590 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:27:36.590 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "sync-flow1", 2024-08-18T17:27:36.591 INFO:teuthology.orchestra.run.smithi134.stdout: "zones": [ 2024-08-18T17:27:36.591 INFO:teuthology.orchestra.run.smithi134.stdout: "10090ccf-aad6-416d-9775-df8b7b27345a", 2024-08-18T17:27:36.591 INFO:teuthology.orchestra.run.smithi134.stdout: "96bf8502-2f39-4d53-8196-3378c28da3b4" 2024-08-18T17:27:36.591 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:27:36.591 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:27:36.591 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:27:36.591 INFO:teuthology.orchestra.run.smithi134.stdout: "directional": [ 2024-08-18T17:27:36.591 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:27:36.591 INFO:teuthology.orchestra.run.smithi134.stdout: "source_zone": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T17:27:36.591 INFO:teuthology.orchestra.run.smithi134.stdout: "dest_zone": "10090ccf-aad6-416d-9775-df8b7b27345a" 2024-08-18T17:27:36.591 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:27:36.591 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:27:36.591 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:27:36.592 INFO:teuthology.orchestra.run.smithi134.stdout: "pipes": [ 2024-08-18T17:27:36.592 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:27:36.592 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "sync-pipe", 2024-08-18T17:27:36.592 INFO:teuthology.orchestra.run.smithi134.stdout: "source": { 2024-08-18T17:27:36.592 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket": "*", 2024-08-18T17:27:36.592 INFO:teuthology.orchestra.run.smithi134.stdout: "zones": [ 2024-08-18T17:27:36.592 INFO:teuthology.orchestra.run.smithi134.stdout: "10090ccf-aad6-416d-9775-df8b7b27345a", 2024-08-18T17:27:36.592 INFO:teuthology.orchestra.run.smithi134.stdout: "96bf8502-2f39-4d53-8196-3378c28da3b4" 2024-08-18T17:27:36.592 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:27:36.592 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:27:36.592 INFO:teuthology.orchestra.run.smithi134.stdout: "dest": { 2024-08-18T17:27:36.592 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket": "*", 2024-08-18T17:27:36.592 INFO:teuthology.orchestra.run.smithi134.stdout: "zones": [ 2024-08-18T17:27:36.592 INFO:teuthology.orchestra.run.smithi134.stdout: "10090ccf-aad6-416d-9775-df8b7b27345a", 2024-08-18T17:27:36.593 INFO:teuthology.orchestra.run.smithi134.stdout: "96bf8502-2f39-4d53-8196-3378c28da3b4" 2024-08-18T17:27:36.593 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:27:36.593 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:27:36.593 INFO:teuthology.orchestra.run.smithi134.stdout: "params": { 2024-08-18T17:27:36.593 INFO:teuthology.orchestra.run.smithi134.stdout: "source": { 2024-08-18T17:27:36.593 INFO:teuthology.orchestra.run.smithi134.stdout: "filter": { 2024-08-18T17:27:36.593 INFO:teuthology.orchestra.run.smithi134.stdout: "tags": [] 2024-08-18T17:27:36.593 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:27:36.593 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:27:36.593 INFO:teuthology.orchestra.run.smithi134.stdout: "dest": {}, 2024-08-18T17:27:36.593 INFO:teuthology.orchestra.run.smithi134.stdout: "priority": 0, 2024-08-18T17:27:36.593 INFO:teuthology.orchestra.run.smithi134.stdout: "mode": "system", 2024-08-18T17:27:36.593 INFO:teuthology.orchestra.run.smithi134.stdout: "user": "" 2024-08-18T17:27:36.593 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:27:36.594 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:27:36.594 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:27:36.594 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "sync-pipe-kibgmb-95", 2024-08-18T17:27:36.594 INFO:teuthology.orchestra.run.smithi134.stdout: "source": { 2024-08-18T17:27:36.594 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket": "kibgmb-95", 2024-08-18T17:27:36.594 INFO:teuthology.orchestra.run.smithi134.stdout: "zones": [ 2024-08-18T17:27:36.594 INFO:teuthology.orchestra.run.smithi134.stdout: "96bf8502-2f39-4d53-8196-3378c28da3b4" 2024-08-18T17:27:36.594 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:27:36.594 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:27:36.594 INFO:teuthology.orchestra.run.smithi134.stdout: "dest": { 2024-08-18T17:27:36.594 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket": "kibgmb-96", 2024-08-18T17:27:36.594 INFO:teuthology.orchestra.run.smithi134.stdout: "zones": [ 2024-08-18T17:27:36.594 INFO:teuthology.orchestra.run.smithi134.stdout: "10090ccf-aad6-416d-9775-df8b7b27345a" 2024-08-18T17:27:36.595 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:27:36.595 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:27:36.595 INFO:teuthology.orchestra.run.smithi134.stdout: "params": { 2024-08-18T17:27:36.595 INFO:teuthology.orchestra.run.smithi134.stdout: "source": { 2024-08-18T17:27:36.595 INFO:teuthology.orchestra.run.smithi134.stdout: "filter": { 2024-08-18T17:27:36.595 INFO:teuthology.orchestra.run.smithi134.stdout: "tags": [] 2024-08-18T17:27:36.595 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:27:36.595 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:27:36.595 INFO:teuthology.orchestra.run.smithi134.stdout: "dest": {}, 2024-08-18T17:27:36.595 INFO:teuthology.orchestra.run.smithi134.stdout: "priority": 0, 2024-08-18T17:27:36.595 INFO:teuthology.orchestra.run.smithi134.stdout: "mode": "system", 2024-08-18T17:27:36.595 INFO:teuthology.orchestra.run.smithi134.stdout: "user": "" 2024-08-18T17:27:36.595 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:27:36.595 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:27:36.596 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:27:36.596 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "sync-pipe-kibgmb-96", 2024-08-18T17:27:36.596 INFO:teuthology.orchestra.run.smithi134.stdout: "source": { 2024-08-18T17:27:36.596 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket": "kibgmb-96", 2024-08-18T17:27:36.596 INFO:teuthology.orchestra.run.smithi134.stdout: "zones": [ 2024-08-18T17:27:36.596 INFO:teuthology.orchestra.run.smithi134.stdout: "96bf8502-2f39-4d53-8196-3378c28da3b4" 2024-08-18T17:27:36.596 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:27:36.596 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:27:36.596 INFO:teuthology.orchestra.run.smithi134.stdout: "dest": { 2024-08-18T17:27:36.596 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket": "kibgmb-96", 2024-08-18T17:27:36.596 INFO:teuthology.orchestra.run.smithi134.stdout: "zones": [ 2024-08-18T17:27:36.610 INFO:teuthology.orchestra.run.smithi134.stdout: "10090ccf-aad6-416d-9775-df8b7b27345a" 2024-08-18T17:27:36.610 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:27:36.610 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:27:36.610 INFO:teuthology.orchestra.run.smithi134.stdout: "params": { 2024-08-18T17:27:36.611 INFO:teuthology.orchestra.run.smithi134.stdout: "source": { 2024-08-18T17:27:36.611 INFO:teuthology.orchestra.run.smithi134.stdout: "filter": { 2024-08-18T17:27:36.611 INFO:teuthology.orchestra.run.smithi134.stdout: "tags": [] 2024-08-18T17:27:36.611 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:27:36.611 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:27:36.611 INFO:teuthology.orchestra.run.smithi134.stdout: "dest": {}, 2024-08-18T17:27:36.611 INFO:teuthology.orchestra.run.smithi134.stdout: "priority": 0, 2024-08-18T17:27:36.611 INFO:teuthology.orchestra.run.smithi134.stdout: "mode": "system", 2024-08-18T17:27:36.612 INFO:teuthology.orchestra.run.smithi134.stdout: "user": "" 2024-08-18T17:27:36.612 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:27:36.612 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:27:36.612 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:27:36.612 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "enabled" 2024-08-18T17:27:36.612 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:27:36.612 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:27:36.612 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:27:36.612 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled_features": [ 2024-08-18T17:27:36.612 INFO:teuthology.orchestra.run.smithi134.stdout: "resharding" 2024-08-18T17:27:36.612 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:27:36.612 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:27:36.613 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:27:36.613 INFO:teuthology.orchestra.run.smithi134.stdout: "short_zone_ids": [ 2024-08-18T17:27:36.613 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:27:36.613 INFO:teuthology.orchestra.run.smithi134.stdout: "key": "018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T17:27:36.613 INFO:teuthology.orchestra.run.smithi134.stdout: "val": 462036791 2024-08-18T17:27:36.613 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:27:36.613 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:27:36.613 INFO:teuthology.orchestra.run.smithi134.stdout: "key": "10090ccf-aad6-416d-9775-df8b7b27345a", 2024-08-18T17:27:36.613 INFO:teuthology.orchestra.run.smithi134.stdout: "val": 874289624 2024-08-18T17:27:36.613 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:27:36.613 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:27:36.613 INFO:teuthology.orchestra.run.smithi134.stdout: "key": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T17:27:36.613 INFO:teuthology.orchestra.run.smithi134.stdout: "val": 1721399886 2024-08-18T17:27:36.614 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:27:36.614 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:27:36.614 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:27:36.614 INFO:teuthology.orchestra.run.smithi134.stdout: "master_zonegroup": "test-zonegroup", 2024-08-18T17:27:36.614 INFO:teuthology.orchestra.run.smithi134.stdout: "master_zone": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T17:27:36.614 INFO:teuthology.orchestra.run.smithi134.stdout: "period_config": { 2024-08-18T17:27:36.614 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_quota": { 2024-08-18T17:27:36.614 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false, 2024-08-18T17:27:36.614 INFO:teuthology.orchestra.run.smithi134.stdout: "check_on_raw": false, 2024-08-18T17:27:36.614 INFO:teuthology.orchestra.run.smithi134.stdout: "max_size": -1, 2024-08-18T17:27:36.614 INFO:teuthology.orchestra.run.smithi134.stdout: "max_size_kb": 0, 2024-08-18T17:27:36.614 INFO:teuthology.orchestra.run.smithi134.stdout: "max_objects": -1 2024-08-18T17:27:36.614 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:27:36.614 INFO:teuthology.orchestra.run.smithi134.stdout: "user_quota": { 2024-08-18T17:27:36.615 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false, 2024-08-18T17:27:36.615 INFO:teuthology.orchestra.run.smithi134.stdout: "check_on_raw": false, 2024-08-18T17:27:36.615 INFO:teuthology.orchestra.run.smithi134.stdout: "max_size": -1, 2024-08-18T17:27:36.615 INFO:teuthology.orchestra.run.smithi134.stdout: "max_size_kb": 0, 2024-08-18T17:27:36.615 INFO:teuthology.orchestra.run.smithi134.stdout: "max_objects": -1 2024-08-18T17:27:36.615 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:27:36.615 INFO:teuthology.orchestra.run.smithi134.stdout: "user_ratelimit": { 2024-08-18T17:27:36.615 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_ops": 0, 2024-08-18T17:27:36.615 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_ops": 0, 2024-08-18T17:27:36.615 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_bytes": 0, 2024-08-18T17:27:36.615 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_bytes": 0, 2024-08-18T17:27:36.615 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false 2024-08-18T17:27:36.615 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:27:36.616 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_ratelimit": { 2024-08-18T17:27:36.616 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_ops": 0, 2024-08-18T17:27:36.616 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_ops": 0, 2024-08-18T17:27:36.616 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_bytes": 0, 2024-08-18T17:27:36.616 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_bytes": 0, 2024-08-18T17:27:36.616 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false 2024-08-18T17:27:36.616 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:27:36.616 INFO:teuthology.orchestra.run.smithi134.stdout: "anonymous_ratelimit": { 2024-08-18T17:27:36.616 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_ops": 0, 2024-08-18T17:27:36.616 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_ops": 0, 2024-08-18T17:27:36.616 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_bytes": 0, 2024-08-18T17:27:36.616 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_bytes": 0, 2024-08-18T17:27:36.616 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false 2024-08-18T17:27:36.617 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:27:36.617 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:27:36.617 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_id": "2827c6f0-bf0d-4836-944c-9e7fcf22457b", 2024-08-18T17:27:36.617 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_name": "test-realm", 2024-08-18T17:27:36.617 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:27:36.617 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T17:27:36.618 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:27:36.618 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:27:36.618 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:27:36.662 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:27:36.663 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:27:36.738 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-kibgmb-95","source":{"bucket":"kibgmb-95","zones":["test-zone1"]},"dest":{"bucket":"kibgmb-96","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-kibgmb-96","source":{"bucket":"kibgmb-96","zones":["test-zone1"]},"dest":{"bucket":"kibgmb-96","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-08-18T17:27:37.075 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:27:37.075 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:27:37.075 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:27:37.137 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:27:37.138 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:27:37.232 INFO:teuthology.orchestra.run.smithi134.stderr:No --period given, using current period=01050e30-179b-43be-bdbb-3bb23cf37ae8 2024-08-18T17:27:37.271 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T17:27:37.271 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:27:37.272 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:27:37.272 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T17:27:37.272 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:27:37.272 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:27:37.272 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:27:37.272 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T17:27:37.272 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:27:37.272 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:27:37.273 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:27:37.273 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:42:47.672461Z" 2024-08-18T17:27:37.273 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:27:37.273 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:27:37.273 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1724002045.275297_1660.1", 2024-08-18T17:27:37.273 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T17:27:25.275297Z" 2024-08-18T17:27:37.273 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:27:37.273 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T17:27:37.274 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-08-18T17:27:37.275 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:27:37.275 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:27:37.275 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:27:37.322 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:27:37.323 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:27:37.353 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:27:37.352+0000 7f2e1ba19880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:27:37.354 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:27:37.353+0000 7f2e1ba19880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:27:37.369 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:27:37.370 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:27:37.370 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:27:37.370 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:27:37.370 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:27:37.370 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:27:37.370 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:27:37.370 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:37.370 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:27:37.371 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:27:37.371 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:27:37.371 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:27:37.371 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:27:37.371 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:27:37.371 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:27:37.371 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:27:37.371 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:27:37.371 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:27:37.371 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:27:37.371 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:27:37.371 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:37.371 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:27:37.372 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:27:37.372 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:27:37.372 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:27:37.372 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:27:37.372 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:27:37.372 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:27:37.372 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:27:37.372 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:27:37.372 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:27:37.372 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:27:37.372 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:37.372 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:27:37.372 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:27:37.373 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:27:37.373 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:27:37.373 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:27:37.373 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:27:37.373 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:27:37.373 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:27:37.373 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:27:37.373 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:27:37.373 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:27:37.373 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:37.373 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:27:37.373 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:27:37.373 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:27:37.374 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:27:37.374 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724002045.275297_1660.1", 2024-08-18T17:27:37.374 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:27:37.374 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:27:37.374 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:27:37.374 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:27:25.275297Z", 2024-08-18T17:27:37.374 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:27:37.374 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:27:37.374 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:27:37.374 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:27:37.374 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:37.375 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:27:37.375 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:27:37.375 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:27:37.375 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:37.375 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:27:37Z" 2024-08-18T17:27:37.375 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:27:37.376 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-08-18T17:27:37.376 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-08-18T17:27:37.376 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:27:37.377 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:27:37.377 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:27:37.443 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:27:37.443 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:27:37.513 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:27:37.512+0000 7f415cc11880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:27:37.514 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:27:37.514+0000 7f415cc11880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:27:37.561 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T17:27:37.562 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T17:27:37.562 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T17:27:37.562 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T17:27:37.562 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T17:27:37.562 INFO:teuthology.orchestra.run.smithi134.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:27:37.562 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:27:37.562 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:27:37.562 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T17:27:37.562 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:27:37.562 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T17:27:37.562 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:27:37.563 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:27:37.563 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:27:37.563 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:27:37.563 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 1, 2024-08-18T17:27:37.563 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:27:37.563 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:27:37.563 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T17:27:37.563 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:27:37.563 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:27:37.563 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:27:37.563 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T17:27:37.563 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:27:37.563 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:27:37.564 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:27:37.564 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:27:37.564 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:27:37.564 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:27:37.564 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T17:27:37.564 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T17:27:37.564 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:27:37.564 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:27:37.564 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:27:37.564 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T17:27:37.564 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:27:37.564 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:27:37.564 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:27:37.564 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:27:37.565 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:27:37.565 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:27:37.565 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:27:37.565 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:27:37.565 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:27:37.565 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:27:37.565 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:27:37.565 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T17:27:37.565 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:27:37.565 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:27:37.565 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1724002045.275297_1660.1", 2024-08-18T17:27:37.565 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:27:37.565 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:27:37.565 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:27:37.566 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T17:27:25.275297Z", 2024-08-18T17:27:37.566 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:27:37.566 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:27:37.566 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:27:37.566 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:27:37.566 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:27:37.566 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T17:27:37.566 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 1, 2024-08-18T17:27:37.566 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 1 2024-08-18T17:27:37.566 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:27:37.566 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T17:27:37Z" 2024-08-18T17:27:37.566 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T17:27:37.567 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-08-18T17:27:37.568 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:27:37.568 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:27:37.568 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:27:37.622 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:27:37.622 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:27:37.733 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T17:27:37.733 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:27:37.733 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000052154", 2024-08-18T17:27:37.733 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T17:27:36.982158Z" 2024-08-18T17:27:37.733 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:27:37.733 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:27:37.733 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000037555", 2024-08-18T17:27:37.733 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T17:27:37.043496Z" 2024-08-18T17:27:37.733 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:27:37.733 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:27:37.734 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000059977", 2024-08-18T17:27:37.734 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T17:27:37.022632Z" 2024-08-18T17:27:37.734 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:27:37.734 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:27:37.734 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:27:37.734 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:27:37.734 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:27:37.734 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T17:27:37.735 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-08-18T17:27:37.735 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:27:37.735 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:27:37.736 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:27:37.787 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:27:37.788 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:27:37.834 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:27:37.834 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:27:37.834 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:27:37.834 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:27:37.834 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:27:37.835 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:27:37.835 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:37.835 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:27:37.835 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:27:37.835 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:27:37.835 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:27:37.835 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:27:37.835 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052014", 2024-08-18T17:27:37.835 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:27:37.835 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:27:37.835 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:27:37.835 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:25:57.178435Z" 2024-08-18T17:27:37.836 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:27:37.836 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:37.836 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:27:37.836 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:27:37.836 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:27:37.836 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:27:37.836 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:27:37.836 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:27:37.836 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:27:37.836 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:27:37.836 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:27:37.836 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:27:37.837 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:37.837 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:27:37.837 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:27:37.837 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:27:37.837 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:27:37.837 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000059697", 2024-08-18T17:27:37.837 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:27:37.837 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:27:37.837 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:27:37.837 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:25:57.169082Z" 2024-08-18T17:27:37.837 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:27:37.837 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:37.837 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:27:37.838 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:27:37.838 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:27:37.838 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:27:37.838 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:27:37.838 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:27:37.838 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:27:37.838 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:27:37.838 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:27:37.838 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:27:37.838 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:27:37.838 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:27:37.838 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:37.838 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:27:37.839 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:27:37.839 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:27:37.839 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:37.839 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:27:37Z" 2024-08-18T17:27:37.839 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:27:37.840 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000052154 target=00000000000000000000:00000000000000052014, shard=1 master=00000000000000000000:00000000000000037555 target=00000000000000000000:00000000000000037415, shard=2 master=00000000000000000000:00000000000000059977 target=00000000000000000000:00000000000000059697 2024-08-18T17:27:42.841 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:27:42.842 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:27:42.842 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:27:42.899 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:27:42.899 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:27:42.949 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:27:42.949 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:27:42.949 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:27:42.949 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:27:42.949 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:27:42.950 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:27:42.950 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:42.950 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:27:42.950 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:27:42.950 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:27:42.950 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:27:42.950 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:27:42.950 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052014", 2024-08-18T17:27:42.950 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:27:42.950 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:27:42.951 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:27:42.951 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:25:57.178435Z" 2024-08-18T17:27:42.951 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:27:42.951 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:42.951 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:27:42.951 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:27:42.951 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:27:42.951 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:27:42.951 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:27:42.951 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:27:42.951 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:27:42.952 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:27:42.952 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:27:42.952 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:27:42.952 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:42.952 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:27:42.952 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:27:42.952 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:27:42.952 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:27:42.952 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000059697", 2024-08-18T17:27:42.952 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:27:42.952 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:27:42.952 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:27:42.953 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:25:57.169082Z" 2024-08-18T17:27:42.953 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:27:42.953 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:42.953 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:27:42.953 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:27:42.953 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:27:42.953 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:27:42.953 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:27:42.953 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:27:42.953 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:27:42.953 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:27:42.953 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:27:42.953 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:27:42.954 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:27:42.954 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:27:42.954 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:42.954 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:27:42.954 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:27:42.954 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:27:42.954 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:42.954 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:27:42Z" 2024-08-18T17:27:42.954 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:27:42.955 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000052154 target=00000000000000000000:00000000000000052014, shard=1 master=00000000000000000000:00000000000000037555 target=00000000000000000000:00000000000000037415, shard=2 master=00000000000000000000:00000000000000059977 target=00000000000000000000:00000000000000059697 2024-08-18T17:27:47.956 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:27:47.956 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:27:47.957 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:27:48.014 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:27:48.014 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:27:48.065 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:27:48.065 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:27:48.065 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:27:48.066 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:27:48.066 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:27:48.066 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:27:48.066 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:48.066 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:27:48.066 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:27:48.067 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:27:48.067 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:27:48.067 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:27:48.067 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052014", 2024-08-18T17:27:48.067 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:27:48.068 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:27:48.068 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:27:48.068 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:25:57.178435Z" 2024-08-18T17:27:48.068 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:27:48.068 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:48.068 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:27:48.069 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:27:48.069 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:27:48.069 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:27:48.069 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:27:48.069 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:27:48.070 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:27:48.070 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:27:48.070 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:27:48.070 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:27:48.071 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:48.071 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:27:48.071 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:27:48.071 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:27:48.071 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:27:48.071 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000059697", 2024-08-18T17:27:48.072 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:27:48.072 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:27:48.072 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:27:48.072 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:25:57.169082Z" 2024-08-18T17:27:48.072 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:27:48.073 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:48.073 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:27:48.073 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:27:48.073 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:27:48.074 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:27:48.074 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:27:48.074 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:27:48.074 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:27:48.074 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:27:48.074 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:27:48.075 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:27:48.075 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:27:48.075 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:27:48.075 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:48.075 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:27:48.075 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:27:48.076 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:27:48.076 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:48.076 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:27:48Z" 2024-08-18T17:27:48.076 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:27:48.078 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000052154 target=00000000000000000000:00000000000000052014, shard=1 master=00000000000000000000:00000000000000037555 target=00000000000000000000:00000000000000037415, shard=2 master=00000000000000000000:00000000000000059977 target=00000000000000000000:00000000000000059697 2024-08-18T17:27:53.079 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:27:53.079 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:27:53.080 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:27:53.136 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:27:53.136 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:27:53.187 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:27:53.187 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:27:53.187 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:27:53.187 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:27:53.187 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:27:53.187 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:27:53.187 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:53.187 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:27:53.187 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:27:53.187 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:27:53.188 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:27:53.188 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:27:53.188 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052014", 2024-08-18T17:27:53.188 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:27:53.188 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:27:53.188 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:27:53.188 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:25:57.178435Z" 2024-08-18T17:27:53.188 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:27:53.188 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:53.188 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:27:53.188 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:27:53.188 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:27:53.189 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:27:53.189 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:27:53.189 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:27:53.189 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:27:53.189 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:27:53.189 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:27:53.189 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:27:53.189 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:53.189 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:27:53.189 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:27:53.189 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:27:53.189 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:27:53.189 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000059697", 2024-08-18T17:27:53.189 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:27:53.190 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:27:53.190 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:27:53.190 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:25:57.169082Z" 2024-08-18T17:27:53.190 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:27:53.190 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:53.190 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:27:53.190 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:27:53.190 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:27:53.190 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:27:53.190 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:27:53.190 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:27:53.190 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:27:53.190 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:27:53.190 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:27:53.191 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:27:53.191 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:27:53.191 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:27:53.191 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:53.191 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:27:53.191 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:27:53.191 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:27:53.191 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:53.191 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:27:53Z" 2024-08-18T17:27:53.191 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:27:53.193 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000052154 target=00000000000000000000:00000000000000052014, shard=1 master=00000000000000000000:00000000000000037555 target=00000000000000000000:00000000000000037415, shard=2 master=00000000000000000000:00000000000000059977 target=00000000000000000000:00000000000000059697 2024-08-18T17:27:58.194 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:27:58.194 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:27:58.195 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:27:58.255 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:27:58.256 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:27:58.303 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:27:58.303 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:27:58.303 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:27:58.303 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:27:58.303 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:27:58.303 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:27:58.304 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:58.304 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:27:58.304 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:27:58.304 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:27:58.304 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:27:58.304 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:27:58.304 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052014", 2024-08-18T17:27:58.304 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:27:58.304 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:27:58.304 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:27:58.304 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:25:57.178435Z" 2024-08-18T17:27:58.304 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:27:58.304 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:58.305 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:27:58.305 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:27:58.305 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:27:58.305 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:27:58.305 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:27:58.305 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:27:58.305 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:27:58.305 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:27:58.305 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:27:58.305 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:27:58.305 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:58.305 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:27:58.305 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:27:58.306 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:27:58.306 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:27:58.306 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000059697", 2024-08-18T17:27:58.306 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:27:58.306 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:27:58.306 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:27:58.306 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:25:57.169082Z" 2024-08-18T17:27:58.306 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:27:58.306 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:58.306 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:27:58.306 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:27:58.306 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:27:58.307 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:27:58.307 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:27:58.307 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:27:58.307 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:27:58.307 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:27:58.307 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:27:58.307 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:27:58.307 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:27:58.307 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:27:58.307 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:58.307 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:27:58.307 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:27:58.307 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:27:58.308 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:27:58.308 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:27:58Z" 2024-08-18T17:27:58.308 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:27:58.309 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000052154 target=00000000000000000000:00000000000000052014, shard=1 master=00000000000000000000:00000000000000037555 target=00000000000000000000:00000000000000037415, shard=2 master=00000000000000000000:00000000000000059977 target=00000000000000000000:00000000000000059697 2024-08-18T17:28:03.310 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:28:03.310 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:28:03.310 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:28:03.367 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:28:03.367 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:28:03.419 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:28:03.419 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:28:03.419 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:28:03.419 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:28:03.419 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:28:03.419 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:28:03.419 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:28:03.419 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:28:03.419 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:28:03.419 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:28:03.420 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:28:03.420 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:28:03.420 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052014", 2024-08-18T17:28:03.420 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:28:03.420 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:28:03.420 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:28:03.420 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:25:57.178435Z" 2024-08-18T17:28:03.420 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:28:03.420 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:28:03.420 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:28:03.420 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:28:03.420 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:28:03.420 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:28:03.421 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:28:03.421 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:28:03.421 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:28:03.421 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:28:03.421 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:28:03.421 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:28:03.421 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:28:03.421 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:28:03.421 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:28:03.421 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:28:03.421 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:28:03.421 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000059697", 2024-08-18T17:28:03.421 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:28:03.422 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:28:03.422 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:28:03.422 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:25:57.169082Z" 2024-08-18T17:28:03.422 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:28:03.422 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:28:03.422 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:28:03.422 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:28:03.422 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:28:03.422 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:28:03.422 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:28:03.422 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:28:03.422 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:28:03.423 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:28:03.423 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:28:03.423 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:28:03.423 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:28:03.423 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:28:03.423 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:28:03.423 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:28:03.423 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:28:03.423 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:28:03.423 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:28:03.423 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:28:03Z" 2024-08-18T17:28:03.423 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:28:03.424 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000052154 target=00000000000000000000:00000000000000052014, shard=1 master=00000000000000000000:00000000000000037555 target=00000000000000000000:00000000000000037415, shard=2 master=00000000000000000000:00000000000000059977 target=00000000000000000000:00000000000000059697 2024-08-18T17:28:08.425 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:28:08.426 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:28:08.426 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:28:08.483 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:28:08.483 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:28:08.534 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:28:08.534 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:28:08.534 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:28:08.534 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:28:08.534 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:28:08.534 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:28:08.535 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:28:08.535 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:28:08.535 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:28:08.535 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:28:08.535 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:28:08.535 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:28:08.535 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052014", 2024-08-18T17:28:08.535 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:28:08.535 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:28:08.535 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:28:08.535 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:25:57.178435Z" 2024-08-18T17:28:08.535 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:28:08.535 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:28:08.536 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:28:08.536 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:28:08.536 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:28:08.536 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:28:08.536 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:28:08.536 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:28:08.536 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:28:08.536 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:28:08.536 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:28:08.536 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:28:08.536 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:28:08.536 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:28:08.536 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:28:08.537 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:28:08.537 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:28:08.537 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000059697", 2024-08-18T17:28:08.537 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:28:08.537 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:28:08.537 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:28:08.537 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:25:57.169082Z" 2024-08-18T17:28:08.537 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:28:08.537 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:28:08.537 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:28:08.537 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:28:08.537 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:28:08.538 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:28:08.538 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:28:08.538 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:28:08.538 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:28:08.538 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:28:08.538 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:28:08.538 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:28:08.538 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:28:08.538 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:28:08.538 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:28:08.538 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:28:08.538 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:28:08.538 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:28:08.539 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:28:08.539 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:28:08Z" 2024-08-18T17:28:08.539 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:28:08.540 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000052154 target=00000000000000000000:00000000000000052014, shard=1 master=00000000000000000000:00000000000000037555 target=00000000000000000000:00000000000000037415, shard=2 master=00000000000000000000:00000000000000059977 target=00000000000000000000:00000000000000059697 2024-08-18T17:28:13.541 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:28:13.541 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:28:13.541 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:28:13.599 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:28:13.599 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:28:13.651 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:28:13.651 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:28:13.651 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:28:13.651 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:28:13.651 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:28:13.651 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:28:13.651 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:28:13.651 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:28:13.651 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:28:13.651 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:28:13.652 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:28:13.652 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:28:13.652 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052014", 2024-08-18T17:28:13.652 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:28:13.652 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:28:13.652 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:28:13.652 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:25:57.178435Z" 2024-08-18T17:28:13.652 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:28:13.652 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:28:13.652 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:28:13.652 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:28:13.652 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:28:13.652 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:28:13.653 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:28:13.653 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:28:13.653 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:28:13.653 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:28:13.653 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:28:13.653 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:28:13.653 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:28:13.653 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:28:13.653 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:28:13.653 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:28:13.653 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:28:13.653 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000059697", 2024-08-18T17:28:13.653 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:28:13.653 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:28:13.654 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:28:13.654 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:25:57.169082Z" 2024-08-18T17:28:13.654 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:28:13.654 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:28:13.654 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:28:13.654 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:28:13.654 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:28:13.654 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:28:13.654 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:28:13.654 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:28:13.654 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:28:13.654 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:28:13.654 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:28:13.655 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:28:13.655 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:28:13.655 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:28:13.655 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:28:13.655 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:28:13.655 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:28:13.655 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:28:13.655 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:28:13.655 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:28:13Z" 2024-08-18T17:28:13.655 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:28:13.656 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000052154 target=00000000000000000000:00000000000000052014, shard=1 master=00000000000000000000:00000000000000037555 target=00000000000000000000:00000000000000037415, shard=2 master=00000000000000000000:00000000000000059977 target=00000000000000000000:00000000000000059697 2024-08-18T17:28:18.657 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:28:18.658 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:28:18.658 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:28:18.715 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:28:18.715 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:28:18.766 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:28:18.766 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:28:18.767 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:28:18.767 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:28:18.767 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:28:18.767 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:28:18.767 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:28:18.767 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:28:18.767 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:28:18.767 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:28:18.767 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:28:18.767 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:28:18.767 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052014", 2024-08-18T17:28:18.767 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:28:18.768 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:28:18.768 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:28:18.768 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:25:57.178435Z" 2024-08-18T17:28:18.768 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:28:18.768 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:28:18.768 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:28:18.768 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:28:18.768 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:28:18.768 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:28:18.768 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:28:18.768 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:28:18.768 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:28:18.768 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:28:18.769 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:28:18.769 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:28:18.769 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:28:18.769 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:28:18.769 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:28:18.769 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:28:18.769 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:28:18.769 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000059697", 2024-08-18T17:28:18.769 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:28:18.769 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:28:18.769 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:28:18.769 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:25:57.169082Z" 2024-08-18T17:28:18.769 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:28:18.770 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:28:18.770 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:28:18.770 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:28:18.770 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:28:18.770 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:28:18.770 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:28:18.770 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:28:18.770 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:28:18.770 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:28:18.770 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:28:18.770 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:28:18.770 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:28:18.771 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:28:18.771 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:28:18.771 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:28:18.771 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:28:18.771 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:28:18.771 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:28:18.771 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:28:18Z" 2024-08-18T17:28:18.771 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:28:18.772 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000052154 target=00000000000000000000:00000000000000052014, shard=1 master=00000000000000000000:00000000000000037555 target=00000000000000000000:00000000000000037415, shard=2 master=00000000000000000000:00000000000000059977 target=00000000000000000000:00000000000000059697 2024-08-18T17:28:23.773 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:28:23.774 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:28:23.774 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:28:23.831 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:28:23.832 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:28:23.882 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:28:23.883 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:28:23.883 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:28:23.883 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:28:23.883 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:28:23.883 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:28:23.883 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:28:23.883 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:28:23.883 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:28:23.883 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:28:23.884 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:28:23.884 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:28:23.884 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052014", 2024-08-18T17:28:23.884 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:28:23.884 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:28:23.884 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:28:23.884 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:25:57.178435Z" 2024-08-18T17:28:23.884 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:28:23.884 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:28:23.884 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:28:23.884 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:28:23.884 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:28:23.885 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:28:23.885 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:28:23.885 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:28:23.885 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:28:23.885 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:28:23.885 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:28:23.885 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:28:23.885 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:28:23.885 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:28:23.885 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:28:23.885 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:28:23.885 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:28:23.885 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000059697", 2024-08-18T17:28:23.886 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:28:23.886 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:28:23.886 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:28:23.886 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:25:57.169082Z" 2024-08-18T17:28:23.886 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:28:23.886 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:28:23.886 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:28:23.886 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:28:23.886 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:28:23.886 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:28:23.886 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:28:23.886 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:28:23.887 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:28:23.887 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:28:23.887 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:28:23.887 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:28:23.887 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:28:23.887 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:28:23.887 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:28:23.887 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:28:23.887 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:28:23.887 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:28:23.887 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:28:23.887 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:28:23Z" 2024-08-18T17:28:23.887 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:28:23.888 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000052154 target=00000000000000000000:00000000000000052014, shard=1 master=00000000000000000000:00000000000000037555 target=00000000000000000000:00000000000000037415, shard=2 master=00000000000000000000:00000000000000059977 target=00000000000000000000:00000000000000059697 2024-08-18T17:28:28.889 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:28:28.889 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:28:28.890 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:28:28.947 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:28:28.947 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:28:28.998 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:28:28.998 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:28:28.998 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:28:28.998 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:28:28.998 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:28:28.998 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:28:28.999 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:28:28.999 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:28:28.999 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:28:28.999 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:28:28.999 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:28:28.999 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:28:28.999 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052014", 2024-08-18T17:28:28.999 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:28:28.999 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:28:29.000 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:28:29.000 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:25:57.178435Z" 2024-08-18T17:28:29.000 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:28:29.000 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:28:29.000 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:28:29.000 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:28:29.000 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:28:29.000 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:28:29.000 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:28:29.000 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:28:29.000 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:28:29.001 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:28:29.001 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:28:29.001 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:28:29.001 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:28:29.001 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:28:29.001 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:28:29.001 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:28:29.001 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:28:29.001 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000059697", 2024-08-18T17:28:29.001 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:28:29.001 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:28:29.002 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:28:29.002 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:25:57.169082Z" 2024-08-18T17:28:29.002 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:28:29.002 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:28:29.002 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:28:29.002 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:28:29.002 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:28:29.002 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:28:29.002 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:28:29.002 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:28:29.003 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:28:29.003 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:28:29.003 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:28:29.003 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:28:29.003 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:28:29.003 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:28:29.003 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:28:29.003 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:28:29.003 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:28:29.003 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:28:29.003 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:28:29.003 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:28:28Z" 2024-08-18T17:28:29.004 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:28:29.005 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000052154 target=00000000000000000000:00000000000000052014, shard=1 master=00000000000000000000:00000000000000037555 target=00000000000000000000:00000000000000037415, shard=2 master=00000000000000000000:00000000000000059977 target=00000000000000000000:00000000000000059697 2024-08-18T17:28:34.007 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:28:34.007 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:28:34.007 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:28:34.066 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:28:34.066 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:28:34.119 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:28:34.119 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:28:34.119 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:28:34.119 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:28:34.119 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:28:34.120 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:28:34.120 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:28:34.120 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:28:34.120 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:28:34.120 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:28:34.120 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:28:34.120 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:28:34.120 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052014", 2024-08-18T17:28:34.120 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:28:34.120 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:28:34.120 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:28:34.120 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:25:57.178435Z" 2024-08-18T17:28:34.121 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:28:34.121 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:28:34.121 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:28:34.121 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:28:34.121 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:28:34.121 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:28:34.121 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:28:34.121 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:28:34.121 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:28:34.121 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:28:34.121 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:28:34.121 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:28:34.121 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:28:34.122 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:28:34.122 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:28:34.122 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:28:34.122 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:28:34.122 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000059697", 2024-08-18T17:28:34.122 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:28:34.122 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:28:34.122 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:28:34.122 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:25:57.169082Z" 2024-08-18T17:28:34.122 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:28:34.122 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:28:34.122 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:28:34.122 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:28:34.123 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:28:34.123 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:28:34.123 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:28:34.123 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:28:34.123 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:28:34.123 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:28:34.123 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:28:34.123 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:28:34.123 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:28:34.123 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:28:34.123 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:28:34.123 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:28:34.123 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:28:34.124 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:28:34.124 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:28:34.124 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:28:34Z" 2024-08-18T17:28:34.124 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:28:34.125 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000052154 target=00000000000000000000:00000000000000052014, shard=1 master=00000000000000000000:00000000000000037555 target=00000000000000000000:00000000000000037415, shard=2 master=00000000000000000000:00000000000000059977 target=00000000000000000000:00000000000000059697 2024-08-18T17:28:39.125 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:28:39.126 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:28:39.126 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:28:39.183 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:28:39.183 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:28:39.235 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:28:39.236 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:28:39.236 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:28:39.236 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:28:39.236 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:28:39.236 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:28:39.236 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:28:39.236 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:28:39.236 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:28:39.236 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:28:39.236 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:28:39.236 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:28:39.237 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052014", 2024-08-18T17:28:39.237 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:28:39.237 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:28:39.237 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:28:39.237 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:25:57.178435Z" 2024-08-18T17:28:39.237 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:28:39.237 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:28:39.237 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:28:39.237 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:28:39.237 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:28:39.237 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:28:39.237 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:28:39.237 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:28:39.237 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:28:39.238 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:28:39.238 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:28:39.238 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:28:39.238 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:28:39.238 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:28:39.238 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:28:39.238 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:28:39.238 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:28:39.238 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000059697", 2024-08-18T17:28:39.238 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:28:39.238 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:28:39.238 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:28:39.239 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:25:57.169082Z" 2024-08-18T17:28:39.239 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:28:39.239 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:28:39.239 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:28:39.239 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:28:39.239 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:28:39.239 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:28:39.239 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:28:39.239 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:28:39.239 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:28:39.239 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:28:39.239 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:28:39.239 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:28:39.239 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:28:39.240 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:28:39.240 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:28:39.240 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:28:39.240 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:28:39.240 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:28:39.240 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:28:39.240 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:28:39Z" 2024-08-18T17:28:39.240 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:28:39.241 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000052154 target=00000000000000000000:00000000000000052014, shard=1 master=00000000000000000000:00000000000000037555 target=00000000000000000000:00000000000000037415, shard=2 master=00000000000000000000:00000000000000059977 target=00000000000000000000:00000000000000059697 2024-08-18T17:28:44.242 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:28:44.243 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:28:44.243 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:28:44.301 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:28:44.301 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:28:44.351 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:28:44.351 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:28:44.352 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:28:44.352 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:28:44.352 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:28:44.352 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:28:44.352 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:28:44.352 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:28:44.352 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:28:44.352 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:28:44.352 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:28:44.352 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:28:44.352 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052014", 2024-08-18T17:28:44.352 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:28:44.353 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:28:44.353 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:28:44.353 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:25:57.178435Z" 2024-08-18T17:28:44.353 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:28:44.353 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:28:44.353 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:28:44.353 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:28:44.353 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:28:44.353 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:28:44.353 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:28:44.353 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:28:44.353 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:28:44.353 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:28:44.354 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:28:44.354 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:28:44.354 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:28:44.354 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:28:44.354 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:28:44.354 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:28:44.354 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:28:44.354 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000059697", 2024-08-18T17:28:44.354 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:28:44.354 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:28:44.354 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:28:44.354 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:25:57.169082Z" 2024-08-18T17:28:44.354 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:28:44.355 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:28:44.355 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:28:44.355 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:28:44.355 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:28:44.355 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:28:44.355 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:28:44.355 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:28:44.355 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:28:44.355 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:28:44.355 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:28:44.355 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:28:44.355 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:28:44.355 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:28:44.356 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:28:44.356 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:28:44.356 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:28:44.356 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:28:44.356 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:28:44.356 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:28:44Z" 2024-08-18T17:28:44.356 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:28:44.357 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000052154 target=00000000000000000000:00000000000000052014, shard=1 master=00000000000000000000:00000000000000037555 target=00000000000000000000:00000000000000037415, shard=2 master=00000000000000000000:00000000000000059977 target=00000000000000000000:00000000000000059697 2024-08-18T17:28:49.358 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:28:49.358 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:28:49.359 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:28:49.417 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:28:49.417 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:28:49.469 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:28:49.469 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:28:49.469 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:28:49.469 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:28:49.469 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:28:49.469 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:28:49.469 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:28:49.469 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:28:49.469 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:28:49.469 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:28:49.470 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:28:49.470 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:28:49.470 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052014", 2024-08-18T17:28:49.470 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:28:49.470 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:28:49.470 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:28:49.470 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:25:57.178435Z" 2024-08-18T17:28:49.470 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:28:49.470 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:28:49.470 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:28:49.470 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:28:49.470 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:28:49.470 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:28:49.470 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:28:49.471 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:28:49.471 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:28:49.471 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:28:49.471 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:28:49.471 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:28:49.471 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:28:49.471 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:28:49.471 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:28:49.471 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:28:49.471 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:28:49.471 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000059697", 2024-08-18T17:28:49.471 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:28:49.471 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:28:49.472 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:28:49.472 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:25:57.169082Z" 2024-08-18T17:28:49.472 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:28:49.472 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:28:49.472 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:28:49.472 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:28:49.472 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:28:49.472 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:28:49.472 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:28:49.472 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:28:49.472 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:28:49.472 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:28:49.472 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:28:49.472 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:28:49.473 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:28:49.473 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:28:49.473 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:28:49.473 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:28:49.473 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:28:49.473 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:28:49.473 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:28:49.473 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:28:49Z" 2024-08-18T17:28:49.473 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:28:49.474 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000052154 target=00000000000000000000:00000000000000052014, shard=1 master=00000000000000000000:00000000000000037555 target=00000000000000000000:00000000000000037415, shard=2 master=00000000000000000000:00000000000000059977 target=00000000000000000000:00000000000000059697 2024-08-18T17:28:54.475 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:28:54.476 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:28:54.476 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:28:54.532 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:28:54.532 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:28:54.584 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:28:54.584 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:28:54.585 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:28:54.585 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:28:54.585 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:28:54.585 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:28:54.585 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:28:54.585 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:28:54.585 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:28:54.585 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:28:54.585 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:28:54.586 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:28:54.586 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052014", 2024-08-18T17:28:54.586 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:28:54.586 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:28:54.586 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:28:54.586 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:25:57.178435Z" 2024-08-18T17:28:54.586 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:28:54.586 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:28:54.586 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:28:54.586 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:28:54.587 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:28:54.587 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:28:54.587 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:28:54.587 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:28:54.587 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:28:54.587 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:28:54.587 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:28:54.588 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:28:54.588 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:28:54.588 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:28:54.588 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:28:54.588 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:28:54.588 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:28:54.589 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000059697", 2024-08-18T17:28:54.589 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:28:54.589 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:28:54.589 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:28:54.589 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:25:57.169082Z" 2024-08-18T17:28:54.589 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:28:54.589 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:28:54.589 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:28:54.589 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:28:54.589 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:28:54.589 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:28:54.589 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:28:54.590 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:28:54.590 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:28:54.590 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:28:54.590 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:28:54.590 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:28:54.590 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:28:54.590 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:28:54.590 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:28:54.590 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:28:54.590 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:28:54.590 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:28:54.590 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:28:54.590 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:28:54Z" 2024-08-18T17:28:54.591 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:28:54.592 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000052154 target=00000000000000000000:00000000000000052014, shard=1 master=00000000000000000000:00000000000000037555 target=00000000000000000000:00000000000000037415, shard=2 master=00000000000000000000:00000000000000059977 target=00000000000000000000:00000000000000059697 2024-08-18T17:28:59.593 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:28:59.593 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:28:59.593 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:28:59.650 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:28:59.651 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:28:59.701 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:28:59.701 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:28:59.701 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:28:59.701 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:28:59.701 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:28:59.701 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:28:59.701 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:28:59.701 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:28:59.702 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:28:59.702 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:28:59.702 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:28:59.702 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:28:59.702 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052014", 2024-08-18T17:28:59.702 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:28:59.702 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:28:59.702 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:28:59.702 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:25:57.178435Z" 2024-08-18T17:28:59.702 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:28:59.702 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:28:59.702 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:28:59.703 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:28:59.703 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:28:59.703 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:28:59.703 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:28:59.703 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:28:59.703 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:28:59.703 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:28:59.703 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:28:59.703 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:28:59.703 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:28:59.703 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:28:59.703 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:28:59.703 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:28:59.704 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:28:59.704 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000059697", 2024-08-18T17:28:59.704 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:28:59.704 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:28:59.704 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:28:59.704 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:25:57.169082Z" 2024-08-18T17:28:59.704 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:28:59.704 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:28:59.704 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:28:59.705 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:28:59.705 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:28:59.705 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:28:59.705 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:28:59.705 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:28:59.705 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:28:59.705 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:28:59.705 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:28:59.705 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:28:59.705 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:28:59.705 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:28:59.706 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:28:59.706 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:28:59.706 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:28:59.706 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:28:59.706 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:28:59.706 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:28:59Z" 2024-08-18T17:28:59.706 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:28:59.707 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000052154 target=00000000000000000000:00000000000000052014, shard=1 master=00000000000000000000:00000000000000037555 target=00000000000000000000:00000000000000037415, shard=2 master=00000000000000000000:00000000000000059977 target=00000000000000000000:00000000000000059697 2024-08-18T17:29:04.709 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:29:04.709 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:29:04.709 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:29:04.765 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:29:04.766 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:29:04.817 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:29:04.818 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:29:04.818 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:29:04.818 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:29:04.818 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:29:04.818 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:29:04.818 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:04.818 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:29:04.818 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:29:04.818 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:29:04.818 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:29:04.818 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:29:04.818 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052014", 2024-08-18T17:29:04.818 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:29:04.819 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:29:04.819 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:29:04.819 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:25:57.178435Z" 2024-08-18T17:29:04.819 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:29:04.819 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:04.819 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:29:04.819 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:29:04.819 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:29:04.819 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:29:04.819 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:29:04.819 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:29:04.819 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:29:04.820 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:29:04.820 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:29:04.820 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:29:04.820 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:04.820 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:29:04.820 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:29:04.820 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:29:04.820 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:29:04.820 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000059697", 2024-08-18T17:29:04.820 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:29:04.820 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:29:04.820 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:29:04.821 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:25:57.169082Z" 2024-08-18T17:29:04.821 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:29:04.821 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:04.821 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:29:04.821 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:29:04.821 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:29:04.821 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:29:04.821 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:29:04.821 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:29:04.821 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:29:04.821 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:29:04.821 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:29:04.821 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:29:04.821 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:29:04.822 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:29:04.822 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:04.822 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:29:04.822 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:29:04.822 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:29:04.822 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:04.822 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:29:04Z" 2024-08-18T17:29:04.822 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:29:04.823 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000052154 target=00000000000000000000:00000000000000052014, shard=1 master=00000000000000000000:00000000000000037555 target=00000000000000000000:00000000000000037415, shard=2 master=00000000000000000000:00000000000000059977 target=00000000000000000000:00000000000000059697 2024-08-18T17:29:09.824 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:29:09.824 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:29:09.824 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:29:09.881 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:29:09.881 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:29:09.932 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:29:09.932 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:29:09.932 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:29:09.932 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:29:09.932 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:29:09.932 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:29:09.932 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:09.932 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:29:09.933 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:29:09.933 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:29:09.933 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:29:09.933 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:29:09.933 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052014", 2024-08-18T17:29:09.933 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:29:09.933 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:29:09.933 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:29:09.933 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:25:57.178435Z" 2024-08-18T17:29:09.933 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:29:09.933 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:09.933 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:29:09.933 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:29:09.933 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:29:09.934 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:29:09.934 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:29:09.934 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:29:09.934 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:29:09.934 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:29:09.934 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:29:09.934 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:29:09.934 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:09.934 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:29:09.934 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:29:09.934 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:29:09.934 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:29:09.934 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000059697", 2024-08-18T17:29:09.935 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:29:09.935 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:29:09.935 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:29:09.935 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:25:57.169082Z" 2024-08-18T17:29:09.935 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:29:09.935 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:09.935 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:29:09.935 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:29:09.935 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:29:09.935 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:29:09.935 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:29:09.935 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:29:09.935 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:29:09.936 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:29:09.936 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:29:09.936 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:29:09.936 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:29:09.936 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:29:09.936 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:09.936 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:29:09.936 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:29:09.936 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:29:09.936 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:09.936 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:29:09Z" 2024-08-18T17:29:09.936 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:29:09.937 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000052154 target=00000000000000000000:00000000000000052014, shard=1 master=00000000000000000000:00000000000000037555 target=00000000000000000000:00000000000000037415, shard=2 master=00000000000000000000:00000000000000059977 target=00000000000000000000:00000000000000059697 2024-08-18T17:29:14.939 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:29:14.941 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:29:14.941 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:29:14.998 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:29:14.998 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:29:15.049 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:29:15.049 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:29:15.049 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:29:15.049 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:29:15.049 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:29:15.050 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:29:15.050 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:15.050 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:29:15.050 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:29:15.050 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:29:15.050 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:29:15.050 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:29:15.050 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052014", 2024-08-18T17:29:15.050 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:29:15.050 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:29:15.050 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:29:15.050 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:25:57.178435Z" 2024-08-18T17:29:15.050 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:29:15.051 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:15.051 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:29:15.051 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:29:15.051 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:29:15.051 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:29:15.051 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:29:15.051 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:29:15.051 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:29:15.051 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:29:15.051 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:29:15.051 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:29:15.052 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:15.052 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:29:15.052 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:29:15.052 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:29:15.052 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:29:15.052 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000059697", 2024-08-18T17:29:15.052 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:29:15.052 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:29:15.052 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:29:15.052 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:25:57.169082Z" 2024-08-18T17:29:15.052 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:29:15.052 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:15.053 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:29:15.053 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:29:15.053 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:29:15.053 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:29:15.053 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:29:15.053 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:29:15.053 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:29:15.053 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:29:15.053 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:29:15.053 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:29:15.053 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:29:15.053 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:29:15.053 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:15.053 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:29:15.054 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:29:15.054 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:29:15.054 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:15.054 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:29:15Z" 2024-08-18T17:29:15.054 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:29:15.055 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000052154 target=00000000000000000000:00000000000000052014, shard=1 master=00000000000000000000:00000000000000037555 target=00000000000000000000:00000000000000037415, shard=2 master=00000000000000000000:00000000000000059977 target=00000000000000000000:00000000000000059697 2024-08-18T17:29:20.055 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:29:20.056 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:29:20.056 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:29:20.113 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:29:20.113 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:29:20.165 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:29:20.165 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:29:20.165 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:29:20.165 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:29:20.165 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:29:20.165 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:29:20.165 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:20.165 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:29:20.166 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:29:20.166 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:29:20.166 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:29:20.166 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:29:20.166 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052014", 2024-08-18T17:29:20.166 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:29:20.166 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:29:20.166 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:29:20.166 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:25:57.178435Z" 2024-08-18T17:29:20.166 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:29:20.166 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:20.166 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:29:20.166 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:29:20.167 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:29:20.167 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:29:20.167 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:29:20.167 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:29:20.167 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:29:20.167 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:29:20.167 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:29:20.167 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:29:20.167 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:20.167 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:29:20.167 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:29:20.167 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:29:20.167 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:29:20.168 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000059697", 2024-08-18T17:29:20.168 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:29:20.168 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:29:20.168 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:29:20.168 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:25:57.169082Z" 2024-08-18T17:29:20.168 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:29:20.168 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:20.168 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:29:20.168 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:29:20.168 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:29:20.168 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:29:20.168 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:29:20.168 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:29:20.169 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:29:20.169 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:29:20.169 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:29:20.169 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:29:20.169 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:29:20.169 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:29:20.169 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:20.169 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:29:20.169 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:29:20.170 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:29:20.170 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:20.170 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:29:20Z" 2024-08-18T17:29:20.170 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:29:20.171 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000052154 target=00000000000000000000:00000000000000052014, shard=1 master=00000000000000000000:00000000000000037555 target=00000000000000000000:00000000000000037415, shard=2 master=00000000000000000000:00000000000000059977 target=00000000000000000000:00000000000000059697 2024-08-18T17:29:25.172 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:29:25.172 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:29:25.172 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:29:25.229 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:29:25.230 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:29:25.280 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:29:25.281 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:29:25.281 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:29:25.281 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:29:25.281 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:29:25.281 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:29:25.281 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:25.281 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:29:25.281 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:29:25.281 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:29:25.281 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:29:25.281 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:29:25.281 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052014", 2024-08-18T17:29:25.282 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:29:25.282 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:29:25.282 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:29:25.282 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:25:57.178435Z" 2024-08-18T17:29:25.282 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:29:25.282 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:25.282 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:29:25.282 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:29:25.282 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:29:25.282 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:29:25.282 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:29:25.283 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:29:25.283 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:29:25.283 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:29:25.283 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:29:25.283 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:29:25.283 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:25.283 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:29:25.283 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:29:25.283 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:29:25.283 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:29:25.283 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000059697", 2024-08-18T17:29:25.283 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:29:25.283 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:29:25.284 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:29:25.284 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:25:57.169082Z" 2024-08-18T17:29:25.284 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:29:25.284 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:25.284 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:29:25.284 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:29:25.284 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:29:25.284 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:29:25.284 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:29:25.284 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:29:25.284 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:29:25.284 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:29:25.284 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:29:25.285 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:29:25.285 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:29:25.285 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:29:25.285 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:25.285 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:29:25.285 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:29:25.285 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:29:25.285 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:25.285 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:29:25Z" 2024-08-18T17:29:25.285 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:29:25.287 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000052154 target=00000000000000000000:00000000000000052014, shard=1 master=00000000000000000000:00000000000000037555 target=00000000000000000000:00000000000000037415, shard=2 master=00000000000000000000:00000000000000059977 target=00000000000000000000:00000000000000059697 2024-08-18T17:29:30.287 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:29:30.287 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:29:30.288 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:29:30.346 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:29:30.346 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:29:30.395 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:29:30.395 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:29:30.395 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:29:30.396 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:29:30.396 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:29:30.396 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:29:30.396 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:30.396 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:29:30.396 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:29:30.396 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:29:30.396 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:29:30.396 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:29:30.396 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052014", 2024-08-18T17:29:30.396 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:29:30.396 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:29:30.397 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:29:30.397 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:25:57.178435Z" 2024-08-18T17:29:30.397 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:29:30.397 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:30.397 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:29:30.397 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:29:30.397 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:29:30.397 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:29:30.397 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:29:30.397 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:29:30.397 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:29:30.397 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:29:30.397 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:29:30.398 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:29:30.398 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:30.398 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:29:30.398 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:29:30.398 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:29:30.398 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:29:30.398 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000059697", 2024-08-18T17:29:30.398 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:29:30.398 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:29:30.398 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:29:30.398 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:25:57.169082Z" 2024-08-18T17:29:30.399 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:29:30.399 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:30.399 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:29:30.399 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:29:30.399 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:29:30.399 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:29:30.399 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:29:30.399 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:29:30.399 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:29:30.399 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:29:30.399 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:29:30.399 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:29:30.400 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:29:30.400 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:29:30.400 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:30.400 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:29:30.401 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:29:30.401 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:29:30.401 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:30.401 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:29:30Z" 2024-08-18T17:29:30.401 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:29:30.402 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000052154 target=00000000000000000000:00000000000000052014, shard=1 master=00000000000000000000:00000000000000037555 target=00000000000000000000:00000000000000037415, shard=2 master=00000000000000000000:00000000000000059977 target=00000000000000000000:00000000000000059697 2024-08-18T17:29:35.403 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:29:35.403 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:29:35.404 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:29:35.461 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:29:35.462 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:29:35.512 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:29:35.512 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:29:35.512 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:29:35.513 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:29:35.513 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:29:35.513 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:29:35.513 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:35.513 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:29:35.513 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:29:35.513 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:29:35.513 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:29:35.513 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:29:35.513 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052014", 2024-08-18T17:29:35.513 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:29:35.513 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:29:35.514 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:29:35.514 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:25:57.178435Z" 2024-08-18T17:29:35.514 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:29:35.514 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:35.514 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:29:35.514 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:29:35.514 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:29:35.514 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:29:35.514 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037415", 2024-08-18T17:29:35.514 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:29:35.514 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:29:35.514 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:29:35.514 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:16:29.301541Z" 2024-08-18T17:29:35.515 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:29:35.515 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:35.515 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:29:35.515 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:29:35.515 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:29:35.515 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:29:35.515 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000059697", 2024-08-18T17:29:35.515 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:29:35.515 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:29:35.515 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:29:35.515 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:25:57.169082Z" 2024-08-18T17:29:35.515 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:29:35.515 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:35.515 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:29:35.516 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:29:35.516 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:29:35.516 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:29:35.516 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:29:35.516 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:29:35.516 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:29:35.516 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:29:35.516 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:29:35.518 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:29:35.518 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:29:35.518 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:29:35.518 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:35.518 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:29:35.519 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:29:35.519 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:29:35.519 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:35.519 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:29:35Z" 2024-08-18T17:29:35.519 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:29:35.521 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000052154 target=00000000000000000000:00000000000000052014, shard=1 master=00000000000000000000:00000000000000037555 target=00000000000000000000:00000000000000037415, shard=2 master=00000000000000000000:00000000000000059977 target=00000000000000000000:00000000000000059697 2024-08-18T17:29:40.521 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:29:40.522 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:29:40.522 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:29:40.579 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:29:40.582 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:29:40.629 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:29:40.630 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:29:40.630 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:29:40.630 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:29:40.630 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:29:40.630 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:29:40.630 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:40.630 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:29:40.630 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:29:40.630 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:29:40.630 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:29:40.630 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:29:40.630 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052154", 2024-08-18T17:29:40.631 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:29:40.631 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:29:40.631 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:29:40.631 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:27:36.982158Z" 2024-08-18T17:29:40.631 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:29:40.631 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:40.631 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:29:40.631 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:29:40.631 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:29:40.631 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:29:40.631 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037555", 2024-08-18T17:29:40.631 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:29:40.631 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:29:40.631 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:29:40.632 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:27:37.043496Z" 2024-08-18T17:29:40.632 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:29:40.632 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:40.632 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:29:40.632 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:29:40.632 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:29:40.632 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:29:40.632 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000059977", 2024-08-18T17:29:40.632 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:29:40.632 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:29:40.632 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:29:40.632 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:27:37.022632Z" 2024-08-18T17:29:40.632 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:29:40.632 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:40.633 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:29:40.633 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:29:40.633 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:29:40.633 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:29:40.633 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:29:40.633 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:29:40.633 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:29:40.633 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:29:40.633 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:29:40.633 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:29:40.633 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:29:40.633 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:29:40.633 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:40.634 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:29:40.634 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:29:40.634 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:29:40.634 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:40.634 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:29:40Z" 2024-08-18T17:29:40.634 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:29:40.636 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-08-18T17:29:40.645 INFO:tasks.rgw_multisite_tests:test_sync_multiple_buckets_to_single: ... FAIL 2024-08-18T17:29:40.646 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:29:40.646 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:29:40.646 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:29:40.700 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:29:40.700 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:29:40.749 INFO:teuthology.orchestra.run.smithi134.stderr:No --period given, using current period=01050e30-179b-43be-bdbb-3bb23cf37ae8 2024-08-18T17:29:40.769 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T17:29:40.769 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:29:40.769 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:29:40.770 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T17:29:40.770 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:29:40.770 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:29:40.770 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:29:40.770 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T17:29:40.770 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:29:40.770 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:29:40.770 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:29:40.770 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:42:47.672461Z" 2024-08-18T17:29:40.770 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:29:40.770 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:29:40.770 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1724002045.275297_1660.1", 2024-08-18T17:29:40.770 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T17:27:25.275297Z" 2024-08-18T17:29:40.771 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:29:40.771 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T17:29:40.771 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-08-18T17:29:40.772 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:29:40.772 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:29:40.772 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:29:40.821 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:29:40.821 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:29:40.852 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:29:40.851+0000 7f932b20f880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:29:40.852 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:29:40.852+0000 7f932b20f880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:29:40.867 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:29:40.867 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:29:40.867 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:29:40.867 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:29:40.867 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:29:40.867 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:29:40.867 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:29:40.867 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:40.868 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:29:40.868 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:29:40.868 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:29:40.868 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:29:40.868 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:29:40.868 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:29:40.868 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:29:40.868 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:29:40.868 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:29:40.868 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:29:40.868 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:29:40.868 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:29:40.868 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:40.869 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:29:40.869 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:29:40.869 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:29:40.869 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:29:40.869 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:29:40.869 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:29:40.869 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:29:40.869 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:29:40.869 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:29:40.869 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:29:40.869 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:29:40.869 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:40.870 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:29:40.870 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:29:40.870 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:29:40.870 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:29:40.870 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:29:40.870 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:29:40.870 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:29:40.870 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:29:40.870 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:29:40.870 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:29:40.870 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:29:40.870 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:40.870 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:29:40.871 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:29:40.871 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:29:40.871 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:29:40.871 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724002045.275297_1660.1", 2024-08-18T17:29:40.871 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:29:40.871 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:29:40.871 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:29:40.871 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:27:25.275297Z", 2024-08-18T17:29:40.871 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:29:40.871 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:29:40.871 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:29:40.871 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:29:40.871 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:40.872 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:29:40.872 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:29:40.872 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:29:40.872 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:40.872 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:29:40Z" 2024-08-18T17:29:40.872 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:29:40.874 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-08-18T17:29:40.874 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-08-18T17:29:40.874 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:29:40.874 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:29:40.874 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:29:40.922 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:29:40.923 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:29:40.976 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:29:40.975+0000 7f3f6fcbb880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:29:40.976 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:29:40.976+0000 7f3f6fcbb880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:29:40.997 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T17:29:40.997 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T17:29:40.997 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T17:29:40.997 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T17:29:40.998 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T17:29:40.998 INFO:teuthology.orchestra.run.smithi134.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:29:40.998 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:29:40.998 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:29:40.998 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T17:29:40.998 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:29:40.998 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T17:29:40.998 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:29:40.998 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:29:40.998 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:29:40.998 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:29:40.998 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 1, 2024-08-18T17:29:40.998 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:29:40.999 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:29:40.999 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T17:29:40.999 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:29:40.999 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:29:40.999 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:29:40.999 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T17:29:40.999 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:29:40.999 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:29:40.999 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:29:40.999 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:29:40.999 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:29:40.999 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:29:40.999 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T17:29:41.000 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T17:29:41.000 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:29:41.000 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:29:41.000 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:29:41.000 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T17:29:41.000 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:29:41.000 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:29:41.000 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:29:41.000 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:29:41.000 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:29:41.000 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:29:41.000 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:29:41.000 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:29:41.000 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:29:41.001 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:29:41.001 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:29:41.001 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T17:29:41.001 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:29:41.001 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:29:41.001 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1724002045.275297_1660.1", 2024-08-18T17:29:41.001 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:29:41.001 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:29:41.001 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:29:41.001 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T17:27:25.275297Z", 2024-08-18T17:29:41.001 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:29:41.001 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:29:41.001 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:29:41.001 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:29:41.002 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:29:41.002 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T17:29:41.002 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 1, 2024-08-18T17:29:41.002 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 1 2024-08-18T17:29:41.002 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:29:41.002 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T17:29:40Z" 2024-08-18T17:29:41.002 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T17:29:41.004 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-08-18T17:29:41.004 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:29:41.004 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:29:41.004 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:29:41.047 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:29:41.048 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:29:41.112 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-kibgmb-95","source":{"bucket":"kibgmb-95","zones":["test-zone1"]},"dest":{"bucket":"kibgmb-96","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-kibgmb-96","source":{"bucket":"kibgmb-96","zones":["test-zone1"]},"dest":{"bucket":"kibgmb-96","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-08-18T17:29:41.112 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=kibgmb-97 2024-08-18T17:29:41.133 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=kibgmb-98 2024-08-18T17:29:41.158 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:29:41.159 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:29:41.159 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:29:41.211 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:29:41.211 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:29:41.256 INFO:teuthology.orchestra.run.smithi134.stderr:No --period given, using current period=01050e30-179b-43be-bdbb-3bb23cf37ae8 2024-08-18T17:29:41.271 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T17:29:41.271 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:29:41.271 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:29:41.272 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T17:29:41.272 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:29:41.272 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:29:41.272 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:29:41.272 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T17:29:41.272 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:29:41.272 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:29:41.272 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:29:41.272 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:42:47.672461Z" 2024-08-18T17:29:41.272 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:29:41.272 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:29:41.272 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1724002181.154947_1688.1", 2024-08-18T17:29:41.272 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T17:29:41.154947Z" 2024-08-18T17:29:41.273 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:29:41.273 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T17:29:41.274 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-08-18T17:29:41.274 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:29:41.274 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:29:41.274 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:29:41.323 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:29:41.323 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:29:41.354 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:29:41.352+0000 7f33fd42f880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:29:41.354 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:29:41.354+0000 7f33fd42f880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:29:41.372 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:29:41.372 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:29:41.372 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:29:41.372 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:29:41.372 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:29:41.373 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:29:41.373 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:29:41.373 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:41.373 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:29:41.373 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:29:41.373 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:29:41.373 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:29:41.373 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:29:41.373 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:29:41.373 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:29:41.373 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:29:41.374 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:29:41.374 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:29:41.374 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:29:41.374 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:29:41.374 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:41.374 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:29:41.375 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:29:41.375 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:29:41.375 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:29:41.375 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:29:41.375 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:29:41.375 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:29:41.375 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:29:41.375 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:29:41.375 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:29:41.375 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:29:41.375 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:41.375 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:29:41.376 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:29:41.376 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:29:41.376 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:29:41.376 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:29:41.376 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:29:41.376 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:29:41.376 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:29:41.376 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:29:41.376 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:29:41.376 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:29:41.376 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:41.377 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:29:41.377 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:29:41.377 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:29:41.377 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:29:41.377 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724002181.125575_1681.1", 2024-08-18T17:29:41.377 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:29:41.377 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:29:41.377 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:29:41.377 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:29:41.125575Z", 2024-08-18T17:29:41.377 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:29:41.377 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:29:41.378 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:29:41.378 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:29:41.378 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:41.378 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:29:41.378 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:29:41.378 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:29:41.378 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:41.378 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:29:41Z" 2024-08-18T17:29:41.378 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:29:41.379 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724002181.154947_1688.1 target=1_1724002181.125575_1681.1 2024-08-18T17:29:46.379 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:29:46.380 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:29:46.380 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:29:46.436 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:29:46.436 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:29:46.470 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:29:46.469+0000 7fa87ad31880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:29:46.488 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:29:46.470+0000 7fa87ad31880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:29:46.489 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:29:46.489 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:29:46.489 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:29:46.489 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:29:46.489 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:29:46.489 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:29:46.489 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:29:46.489 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:46.489 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:29:46.490 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:29:46.490 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:29:46.490 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:29:46.490 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:29:46.490 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:29:46.490 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:29:46.490 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:29:46.490 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:29:46.490 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:29:46.490 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:29:46.490 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:29:46.491 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:46.491 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:29:46.491 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:29:46.491 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:29:46.491 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:29:46.491 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:29:46.491 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:29:46.491 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:29:46.491 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:29:46.491 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:29:46.491 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:29:46.491 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:29:46.491 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:46.491 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:29:46.492 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:29:46.492 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:29:46.492 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:29:46.494 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:29:46.494 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:29:46.494 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:29:46.494 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:29:46.494 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:29:46.495 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:29:46.495 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:29:46.495 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:46.495 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:29:46.495 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:29:46.495 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:29:46.495 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:29:46.495 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724002181.154947_1688.1", 2024-08-18T17:29:46.495 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:29:46.495 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:29:46.496 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:29:46.496 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:29:41.154947Z", 2024-08-18T17:29:46.496 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:29:46.496 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:29:46.497 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:29:46.497 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:29:46.497 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:46.497 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:29:46.497 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:29:46.497 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:29:46.497 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:46.497 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:29:46Z" 2024-08-18T17:29:46.497 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:29:46.497 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-08-18T17:29:46.498 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-08-18T17:29:46.498 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:29:46.498 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:29:46.498 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:29:46.553 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:29:46.553 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:29:46.597 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:29:46.596+0000 7f5c77722880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:29:46.597 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:29:46.597+0000 7f5c77722880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:29:46.612 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T17:29:46.612 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T17:29:46.612 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T17:29:46.612 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T17:29:46.613 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T17:29:46.613 INFO:teuthology.orchestra.run.smithi134.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:29:46.613 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:29:46.613 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:29:46.613 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T17:29:46.613 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:29:46.613 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T17:29:46.613 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:29:46.613 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:29:46.613 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:29:46.613 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:29:46.613 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 1, 2024-08-18T17:29:46.614 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:29:46.614 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:29:46.614 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T17:29:46.614 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:29:46.614 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:29:46.614 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:29:46.614 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T17:29:46.614 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:29:46.614 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:29:46.614 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:29:46.614 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:29:46.614 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:29:46.614 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:29:46.615 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T17:29:46.615 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T17:29:46.615 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:29:46.615 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:29:46.615 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:29:46.615 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T17:29:46.615 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:29:46.615 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:29:46.615 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:29:46.615 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:29:46.615 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:29:46.615 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:29:46.615 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:29:46.615 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:29:46.616 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:29:46.616 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:29:46.616 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:29:46.616 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T17:29:46.616 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:29:46.616 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:29:46.616 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1724002181.154947_1688.1", 2024-08-18T17:29:46.616 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:29:46.616 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:29:46.616 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:29:46.616 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T17:29:41.154947Z", 2024-08-18T17:29:46.616 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:29:46.616 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:29:46.616 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:29:46.617 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:29:46.617 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:29:46.617 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T17:29:46.617 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 1, 2024-08-18T17:29:46.617 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 1 2024-08-18T17:29:46.617 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:29:46.617 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T17:29:46Z" 2024-08-18T17:29:46.617 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T17:29:46.618 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-08-18T17:29:46.618 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:29:46.619 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:29:46.619 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:29:46.667 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:29:46.667 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:29:46.735 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-kibgmb-95","source":{"bucket":"kibgmb-95","zones":["test-zone1"]},"dest":{"bucket":"kibgmb-96","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-kibgmb-96","source":{"bucket":"kibgmb-96","zones":["test-zone1"]},"dest":{"bucket":"kibgmb-96","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-08-18T17:29:46.735 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:29:46.735 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:29:46.736 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=test-zone1,test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:29:46.786 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:29:46.786 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:29:46.857 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-kibgmb-95","source":{"bucket":"kibgmb-95","zones":["test-zone1"]},"dest":{"bucket":"kibgmb-96","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-kibgmb-96","source":{"bucket":"kibgmb-96","zones":["test-zone1"]},"dest":{"bucket":"kibgmb-96","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-08-18T17:29:46.858 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-kibgmb-97', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--source-bucket=kibgmb-97', '--dest-bucket=kibgmb-97', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:29:46.858 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-kibgmb-97', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--source-bucket=kibgmb-97', '--dest-bucket=kibgmb-97', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:29:46.858 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe-kibgmb-97 --source-zones=test-zone1 --dest-zones=test-zone2 --source-bucket=kibgmb-97 --dest-bucket=kibgmb-97 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:29:46.909 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:29:46.910 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:29:46.978 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-kibgmb-95","source":{"bucket":"kibgmb-95","zones":["test-zone1"]},"dest":{"bucket":"kibgmb-96","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-kibgmb-96","source":{"bucket":"kibgmb-96","zones":["test-zone1"]},"dest":{"bucket":"kibgmb-96","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-kibgmb-97","source":{"bucket":"kibgmb-97","zones":["test-zone1"]},"dest":{"bucket":"kibgmb-97","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-08-18T17:29:46.979 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-kibgmb-98', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--source-bucket=kibgmb-97', '--dest-bucket=kibgmb-98', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:29:46.979 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-kibgmb-98', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--source-bucket=kibgmb-97', '--dest-bucket=kibgmb-98', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:29:46.979 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe-kibgmb-98 --source-zones=test-zone1 --dest-zones=test-zone2 --source-bucket=kibgmb-97 --dest-bucket=kibgmb-98 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:29:47.030 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:29:47.031 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:29:47.086 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-kibgmb-95","source":{"bucket":"kibgmb-95","zones":["test-zone1"]},"dest":{"bucket":"kibgmb-96","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-kibgmb-96","source":{"bucket":"kibgmb-96","zones":["test-zone1"]},"dest":{"bucket":"kibgmb-96","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-kibgmb-97","source":{"bucket":"kibgmb-97","zones":["test-zone1"]},"dest":{"bucket":"kibgmb-97","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-kibgmb-98","source":{"bucket":"kibgmb-97","zones":["test-zone1"]},"dest":{"bucket":"kibgmb-98","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-08-18T17:29:47.087 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--source-bucket=kibgmb-97', '--dest-bucket=kibgmb-98', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:29:47.087 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--source-bucket=kibgmb-97', '--dest-bucket=kibgmb-98', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:29:47.087 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=test-zone1 --dest-zones=test-zone2 --source-bucket=kibgmb-97 --dest-bucket=kibgmb-98 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:29:47.139 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:29:47.139 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:29:47.221 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"kibgmb-97","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"kibgmb-98","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-kibgmb-95","source":{"bucket":"kibgmb-95","zones":["test-zone1"]},"dest":{"bucket":"kibgmb-96","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-kibgmb-96","source":{"bucket":"kibgmb-96","zones":["test-zone1"]},"dest":{"bucket":"kibgmb-96","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-kibgmb-97","source":{"bucket":"kibgmb-97","zones":["test-zone1"]},"dest":{"bucket":"kibgmb-97","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-kibgmb-98","source":{"bucket":"kibgmb-97","zones":["test-zone1"]},"dest":{"bucket":"kibgmb-98","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-08-18T17:29:47.222 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:29:47.222 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:29:47.222 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:29:47.269 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:29:47.269 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:29:47.331 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"kibgmb-97","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"kibgmb-98","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-kibgmb-95","source":{"bucket":"kibgmb-95","zones":["test-zone1"]},"dest":{"bucket":"kibgmb-96","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-kibgmb-96","source":{"bucket":"kibgmb-96","zones":["test-zone1"]},"dest":{"bucket":"kibgmb-96","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-kibgmb-97","source":{"bucket":"kibgmb-97","zones":["test-zone1"]},"dest":{"bucket":"kibgmb-97","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-kibgmb-98","source":{"bucket":"kibgmb-97","zones":["test-zone1"]},"dest":{"bucket":"kibgmb-98","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-08-18T17:29:47.332 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:29:47.332 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:29:47.332 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:29:47.379 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:29:47.379 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:29:47.451 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T17:29:47.452 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:29:47.452 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 20, 2024-08-18T17:29:47.452 INFO:teuthology.orchestra.run.smithi134.stdout: "predecessor_uuid": "64309209-7567-46e5-8881-65783d84cf3c", 2024-08-18T17:29:47.452 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": [ 2024-08-18T17:29:47.452 INFO:teuthology.orchestra.run.smithi134.stdout: "", 2024-08-18T17:29:47.452 INFO:teuthology.orchestra.run.smithi134.stdout: "", 2024-08-18T17:29:47.452 INFO:teuthology.orchestra.run.smithi134.stdout: "", 2024-08-18T17:29:47.452 INFO:teuthology.orchestra.run.smithi134.stdout: "1_1723995848.930494_187.1" 2024-08-18T17:29:47.452 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:29:47.452 INFO:teuthology.orchestra.run.smithi134.stdout: "period_map": { 2024-08-18T17:29:47.452 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:29:47.452 INFO:teuthology.orchestra.run.smithi134.stdout: "zonegroups": [ 2024-08-18T17:29:47.453 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:29:47.453 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "test-zonegroup", 2024-08-18T17:29:47.453 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "test-zonegroup", 2024-08-18T17:29:47.453 INFO:teuthology.orchestra.run.smithi134.stdout: "api_name": "", 2024-08-18T17:29:47.453 INFO:teuthology.orchestra.run.smithi134.stdout: "is_master": true, 2024-08-18T17:29:47.453 INFO:teuthology.orchestra.run.smithi134.stdout: "endpoints": [ 2024-08-18T17:29:47.453 INFO:teuthology.orchestra.run.smithi134.stdout: "http://smithi134.front.sepia.ceph.com:8000" 2024-08-18T17:29:47.453 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:29:47.453 INFO:teuthology.orchestra.run.smithi134.stdout: "hostnames": [], 2024-08-18T17:29:47.453 INFO:teuthology.orchestra.run.smithi134.stdout: "hostnames_s3website": [], 2024-08-18T17:29:47.453 INFO:teuthology.orchestra.run.smithi134.stdout: "master_zone": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T17:29:47.453 INFO:teuthology.orchestra.run.smithi134.stdout: "zones": [ 2024-08-18T17:29:47.453 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:29:47.454 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T17:29:47.454 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "test-zone3", 2024-08-18T17:29:47.454 INFO:teuthology.orchestra.run.smithi134.stdout: "endpoints": [ 2024-08-18T17:29:47.454 INFO:teuthology.orchestra.run.smithi134.stdout: "http://smithi134.front.sepia.ceph.com:8001" 2024-08-18T17:29:47.454 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:29:47.454 INFO:teuthology.orchestra.run.smithi134.stdout: "log_meta": false, 2024-08-18T17:29:47.454 INFO:teuthology.orchestra.run.smithi134.stdout: "log_data": true, 2024-08-18T17:29:47.454 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_index_max_shards": 11, 2024-08-18T17:29:47.454 INFO:teuthology.orchestra.run.smithi134.stdout: "read_only": false, 2024-08-18T17:29:47.454 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_type": "", 2024-08-18T17:29:47.454 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from_all": true, 2024-08-18T17:29:47.454 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from": [], 2024-08-18T17:29:47.455 INFO:teuthology.orchestra.run.smithi134.stdout: "redirect_zone": "", 2024-08-18T17:29:47.455 INFO:teuthology.orchestra.run.smithi134.stdout: "supported_features": [ 2024-08-18T17:29:47.455 INFO:teuthology.orchestra.run.smithi134.stdout: "compress-encrypted", 2024-08-18T17:29:47.455 INFO:teuthology.orchestra.run.smithi134.stdout: "resharding" 2024-08-18T17:29:47.455 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:29:47.455 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:29:47.455 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:29:47.455 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "10090ccf-aad6-416d-9775-df8b7b27345a", 2024-08-18T17:29:47.455 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "test-zone2", 2024-08-18T17:29:47.455 INFO:teuthology.orchestra.run.smithi134.stdout: "endpoints": [ 2024-08-18T17:29:47.455 INFO:teuthology.orchestra.run.smithi134.stdout: "http://smithi193.front.sepia.ceph.com:8000" 2024-08-18T17:29:47.455 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:29:47.455 INFO:teuthology.orchestra.run.smithi134.stdout: "log_meta": false, 2024-08-18T17:29:47.456 INFO:teuthology.orchestra.run.smithi134.stdout: "log_data": true, 2024-08-18T17:29:47.456 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_index_max_shards": 11, 2024-08-18T17:29:47.456 INFO:teuthology.orchestra.run.smithi134.stdout: "read_only": false, 2024-08-18T17:29:47.457 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_type": "", 2024-08-18T17:29:47.457 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from_all": true, 2024-08-18T17:29:47.457 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from": [], 2024-08-18T17:29:47.457 INFO:teuthology.orchestra.run.smithi134.stdout: "redirect_zone": "", 2024-08-18T17:29:47.457 INFO:teuthology.orchestra.run.smithi134.stdout: "supported_features": [ 2024-08-18T17:29:47.457 INFO:teuthology.orchestra.run.smithi134.stdout: "compress-encrypted", 2024-08-18T17:29:47.457 INFO:teuthology.orchestra.run.smithi134.stdout: "resharding" 2024-08-18T17:29:47.457 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:29:47.457 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:29:47.457 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:29:47.457 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T17:29:47.457 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "test-zone1", 2024-08-18T17:29:47.458 INFO:teuthology.orchestra.run.smithi134.stdout: "endpoints": [ 2024-08-18T17:29:47.458 INFO:teuthology.orchestra.run.smithi134.stdout: "http://smithi134.front.sepia.ceph.com:8000" 2024-08-18T17:29:47.458 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:29:47.458 INFO:teuthology.orchestra.run.smithi134.stdout: "log_meta": false, 2024-08-18T17:29:47.458 INFO:teuthology.orchestra.run.smithi134.stdout: "log_data": true, 2024-08-18T17:29:47.458 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_index_max_shards": 11, 2024-08-18T17:29:47.458 INFO:teuthology.orchestra.run.smithi134.stdout: "read_only": false, 2024-08-18T17:29:47.458 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_type": "", 2024-08-18T17:29:47.458 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from_all": true, 2024-08-18T17:29:47.458 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from": [], 2024-08-18T17:29:47.458 INFO:teuthology.orchestra.run.smithi134.stdout: "redirect_zone": "", 2024-08-18T17:29:47.458 INFO:teuthology.orchestra.run.smithi134.stdout: "supported_features": [ 2024-08-18T17:29:47.458 INFO:teuthology.orchestra.run.smithi134.stdout: "compress-encrypted", 2024-08-18T17:29:47.459 INFO:teuthology.orchestra.run.smithi134.stdout: "resharding" 2024-08-18T17:29:47.459 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:29:47.459 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:29:47.459 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:29:47.459 INFO:teuthology.orchestra.run.smithi134.stdout: "placement_targets": [ 2024-08-18T17:29:47.459 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:29:47.459 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "default-placement", 2024-08-18T17:29:47.459 INFO:teuthology.orchestra.run.smithi134.stdout: "tags": [], 2024-08-18T17:29:47.459 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_classes": [ 2024-08-18T17:29:47.459 INFO:teuthology.orchestra.run.smithi134.stdout: "STANDARD" 2024-08-18T17:29:47.459 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:29:47.460 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:29:47.460 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:29:47.460 INFO:teuthology.orchestra.run.smithi134.stdout: "default_placement": "default-placement", 2024-08-18T17:29:47.460 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_id": "2827c6f0-bf0d-4836-944c-9e7fcf22457b", 2024-08-18T17:29:47.460 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_policy": { 2024-08-18T17:29:47.460 INFO:teuthology.orchestra.run.smithi134.stdout: "groups": [ 2024-08-18T17:29:47.460 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:29:47.460 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "sync-group", 2024-08-18T17:29:47.460 INFO:teuthology.orchestra.run.smithi134.stdout: "data_flow": { 2024-08-18T17:29:47.460 INFO:teuthology.orchestra.run.smithi134.stdout: "symmetrical": [ 2024-08-18T17:29:47.460 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:29:47.460 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "sync-flow1", 2024-08-18T17:29:47.460 INFO:teuthology.orchestra.run.smithi134.stdout: "zones": [ 2024-08-18T17:29:47.461 INFO:teuthology.orchestra.run.smithi134.stdout: "10090ccf-aad6-416d-9775-df8b7b27345a", 2024-08-18T17:29:47.461 INFO:teuthology.orchestra.run.smithi134.stdout: "96bf8502-2f39-4d53-8196-3378c28da3b4" 2024-08-18T17:29:47.461 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:29:47.461 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:29:47.461 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:29:47.461 INFO:teuthology.orchestra.run.smithi134.stdout: "directional": [ 2024-08-18T17:29:47.461 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:29:47.461 INFO:teuthology.orchestra.run.smithi134.stdout: "source_zone": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T17:29:47.461 INFO:teuthology.orchestra.run.smithi134.stdout: "dest_zone": "10090ccf-aad6-416d-9775-df8b7b27345a" 2024-08-18T17:29:47.461 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:29:47.461 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:29:47.461 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:29:47.461 INFO:teuthology.orchestra.run.smithi134.stdout: "pipes": [ 2024-08-18T17:29:47.462 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:29:47.462 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "sync-pipe", 2024-08-18T17:29:47.462 INFO:teuthology.orchestra.run.smithi134.stdout: "source": { 2024-08-18T17:29:47.462 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket": "kibgmb-97", 2024-08-18T17:29:47.462 INFO:teuthology.orchestra.run.smithi134.stdout: "zones": [ 2024-08-18T17:29:47.462 INFO:teuthology.orchestra.run.smithi134.stdout: "10090ccf-aad6-416d-9775-df8b7b27345a", 2024-08-18T17:29:47.462 INFO:teuthology.orchestra.run.smithi134.stdout: "96bf8502-2f39-4d53-8196-3378c28da3b4" 2024-08-18T17:29:47.462 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:29:47.462 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:29:47.462 INFO:teuthology.orchestra.run.smithi134.stdout: "dest": { 2024-08-18T17:29:47.462 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket": "kibgmb-98", 2024-08-18T17:29:47.462 INFO:teuthology.orchestra.run.smithi134.stdout: "zones": [ 2024-08-18T17:29:47.462 INFO:teuthology.orchestra.run.smithi134.stdout: "10090ccf-aad6-416d-9775-df8b7b27345a", 2024-08-18T17:29:47.463 INFO:teuthology.orchestra.run.smithi134.stdout: "96bf8502-2f39-4d53-8196-3378c28da3b4" 2024-08-18T17:29:47.463 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:29:47.463 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:29:47.463 INFO:teuthology.orchestra.run.smithi134.stdout: "params": { 2024-08-18T17:29:47.463 INFO:teuthology.orchestra.run.smithi134.stdout: "source": { 2024-08-18T17:29:47.463 INFO:teuthology.orchestra.run.smithi134.stdout: "filter": { 2024-08-18T17:29:47.463 INFO:teuthology.orchestra.run.smithi134.stdout: "tags": [] 2024-08-18T17:29:47.463 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:29:47.463 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:29:47.463 INFO:teuthology.orchestra.run.smithi134.stdout: "dest": {}, 2024-08-18T17:29:47.464 INFO:teuthology.orchestra.run.smithi134.stdout: "priority": 0, 2024-08-18T17:29:47.464 INFO:teuthology.orchestra.run.smithi134.stdout: "mode": "system", 2024-08-18T17:29:47.464 INFO:teuthology.orchestra.run.smithi134.stdout: "user": "" 2024-08-18T17:29:47.464 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:29:47.464 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:29:47.464 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:29:47.464 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "sync-pipe-kibgmb-95", 2024-08-18T17:29:47.464 INFO:teuthology.orchestra.run.smithi134.stdout: "source": { 2024-08-18T17:29:47.464 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket": "kibgmb-95", 2024-08-18T17:29:47.464 INFO:teuthology.orchestra.run.smithi134.stdout: "zones": [ 2024-08-18T17:29:47.464 INFO:teuthology.orchestra.run.smithi134.stdout: "96bf8502-2f39-4d53-8196-3378c28da3b4" 2024-08-18T17:29:47.464 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:29:47.464 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:29:47.465 INFO:teuthology.orchestra.run.smithi134.stdout: "dest": { 2024-08-18T17:29:47.465 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket": "kibgmb-96", 2024-08-18T17:29:47.465 INFO:teuthology.orchestra.run.smithi134.stdout: "zones": [ 2024-08-18T17:29:47.465 INFO:teuthology.orchestra.run.smithi134.stdout: "10090ccf-aad6-416d-9775-df8b7b27345a" 2024-08-18T17:29:47.465 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:29:47.465 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:29:47.465 INFO:teuthology.orchestra.run.smithi134.stdout: "params": { 2024-08-18T17:29:47.465 INFO:teuthology.orchestra.run.smithi134.stdout: "source": { 2024-08-18T17:29:47.465 INFO:teuthology.orchestra.run.smithi134.stdout: "filter": { 2024-08-18T17:29:47.465 INFO:teuthology.orchestra.run.smithi134.stdout: "tags": [] 2024-08-18T17:29:47.466 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:29:47.466 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:29:47.466 INFO:teuthology.orchestra.run.smithi134.stdout: "dest": {}, 2024-08-18T17:29:47.466 INFO:teuthology.orchestra.run.smithi134.stdout: "priority": 0, 2024-08-18T17:29:47.466 INFO:teuthology.orchestra.run.smithi134.stdout: "mode": "system", 2024-08-18T17:29:47.466 INFO:teuthology.orchestra.run.smithi134.stdout: "user": "" 2024-08-18T17:29:47.466 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:29:47.466 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:29:47.466 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:29:47.466 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "sync-pipe-kibgmb-96", 2024-08-18T17:29:47.466 INFO:teuthology.orchestra.run.smithi134.stdout: "source": { 2024-08-18T17:29:47.466 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket": "kibgmb-96", 2024-08-18T17:29:47.466 INFO:teuthology.orchestra.run.smithi134.stdout: "zones": [ 2024-08-18T17:29:47.467 INFO:teuthology.orchestra.run.smithi134.stdout: "96bf8502-2f39-4d53-8196-3378c28da3b4" 2024-08-18T17:29:47.467 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:29:47.467 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:29:47.467 INFO:teuthology.orchestra.run.smithi134.stdout: "dest": { 2024-08-18T17:29:47.467 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket": "kibgmb-96", 2024-08-18T17:29:47.467 INFO:teuthology.orchestra.run.smithi134.stdout: "zones": [ 2024-08-18T17:29:47.467 INFO:teuthology.orchestra.run.smithi134.stdout: "10090ccf-aad6-416d-9775-df8b7b27345a" 2024-08-18T17:29:47.467 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:29:47.467 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:29:47.467 INFO:teuthology.orchestra.run.smithi134.stdout: "params": { 2024-08-18T17:29:47.467 INFO:teuthology.orchestra.run.smithi134.stdout: "source": { 2024-08-18T17:29:47.467 INFO:teuthology.orchestra.run.smithi134.stdout: "filter": { 2024-08-18T17:29:47.468 INFO:teuthology.orchestra.run.smithi134.stdout: "tags": [] 2024-08-18T17:29:47.468 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:29:47.468 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:29:47.468 INFO:teuthology.orchestra.run.smithi134.stdout: "dest": {}, 2024-08-18T17:29:47.468 INFO:teuthology.orchestra.run.smithi134.stdout: "priority": 0, 2024-08-18T17:29:47.468 INFO:teuthology.orchestra.run.smithi134.stdout: "mode": "system", 2024-08-18T17:29:47.468 INFO:teuthology.orchestra.run.smithi134.stdout: "user": "" 2024-08-18T17:29:47.468 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:29:47.468 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:29:47.468 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:29:47.468 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "sync-pipe-kibgmb-97", 2024-08-18T17:29:47.468 INFO:teuthology.orchestra.run.smithi134.stdout: "source": { 2024-08-18T17:29:47.468 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket": "kibgmb-97", 2024-08-18T17:29:47.469 INFO:teuthology.orchestra.run.smithi134.stdout: "zones": [ 2024-08-18T17:29:47.469 INFO:teuthology.orchestra.run.smithi134.stdout: "96bf8502-2f39-4d53-8196-3378c28da3b4" 2024-08-18T17:29:47.469 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:29:47.469 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:29:47.469 INFO:teuthology.orchestra.run.smithi134.stdout: "dest": { 2024-08-18T17:29:47.469 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket": "kibgmb-97", 2024-08-18T17:29:47.469 INFO:teuthology.orchestra.run.smithi134.stdout: "zones": [ 2024-08-18T17:29:47.469 INFO:teuthology.orchestra.run.smithi134.stdout: "10090ccf-aad6-416d-9775-df8b7b27345a" 2024-08-18T17:29:47.469 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:29:47.469 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:29:47.469 INFO:teuthology.orchestra.run.smithi134.stdout: "params": { 2024-08-18T17:29:47.469 INFO:teuthology.orchestra.run.smithi134.stdout: "source": { 2024-08-18T17:29:47.469 INFO:teuthology.orchestra.run.smithi134.stdout: "filter": { 2024-08-18T17:29:47.470 INFO:teuthology.orchestra.run.smithi134.stdout: "tags": [] 2024-08-18T17:29:47.470 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:29:47.470 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:29:47.470 INFO:teuthology.orchestra.run.smithi134.stdout: "dest": {}, 2024-08-18T17:29:47.470 INFO:teuthology.orchestra.run.smithi134.stdout: "priority": 0, 2024-08-18T17:29:47.470 INFO:teuthology.orchestra.run.smithi134.stdout: "mode": "system", 2024-08-18T17:29:47.470 INFO:teuthology.orchestra.run.smithi134.stdout: "user": "" 2024-08-18T17:29:47.470 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:29:47.470 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:29:47.470 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:29:47.470 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "sync-pipe-kibgmb-98", 2024-08-18T17:29:47.470 INFO:teuthology.orchestra.run.smithi134.stdout: "source": { 2024-08-18T17:29:47.471 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket": "kibgmb-97", 2024-08-18T17:29:47.471 INFO:teuthology.orchestra.run.smithi134.stdout: "zones": [ 2024-08-18T17:29:47.471 INFO:teuthology.orchestra.run.smithi134.stdout: "96bf8502-2f39-4d53-8196-3378c28da3b4" 2024-08-18T17:29:47.471 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:29:47.471 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:29:47.471 INFO:teuthology.orchestra.run.smithi134.stdout: "dest": { 2024-08-18T17:29:47.471 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket": "kibgmb-98", 2024-08-18T17:29:47.471 INFO:teuthology.orchestra.run.smithi134.stdout: "zones": [ 2024-08-18T17:29:47.471 INFO:teuthology.orchestra.run.smithi134.stdout: "10090ccf-aad6-416d-9775-df8b7b27345a" 2024-08-18T17:29:47.471 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:29:47.471 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:29:47.471 INFO:teuthology.orchestra.run.smithi134.stdout: "params": { 2024-08-18T17:29:47.471 INFO:teuthology.orchestra.run.smithi134.stdout: "source": { 2024-08-18T17:29:47.471 INFO:teuthology.orchestra.run.smithi134.stdout: "filter": { 2024-08-18T17:29:47.472 INFO:teuthology.orchestra.run.smithi134.stdout: "tags": [] 2024-08-18T17:29:47.472 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:29:47.472 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:29:47.472 INFO:teuthology.orchestra.run.smithi134.stdout: "dest": {}, 2024-08-18T17:29:47.472 INFO:teuthology.orchestra.run.smithi134.stdout: "priority": 0, 2024-08-18T17:29:47.472 INFO:teuthology.orchestra.run.smithi134.stdout: "mode": "system", 2024-08-18T17:29:47.472 INFO:teuthology.orchestra.run.smithi134.stdout: "user": "" 2024-08-18T17:29:47.472 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:29:47.472 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:29:47.472 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:29:47.472 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "enabled" 2024-08-18T17:29:47.472 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:29:47.472 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:29:47.473 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:29:47.473 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled_features": [ 2024-08-18T17:29:47.473 INFO:teuthology.orchestra.run.smithi134.stdout: "resharding" 2024-08-18T17:29:47.473 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:29:47.473 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:29:47.473 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:29:47.473 INFO:teuthology.orchestra.run.smithi134.stdout: "short_zone_ids": [ 2024-08-18T17:29:47.473 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:29:47.473 INFO:teuthology.orchestra.run.smithi134.stdout: "key": "018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T17:29:47.473 INFO:teuthology.orchestra.run.smithi134.stdout: "val": 462036791 2024-08-18T17:29:47.473 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:29:47.473 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:29:47.473 INFO:teuthology.orchestra.run.smithi134.stdout: "key": "10090ccf-aad6-416d-9775-df8b7b27345a", 2024-08-18T17:29:47.474 INFO:teuthology.orchestra.run.smithi134.stdout: "val": 874289624 2024-08-18T17:29:47.474 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:29:47.474 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:29:47.474 INFO:teuthology.orchestra.run.smithi134.stdout: "key": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T17:29:47.475 INFO:teuthology.orchestra.run.smithi134.stdout: "val": 1721399886 2024-08-18T17:29:47.475 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:29:47.475 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:29:47.475 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:29:47.475 INFO:teuthology.orchestra.run.smithi134.stdout: "master_zonegroup": "test-zonegroup", 2024-08-18T17:29:47.475 INFO:teuthology.orchestra.run.smithi134.stdout: "master_zone": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T17:29:47.476 INFO:teuthology.orchestra.run.smithi134.stdout: "period_config": { 2024-08-18T17:29:47.476 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_quota": { 2024-08-18T17:29:47.476 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false, 2024-08-18T17:29:47.476 INFO:teuthology.orchestra.run.smithi134.stdout: "check_on_raw": false, 2024-08-18T17:29:47.476 INFO:teuthology.orchestra.run.smithi134.stdout: "max_size": -1, 2024-08-18T17:29:47.476 INFO:teuthology.orchestra.run.smithi134.stdout: "max_size_kb": 0, 2024-08-18T17:29:47.476 INFO:teuthology.orchestra.run.smithi134.stdout: "max_objects": -1 2024-08-18T17:29:47.476 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:29:47.476 INFO:teuthology.orchestra.run.smithi134.stdout: "user_quota": { 2024-08-18T17:29:47.476 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false, 2024-08-18T17:29:47.476 INFO:teuthology.orchestra.run.smithi134.stdout: "check_on_raw": false, 2024-08-18T17:29:47.476 INFO:teuthology.orchestra.run.smithi134.stdout: "max_size": -1, 2024-08-18T17:29:47.476 INFO:teuthology.orchestra.run.smithi134.stdout: "max_size_kb": 0, 2024-08-18T17:29:47.477 INFO:teuthology.orchestra.run.smithi134.stdout: "max_objects": -1 2024-08-18T17:29:47.477 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:29:47.477 INFO:teuthology.orchestra.run.smithi134.stdout: "user_ratelimit": { 2024-08-18T17:29:47.477 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_ops": 0, 2024-08-18T17:29:47.477 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_ops": 0, 2024-08-18T17:29:47.477 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_bytes": 0, 2024-08-18T17:29:47.477 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_bytes": 0, 2024-08-18T17:29:47.477 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false 2024-08-18T17:29:47.477 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:29:47.477 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_ratelimit": { 2024-08-18T17:29:47.477 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_ops": 0, 2024-08-18T17:29:47.477 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_ops": 0, 2024-08-18T17:29:47.478 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_bytes": 0, 2024-08-18T17:29:47.478 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_bytes": 0, 2024-08-18T17:29:47.478 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false 2024-08-18T17:29:47.478 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:29:47.478 INFO:teuthology.orchestra.run.smithi134.stdout: "anonymous_ratelimit": { 2024-08-18T17:29:47.478 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_ops": 0, 2024-08-18T17:29:47.478 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_ops": 0, 2024-08-18T17:29:47.478 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_bytes": 0, 2024-08-18T17:29:47.478 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_bytes": 0, 2024-08-18T17:29:47.478 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false 2024-08-18T17:29:47.478 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:29:47.478 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:29:47.479 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_id": "2827c6f0-bf0d-4836-944c-9e7fcf22457b", 2024-08-18T17:29:47.479 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_name": "test-realm", 2024-08-18T17:29:47.479 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:29:47.479 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T17:29:47.479 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:29:47.479 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:29:47.479 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:29:47.523 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:29:47.523 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:29:47.605 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"kibgmb-97","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"kibgmb-98","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-kibgmb-95","source":{"bucket":"kibgmb-95","zones":["test-zone1"]},"dest":{"bucket":"kibgmb-96","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-kibgmb-96","source":{"bucket":"kibgmb-96","zones":["test-zone1"]},"dest":{"bucket":"kibgmb-96","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-kibgmb-97","source":{"bucket":"kibgmb-97","zones":["test-zone1"]},"dest":{"bucket":"kibgmb-97","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-kibgmb-98","source":{"bucket":"kibgmb-97","zones":["test-zone1"]},"dest":{"bucket":"kibgmb-98","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-08-18T17:29:47.879 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:29:47.879 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:29:47.879 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:29:47.945 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:29:47.945 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:29:48.039 INFO:teuthology.orchestra.run.smithi134.stderr:No --period given, using current period=01050e30-179b-43be-bdbb-3bb23cf37ae8 2024-08-18T17:29:48.088 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T17:29:48.088 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:29:48.088 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:29:48.088 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T17:29:48.088 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:29:48.088 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:29:48.088 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:29:48.088 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T17:29:48.089 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:29:48.089 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:29:48.089 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:29:48.089 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:42:47.672461Z" 2024-08-18T17:29:48.089 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:29:48.089 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:29:48.089 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1724002181.154947_1688.1", 2024-08-18T17:29:48.089 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T17:29:41.154947Z" 2024-08-18T17:29:48.089 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:29:48.089 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T17:29:48.090 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-08-18T17:29:48.090 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:29:48.091 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:29:48.091 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:29:48.140 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:29:48.140 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:29:48.171 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:29:48.171+0000 7f7612219880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:29:48.172 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:29:48.172+0000 7f7612219880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:29:48.186 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:29:48.186 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:29:48.186 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:29:48.186 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:29:48.186 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:29:48.187 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:29:48.187 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:29:48.187 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:48.187 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:29:48.187 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:29:48.187 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:29:48.187 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:29:48.187 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:29:48.187 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:29:48.187 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:29:48.187 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:29:48.187 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:29:48.187 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:29:48.188 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:29:48.188 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:29:48.188 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:48.188 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:29:48.188 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:29:48.188 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:29:48.188 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:29:48.188 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:29:48.188 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:29:48.188 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:29:48.188 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:29:48.188 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:29:48.188 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:29:48.188 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:29:48.189 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:48.189 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:29:48.189 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:29:48.189 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:29:48.189 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:29:48.189 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:29:48.189 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:29:48.189 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:29:48.189 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:29:48.189 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:29:48.189 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:29:48.189 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:29:48.189 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:48.190 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:29:48.190 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:29:48.190 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:29:48.190 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:29:48.190 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724002181.154947_1688.1", 2024-08-18T17:29:48.190 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:29:48.190 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:29:48.190 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:29:48.190 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:29:41.154947Z", 2024-08-18T17:29:48.190 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:29:48.190 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:29:48.190 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:29:48.190 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:29:48.191 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:48.191 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:29:48.191 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:29:48.191 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:29:48.191 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:48.191 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:29:48Z" 2024-08-18T17:29:48.191 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:29:48.192 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-08-18T17:29:48.192 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-08-18T17:29:48.192 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:29:48.192 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:29:48.193 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:29:48.259 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:29:48.259 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:29:48.337 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:29:48.335+0000 7f13dea30880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:29:48.338 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:29:48.337+0000 7f13dea30880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:29:48.376 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T17:29:48.377 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T17:29:48.377 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T17:29:48.377 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T17:29:48.377 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T17:29:48.377 INFO:teuthology.orchestra.run.smithi134.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:29:48.377 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:29:48.377 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:29:48.377 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T17:29:48.377 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:29:48.378 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T17:29:48.378 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:29:48.378 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:29:48.378 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:29:48.378 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:29:48.378 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 1, 2024-08-18T17:29:48.378 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:29:48.378 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:29:48.378 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T17:29:48.379 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:29:48.379 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:29:48.379 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:29:48.379 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T17:29:48.379 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:29:48.379 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:29:48.379 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:29:48.379 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:29:48.379 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:29:48.379 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:29:48.379 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T17:29:48.379 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T17:29:48.380 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:29:48.380 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:29:48.380 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:29:48.380 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T17:29:48.380 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:29:48.380 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:29:48.380 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:29:48.380 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:29:48.380 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:29:48.380 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:29:48.380 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:29:48.381 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:29:48.381 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:29:48.381 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:29:48.381 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:29:48.381 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T17:29:48.381 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:29:48.381 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:29:48.381 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1724002181.154947_1688.1", 2024-08-18T17:29:48.381 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:29:48.381 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:29:48.381 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:29:48.381 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T17:29:41.154947Z", 2024-08-18T17:29:48.381 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:29:48.382 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:29:48.382 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:29:48.382 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:29:48.382 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:29:48.382 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T17:29:48.382 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 1, 2024-08-18T17:29:48.382 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 1 2024-08-18T17:29:48.382 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:29:48.382 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T17:29:48Z" 2024-08-18T17:29:48.382 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T17:29:48.383 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-08-18T17:29:48.384 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:29:48.384 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:29:48.385 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:29:48.437 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:29:48.437 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:29:48.558 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T17:29:48.558 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:29:48.558 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000052574", 2024-08-18T17:29:48.559 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T17:29:47.854757Z" 2024-08-18T17:29:48.559 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:29:48.559 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:29:48.559 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000037555", 2024-08-18T17:29:48.559 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T17:27:37.043496Z" 2024-08-18T17:29:48.559 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:29:48.559 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:29:48.559 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000060117", 2024-08-18T17:29:48.559 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T17:29:47.826846Z" 2024-08-18T17:29:48.559 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:29:48.559 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:29:48.559 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:29:48.560 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:29:48.560 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:29:48.560 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T17:29:48.561 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-08-18T17:29:48.561 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:29:48.561 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:29:48.561 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:29:48.612 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:29:48.612 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:29:48.659 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:29:48.659 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:29:48.659 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:29:48.659 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:29:48.659 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:29:48.659 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:29:48.659 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:48.660 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:29:48.660 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:29:48.660 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:29:48.660 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:29:48.660 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:29:48.660 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052154", 2024-08-18T17:29:48.660 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:29:48.660 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:29:48.660 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:29:48.660 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:27:36.982158Z" 2024-08-18T17:29:48.660 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:29:48.661 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:48.661 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:29:48.661 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:29:48.661 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:29:48.661 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:29:48.661 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037555", 2024-08-18T17:29:48.661 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:29:48.661 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:29:48.661 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:29:48.661 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:27:37.043496Z" 2024-08-18T17:29:48.661 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:29:48.661 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:48.661 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:29:48.662 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:29:48.662 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:29:48.662 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:29:48.662 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000059977", 2024-08-18T17:29:48.662 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:29:48.662 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:29:48.662 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:29:48.662 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:27:37.022632Z" 2024-08-18T17:29:48.662 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:29:48.662 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:48.662 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:29:48.662 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:29:48.663 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:29:48.663 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:29:48.663 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:29:48.663 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:29:48.663 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:29:48.663 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:29:48.663 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:29:48.663 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:29:48.663 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:29:48.663 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:29:48.663 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:48.663 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:29:48.664 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:29:48.664 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:29:48.664 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:48.664 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:29:48Z" 2024-08-18T17:29:48.664 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:29:48.665 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000052574 target=00000000000000000000:00000000000000052154, shard=2 master=00000000000000000000:00000000000000060117 target=00000000000000000000:00000000000000059977 2024-08-18T17:29:53.672 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:29:53.672 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:29:53.673 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:29:53.731 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:29:53.731 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:29:53.782 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:29:53.782 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:29:53.782 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:29:53.782 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:29:53.782 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:29:53.782 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:29:53.782 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:53.782 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:29:53.783 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:29:53.783 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:29:53.783 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:29:53.783 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:29:53.783 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052154", 2024-08-18T17:29:53.783 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:29:53.783 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:29:53.783 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:29:53.783 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:27:36.982158Z" 2024-08-18T17:29:53.783 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:29:53.783 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:53.783 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:29:53.783 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:29:53.783 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:29:53.784 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:29:53.784 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037555", 2024-08-18T17:29:53.784 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:29:53.784 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:29:53.784 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:29:53.784 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:27:37.043496Z" 2024-08-18T17:29:53.784 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:29:53.784 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:53.784 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:29:53.784 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:29:53.784 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:29:53.784 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:29:53.784 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000059977", 2024-08-18T17:29:53.784 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:29:53.785 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:29:53.785 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:29:53.785 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:27:37.022632Z" 2024-08-18T17:29:53.785 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:29:53.785 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:53.785 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:29:53.785 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:29:53.785 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:29:53.785 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:29:53.785 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:29:53.785 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:29:53.785 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:29:53.785 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:29:53.786 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:29:53.786 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:29:53.786 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:29:53.786 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:29:53.786 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:53.786 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:29:53.786 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:29:53.786 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:29:53.786 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:53.787 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:29:53Z" 2024-08-18T17:29:53.787 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:29:53.788 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000052574 target=00000000000000000000:00000000000000052154, shard=2 master=00000000000000000000:00000000000000060117 target=00000000000000000000:00000000000000059977 2024-08-18T17:29:58.789 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:29:58.789 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:29:58.789 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:29:58.846 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:29:58.846 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:29:58.897 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:29:58.897 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:29:58.897 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:29:58.898 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:29:58.898 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:29:58.898 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:29:58.898 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:58.898 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:29:58.898 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:29:58.898 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:29:58.898 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:29:58.898 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:29:58.898 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052154", 2024-08-18T17:29:58.898 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:29:58.898 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:29:58.899 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:29:58.899 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:27:36.982158Z" 2024-08-18T17:29:58.899 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:29:58.899 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:58.899 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:29:58.899 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:29:58.899 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:29:58.899 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:29:58.899 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037555", 2024-08-18T17:29:58.899 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:29:58.899 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:29:58.899 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:29:58.899 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:27:37.043496Z" 2024-08-18T17:29:58.899 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:29:58.900 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:58.900 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:29:58.900 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:29:58.900 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:29:58.900 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:29:58.900 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000059977", 2024-08-18T17:29:58.900 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:29:58.900 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:29:58.900 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:29:58.900 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:27:37.022632Z" 2024-08-18T17:29:58.900 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:29:58.900 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:58.900 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:29:58.900 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:29:58.901 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:29:58.901 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:29:58.901 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:29:58.901 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:29:58.901 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:29:58.901 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:29:58.901 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:29:58.901 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:29:58.901 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:29:58.901 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:29:58.901 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:58.901 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:29:58.901 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:29:58.901 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:29:58.902 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:29:58.902 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:29:58Z" 2024-08-18T17:29:58.902 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:29:58.903 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000052574 target=00000000000000000000:00000000000000052154, shard=2 master=00000000000000000000:00000000000000060117 target=00000000000000000000:00000000000000059977 2024-08-18T17:30:03.903 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:30:03.904 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:30:03.904 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:30:03.961 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:30:03.961 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:30:04.012 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:30:04.012 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:30:04.012 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:30:04.012 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:30:04.012 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:30:04.012 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:30:04.012 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:30:04.012 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:30:04.012 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:30:04.012 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:30:04.013 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:30:04.013 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:30:04.013 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052154", 2024-08-18T17:30:04.013 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:30:04.013 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:30:04.013 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:30:04.013 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:27:36.982158Z" 2024-08-18T17:30:04.013 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:30:04.013 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:30:04.013 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:30:04.013 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:30:04.013 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:30:04.013 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:30:04.014 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037555", 2024-08-18T17:30:04.014 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:30:04.014 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:30:04.014 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:30:04.014 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:27:37.043496Z" 2024-08-18T17:30:04.014 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:30:04.014 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:30:04.014 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:30:04.014 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:30:04.014 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:30:04.014 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:30:04.014 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000059977", 2024-08-18T17:30:04.014 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:30:04.014 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:30:04.015 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:30:04.015 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:27:37.022632Z" 2024-08-18T17:30:04.015 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:30:04.015 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:30:04.015 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:30:04.015 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:30:04.015 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:30:04.015 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:30:04.015 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:30:04.015 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:30:04.015 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:30:04.016 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:30:04.016 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:30:04.016 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:30:04.016 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:30:04.016 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:30:04.016 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:30:04.016 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:30:04.016 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:30:04.016 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:30:04.016 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:30:04.016 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:30:03Z" 2024-08-18T17:30:04.016 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:30:04.017 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000052574 target=00000000000000000000:00000000000000052154, shard=2 master=00000000000000000000:00000000000000060117 target=00000000000000000000:00000000000000059977 2024-08-18T17:30:09.018 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:30:09.018 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:30:09.019 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:30:09.076 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:30:09.076 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:30:09.127 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:30:09.127 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:30:09.128 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:30:09.128 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:30:09.128 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:30:09.128 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:30:09.128 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:30:09.128 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:30:09.128 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:30:09.128 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:30:09.128 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:30:09.128 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:30:09.128 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052154", 2024-08-18T17:30:09.128 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:30:09.129 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:30:09.129 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:30:09.129 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:27:36.982158Z" 2024-08-18T17:30:09.129 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:30:09.129 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:30:09.129 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:30:09.129 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:30:09.129 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:30:09.129 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:30:09.129 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037555", 2024-08-18T17:30:09.129 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:30:09.129 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:30:09.129 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:30:09.130 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:27:37.043496Z" 2024-08-18T17:30:09.130 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:30:09.130 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:30:09.130 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:30:09.130 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:30:09.130 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:30:09.130 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:30:09.130 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000059977", 2024-08-18T17:30:09.130 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:30:09.130 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:30:09.130 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:30:09.130 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:27:37.022632Z" 2024-08-18T17:30:09.131 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:30:09.131 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:30:09.131 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:30:09.131 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:30:09.131 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:30:09.131 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:30:09.131 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:30:09.131 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:30:09.131 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:30:09.131 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:30:09.131 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:30:09.131 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:30:09.131 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:30:09.132 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:30:09.132 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:30:09.132 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:30:09.132 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:30:09.132 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:30:09.132 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:30:09.132 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:30:09Z" 2024-08-18T17:30:09.132 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:30:09.133 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000052574 target=00000000000000000000:00000000000000052154, shard=2 master=00000000000000000000:00000000000000060117 target=00000000000000000000:00000000000000059977 2024-08-18T17:30:14.134 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:30:14.135 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:30:14.135 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:30:14.194 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:30:14.194 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:30:14.248 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:30:14.249 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:30:14.249 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:30:14.249 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:30:14.249 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:30:14.249 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:30:14.249 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:30:14.249 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:30:14.249 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:30:14.249 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:30:14.249 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:30:14.249 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:30:14.249 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052154", 2024-08-18T17:30:14.250 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:30:14.250 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:30:14.250 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:30:14.250 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:27:36.982158Z" 2024-08-18T17:30:14.250 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:30:14.250 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:30:14.250 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:30:14.250 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:30:14.250 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:30:14.250 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:30:14.250 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037555", 2024-08-18T17:30:14.250 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:30:14.251 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:30:14.251 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:30:14.251 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:27:37.043496Z" 2024-08-18T17:30:14.251 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:30:14.251 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:30:14.251 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:30:14.251 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:30:14.251 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:30:14.251 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:30:14.251 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000059977", 2024-08-18T17:30:14.251 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:30:14.251 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:30:14.251 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:30:14.251 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:27:37.022632Z" 2024-08-18T17:30:14.252 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:30:14.252 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:30:14.252 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:30:14.252 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:30:14.252 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:30:14.252 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:30:14.252 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:30:14.252 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:30:14.252 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:30:14.252 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:30:14.252 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:30:14.252 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:30:14.252 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:30:14.253 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:30:14.253 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:30:14.253 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:30:14.253 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:30:14.253 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:30:14.253 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:30:14.253 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:30:14Z" 2024-08-18T17:30:14.253 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:30:14.254 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000052574 target=00000000000000000000:00000000000000052154, shard=2 master=00000000000000000000:00000000000000060117 target=00000000000000000000:00000000000000059977 2024-08-18T17:30:19.255 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:30:19.256 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:30:19.256 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:30:19.314 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:30:19.314 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:30:19.364 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:30:19.364 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:30:19.364 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:30:19.364 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:30:19.364 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:30:19.364 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:30:19.364 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:30:19.364 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:30:19.364 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:30:19.365 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:30:19.365 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:30:19.365 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:30:19.365 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052154", 2024-08-18T17:30:19.365 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:30:19.365 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:30:19.365 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:30:19.365 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:27:36.982158Z" 2024-08-18T17:30:19.365 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:30:19.365 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:30:19.365 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:30:19.365 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:30:19.365 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:30:19.366 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:30:19.366 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037555", 2024-08-18T17:30:19.366 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:30:19.366 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:30:19.366 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:30:19.366 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:27:37.043496Z" 2024-08-18T17:30:19.366 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:30:19.366 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:30:19.366 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:30:19.366 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:30:19.366 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:30:19.366 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:30:19.366 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000059977", 2024-08-18T17:30:19.367 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:30:19.367 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:30:19.367 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:30:19.367 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:27:37.022632Z" 2024-08-18T17:30:19.367 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:30:19.367 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:30:19.367 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:30:19.367 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:30:19.367 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:30:19.367 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:30:19.367 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:30:19.367 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:30:19.367 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:30:19.367 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:30:19.368 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:30:19.368 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:30:19.368 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:30:19.368 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:30:19.368 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:30:19.368 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:30:19.368 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:30:19.368 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:30:19.368 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:30:19.368 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:30:19Z" 2024-08-18T17:30:19.368 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:30:19.369 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000052574 target=00000000000000000000:00000000000000052154, shard=2 master=00000000000000000000:00000000000000060117 target=00000000000000000000:00000000000000059977 2024-08-18T17:30:24.370 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:30:24.370 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:30:24.371 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:30:24.430 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:30:24.430 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:30:24.479 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:30:24.479 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:30:24.479 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:30:24.479 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:30:24.479 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:30:24.479 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:30:24.479 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:30:24.479 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:30:24.479 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:30:24.479 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:30:24.480 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:30:24.480 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:30:24.480 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052154", 2024-08-18T17:30:24.480 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:30:24.480 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:30:24.480 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:30:24.480 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:27:36.982158Z" 2024-08-18T17:30:24.480 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:30:24.480 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:30:24.480 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:30:24.480 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:30:24.480 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:30:24.480 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:30:24.481 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037555", 2024-08-18T17:30:24.481 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:30:24.481 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:30:24.481 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:30:24.481 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:27:37.043496Z" 2024-08-18T17:30:24.481 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:30:24.481 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:30:24.481 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:30:24.481 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:30:24.481 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:30:24.481 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:30:24.481 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000059977", 2024-08-18T17:30:24.481 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:30:24.481 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:30:24.482 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:30:24.482 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:27:37.022632Z" 2024-08-18T17:30:24.482 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:30:24.482 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:30:24.482 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:30:24.482 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:30:24.482 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:30:24.482 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:30:24.482 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:30:24.482 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:30:24.482 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:30:24.482 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:30:24.482 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:30:24.483 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:30:24.483 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:30:24.483 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:30:24.483 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:30:24.483 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:30:24.483 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:30:24.483 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:30:24.483 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:30:24.483 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:30:24Z" 2024-08-18T17:30:24.483 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:30:24.484 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000052574 target=00000000000000000000:00000000000000052154, shard=2 master=00000000000000000000:00000000000000060117 target=00000000000000000000:00000000000000059977 2024-08-18T17:30:29.485 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:30:29.486 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:30:29.486 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:30:29.543 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:30:29.543 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:30:29.595 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:30:29.595 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:30:29.595 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:30:29.595 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:30:29.595 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:30:29.595 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:30:29.595 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:30:29.596 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:30:29.596 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:30:29.596 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:30:29.596 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:30:29.596 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:30:29.596 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052154", 2024-08-18T17:30:29.596 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:30:29.596 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:30:29.596 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:30:29.596 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:27:36.982158Z" 2024-08-18T17:30:29.596 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:30:29.596 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:30:29.596 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:30:29.597 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:30:29.597 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:30:29.597 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:30:29.597 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037555", 2024-08-18T17:30:29.597 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:30:29.597 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:30:29.597 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:30:29.597 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:27:37.043496Z" 2024-08-18T17:30:29.597 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:30:29.597 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:30:29.597 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:30:29.597 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:30:29.597 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:30:29.598 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:30:29.598 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000059977", 2024-08-18T17:30:29.598 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:30:29.598 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:30:29.598 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:30:29.598 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:27:37.022632Z" 2024-08-18T17:30:29.598 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:30:29.598 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:30:29.598 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:30:29.598 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:30:29.598 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:30:29.598 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:30:29.599 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:30:29.599 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:30:29.599 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:30:29.599 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:30:29.599 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:30:29.599 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:30:29.599 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:30:29.599 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:30:29.599 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:30:29.599 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:30:29.599 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:30:29.599 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:30:29.599 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:30:29.599 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:30:29Z" 2024-08-18T17:30:29.600 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:30:29.601 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000052574 target=00000000000000000000:00000000000000052154, shard=2 master=00000000000000000000:00000000000000060117 target=00000000000000000000:00000000000000059977 2024-08-18T17:30:34.601 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:30:34.601 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:30:34.602 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:30:34.662 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:30:34.662 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:30:34.712 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:30:34.712 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:30:34.712 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:30:34.712 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:30:34.712 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:30:34.712 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:30:34.713 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:30:34.713 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:30:34.713 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:30:34.713 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:30:34.713 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:30:34.713 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:30:34.713 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052154", 2024-08-18T17:30:34.713 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:30:34.713 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:30:34.713 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:30:34.713 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:27:36.982158Z" 2024-08-18T17:30:34.713 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:30:34.713 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:30:34.714 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:30:34.714 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:30:34.714 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:30:34.714 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:30:34.714 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037555", 2024-08-18T17:30:34.714 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:30:34.714 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:30:34.714 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:30:34.714 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:27:37.043496Z" 2024-08-18T17:30:34.714 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:30:34.714 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:30:34.714 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:30:34.715 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:30:34.715 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:30:34.715 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:30:34.715 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000059977", 2024-08-18T17:30:34.715 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:30:34.715 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:30:34.715 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:30:34.715 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:27:37.022632Z" 2024-08-18T17:30:34.715 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:30:34.715 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:30:34.715 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:30:34.715 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:30:34.715 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:30:34.715 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:30:34.716 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:30:34.716 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:30:34.716 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:30:34.716 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:30:34.716 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:30:34.716 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:30:34.716 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:30:34.716 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:30:34.716 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:30:34.716 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:30:34.716 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:30:34.716 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:30:34.716 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:30:34.716 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:30:34Z" 2024-08-18T17:30:34.717 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:30:34.718 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000052574 target=00000000000000000000:00000000000000052154, shard=2 master=00000000000000000000:00000000000000060117 target=00000000000000000000:00000000000000059977 2024-08-18T17:30:39.718 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:30:39.719 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:30:39.719 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:30:39.777 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:30:39.777 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:30:39.827 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:30:39.827 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:30:39.828 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:30:39.828 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:30:39.828 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:30:39.828 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:30:39.828 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:30:39.828 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:30:39.828 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:30:39.828 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:30:39.828 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:30:39.828 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:30:39.828 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052154", 2024-08-18T17:30:39.828 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:30:39.828 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:30:39.829 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:30:39.829 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:27:36.982158Z" 2024-08-18T17:30:39.829 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:30:39.829 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:30:39.829 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:30:39.829 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:30:39.829 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:30:39.829 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:30:39.829 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037555", 2024-08-18T17:30:39.829 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:30:39.829 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:30:39.829 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:30:39.829 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:27:37.043496Z" 2024-08-18T17:30:39.830 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:30:39.830 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:30:39.830 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:30:39.830 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:30:39.830 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:30:39.830 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:30:39.830 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000059977", 2024-08-18T17:30:39.830 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:30:39.830 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:30:39.830 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:30:39.830 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:27:37.022632Z" 2024-08-18T17:30:39.830 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:30:39.831 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:30:39.831 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:30:39.831 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:30:39.831 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:30:39.831 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:30:39.831 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:30:39.831 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:30:39.831 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:30:39.831 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:30:39.831 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:30:39.831 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:30:39.831 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:30:39.831 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:30:39.831 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:30:39.832 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:30:39.832 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:30:39.832 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:30:39.832 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:30:39.832 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:30:39Z" 2024-08-18T17:30:39.832 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:30:39.833 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000052574 target=00000000000000000000:00000000000000052154, shard=2 master=00000000000000000000:00000000000000060117 target=00000000000000000000:00000000000000059977 2024-08-18T17:30:44.834 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:30:44.834 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:30:44.834 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:30:44.892 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:30:44.892 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:30:44.943 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:30:44.943 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:30:44.944 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:30:44.944 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:30:44.944 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:30:44.944 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:30:44.944 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:30:44.944 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:30:44.944 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:30:44.944 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:30:44.944 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:30:44.944 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:30:44.944 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052154", 2024-08-18T17:30:44.944 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:30:44.945 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:30:44.945 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:30:44.945 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:27:36.982158Z" 2024-08-18T17:30:44.945 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:30:44.945 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:30:44.945 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:30:44.945 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:30:44.945 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:30:44.945 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:30:44.945 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037555", 2024-08-18T17:30:44.945 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:30:44.945 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:30:44.945 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:30:44.945 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:27:37.043496Z" 2024-08-18T17:30:44.946 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:30:44.946 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:30:44.946 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:30:44.946 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:30:44.946 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:30:44.946 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:30:44.946 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000059977", 2024-08-18T17:30:44.946 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:30:44.946 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:30:44.946 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:30:44.946 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:27:37.022632Z" 2024-08-18T17:30:44.946 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:30:44.946 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:30:44.947 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:30:44.947 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:30:44.947 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:30:44.947 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:30:44.947 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:30:44.947 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:30:44.947 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:30:44.947 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:30:44.947 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:30:44.947 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:30:44.947 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:30:44.947 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:30:44.948 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:30:44.948 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:30:44.948 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:30:44.948 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:30:44.948 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:30:44.948 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:30:44Z" 2024-08-18T17:30:44.948 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:30:44.949 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000052574 target=00000000000000000000:00000000000000052154, shard=2 master=00000000000000000000:00000000000000060117 target=00000000000000000000:00000000000000059977 2024-08-18T17:30:49.950 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:30:49.950 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:30:49.950 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:30:50.007 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:30:50.007 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:30:50.059 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:30:50.059 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:30:50.059 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:30:50.059 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:30:50.059 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:30:50.059 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:30:50.060 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:30:50.060 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:30:50.060 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:30:50.060 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:30:50.060 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:30:50.060 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:30:50.060 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052154", 2024-08-18T17:30:50.060 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:30:50.060 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:30:50.060 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:30:50.061 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:27:36.982158Z" 2024-08-18T17:30:50.061 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:30:50.061 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:30:50.061 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:30:50.061 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:30:50.061 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:30:50.061 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:30:50.061 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037555", 2024-08-18T17:30:50.061 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:30:50.062 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:30:50.062 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:30:50.062 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:27:37.043496Z" 2024-08-18T17:30:50.062 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:30:50.062 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:30:50.062 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:30:50.062 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:30:50.062 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:30:50.062 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:30:50.062 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000059977", 2024-08-18T17:30:50.063 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:30:50.063 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:30:50.063 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:30:50.063 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:27:37.022632Z" 2024-08-18T17:30:50.063 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:30:50.063 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:30:50.063 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:30:50.063 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:30:50.063 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:30:50.063 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:30:50.064 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:30:50.064 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:30:50.064 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:30:50.064 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:30:50.064 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:30:50.064 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:30:50.064 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:30:50.064 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:30:50.064 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:30:50.064 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:30:50.064 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:30:50.064 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:30:50.065 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:30:50.065 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:30:50Z" 2024-08-18T17:30:50.065 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:30:50.066 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000052574 target=00000000000000000000:00000000000000052154, shard=2 master=00000000000000000000:00000000000000060117 target=00000000000000000000:00000000000000059977 2024-08-18T17:30:55.067 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:30:55.067 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:30:55.067 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:30:55.124 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:30:55.124 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:30:55.176 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:30:55.176 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:30:55.176 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:30:55.176 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:30:55.176 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:30:55.176 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:30:55.176 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:30:55.177 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:30:55.177 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:30:55.177 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:30:55.177 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:30:55.177 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:30:55.177 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052154", 2024-08-18T17:30:55.177 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:30:55.177 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:30:55.177 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:30:55.177 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:27:36.982158Z" 2024-08-18T17:30:55.177 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:30:55.177 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:30:55.177 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:30:55.178 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:30:55.178 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:30:55.178 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:30:55.178 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037555", 2024-08-18T17:30:55.178 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:30:55.178 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:30:55.178 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:30:55.178 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:27:37.043496Z" 2024-08-18T17:30:55.178 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:30:55.178 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:30:55.178 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:30:55.178 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:30:55.178 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:30:55.179 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:30:55.179 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000059977", 2024-08-18T17:30:55.179 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:30:55.179 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:30:55.179 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:30:55.179 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:27:37.022632Z" 2024-08-18T17:30:55.179 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:30:55.180 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:30:55.180 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:30:55.180 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:30:55.180 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:30:55.180 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:30:55.180 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:30:55.180 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:30:55.180 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:30:55.180 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:30:55.180 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:30:55.180 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:30:55.180 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:30:55.181 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:30:55.181 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:30:55.181 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:30:55.181 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:30:55.181 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:30:55.181 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:30:55.181 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:30:55Z" 2024-08-18T17:30:55.181 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:30:55.182 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000052574 target=00000000000000000000:00000000000000052154, shard=2 master=00000000000000000000:00000000000000060117 target=00000000000000000000:00000000000000059977 2024-08-18T17:31:00.183 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:31:00.184 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:31:00.184 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:31:00.240 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:31:00.240 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:31:00.291 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:31:00.291 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:31:00.291 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:31:00.291 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:31:00.291 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:31:00.291 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:31:00.291 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:31:00.291 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:31:00.292 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:31:00.292 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:31:00.292 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:31:00.292 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:31:00.292 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052154", 2024-08-18T17:31:00.292 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:31:00.292 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:31:00.292 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:31:00.292 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:27:36.982158Z" 2024-08-18T17:31:00.292 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:31:00.293 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:31:00.293 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:31:00.293 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:31:00.293 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:31:00.293 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:31:00.293 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037555", 2024-08-18T17:31:00.293 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:31:00.293 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:31:00.293 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:31:00.293 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:27:37.043496Z" 2024-08-18T17:31:00.294 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:31:00.294 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:31:00.294 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:31:00.294 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:31:00.294 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:31:00.294 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:31:00.294 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000059977", 2024-08-18T17:31:00.294 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:31:00.294 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:31:00.294 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:31:00.295 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:27:37.022632Z" 2024-08-18T17:31:00.295 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:31:00.295 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:31:00.295 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:31:00.295 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:31:00.295 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:31:00.295 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:31:00.296 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:31:00.296 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:31:00.296 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:31:00.296 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:31:00.296 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:31:00.297 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:31:00.297 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:31:00.297 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:31:00.297 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:31:00.297 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:31:00.297 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:31:00.298 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:31:00.298 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:31:00.298 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:31:00Z" 2024-08-18T17:31:00.298 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:31:00.300 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000052574 target=00000000000000000000:00000000000000052154, shard=2 master=00000000000000000000:00000000000000060117 target=00000000000000000000:00000000000000059977 2024-08-18T17:31:05.300 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:31:05.301 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:31:05.301 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:31:05.358 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:31:05.359 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:31:05.412 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:31:05.413 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:31:05.413 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:31:05.413 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:31:05.413 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:31:05.413 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:31:05.413 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:31:05.413 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:31:05.413 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:31:05.413 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:31:05.413 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:31:05.414 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:31:05.414 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052154", 2024-08-18T17:31:05.414 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:31:05.414 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:31:05.414 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:31:05.414 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:27:36.982158Z" 2024-08-18T17:31:05.414 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:31:05.414 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:31:05.414 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:31:05.414 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:31:05.414 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:31:05.414 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:31:05.415 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037555", 2024-08-18T17:31:05.415 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:31:05.415 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:31:05.415 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:31:05.415 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:27:37.043496Z" 2024-08-18T17:31:05.415 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:31:05.415 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:31:05.415 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:31:05.415 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:31:05.415 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:31:05.415 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:31:05.415 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000059977", 2024-08-18T17:31:05.415 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:31:05.416 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:31:05.416 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:31:05.416 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:27:37.022632Z" 2024-08-18T17:31:05.416 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:31:05.416 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:31:05.416 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:31:05.416 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:31:05.416 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:31:05.416 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:31:05.416 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:31:05.416 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:31:05.417 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:31:05.417 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:31:05.417 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:31:05.417 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:31:05.417 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:31:05.417 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:31:05.417 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:31:05.417 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:31:05.418 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:31:05.418 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:31:05.418 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:31:05.418 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:31:05Z" 2024-08-18T17:31:05.418 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:31:05.419 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000052574 target=00000000000000000000:00000000000000052154, shard=2 master=00000000000000000000:00000000000000060117 target=00000000000000000000:00000000000000059977 2024-08-18T17:31:10.420 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:31:10.421 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:31:10.421 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:31:10.481 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:31:10.481 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:31:10.533 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:31:10.533 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:31:10.533 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:31:10.533 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:31:10.533 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:31:10.533 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:31:10.534 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:31:10.534 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:31:10.534 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:31:10.534 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:31:10.534 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:31:10.534 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:31:10.534 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052154", 2024-08-18T17:31:10.534 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:31:10.534 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:31:10.534 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:31:10.534 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:27:36.982158Z" 2024-08-18T17:31:10.534 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:31:10.535 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:31:10.535 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:31:10.535 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:31:10.535 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:31:10.535 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:31:10.535 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037555", 2024-08-18T17:31:10.535 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:31:10.535 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:31:10.535 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:31:10.535 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:27:37.043496Z" 2024-08-18T17:31:10.535 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:31:10.535 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:31:10.535 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:31:10.535 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:31:10.536 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:31:10.536 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:31:10.536 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000059977", 2024-08-18T17:31:10.536 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:31:10.538 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:31:10.538 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:31:10.538 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:27:37.022632Z" 2024-08-18T17:31:10.538 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:31:10.538 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:31:10.538 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:31:10.538 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:31:10.538 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:31:10.538 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:31:10.538 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:31:10.539 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:31:10.539 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:31:10.539 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:31:10.539 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:31:10.539 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:31:10.539 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:31:10.539 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:31:10.539 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:31:10.539 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:31:10.539 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:31:10.539 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:31:10.539 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:31:10.539 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:31:10Z" 2024-08-18T17:31:10.540 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:31:10.541 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000052574 target=00000000000000000000:00000000000000052154, shard=2 master=00000000000000000000:00000000000000060117 target=00000000000000000000:00000000000000059977 2024-08-18T17:31:15.541 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:31:15.542 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:31:15.542 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:31:15.599 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:31:15.599 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:31:15.650 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:31:15.651 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:31:15.651 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:31:15.651 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:31:15.651 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:31:15.651 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:31:15.651 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:31:15.651 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:31:15.651 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:31:15.651 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:31:15.651 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:31:15.651 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:31:15.651 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052154", 2024-08-18T17:31:15.652 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:31:15.652 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:31:15.652 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:31:15.652 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:27:36.982158Z" 2024-08-18T17:31:15.652 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:31:15.652 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:31:15.652 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:31:15.652 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:31:15.652 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:31:15.652 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:31:15.652 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037555", 2024-08-18T17:31:15.652 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:31:15.652 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:31:15.653 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:31:15.653 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:27:37.043496Z" 2024-08-18T17:31:15.653 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:31:15.653 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:31:15.653 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:31:15.653 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:31:15.653 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:31:15.653 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:31:15.653 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000059977", 2024-08-18T17:31:15.653 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:31:15.653 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:31:15.653 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:31:15.653 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:27:37.022632Z" 2024-08-18T17:31:15.654 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:31:15.654 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:31:15.654 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:31:15.654 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:31:15.654 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:31:15.654 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:31:15.654 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:31:15.654 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:31:15.654 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:31:15.655 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:31:15.655 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:31:15.655 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:31:15.655 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:31:15.655 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:31:15.655 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:31:15.655 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:31:15.655 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:31:15.655 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:31:15.655 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:31:15.655 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:31:15Z" 2024-08-18T17:31:15.655 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:31:15.656 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000052574 target=00000000000000000000:00000000000000052154, shard=2 master=00000000000000000000:00000000000000060117 target=00000000000000000000:00000000000000059977 2024-08-18T17:31:20.657 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:31:20.658 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:31:20.658 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:31:20.715 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:31:20.715 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:31:20.765 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:31:20.765 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:31:20.766 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:31:20.766 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:31:20.766 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:31:20.766 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:31:20.766 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:31:20.766 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:31:20.766 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:31:20.766 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:31:20.766 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:31:20.766 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:31:20.766 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052154", 2024-08-18T17:31:20.767 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:31:20.767 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:31:20.767 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:31:20.767 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:27:36.982158Z" 2024-08-18T17:31:20.767 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:31:20.767 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:31:20.767 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:31:20.767 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:31:20.768 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:31:20.768 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:31:20.768 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037555", 2024-08-18T17:31:20.768 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:31:20.768 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:31:20.769 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:31:20.769 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:27:37.043496Z" 2024-08-18T17:31:20.769 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:31:20.769 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:31:20.769 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:31:20.769 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:31:20.769 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:31:20.769 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:31:20.769 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000059977", 2024-08-18T17:31:20.769 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:31:20.770 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:31:20.770 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:31:20.770 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:27:37.022632Z" 2024-08-18T17:31:20.770 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:31:20.770 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:31:20.770 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:31:20.770 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:31:20.770 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:31:20.771 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:31:20.771 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:31:20.771 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:31:20.771 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:31:20.771 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:31:20.771 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:31:20.771 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:31:20.771 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:31:20.771 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:31:20.771 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:31:20.771 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:31:20.772 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:31:20.772 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:31:20.772 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:31:20.772 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:31:20Z" 2024-08-18T17:31:20.772 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:31:20.773 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000052574 target=00000000000000000000:00000000000000052154, shard=2 master=00000000000000000000:00000000000000060117 target=00000000000000000000:00000000000000059977 2024-08-18T17:31:25.774 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:31:25.774 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:31:25.775 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:31:25.832 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:31:25.832 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:31:25.884 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:31:25.884 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:31:25.884 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:31:25.884 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:31:25.884 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:31:25.884 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:31:25.884 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:31:25.884 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:31:25.885 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:31:25.885 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:31:25.885 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:31:25.885 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:31:25.885 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052154", 2024-08-18T17:31:25.885 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:31:25.885 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:31:25.885 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:31:25.885 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:27:36.982158Z" 2024-08-18T17:31:25.885 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:31:25.885 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:31:25.885 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:31:25.885 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:31:25.885 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:31:25.886 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:31:25.886 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037555", 2024-08-18T17:31:25.886 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:31:25.886 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:31:25.886 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:31:25.886 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:27:37.043496Z" 2024-08-18T17:31:25.886 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:31:25.886 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:31:25.886 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:31:25.886 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:31:25.886 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:31:25.886 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:31:25.887 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000059977", 2024-08-18T17:31:25.887 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:31:25.887 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:31:25.887 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:31:25.887 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:27:37.022632Z" 2024-08-18T17:31:25.887 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:31:25.887 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:31:25.887 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:31:25.887 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:31:25.887 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:31:25.887 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:31:25.887 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:31:25.887 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:31:25.887 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:31:25.888 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:31:25.888 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:31:25.888 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:31:25.888 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:31:25.888 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:31:25.888 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:31:25.888 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:31:25.888 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:31:25.888 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:31:25.888 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:31:25.888 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:31:25Z" 2024-08-18T17:31:25.888 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:31:25.889 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000052574 target=00000000000000000000:00000000000000052154, shard=2 master=00000000000000000000:00000000000000060117 target=00000000000000000000:00000000000000059977 2024-08-18T17:31:30.890 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:31:30.890 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:31:30.891 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:31:30.950 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:31:30.950 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:31:31.004 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:31:31.005 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:31:31.005 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:31:31.005 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:31:31.005 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:31:31.005 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:31:31.005 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:31:31.005 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:31:31.005 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:31:31.005 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:31:31.005 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:31:31.005 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:31:31.005 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052154", 2024-08-18T17:31:31.005 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:31:31.006 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:31:31.006 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:31:31.006 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:27:36.982158Z" 2024-08-18T17:31:31.006 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:31:31.006 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:31:31.006 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:31:31.006 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:31:31.006 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:31:31.006 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:31:31.006 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037555", 2024-08-18T17:31:31.006 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:31:31.006 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:31:31.006 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:31:31.007 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:27:37.043496Z" 2024-08-18T17:31:31.007 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:31:31.007 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:31:31.007 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:31:31.007 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:31:31.007 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:31:31.007 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:31:31.007 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000059977", 2024-08-18T17:31:31.007 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:31:31.007 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:31:31.007 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:31:31.007 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:27:37.022632Z" 2024-08-18T17:31:31.007 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:31:31.007 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:31:31.008 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:31:31.008 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:31:31.008 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:31:31.008 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:31:31.008 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:31:31.008 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:31:31.008 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:31:31.008 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:31:31.008 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:31:31.008 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:31:31.008 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:31:31.008 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:31:31.008 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:31:31.009 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:31:31.009 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:31:31.009 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:31:31.009 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:31:31.009 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:31:30Z" 2024-08-18T17:31:31.009 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:31:31.010 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000052574 target=00000000000000000000:00000000000000052154, shard=2 master=00000000000000000000:00000000000000060117 target=00000000000000000000:00000000000000059977 2024-08-18T17:31:36.011 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:31:36.011 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:31:36.012 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:31:36.069 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:31:36.069 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:31:36.119 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:31:36.120 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:31:36.120 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:31:36.120 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:31:36.120 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:31:36.120 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:31:36.120 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:31:36.120 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:31:36.120 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:31:36.120 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:31:36.120 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:31:36.120 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:31:36.120 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052154", 2024-08-18T17:31:36.120 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:31:36.121 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:31:36.121 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:31:36.121 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:27:36.982158Z" 2024-08-18T17:31:36.121 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:31:36.121 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:31:36.121 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:31:36.121 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:31:36.121 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:31:36.121 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:31:36.121 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037555", 2024-08-18T17:31:36.121 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:31:36.121 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:31:36.121 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:31:36.122 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:27:37.043496Z" 2024-08-18T17:31:36.122 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:31:36.122 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:31:36.122 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:31:36.122 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:31:36.122 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:31:36.122 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:31:36.122 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000059977", 2024-08-18T17:31:36.122 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:31:36.122 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:31:36.122 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:31:36.122 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:27:37.022632Z" 2024-08-18T17:31:36.122 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:31:36.123 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:31:36.123 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:31:36.123 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:31:36.123 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:31:36.123 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:31:36.123 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:31:36.123 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:31:36.123 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:31:36.123 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:31:36.123 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:31:36.123 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:31:36.123 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:31:36.123 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:31:36.123 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:31:36.124 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:31:36.124 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:31:36.124 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:31:36.124 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:31:36.124 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:31:36Z" 2024-08-18T17:31:36.124 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:31:36.125 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000052574 target=00000000000000000000:00000000000000052154, shard=2 master=00000000000000000000:00000000000000060117 target=00000000000000000000:00000000000000059977 2024-08-18T17:31:41.125 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:31:41.126 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:31:41.126 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:31:41.183 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:31:41.183 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:31:41.233 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:31:41.234 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:31:41.234 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:31:41.234 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:31:41.234 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:31:41.234 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:31:41.234 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:31:41.234 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:31:41.234 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:31:41.234 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:31:41.234 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:31:41.235 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:31:41.235 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052154", 2024-08-18T17:31:41.235 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:31:41.235 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:31:41.235 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:31:41.235 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:27:36.982158Z" 2024-08-18T17:31:41.235 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:31:41.235 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:31:41.235 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:31:41.235 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:31:41.235 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:31:41.235 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:31:41.235 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037555", 2024-08-18T17:31:41.236 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:31:41.236 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:31:41.236 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:31:41.236 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:27:37.043496Z" 2024-08-18T17:31:41.236 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:31:41.236 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:31:41.236 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:31:41.236 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:31:41.236 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:31:41.236 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:31:41.236 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000059977", 2024-08-18T17:31:41.236 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:31:41.236 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:31:41.237 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:31:41.237 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:27:37.022632Z" 2024-08-18T17:31:41.237 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:31:41.237 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:31:41.237 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:31:41.237 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:31:41.237 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:31:41.237 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:31:41.237 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:31:41.237 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:31:41.238 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:31:41.238 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:31:41.238 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:31:41.238 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:31:41.238 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:31:41.238 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:31:41.238 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:31:41.238 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:31:41.238 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:31:41.238 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:31:41.238 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:31:41.238 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:31:41Z" 2024-08-18T17:31:41.238 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:31:41.240 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000052574 target=00000000000000000000:00000000000000052154, shard=2 master=00000000000000000000:00000000000000060117 target=00000000000000000000:00000000000000059977 2024-08-18T17:31:46.241 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:31:46.241 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:31:46.241 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:31:46.297 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:31:46.298 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:31:46.347 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:31:46.347 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:31:46.347 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:31:46.348 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:31:46.348 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:31:46.348 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:31:46.348 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:31:46.348 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:31:46.348 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:31:46.348 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:31:46.348 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:31:46.348 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:31:46.348 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052154", 2024-08-18T17:31:46.348 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:31:46.348 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:31:46.349 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:31:46.349 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:27:36.982158Z" 2024-08-18T17:31:46.349 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:31:46.349 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:31:46.349 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:31:46.349 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:31:46.349 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:31:46.349 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:31:46.349 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037555", 2024-08-18T17:31:46.349 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:31:46.350 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:31:46.350 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:31:46.350 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:27:37.043496Z" 2024-08-18T17:31:46.350 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:31:46.350 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:31:46.350 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:31:46.350 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:31:46.350 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:31:46.350 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:31:46.350 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000059977", 2024-08-18T17:31:46.350 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:31:46.350 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:31:46.351 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:31:46.351 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:27:37.022632Z" 2024-08-18T17:31:46.351 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:31:46.351 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:31:46.351 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:31:46.351 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:31:46.351 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:31:46.351 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:31:46.351 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:31:46.351 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:31:46.351 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:31:46.351 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:31:46.351 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:31:46.352 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:31:46.352 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:31:46.352 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:31:46.352 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:31:46.352 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:31:46.352 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:31:46.352 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:31:46.352 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:31:46.352 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:31:46Z" 2024-08-18T17:31:46.352 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:31:46.353 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000052574 target=00000000000000000000:00000000000000052154, shard=2 master=00000000000000000000:00000000000000060117 target=00000000000000000000:00000000000000059977 2024-08-18T17:31:51.355 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:31:51.355 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:31:51.355 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:31:51.414 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:31:51.415 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:31:51.466 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:31:51.466 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:31:51.466 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:31:51.466 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:31:51.466 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:31:51.467 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:31:51.467 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:31:51.467 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:31:51.467 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:31:51.467 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:31:51.467 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:31:51.468 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:31:51.468 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052574", 2024-08-18T17:31:51.468 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:31:51.468 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:31:51.468 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:31:51.468 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:29:47.854757Z" 2024-08-18T17:31:51.468 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:31:51.468 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:31:51.468 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:31:51.468 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:31:51.468 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:31:51.469 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:31:51.469 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037555", 2024-08-18T17:31:51.469 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:31:51.469 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:31:51.469 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:31:51.469 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:27:37.043496Z" 2024-08-18T17:31:51.469 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:31:51.469 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:31:51.469 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:31:51.469 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:31:51.469 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:31:51.470 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:31:51.470 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000060117", 2024-08-18T17:31:51.470 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:31:51.470 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:31:51.470 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:31:51.470 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:29:47.826846Z" 2024-08-18T17:31:51.470 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:31:51.470 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:31:51.470 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:31:51.470 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:31:51.470 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:31:51.471 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:31:51.471 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:31:51.471 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:31:51.471 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:31:51.471 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:31:51.471 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:31:51.471 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:31:51.471 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:31:51.471 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:31:51.471 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:31:51.471 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:31:51.472 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:31:51.472 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:31:51.472 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:31:51.472 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:31:51Z" 2024-08-18T17:31:51.472 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:31:51.473 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-08-18T17:31:51.538 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:31:51.538 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:31:51.539 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-group --pipe-id sync-pipe --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:31:51.596 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:31:51.596 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:31:51.683 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe-kibgmb-95","source":{"bucket":"kibgmb-95","zones":["test-zone1"]},"dest":{"bucket":"kibgmb-96","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-kibgmb-96","source":{"bucket":"kibgmb-96","zones":["test-zone1"]},"dest":{"bucket":"kibgmb-96","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-kibgmb-97","source":{"bucket":"kibgmb-97","zones":["test-zone1"]},"dest":{"bucket":"kibgmb-97","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-kibgmb-98","source":{"bucket":"kibgmb-97","zones":["test-zone1"]},"dest":{"bucket":"kibgmb-98","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-08-18T17:31:51.683 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=*', '--dest-zones=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:31:51.683 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=*', '--dest-zones=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:31:51.684 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe '--source-zones=*' '--dest-zones=*' --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:31:51.734 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:31:51.734 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:31:51.767 INFO:teuthology.orchestra.run.smithi134.stderr:WARNING: cannot find source zone id for name=* 2024-08-18T17:31:51.767 INFO:teuthology.orchestra.run.smithi134.stderr:WARNING: cannot find source zone id for name=* 2024-08-18T17:31:51.784 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe-kibgmb-95","source":{"bucket":"kibgmb-95","zones":["test-zone1"]},"dest":{"bucket":"kibgmb-96","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-kibgmb-96","source":{"bucket":"kibgmb-96","zones":["test-zone1"]},"dest":{"bucket":"kibgmb-96","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-kibgmb-97","source":{"bucket":"kibgmb-97","zones":["test-zone1"]},"dest":{"bucket":"kibgmb-97","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-kibgmb-98","source":{"bucket":"kibgmb-97","zones":["test-zone1"]},"dest":{"bucket":"kibgmb-98","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"*","zones":["*"]},"dest":{"bucket":"*","zones":["*"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-08-18T17:31:51.785 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:31:51.785 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:31:51.785 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:31:51.833 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:31:51.833 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:31:51.916 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe-kibgmb-95","source":{"bucket":"kibgmb-95","zones":["test-zone1"]},"dest":{"bucket":"kibgmb-96","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-kibgmb-96","source":{"bucket":"kibgmb-96","zones":["test-zone1"]},"dest":{"bucket":"kibgmb-96","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-kibgmb-97","source":{"bucket":"kibgmb-97","zones":["test-zone1"]},"dest":{"bucket":"kibgmb-97","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-kibgmb-98","source":{"bucket":"kibgmb-97","zones":["test-zone1"]},"dest":{"bucket":"kibgmb-98","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"*","zones":["*"]},"dest":{"bucket":"*","zones":["*"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-08-18T17:31:51.916 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:31:51.916 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:31:51.916 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:31:51.962 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:31:51.962 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:31:52.041 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T17:31:52.041 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:31:52.041 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 21, 2024-08-18T17:31:52.041 INFO:teuthology.orchestra.run.smithi134.stdout: "predecessor_uuid": "64309209-7567-46e5-8881-65783d84cf3c", 2024-08-18T17:31:52.042 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": [ 2024-08-18T17:31:52.042 INFO:teuthology.orchestra.run.smithi134.stdout: "", 2024-08-18T17:31:52.042 INFO:teuthology.orchestra.run.smithi134.stdout: "", 2024-08-18T17:31:52.042 INFO:teuthology.orchestra.run.smithi134.stdout: "", 2024-08-18T17:31:52.042 INFO:teuthology.orchestra.run.smithi134.stdout: "1_1723995848.930494_187.1" 2024-08-18T17:31:52.042 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:31:52.042 INFO:teuthology.orchestra.run.smithi134.stdout: "period_map": { 2024-08-18T17:31:52.042 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:31:52.042 INFO:teuthology.orchestra.run.smithi134.stdout: "zonegroups": [ 2024-08-18T17:31:52.042 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:31:52.042 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "test-zonegroup", 2024-08-18T17:31:52.042 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "test-zonegroup", 2024-08-18T17:31:52.043 INFO:teuthology.orchestra.run.smithi134.stdout: "api_name": "", 2024-08-18T17:31:52.043 INFO:teuthology.orchestra.run.smithi134.stdout: "is_master": true, 2024-08-18T17:31:52.043 INFO:teuthology.orchestra.run.smithi134.stdout: "endpoints": [ 2024-08-18T17:31:52.043 INFO:teuthology.orchestra.run.smithi134.stdout: "http://smithi134.front.sepia.ceph.com:8000" 2024-08-18T17:31:52.043 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:31:52.043 INFO:teuthology.orchestra.run.smithi134.stdout: "hostnames": [], 2024-08-18T17:31:52.043 INFO:teuthology.orchestra.run.smithi134.stdout: "hostnames_s3website": [], 2024-08-18T17:31:52.043 INFO:teuthology.orchestra.run.smithi134.stdout: "master_zone": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T17:31:52.043 INFO:teuthology.orchestra.run.smithi134.stdout: "zones": [ 2024-08-18T17:31:52.043 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:31:52.043 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T17:31:52.043 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "test-zone3", 2024-08-18T17:31:52.043 INFO:teuthology.orchestra.run.smithi134.stdout: "endpoints": [ 2024-08-18T17:31:52.043 INFO:teuthology.orchestra.run.smithi134.stdout: "http://smithi134.front.sepia.ceph.com:8001" 2024-08-18T17:31:52.044 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:31:52.044 INFO:teuthology.orchestra.run.smithi134.stdout: "log_meta": false, 2024-08-18T17:31:52.044 INFO:teuthology.orchestra.run.smithi134.stdout: "log_data": true, 2024-08-18T17:31:52.044 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_index_max_shards": 11, 2024-08-18T17:31:52.044 INFO:teuthology.orchestra.run.smithi134.stdout: "read_only": false, 2024-08-18T17:31:52.044 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_type": "", 2024-08-18T17:31:52.044 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from_all": true, 2024-08-18T17:31:52.044 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from": [], 2024-08-18T17:31:52.044 INFO:teuthology.orchestra.run.smithi134.stdout: "redirect_zone": "", 2024-08-18T17:31:52.044 INFO:teuthology.orchestra.run.smithi134.stdout: "supported_features": [ 2024-08-18T17:31:52.044 INFO:teuthology.orchestra.run.smithi134.stdout: "compress-encrypted", 2024-08-18T17:31:52.044 INFO:teuthology.orchestra.run.smithi134.stdout: "resharding" 2024-08-18T17:31:52.044 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:31:52.045 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:31:52.045 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:31:52.045 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "10090ccf-aad6-416d-9775-df8b7b27345a", 2024-08-18T17:31:52.045 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "test-zone2", 2024-08-18T17:31:52.045 INFO:teuthology.orchestra.run.smithi134.stdout: "endpoints": [ 2024-08-18T17:31:52.045 INFO:teuthology.orchestra.run.smithi134.stdout: "http://smithi193.front.sepia.ceph.com:8000" 2024-08-18T17:31:52.046 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:31:52.046 INFO:teuthology.orchestra.run.smithi134.stdout: "log_meta": false, 2024-08-18T17:31:52.046 INFO:teuthology.orchestra.run.smithi134.stdout: "log_data": true, 2024-08-18T17:31:52.046 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_index_max_shards": 11, 2024-08-18T17:31:52.046 INFO:teuthology.orchestra.run.smithi134.stdout: "read_only": false, 2024-08-18T17:31:52.046 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_type": "", 2024-08-18T17:31:52.047 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from_all": true, 2024-08-18T17:31:52.047 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from": [], 2024-08-18T17:31:52.047 INFO:teuthology.orchestra.run.smithi134.stdout: "redirect_zone": "", 2024-08-18T17:31:52.047 INFO:teuthology.orchestra.run.smithi134.stdout: "supported_features": [ 2024-08-18T17:31:52.047 INFO:teuthology.orchestra.run.smithi134.stdout: "compress-encrypted", 2024-08-18T17:31:52.047 INFO:teuthology.orchestra.run.smithi134.stdout: "resharding" 2024-08-18T17:31:52.047 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:31:52.047 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:31:52.047 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:31:52.047 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T17:31:52.047 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "test-zone1", 2024-08-18T17:31:52.047 INFO:teuthology.orchestra.run.smithi134.stdout: "endpoints": [ 2024-08-18T17:31:52.048 INFO:teuthology.orchestra.run.smithi134.stdout: "http://smithi134.front.sepia.ceph.com:8000" 2024-08-18T17:31:52.048 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:31:52.048 INFO:teuthology.orchestra.run.smithi134.stdout: "log_meta": false, 2024-08-18T17:31:52.048 INFO:teuthology.orchestra.run.smithi134.stdout: "log_data": true, 2024-08-18T17:31:52.048 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_index_max_shards": 11, 2024-08-18T17:31:52.048 INFO:teuthology.orchestra.run.smithi134.stdout: "read_only": false, 2024-08-18T17:31:52.048 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_type": "", 2024-08-18T17:31:52.048 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from_all": true, 2024-08-18T17:31:52.048 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_from": [], 2024-08-18T17:31:52.048 INFO:teuthology.orchestra.run.smithi134.stdout: "redirect_zone": "", 2024-08-18T17:31:52.048 INFO:teuthology.orchestra.run.smithi134.stdout: "supported_features": [ 2024-08-18T17:31:52.048 INFO:teuthology.orchestra.run.smithi134.stdout: "compress-encrypted", 2024-08-18T17:31:52.048 INFO:teuthology.orchestra.run.smithi134.stdout: "resharding" 2024-08-18T17:31:52.048 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:31:52.049 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:31:52.049 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:31:52.049 INFO:teuthology.orchestra.run.smithi134.stdout: "placement_targets": [ 2024-08-18T17:31:52.049 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:31:52.049 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "default-placement", 2024-08-18T17:31:52.049 INFO:teuthology.orchestra.run.smithi134.stdout: "tags": [], 2024-08-18T17:31:52.049 INFO:teuthology.orchestra.run.smithi134.stdout: "storage_classes": [ 2024-08-18T17:31:52.049 INFO:teuthology.orchestra.run.smithi134.stdout: "STANDARD" 2024-08-18T17:31:52.049 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:31:52.049 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:31:52.049 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:31:52.049 INFO:teuthology.orchestra.run.smithi134.stdout: "default_placement": "default-placement", 2024-08-18T17:31:52.049 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_id": "2827c6f0-bf0d-4836-944c-9e7fcf22457b", 2024-08-18T17:31:52.050 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_policy": { 2024-08-18T17:31:52.050 INFO:teuthology.orchestra.run.smithi134.stdout: "groups": [ 2024-08-18T17:31:52.050 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:31:52.050 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "sync-group", 2024-08-18T17:31:52.050 INFO:teuthology.orchestra.run.smithi134.stdout: "data_flow": { 2024-08-18T17:31:52.050 INFO:teuthology.orchestra.run.smithi134.stdout: "symmetrical": [ 2024-08-18T17:31:52.050 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:31:52.050 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "sync-flow1", 2024-08-18T17:31:52.050 INFO:teuthology.orchestra.run.smithi134.stdout: "zones": [ 2024-08-18T17:31:52.051 INFO:teuthology.orchestra.run.smithi134.stdout: "10090ccf-aad6-416d-9775-df8b7b27345a", 2024-08-18T17:31:52.051 INFO:teuthology.orchestra.run.smithi134.stdout: "96bf8502-2f39-4d53-8196-3378c28da3b4" 2024-08-18T17:31:52.051 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:31:52.051 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:31:52.051 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:31:52.051 INFO:teuthology.orchestra.run.smithi134.stdout: "directional": [ 2024-08-18T17:31:52.051 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:31:52.051 INFO:teuthology.orchestra.run.smithi134.stdout: "source_zone": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T17:31:52.051 INFO:teuthology.orchestra.run.smithi134.stdout: "dest_zone": "10090ccf-aad6-416d-9775-df8b7b27345a" 2024-08-18T17:31:52.051 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:31:52.051 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:31:52.051 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:31:52.051 INFO:teuthology.orchestra.run.smithi134.stdout: "pipes": [ 2024-08-18T17:31:52.052 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:31:52.052 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "sync-pipe-kibgmb-95", 2024-08-18T17:31:52.052 INFO:teuthology.orchestra.run.smithi134.stdout: "source": { 2024-08-18T17:31:52.052 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket": "kibgmb-95", 2024-08-18T17:31:52.052 INFO:teuthology.orchestra.run.smithi134.stdout: "zones": [ 2024-08-18T17:31:52.052 INFO:teuthology.orchestra.run.smithi134.stdout: "96bf8502-2f39-4d53-8196-3378c28da3b4" 2024-08-18T17:31:52.052 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:31:52.052 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:31:52.052 INFO:teuthology.orchestra.run.smithi134.stdout: "dest": { 2024-08-18T17:31:52.052 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket": "kibgmb-96", 2024-08-18T17:31:52.052 INFO:teuthology.orchestra.run.smithi134.stdout: "zones": [ 2024-08-18T17:31:52.052 INFO:teuthology.orchestra.run.smithi134.stdout: "10090ccf-aad6-416d-9775-df8b7b27345a" 2024-08-18T17:31:52.052 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:31:52.052 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:31:52.053 INFO:teuthology.orchestra.run.smithi134.stdout: "params": { 2024-08-18T17:31:52.053 INFO:teuthology.orchestra.run.smithi134.stdout: "source": { 2024-08-18T17:31:52.053 INFO:teuthology.orchestra.run.smithi134.stdout: "filter": { 2024-08-18T17:31:52.053 INFO:teuthology.orchestra.run.smithi134.stdout: "tags": [] 2024-08-18T17:31:52.053 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:31:52.053 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:31:52.053 INFO:teuthology.orchestra.run.smithi134.stdout: "dest": {}, 2024-08-18T17:31:52.053 INFO:teuthology.orchestra.run.smithi134.stdout: "priority": 0, 2024-08-18T17:31:52.053 INFO:teuthology.orchestra.run.smithi134.stdout: "mode": "system", 2024-08-18T17:31:52.053 INFO:teuthology.orchestra.run.smithi134.stdout: "user": "" 2024-08-18T17:31:52.053 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:31:52.053 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:31:52.053 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:31:52.054 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "sync-pipe-kibgmb-96", 2024-08-18T17:31:52.054 INFO:teuthology.orchestra.run.smithi134.stdout: "source": { 2024-08-18T17:31:52.054 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket": "kibgmb-96", 2024-08-18T17:31:52.054 INFO:teuthology.orchestra.run.smithi134.stdout: "zones": [ 2024-08-18T17:31:52.054 INFO:teuthology.orchestra.run.smithi134.stdout: "96bf8502-2f39-4d53-8196-3378c28da3b4" 2024-08-18T17:31:52.054 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:31:52.054 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:31:52.054 INFO:teuthology.orchestra.run.smithi134.stdout: "dest": { 2024-08-18T17:31:52.054 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket": "kibgmb-96", 2024-08-18T17:31:52.054 INFO:teuthology.orchestra.run.smithi134.stdout: "zones": [ 2024-08-18T17:31:52.054 INFO:teuthology.orchestra.run.smithi134.stdout: "10090ccf-aad6-416d-9775-df8b7b27345a" 2024-08-18T17:31:52.054 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:31:52.054 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:31:52.055 INFO:teuthology.orchestra.run.smithi134.stdout: "params": { 2024-08-18T17:31:52.055 INFO:teuthology.orchestra.run.smithi134.stdout: "source": { 2024-08-18T17:31:52.055 INFO:teuthology.orchestra.run.smithi134.stdout: "filter": { 2024-08-18T17:31:52.055 INFO:teuthology.orchestra.run.smithi134.stdout: "tags": [] 2024-08-18T17:31:52.055 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:31:52.055 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:31:52.055 INFO:teuthology.orchestra.run.smithi134.stdout: "dest": {}, 2024-08-18T17:31:52.056 INFO:teuthology.orchestra.run.smithi134.stdout: "priority": 0, 2024-08-18T17:31:52.056 INFO:teuthology.orchestra.run.smithi134.stdout: "mode": "system", 2024-08-18T17:31:52.056 INFO:teuthology.orchestra.run.smithi134.stdout: "user": "" 2024-08-18T17:31:52.056 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:31:52.056 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:31:52.056 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:31:52.056 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "sync-pipe-kibgmb-97", 2024-08-18T17:31:52.056 INFO:teuthology.orchestra.run.smithi134.stdout: "source": { 2024-08-18T17:31:52.056 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket": "kibgmb-97", 2024-08-18T17:31:52.056 INFO:teuthology.orchestra.run.smithi134.stdout: "zones": [ 2024-08-18T17:31:52.056 INFO:teuthology.orchestra.run.smithi134.stdout: "96bf8502-2f39-4d53-8196-3378c28da3b4" 2024-08-18T17:31:52.056 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:31:52.057 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:31:52.057 INFO:teuthology.orchestra.run.smithi134.stdout: "dest": { 2024-08-18T17:31:52.057 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket": "kibgmb-97", 2024-08-18T17:31:52.057 INFO:teuthology.orchestra.run.smithi134.stdout: "zones": [ 2024-08-18T17:31:52.057 INFO:teuthology.orchestra.run.smithi134.stdout: "10090ccf-aad6-416d-9775-df8b7b27345a" 2024-08-18T17:31:52.057 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:31:52.057 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:31:52.057 INFO:teuthology.orchestra.run.smithi134.stdout: "params": { 2024-08-18T17:31:52.057 INFO:teuthology.orchestra.run.smithi134.stdout: "source": { 2024-08-18T17:31:52.057 INFO:teuthology.orchestra.run.smithi134.stdout: "filter": { 2024-08-18T17:31:52.057 INFO:teuthology.orchestra.run.smithi134.stdout: "tags": [] 2024-08-18T17:31:52.057 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:31:52.058 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:31:52.058 INFO:teuthology.orchestra.run.smithi134.stdout: "dest": {}, 2024-08-18T17:31:52.058 INFO:teuthology.orchestra.run.smithi134.stdout: "priority": 0, 2024-08-18T17:31:52.058 INFO:teuthology.orchestra.run.smithi134.stdout: "mode": "system", 2024-08-18T17:31:52.058 INFO:teuthology.orchestra.run.smithi134.stdout: "user": "" 2024-08-18T17:31:52.058 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:31:52.058 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:31:52.058 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:31:52.058 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "sync-pipe-kibgmb-98", 2024-08-18T17:31:52.058 INFO:teuthology.orchestra.run.smithi134.stdout: "source": { 2024-08-18T17:31:52.058 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket": "kibgmb-97", 2024-08-18T17:31:52.058 INFO:teuthology.orchestra.run.smithi134.stdout: "zones": [ 2024-08-18T17:31:52.058 INFO:teuthology.orchestra.run.smithi134.stdout: "96bf8502-2f39-4d53-8196-3378c28da3b4" 2024-08-18T17:31:52.059 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:31:52.059 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:31:52.059 INFO:teuthology.orchestra.run.smithi134.stdout: "dest": { 2024-08-18T17:31:52.059 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket": "kibgmb-98", 2024-08-18T17:31:52.059 INFO:teuthology.orchestra.run.smithi134.stdout: "zones": [ 2024-08-18T17:31:52.059 INFO:teuthology.orchestra.run.smithi134.stdout: "10090ccf-aad6-416d-9775-df8b7b27345a" 2024-08-18T17:31:52.059 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:31:52.059 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:31:52.059 INFO:teuthology.orchestra.run.smithi134.stdout: "params": { 2024-08-18T17:31:52.059 INFO:teuthology.orchestra.run.smithi134.stdout: "source": { 2024-08-18T17:31:52.059 INFO:teuthology.orchestra.run.smithi134.stdout: "filter": { 2024-08-18T17:31:52.059 INFO:teuthology.orchestra.run.smithi134.stdout: "tags": [] 2024-08-18T17:31:52.059 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:31:52.060 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:31:52.060 INFO:teuthology.orchestra.run.smithi134.stdout: "dest": {}, 2024-08-18T17:31:52.060 INFO:teuthology.orchestra.run.smithi134.stdout: "priority": 0, 2024-08-18T17:31:52.060 INFO:teuthology.orchestra.run.smithi134.stdout: "mode": "system", 2024-08-18T17:31:52.060 INFO:teuthology.orchestra.run.smithi134.stdout: "user": "" 2024-08-18T17:31:52.060 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:31:52.060 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:31:52.060 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:31:52.060 INFO:teuthology.orchestra.run.smithi134.stdout: "id": "sync-pipe", 2024-08-18T17:31:52.060 INFO:teuthology.orchestra.run.smithi134.stdout: "source": { 2024-08-18T17:31:52.060 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket": "*", 2024-08-18T17:31:52.060 INFO:teuthology.orchestra.run.smithi134.stdout: "zones": [ 2024-08-18T17:31:52.060 INFO:teuthology.orchestra.run.smithi134.stdout: "*" 2024-08-18T17:31:52.061 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:31:52.061 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:31:52.061 INFO:teuthology.orchestra.run.smithi134.stdout: "dest": { 2024-08-18T17:31:52.061 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket": "*", 2024-08-18T17:31:52.061 INFO:teuthology.orchestra.run.smithi134.stdout: "zones": [ 2024-08-18T17:31:52.061 INFO:teuthology.orchestra.run.smithi134.stdout: "*" 2024-08-18T17:31:52.061 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:31:52.061 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:31:52.061 INFO:teuthology.orchestra.run.smithi134.stdout: "params": { 2024-08-18T17:31:52.061 INFO:teuthology.orchestra.run.smithi134.stdout: "source": { 2024-08-18T17:31:52.061 INFO:teuthology.orchestra.run.smithi134.stdout: "filter": { 2024-08-18T17:31:52.061 INFO:teuthology.orchestra.run.smithi134.stdout: "tags": [] 2024-08-18T17:31:52.061 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:31:52.062 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:31:52.062 INFO:teuthology.orchestra.run.smithi134.stdout: "dest": {}, 2024-08-18T17:31:52.062 INFO:teuthology.orchestra.run.smithi134.stdout: "priority": 0, 2024-08-18T17:31:52.062 INFO:teuthology.orchestra.run.smithi134.stdout: "mode": "system", 2024-08-18T17:31:52.062 INFO:teuthology.orchestra.run.smithi134.stdout: "user": "" 2024-08-18T17:31:52.062 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:31:52.062 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:31:52.062 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:31:52.062 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "allowed" 2024-08-18T17:31:52.062 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:31:52.062 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:31:52.062 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:31:52.062 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled_features": [ 2024-08-18T17:31:52.063 INFO:teuthology.orchestra.run.smithi134.stdout: "resharding" 2024-08-18T17:31:52.063 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:31:52.063 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:31:52.063 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-08-18T17:31:52.063 INFO:teuthology.orchestra.run.smithi134.stdout: "short_zone_ids": [ 2024-08-18T17:31:52.063 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:31:52.063 INFO:teuthology.orchestra.run.smithi134.stdout: "key": "018ea73b-7528-48da-aafa-0118123784f7", 2024-08-18T17:31:52.063 INFO:teuthology.orchestra.run.smithi134.stdout: "val": 462036791 2024-08-18T17:31:52.063 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:31:52.063 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:31:52.063 INFO:teuthology.orchestra.run.smithi134.stdout: "key": "10090ccf-aad6-416d-9775-df8b7b27345a", 2024-08-18T17:31:52.063 INFO:teuthology.orchestra.run.smithi134.stdout: "val": 874289624 2024-08-18T17:31:52.064 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:31:52.064 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:31:52.064 INFO:teuthology.orchestra.run.smithi134.stdout: "key": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T17:31:52.064 INFO:teuthology.orchestra.run.smithi134.stdout: "val": 1721399886 2024-08-18T17:31:52.064 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:31:52.064 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:31:52.064 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:31:52.064 INFO:teuthology.orchestra.run.smithi134.stdout: "master_zonegroup": "test-zonegroup", 2024-08-18T17:31:52.064 INFO:teuthology.orchestra.run.smithi134.stdout: "master_zone": "96bf8502-2f39-4d53-8196-3378c28da3b4", 2024-08-18T17:31:52.064 INFO:teuthology.orchestra.run.smithi134.stdout: "period_config": { 2024-08-18T17:31:52.064 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_quota": { 2024-08-18T17:31:52.064 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false, 2024-08-18T17:31:52.064 INFO:teuthology.orchestra.run.smithi134.stdout: "check_on_raw": false, 2024-08-18T17:31:52.070 INFO:teuthology.orchestra.run.smithi134.stdout: "max_size": -1, 2024-08-18T17:31:52.071 INFO:teuthology.orchestra.run.smithi134.stdout: "max_size_kb": 0, 2024-08-18T17:31:52.071 INFO:teuthology.orchestra.run.smithi134.stdout: "max_objects": -1 2024-08-18T17:31:52.071 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:31:52.071 INFO:teuthology.orchestra.run.smithi134.stdout: "user_quota": { 2024-08-18T17:31:52.071 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false, 2024-08-18T17:31:52.071 INFO:teuthology.orchestra.run.smithi134.stdout: "check_on_raw": false, 2024-08-18T17:31:52.071 INFO:teuthology.orchestra.run.smithi134.stdout: "max_size": -1, 2024-08-18T17:31:52.071 INFO:teuthology.orchestra.run.smithi134.stdout: "max_size_kb": 0, 2024-08-18T17:31:52.071 INFO:teuthology.orchestra.run.smithi134.stdout: "max_objects": -1 2024-08-18T17:31:52.071 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:31:52.071 INFO:teuthology.orchestra.run.smithi134.stdout: "user_ratelimit": { 2024-08-18T17:31:52.071 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_ops": 0, 2024-08-18T17:31:52.072 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_ops": 0, 2024-08-18T17:31:52.072 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_bytes": 0, 2024-08-18T17:31:52.072 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_bytes": 0, 2024-08-18T17:31:52.072 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false 2024-08-18T17:31:52.072 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:31:52.072 INFO:teuthology.orchestra.run.smithi134.stdout: "bucket_ratelimit": { 2024-08-18T17:31:52.072 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_ops": 0, 2024-08-18T17:31:52.072 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_ops": 0, 2024-08-18T17:31:52.072 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_bytes": 0, 2024-08-18T17:31:52.072 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_bytes": 0, 2024-08-18T17:31:52.072 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false 2024-08-18T17:31:52.072 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:31:52.072 INFO:teuthology.orchestra.run.smithi134.stdout: "anonymous_ratelimit": { 2024-08-18T17:31:52.072 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_ops": 0, 2024-08-18T17:31:52.073 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_ops": 0, 2024-08-18T17:31:52.073 INFO:teuthology.orchestra.run.smithi134.stdout: "max_read_bytes": 0, 2024-08-18T17:31:52.073 INFO:teuthology.orchestra.run.smithi134.stdout: "max_write_bytes": 0, 2024-08-18T17:31:52.073 INFO:teuthology.orchestra.run.smithi134.stdout: "enabled": false 2024-08-18T17:31:52.073 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:31:52.073 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:31:52.073 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_id": "2827c6f0-bf0d-4836-944c-9e7fcf22457b", 2024-08-18T17:31:52.073 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_name": "test-realm", 2024-08-18T17:31:52.073 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:31:52.073 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T17:31:52.074 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:31:52.074 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:31:52.075 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:31:52.122 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:31:52.123 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:31:52.199 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe-kibgmb-95","source":{"bucket":"kibgmb-95","zones":["test-zone1"]},"dest":{"bucket":"kibgmb-96","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-kibgmb-96","source":{"bucket":"kibgmb-96","zones":["test-zone1"]},"dest":{"bucket":"kibgmb-96","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-kibgmb-97","source":{"bucket":"kibgmb-97","zones":["test-zone1"]},"dest":{"bucket":"kibgmb-97","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-kibgmb-98","source":{"bucket":"kibgmb-97","zones":["test-zone1"]},"dest":{"bucket":"kibgmb-98","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"*","zones":["*"]},"dest":{"bucket":"*","zones":["*"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-08-18T17:31:52.199 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'kibgmb-97', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:31:52.199 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'kibgmb-97', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:31:52.200 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-bucket --status allowed --bucket kibgmb-97 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:31:52.267 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:31:52.267 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:31:52.378 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"allowed"}]} 2024-08-18T17:31:52.379 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--bucket', 'kibgmb-97', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:31:52.379 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--bucket', 'kibgmb-97', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:31:52.379 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flowA --flow-type symmetrical --zones=test-zone1,test-zone2 --bucket kibgmb-97 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:31:52.447 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:31:52.447 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:31:52.584 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["test-zone2","test-zone1"]}]},"pipes":[],"status":"allowed"}]} 2024-08-18T17:31:52.584 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-kibgmb-97', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--bucket=kibgmb-97', '--source-bucket=*', '--dest-bucket=kibgmb-97', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:31:52.585 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-kibgmb-97', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--bucket=kibgmb-97', '--source-bucket=*', '--dest-bucket=kibgmb-97', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:31:52.585 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe-kibgmb-97 --source-zones=test-zone1 --dest-zones=test-zone2 --bucket=kibgmb-97 '--source-bucket=*' --dest-bucket=kibgmb-97 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:31:52.648 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:31:52.648 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:31:52.766 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["test-zone2","test-zone1"]}]},"pipes":[{"id":"sync-pipe-kibgmb-97","source":{"bucket":"*","zones":["test-zone1"]},"dest":{"bucket":"kibgmb-97","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2024-08-18T17:31:52.766 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-kibgmb-98', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--bucket=kibgmb-97', '--source-bucket=*', '--dest-bucket=kibgmb-98', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:31:52.766 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-kibgmb-98', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--bucket=kibgmb-97', '--source-bucket=*', '--dest-bucket=kibgmb-98', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:31:52.766 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe-kibgmb-98 --source-zones=test-zone1 --dest-zones=test-zone2 --bucket=kibgmb-97 '--source-bucket=*' --dest-bucket=kibgmb-98 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:31:52.825 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:31:52.825 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:31:52.946 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["test-zone2","test-zone1"]}]},"pipes":[{"id":"sync-pipe-kibgmb-97","source":{"bucket":"*","zones":["test-zone1"]},"dest":{"bucket":"kibgmb-97","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}},{"id":"sync-pipe-kibgmb-98","source":{"bucket":"*","zones":["test-zone1"]},"dest":{"bucket":"kibgmb-98","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2024-08-18T17:31:52.947 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'kibgmb-97', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:31:52.947 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'kibgmb-97', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:31:52.947 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-bucket --status enabled --bucket kibgmb-97 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:31:53.004 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:31:53.004 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:31:53.098 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["test-zone2","test-zone1"]}]},"pipes":[{"id":"sync-pipe-kibgmb-97","source":{"bucket":"*","zones":["test-zone1"]},"dest":{"bucket":"kibgmb-97","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}},{"id":"sync-pipe-kibgmb-98","source":{"bucket":"*","zones":["test-zone1"]},"dest":{"bucket":"kibgmb-98","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2024-08-18T17:31:53.098 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:31:53.098 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:31:53.099 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:31:53.144 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:31:53.145 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:31:53.212 INFO:teuthology.orchestra.run.smithi134.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe-kibgmb-95","source":{"bucket":"kibgmb-95","zones":["test-zone1"]},"dest":{"bucket":"kibgmb-96","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-kibgmb-96","source":{"bucket":"kibgmb-96","zones":["test-zone1"]},"dest":{"bucket":"kibgmb-96","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-kibgmb-97","source":{"bucket":"kibgmb-97","zones":["test-zone1"]},"dest":{"bucket":"kibgmb-97","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-kibgmb-98","source":{"bucket":"kibgmb-97","zones":["test-zone1"]},"dest":{"bucket":"kibgmb-98","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"*","zones":["*"]},"dest":{"bucket":"*","zones":["*"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-08-18T17:31:53.213 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:31:53.213 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:31:53.214 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:31:53.260 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:31:53.260 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:31:53.307 INFO:teuthology.orchestra.run.smithi134.stderr:No --period given, using current period=01050e30-179b-43be-bdbb-3bb23cf37ae8 2024-08-18T17:31:53.326 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T17:31:53.326 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:31:53.326 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:31:53.326 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T17:31:53.326 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:31:53.326 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:31:53.327 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:31:53.327 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T17:31:53.327 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:31:53.327 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:31:53.327 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:31:53.327 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:42:47.672461Z" 2024-08-18T17:31:53.327 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:31:53.327 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:31:53.327 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1724002313.076576_1734.1", 2024-08-18T17:31:53.327 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T17:31:53.076576Z" 2024-08-18T17:31:53.327 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:31:53.327 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T17:31:53.328 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-08-18T17:31:53.328 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:31:53.328 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:31:53.329 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:31:53.381 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:31:53.382 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:31:53.414 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:31:53.413+0000 7f1fabd28880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:31:53.415 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:31:53.414+0000 7f1fabd28880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:31:53.430 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:31:53.430 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:31:53.430 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:31:53.430 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:31:53.430 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:31:53.430 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:31:53.430 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:31:53.431 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:31:53.431 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:31:53.431 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:31:53.431 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:31:53.431 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:31:53.431 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:31:53.431 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:31:53.431 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:31:53.431 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:31:53.431 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:31:53.431 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:31:53.431 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:31:53.431 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:31:53.431 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:31:53.432 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:31:53.432 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:31:53.432 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:31:53.432 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:31:53.432 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:31:53.432 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:31:53.432 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:31:53.432 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:31:53.432 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:31:53.432 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:31:53.432 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:31:53.432 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:31:53.432 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:31:53.432 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:31:53.433 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:31:53.433 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:31:53.433 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:31:53.433 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:31:53.433 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:31:53.433 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:31:53.433 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:31:53.433 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:31:53.433 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:31:53.433 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:31:53.433 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:31:53.433 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:31:53.433 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:31:53.434 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:31:53.434 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724002181.154947_1688.1", 2024-08-18T17:31:53.434 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:31:53.434 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:31:53.434 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:31:53.434 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:29:41.154947Z", 2024-08-18T17:31:53.434 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:31:53.434 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:31:53.434 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:31:53.434 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:31:53.434 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:31:53.434 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:31:53.434 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:31:53.434 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:31:53.435 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:31:53.435 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:31:53Z" 2024-08-18T17:31:53.435 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:31:53.436 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724002313.076576_1734.1 target=1_1724002181.154947_1688.1 2024-08-18T17:31:58.436 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:31:58.436 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:31:58.437 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:31:58.492 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:31:58.492 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:31:58.525 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:31:58.524+0000 7f0f81942880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:31:58.526 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:31:58.525+0000 7f0f81942880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:31:58.541 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:31:58.541 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:31:58.541 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:31:58.541 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:31:58.541 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:31:58.541 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:31:58.541 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:31:58.542 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:31:58.542 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:31:58.542 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:31:58.542 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:31:58.542 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:31:58.542 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:31:58.542 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:31:58.542 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:31:58.542 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:31:58.542 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:31:58.542 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:31:58.542 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:31:58.543 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:31:58.543 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:31:58.543 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:31:58.543 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:31:58.543 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:31:58.543 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:31:58.543 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:31:58.543 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:31:58.543 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:31:58.543 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:31:58.543 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:31:58.543 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:31:58.543 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:31:58.544 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:31:58.544 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:31:58.544 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:31:58.544 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:31:58.544 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:31:58.544 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:31:58.544 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:31:58.544 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:31:58.544 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:31:58.544 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:31:58.544 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:31:58.545 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:31:58.545 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:31:58.545 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:31:58.545 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:31:58.545 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:31:58.545 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:31:58.546 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724002181.154947_1688.1", 2024-08-18T17:31:58.546 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:31:58.546 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:31:58.546 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:31:58.546 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:29:41.154947Z", 2024-08-18T17:31:58.546 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:31:58.546 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:31:58.546 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:31:58.546 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:31:58.546 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:31:58.546 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:31:58.546 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:31:58.547 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:31:58.547 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:31:58.547 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:31:58Z" 2024-08-18T17:31:58.547 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:31:58.548 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724002313.076576_1734.1 target=1_1724002181.154947_1688.1 2024-08-18T17:32:03.548 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:32:03.549 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:32:03.549 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:32:03.604 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:32:03.605 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:32:03.639 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:32:03.638+0000 7f9e66012880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:32:03.639 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:32:03.639+0000 7f9e66012880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:32:03.654 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:32:03.654 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:32:03.654 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:32:03.654 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:32:03.654 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:32:03.654 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:32:03.654 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:32:03.655 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:32:03.655 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:32:03.655 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:32:03.655 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:32:03.655 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:32:03.655 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:32:03.655 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:32:03.655 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:32:03.655 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:32:03.655 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:32:03.655 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:32:03.655 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:32:03.655 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:32:03.656 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:32:03.656 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:32:03.656 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:32:03.656 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:32:03.656 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:32:03.656 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:32:03.656 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:32:03.656 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:32:03.656 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:32:03.656 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:32:03.656 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:32:03.656 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:32:03.656 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:32:03.656 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:32:03.657 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:32:03.657 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:32:03.657 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:32:03.657 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:32:03.657 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:32:03.657 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:32:03.657 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:32:03.657 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:32:03.657 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:32:03.657 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:32:03.657 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:32:03.657 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:32:03.657 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:32:03.658 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:32:03.658 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:32:03.658 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724002181.154947_1688.1", 2024-08-18T17:32:03.658 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:32:03.658 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:32:03.658 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:32:03.658 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:29:41.154947Z", 2024-08-18T17:32:03.658 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:32:03.658 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:32:03.659 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:32:03.659 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:32:03.659 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:32:03.659 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:32:03.659 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:32:03.659 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:32:03.659 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:32:03.659 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:32:03Z" 2024-08-18T17:32:03.659 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:32:03.660 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724002313.076576_1734.1 target=1_1724002181.154947_1688.1 2024-08-18T17:32:08.661 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:32:08.661 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:32:08.661 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:32:08.716 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:32:08.716 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:32:08.750 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:32:08.749+0000 7f2f28086880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:32:08.751 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:32:08.750+0000 7f2f28086880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:32:08.766 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:32:08.766 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:32:08.766 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:32:08.766 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:32:08.766 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:32:08.766 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:32:08.767 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:32:08.767 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:32:08.767 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:32:08.767 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:32:08.767 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:32:08.767 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:32:08.767 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:32:08.767 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:32:08.767 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:32:08.767 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:32:08.767 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:32:08.767 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:32:08.767 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:32:08.768 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:32:08.768 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:32:08.768 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:32:08.768 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:32:08.768 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:32:08.768 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:32:08.768 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:32:08.768 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:32:08.768 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:32:08.768 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:32:08.768 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:32:08.768 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:32:08.768 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:32:08.769 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:32:08.769 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:32:08.769 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:32:08.769 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:32:08.769 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:32:08.769 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:32:08.769 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:32:08.769 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:32:08.769 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:32:08.769 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:32:08.769 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:32:08.769 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:32:08.769 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:32:08.770 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:32:08.770 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:32:08.770 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:32:08.770 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:32:08.770 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724002181.154947_1688.1", 2024-08-18T17:32:08.770 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:32:08.770 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:32:08.770 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:32:08.770 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:29:41.154947Z", 2024-08-18T17:32:08.770 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:32:08.770 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:32:08.770 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:32:08.770 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:32:08.770 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:32:08.771 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:32:08.771 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:32:08.771 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:32:08.771 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:32:08.771 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:32:08Z" 2024-08-18T17:32:08.771 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:32:08.772 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724002313.076576_1734.1 target=1_1724002181.154947_1688.1 2024-08-18T17:32:13.773 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:32:13.774 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:32:13.774 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:32:13.829 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:32:13.829 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:32:13.863 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:32:13.862+0000 7f1690b19880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:32:13.864 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:32:13.863+0000 7f1690b19880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:32:13.878 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:32:13.879 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:32:13.879 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:32:13.879 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:32:13.879 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:32:13.879 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:32:13.879 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:32:13.879 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:32:13.879 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:32:13.879 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:32:13.879 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:32:13.879 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:32:13.879 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:32:13.880 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:32:13.880 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:32:13.880 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:32:13.880 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:32:13.880 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:32:13.880 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:32:13.880 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:32:13.880 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:32:13.880 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:32:13.880 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:32:13.880 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:32:13.880 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:32:13.880 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:32:13.881 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:32:13.881 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:32:13.881 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:32:13.881 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:32:13.881 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:32:13.881 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:32:13.881 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:32:13.881 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:32:13.881 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:32:13.881 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:32:13.881 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:32:13.881 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:32:13.882 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:32:13.882 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:32:13.882 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:32:13.882 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:32:13.882 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:32:13.882 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:32:13.882 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:32:13.882 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:32:13.882 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:32:13.882 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:32:13.882 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:32:13.882 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724002181.154947_1688.1", 2024-08-18T17:32:13.882 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:32:13.883 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:32:13.883 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:32:13.883 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:29:41.154947Z", 2024-08-18T17:32:13.883 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:32:13.883 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:32:13.883 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:32:13.883 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:32:13.883 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:32:13.883 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:32:13.883 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:32:13.883 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:32:13.883 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:32:13.883 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:32:13Z" 2024-08-18T17:32:13.884 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:32:13.884 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724002313.076576_1734.1 target=1_1724002181.154947_1688.1 2024-08-18T17:32:18.885 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:32:18.886 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:32:18.886 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:32:18.941 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:32:18.941 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:32:18.975 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:32:18.974+0000 7f4b240bb880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:32:18.975 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:32:18.975+0000 7f4b240bb880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:32:18.990 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:32:18.990 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:32:18.991 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:32:18.991 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:32:18.991 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:32:18.991 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:32:18.991 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:32:18.991 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:32:18.991 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:32:18.991 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:32:18.991 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:32:18.991 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:32:18.991 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:32:18.991 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:32:18.991 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:32:18.991 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:32:18.992 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:32:18.992 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:32:18.992 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:32:18.992 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:32:18.992 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:32:18.992 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:32:18.992 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:32:18.992 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:32:18.992 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:32:18.992 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:32:18.992 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:32:18.992 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:32:18.992 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:32:18.993 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:32:18.993 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:32:18.993 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:32:18.993 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:32:18.993 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:32:18.993 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:32:18.993 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:32:18.993 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:32:18.993 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:32:18.993 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:32:18.993 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:32:18.993 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:32:18.993 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:32:18.993 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:32:18.994 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:32:18.994 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:32:18.994 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:32:18.994 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:32:18.994 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:32:18.994 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:32:18.994 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724002181.154947_1688.1", 2024-08-18T17:32:18.994 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:32:18.994 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:32:18.994 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:32:18.994 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:29:41.154947Z", 2024-08-18T17:32:18.994 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:32:18.995 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:32:18.995 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:32:18.995 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:32:18.995 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:32:18.995 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:32:18.995 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:32:18.995 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:32:18.995 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:32:18.995 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:32:18Z" 2024-08-18T17:32:18.996 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:32:18.996 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724002313.076576_1734.1 target=1_1724002181.154947_1688.1 2024-08-18T17:32:23.997 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:32:23.998 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:32:23.998 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:32:24.053 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:32:24.053 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:32:24.087 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:32:24.086+0000 7f39cb6bb880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:32:24.088 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:32:24.087+0000 7f39cb6bb880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:32:24.103 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:32:24.103 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:32:24.103 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:32:24.103 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:32:24.103 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:32:24.103 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:32:24.103 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:32:24.103 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:32:24.103 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:32:24.103 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:32:24.104 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:32:24.104 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:32:24.104 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:32:24.104 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:32:24.104 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:32:24.104 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:32:24.104 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:32:24.104 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:32:24.104 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:32:24.104 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:32:24.104 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:32:24.104 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:32:24.104 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:32:24.105 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:32:24.105 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:32:24.105 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:32:24.105 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:32:24.105 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:32:24.105 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:32:24.105 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:32:24.105 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:32:24.105 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:32:24.105 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:32:24.105 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:32:24.105 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:32:24.105 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:32:24.105 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:32:24.106 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:32:24.106 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:32:24.106 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:32:24.106 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:32:24.106 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:32:24.106 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:32:24.106 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:32:24.106 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:32:24.106 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:32:24.106 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:32:24.106 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:32:24.106 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:32:24.106 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724002181.154947_1688.1", 2024-08-18T17:32:24.107 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:32:24.107 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:32:24.107 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:32:24.107 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:29:41.154947Z", 2024-08-18T17:32:24.107 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:32:24.107 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:32:24.107 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:32:24.107 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:32:24.107 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:32:24.108 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:32:24.108 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:32:24.108 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:32:24.108 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:32:24.108 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:32:24Z" 2024-08-18T17:32:24.108 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:32:24.109 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724002313.076576_1734.1 target=1_1724002181.154947_1688.1 2024-08-18T17:32:29.109 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:32:29.110 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:32:29.110 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:32:29.165 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:32:29.165 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:32:29.199 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:32:29.198+0000 7f68a5815880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:32:29.199 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:32:29.199+0000 7f68a5815880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:32:29.215 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:32:29.215 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:32:29.215 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:32:29.215 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:32:29.215 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:32:29.215 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:32:29.215 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:32:29.215 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:32:29.216 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:32:29.216 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:32:29.216 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:32:29.216 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:32:29.216 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:32:29.216 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:32:29.216 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:32:29.216 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:32:29.216 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:32:29.216 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:32:29.216 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:32:29.216 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:32:29.216 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:32:29.217 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:32:29.217 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:32:29.217 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:32:29.217 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:32:29.217 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:32:29.217 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:32:29.217 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:32:29.217 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:32:29.217 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:32:29.218 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:32:29.218 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:32:29.218 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:32:29.218 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:32:29.218 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:32:29.218 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:32:29.218 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:32:29.218 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:32:29.218 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:32:29.218 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:32:29.218 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:32:29.218 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:32:29.219 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:32:29.219 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:32:29.219 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:32:29.219 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:32:29.219 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:32:29.219 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:32:29.219 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:32:29.219 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724002181.154947_1688.1", 2024-08-18T17:32:29.219 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:32:29.220 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:32:29.220 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:32:29.220 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:29:41.154947Z", 2024-08-18T17:32:29.220 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:32:29.220 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:32:29.220 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:32:29.220 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:32:29.221 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:32:29.221 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:32:29.221 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:32:29.221 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:32:29.221 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:32:29.221 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:32:29Z" 2024-08-18T17:32:29.221 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:32:29.223 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724002313.076576_1734.1 target=1_1724002181.154947_1688.1 2024-08-18T17:32:34.224 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:32:34.224 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:32:34.224 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:32:34.279 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:32:34.279 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:32:34.313 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:32:34.311+0000 7f50c7538880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:32:34.313 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:32:34.311+0000 7f50c7538880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:32:34.328 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:32:34.329 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:32:34.329 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:32:34.329 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:32:34.329 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:32:34.329 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:32:34.329 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:32:34.329 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:32:34.329 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:32:34.330 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:32:34.330 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:32:34.330 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:32:34.330 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:32:34.330 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:32:34.330 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:32:34.330 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:32:34.330 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:32:34.330 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:32:34.331 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:32:34.331 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:32:34.331 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:32:34.331 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:32:34.331 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:32:34.331 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:32:34.332 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:32:34.332 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:32:34.332 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:32:34.332 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:32:34.332 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:32:34.333 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:32:34.333 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:32:34.333 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:32:34.333 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:32:34.333 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:32:34.333 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:32:34.333 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:32:34.333 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:32:34.333 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:32:34.333 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:32:34.334 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:32:34.334 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:32:34.334 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:32:34.334 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:32:34.334 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:32:34.334 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:32:34.335 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:32:34.335 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:32:34.335 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:32:34.335 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:32:34.335 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724002181.154947_1688.1", 2024-08-18T17:32:34.335 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:32:34.335 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:32:34.335 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:32:34.335 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:29:41.154947Z", 2024-08-18T17:32:34.336 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:32:34.336 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:32:34.336 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:32:34.336 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:32:34.336 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:32:34.336 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:32:34.336 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:32:34.336 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:32:34.336 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:32:34.337 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:32:34Z" 2024-08-18T17:32:34.337 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:32:34.338 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724002313.076576_1734.1 target=1_1724002181.154947_1688.1 2024-08-18T17:32:39.339 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:32:39.339 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:32:39.339 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:32:39.397 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:32:39.397 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:32:39.431 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:32:39.429+0000 7fb21352a880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:32:39.432 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:32:39.430+0000 7fb21352a880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:32:39.446 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:32:39.446 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:32:39.447 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:32:39.447 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:32:39.447 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:32:39.447 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:32:39.447 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:32:39.447 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:32:39.447 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:32:39.447 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:32:39.447 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:32:39.447 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:32:39.447 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:32:39.447 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:32:39.448 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:32:39.448 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:32:39.448 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:32:39.448 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:32:39.448 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:32:39.448 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:32:39.448 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:32:39.448 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:32:39.448 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:32:39.448 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:32:39.449 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:32:39.449 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:32:39.449 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:32:39.449 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:32:39.449 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:32:39.449 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:32:39.449 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:32:39.449 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:32:39.449 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:32:39.449 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:32:39.449 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:32:39.449 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:32:39.449 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:32:39.450 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:32:39.450 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:32:39.450 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:32:39.450 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:32:39.450 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:32:39.450 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:32:39.450 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:32:39.450 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:32:39.450 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:32:39.450 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:32:39.450 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:32:39.450 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:32:39.450 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724002181.154947_1688.1", 2024-08-18T17:32:39.451 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:32:39.451 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:32:39.451 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:32:39.451 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:29:41.154947Z", 2024-08-18T17:32:39.451 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:32:39.451 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:32:39.451 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:32:39.451 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:32:39.451 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:32:39.451 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:32:39.451 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:32:39.451 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:32:39.451 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:32:39.452 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:32:39Z" 2024-08-18T17:32:39.452 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:32:39.453 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724002313.076576_1734.1 target=1_1724002181.154947_1688.1 2024-08-18T17:32:44.454 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:32:44.454 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:32:44.454 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:32:44.508 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:32:44.509 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:32:44.542 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:32:44.541+0000 7f2494f20880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:32:44.542 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:32:44.542+0000 7f2494f20880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:32:44.557 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:32:44.557 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:32:44.557 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:32:44.557 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:32:44.557 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:32:44.557 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:32:44.558 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:32:44.558 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:32:44.558 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:32:44.558 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:32:44.558 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:32:44.558 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:32:44.558 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:32:44.558 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:32:44.558 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:32:44.559 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:32:44.559 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:32:44.559 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:32:44.559 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:32:44.559 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:32:44.559 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:32:44.559 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:32:44.559 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:32:44.559 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:32:44.559 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:32:44.560 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:32:44.560 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:32:44.560 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:32:44.560 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:32:44.560 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:32:44.560 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:32:44.560 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:32:44.560 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:32:44.560 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:32:44.560 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:32:44.560 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:32:44.561 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:32:44.561 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:32:44.561 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:32:44.561 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:32:44.561 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:32:44.561 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:32:44.561 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:32:44.561 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:32:44.561 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:32:44.561 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:32:44.561 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:32:44.562 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:32:44.562 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:32:44.562 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724002181.154947_1688.1", 2024-08-18T17:32:44.562 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:32:44.562 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:32:44.562 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:32:44.562 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:29:41.154947Z", 2024-08-18T17:32:44.562 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:32:44.562 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:32:44.562 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:32:44.562 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:32:44.563 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:32:44.563 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:32:44.563 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:32:44.563 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:32:44.563 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:32:44.563 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:32:44Z" 2024-08-18T17:32:44.563 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:32:44.564 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724002313.076576_1734.1 target=1_1724002181.154947_1688.1 2024-08-18T17:32:49.566 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:32:49.566 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:32:49.566 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:32:49.621 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:32:49.621 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:32:49.656 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:32:49.655+0000 7f9f9e6bb880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:32:49.656 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:32:49.656+0000 7f9f9e6bb880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:32:49.671 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:32:49.671 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:32:49.671 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:32:49.671 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:32:49.672 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:32:49.672 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:32:49.672 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:32:49.672 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:32:49.672 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:32:49.672 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:32:49.672 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:32:49.672 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:32:49.672 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:32:49.672 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:32:49.672 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:32:49.672 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:32:49.672 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:32:49.673 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:32:49.673 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:32:49.673 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:32:49.673 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:32:49.673 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:32:49.673 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:32:49.673 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:32:49.673 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:32:49.673 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:32:49.673 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:32:49.673 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:32:49.673 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:32:49.673 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:32:49.674 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:32:49.674 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:32:49.674 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:32:49.674 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:32:49.674 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:32:49.674 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:32:49.674 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:32:49.674 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:32:49.674 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:32:49.674 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:32:49.674 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:32:49.674 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:32:49.674 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:32:49.675 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:32:49.675 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:32:49.675 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:32:49.675 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:32:49.675 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:32:49.675 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:32:49.675 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724002181.154947_1688.1", 2024-08-18T17:32:49.675 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:32:49.675 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:32:49.675 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:32:49.675 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:29:41.154947Z", 2024-08-18T17:32:49.675 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:32:49.675 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:32:49.676 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:32:49.676 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:32:49.676 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:32:49.676 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:32:49.676 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:32:49.676 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:32:49.676 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:32:49.676 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:32:49Z" 2024-08-18T17:32:49.676 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:32:49.678 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724002313.076576_1734.1 target=1_1724002181.154947_1688.1 2024-08-18T17:32:54.678 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:32:54.679 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:32:54.679 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:32:54.734 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:32:54.735 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:32:54.768 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:32:54.767+0000 7f921d936880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:32:54.769 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:32:54.768+0000 7f921d936880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:32:54.783 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:32:54.784 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:32:54.784 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:32:54.784 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:32:54.784 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:32:54.784 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:32:54.784 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:32:54.784 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:32:54.784 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:32:54.784 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:32:54.784 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:32:54.785 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:32:54.785 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:32:54.785 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:32:54.785 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:32:54.785 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:32:54.785 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:32:54.785 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:32:54.785 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:32:54.785 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:32:54.785 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:32:54.786 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:32:54.786 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:32:54.786 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:32:54.786 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:32:54.786 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:32:54.786 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:32:54.786 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:32:54.786 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:32:54.786 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:32:54.787 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:32:54.787 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:32:54.787 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:32:54.787 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:32:54.787 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:32:54.787 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:32:54.787 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:32:54.787 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:32:54.787 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:32:54.787 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:32:54.787 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:32:54.787 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:32:54.787 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:32:54.788 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:32:54.788 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:32:54.788 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:32:54.788 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:32:54.788 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:32:54.788 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:32:54.788 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724002181.154947_1688.1", 2024-08-18T17:32:54.788 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:32:54.788 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:32:54.788 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:32:54.788 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:29:41.154947Z", 2024-08-18T17:32:54.788 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:32:54.789 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:32:54.789 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:32:54.789 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:32:54.789 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:32:54.789 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:32:54.789 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:32:54.789 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:32:54.789 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:32:54.789 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:32:54Z" 2024-08-18T17:32:54.789 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:32:54.790 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724002313.076576_1734.1 target=1_1724002181.154947_1688.1 2024-08-18T17:32:59.791 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:32:59.791 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:32:59.792 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:32:59.847 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:32:59.847 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:32:59.881 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:32:59.880+0000 7f6179933880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:32:59.881 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:32:59.881+0000 7f6179933880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:32:59.896 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:32:59.897 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:32:59.897 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:32:59.897 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:32:59.897 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:32:59.897 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:32:59.897 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:32:59.897 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:32:59.897 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:32:59.897 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:32:59.897 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:32:59.897 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:32:59.897 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:32:59.898 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:32:59.898 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:32:59.898 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:32:59.898 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:32:59.898 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:32:59.898 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:32:59.898 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:32:59.898 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:32:59.898 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:32:59.898 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:32:59.898 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:32:59.898 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:32:59.898 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:32:59.899 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:32:59.899 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:32:59.899 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:32:59.899 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:32:59.899 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:32:59.899 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:32:59.899 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:32:59.899 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:32:59.899 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:32:59.899 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:32:59.899 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:32:59.899 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:32:59.899 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:32:59.899 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:32:59.900 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:32:59.900 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:32:59.900 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:32:59.900 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:32:59.900 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:32:59.900 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:32:59.900 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:32:59.900 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:32:59.900 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:32:59.900 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724002181.154947_1688.1", 2024-08-18T17:32:59.900 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:32:59.901 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:32:59.901 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:32:59.901 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:29:41.154947Z", 2024-08-18T17:32:59.901 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:32:59.901 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:32:59.901 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:32:59.901 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:32:59.901 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:32:59.901 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:32:59.901 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:32:59.902 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:32:59.902 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:32:59.902 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:32:59Z" 2024-08-18T17:32:59.902 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:32:59.903 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724002313.076576_1734.1 target=1_1724002181.154947_1688.1 2024-08-18T17:33:04.904 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:33:04.904 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:33:04.904 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:33:04.959 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:33:04.959 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:33:04.993 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:33:04.992+0000 7f07c4d22880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:33:04.994 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:33:04.993+0000 7f07c4d22880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:33:05.009 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:33:05.009 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:33:05.009 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:33:05.009 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:33:05.009 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:33:05.010 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:33:05.010 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:33:05.010 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:33:05.010 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:33:05.010 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:33:05.010 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:33:05.010 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:33:05.010 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:33:05.010 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:33:05.010 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:33:05.010 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:33:05.011 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:33:05.011 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:33:05.011 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:33:05.011 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:33:05.011 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:33:05.011 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:33:05.011 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:33:05.011 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:33:05.011 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:33:05.011 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:33:05.011 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:33:05.011 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:33:05.011 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:33:05.011 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:33:05.012 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:33:05.012 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:33:05.012 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:33:05.012 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:33:05.012 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:33:05.012 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:33:05.012 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:33:05.012 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:33:05.012 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:33:05.012 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:33:05.012 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:33:05.012 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:33:05.012 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:33:05.013 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:33:05.013 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:33:05.013 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:33:05.013 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:33:05.013 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:33:05.013 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:33:05.013 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724002181.154947_1688.1", 2024-08-18T17:33:05.013 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:33:05.013 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:33:05.013 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:33:05.013 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:29:41.154947Z", 2024-08-18T17:33:05.013 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:33:05.013 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:33:05.014 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:33:05.014 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:33:05.014 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:33:05.014 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:33:05.014 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:33:05.014 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:33:05.014 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:33:05.014 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:33:04Z" 2024-08-18T17:33:05.014 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:33:05.015 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724002313.076576_1734.1 target=1_1724002181.154947_1688.1 2024-08-18T17:33:10.016 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:33:10.016 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:33:10.017 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:33:10.072 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:33:10.072 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:33:10.106 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:33:10.105+0000 7f0000c0e880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:33:10.106 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:33:10.106+0000 7f0000c0e880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:33:10.121 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:33:10.121 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:33:10.121 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:33:10.122 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:33:10.122 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:33:10.122 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:33:10.122 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:33:10.122 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:33:10.122 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:33:10.122 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:33:10.122 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:33:10.122 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:33:10.122 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:33:10.122 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:33:10.122 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:33:10.122 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:33:10.122 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:33:10.123 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:33:10.123 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:33:10.123 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:33:10.123 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:33:10.123 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:33:10.123 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:33:10.123 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:33:10.123 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:33:10.123 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:33:10.123 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:33:10.123 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:33:10.123 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:33:10.123 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:33:10.124 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:33:10.124 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:33:10.124 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:33:10.124 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:33:10.124 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:33:10.124 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:33:10.124 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:33:10.124 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:33:10.124 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:33:10.124 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:33:10.124 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:33:10.124 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:33:10.124 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:33:10.124 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:33:10.125 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:33:10.125 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:33:10.125 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:33:10.125 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:33:10.125 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:33:10.125 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724002181.154947_1688.1", 2024-08-18T17:33:10.125 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:33:10.125 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:33:10.125 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:33:10.125 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:29:41.154947Z", 2024-08-18T17:33:10.125 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:33:10.125 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:33:10.125 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:33:10.125 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:33:10.126 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:33:10.126 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:33:10.126 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:33:10.126 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:33:10.126 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:33:10.126 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:33:10Z" 2024-08-18T17:33:10.126 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:33:10.133 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724002313.076576_1734.1 target=1_1724002181.154947_1688.1 2024-08-18T17:33:15.135 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:33:15.135 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:33:15.135 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:33:15.190 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:33:15.190 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:33:15.224 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:33:15.223+0000 7f4715538880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:33:15.225 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:33:15.224+0000 7f4715538880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:33:15.239 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:33:15.240 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:33:15.240 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:33:15.240 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:33:15.240 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:33:15.240 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:33:15.241 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:33:15.241 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:33:15.241 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:33:15.241 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:33:15.241 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:33:15.241 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:33:15.241 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:33:15.241 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:33:15.242 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:33:15.242 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:33:15.242 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:33:15.242 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:33:15.242 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:33:15.242 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:33:15.242 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:33:15.242 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:33:15.243 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:33:15.243 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:33:15.243 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:33:15.243 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:33:15.243 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:33:15.243 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:33:15.244 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:33:15.245 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:33:15.245 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:33:15.245 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:33:15.245 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:33:15.245 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:33:15.245 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:33:15.245 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:33:15.245 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:33:15.245 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:33:15.245 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:33:15.245 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:33:15.245 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:33:15.245 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:33:15.246 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:33:15.246 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:33:15.246 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:33:15.246 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:33:15.246 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:33:15.246 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:33:15.246 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:33:15.246 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724002181.154947_1688.1", 2024-08-18T17:33:15.246 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:33:15.246 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:33:15.246 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:33:15.246 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:29:41.154947Z", 2024-08-18T17:33:15.247 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:33:15.247 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:33:15.247 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:33:15.247 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:33:15.247 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:33:15.247 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:33:15.247 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:33:15.247 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:33:15.247 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:33:15.247 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:33:15Z" 2024-08-18T17:33:15.247 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:33:15.248 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724002313.076576_1734.1 target=1_1724002181.154947_1688.1 2024-08-18T17:33:20.249 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:33:20.249 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:33:20.249 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:33:20.304 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:33:20.304 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:33:20.338 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:33:20.337+0000 7f31a2813880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:33:20.339 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:33:20.338+0000 7f31a2813880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:33:20.354 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:33:20.354 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:33:20.354 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:33:20.354 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:33:20.354 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:33:20.354 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:33:20.354 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:33:20.354 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:33:20.354 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:33:20.355 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:33:20.355 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:33:20.355 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:33:20.355 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:33:20.355 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:33:20.355 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:33:20.355 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:33:20.355 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:33:20.355 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:33:20.355 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:33:20.355 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:33:20.355 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:33:20.356 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:33:20.356 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:33:20.356 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:33:20.356 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:33:20.356 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:33:20.356 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:33:20.356 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:33:20.356 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:33:20.356 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:33:20.356 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:33:20.356 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:33:20.356 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:33:20.357 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:33:20.357 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:33:20.357 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:33:20.357 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:33:20.357 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:33:20.357 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:33:20.357 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:33:20.357 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:33:20.357 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:33:20.357 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:33:20.357 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:33:20.357 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:33:20.357 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:33:20.358 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:33:20.358 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:33:20.358 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:33:20.358 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724002181.154947_1688.1", 2024-08-18T17:33:20.358 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:33:20.358 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:33:20.358 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:33:20.358 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:29:41.154947Z", 2024-08-18T17:33:20.358 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:33:20.358 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:33:20.359 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:33:20.359 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:33:20.359 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:33:20.359 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:33:20.359 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:33:20.359 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:33:20.359 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:33:20.359 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:33:20Z" 2024-08-18T17:33:20.359 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:33:20.360 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724002313.076576_1734.1 target=1_1724002181.154947_1688.1 2024-08-18T17:33:25.361 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:33:25.362 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:33:25.362 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:33:25.416 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:33:25.417 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:33:25.450 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:33:25.449+0000 7f7fc3526880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:33:25.451 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:33:25.450+0000 7f7fc3526880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:33:25.471 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:33:25.471 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:33:25.471 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:33:25.471 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:33:25.471 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:33:25.471 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:33:25.471 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:33:25.471 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:33:25.471 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:33:25.472 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:33:25.472 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:33:25.472 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:33:25.472 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:33:25.472 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:33:25.472 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:33:25.472 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:33:25.472 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:33:25.472 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:33:25.472 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:33:25.472 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:33:25.472 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:33:25.473 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:33:25.473 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:33:25.473 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:33:25.473 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:33:25.473 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:33:25.473 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:33:25.473 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:33:25.473 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:33:25.473 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:33:25.473 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:33:25.473 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:33:25.473 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:33:25.473 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:33:25.473 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:33:25.474 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:33:25.474 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:33:25.474 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:33:25.474 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:33:25.474 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:33:25.474 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:33:25.474 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:33:25.474 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:33:25.474 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:33:25.474 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:33:25.474 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:33:25.474 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:33:25.474 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:33:25.475 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:33:25.475 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724002181.154947_1688.1", 2024-08-18T17:33:25.475 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:33:25.475 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:33:25.475 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:33:25.475 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:29:41.154947Z", 2024-08-18T17:33:25.475 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:33:25.475 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:33:25.475 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:33:25.475 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:33:25.475 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:33:25.475 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:33:25.475 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:33:25.476 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:33:25.476 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:33:25.476 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:33:25Z" 2024-08-18T17:33:25.476 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:33:25.476 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724002313.076576_1734.1 target=1_1724002181.154947_1688.1 2024-08-18T17:33:30.477 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:33:30.478 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:33:30.478 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:33:30.533 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:33:30.533 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:33:30.567 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:33:30.566+0000 7f4046522880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:33:30.568 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:33:30.567+0000 7f4046522880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:33:30.582 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:33:30.583 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:33:30.583 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:33:30.583 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:33:30.583 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:33:30.583 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:33:30.583 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:33:30.583 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:33:30.583 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:33:30.583 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:33:30.583 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:33:30.583 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:33:30.583 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:33:30.584 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:33:30.584 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:33:30.584 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:33:30.584 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:33:30.584 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:33:30.584 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:33:30.584 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:33:30.584 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:33:30.584 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:33:30.584 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:33:30.584 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:33:30.584 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:33:30.584 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:33:30.585 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:33:30.585 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:33:30.585 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:33:30.585 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:33:30.585 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:33:30.585 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:33:30.585 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:33:30.585 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:33:30.585 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:33:30.585 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:33:30.585 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:33:30.585 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:33:30.585 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:33:30.586 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:33:30.586 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:33:30.586 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:33:30.586 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:33:30.586 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:33:30.586 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:33:30.586 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:33:30.586 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:33:30.586 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:33:30.586 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:33:30.586 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724002181.154947_1688.1", 2024-08-18T17:33:30.586 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:33:30.586 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:33:30.587 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:33:30.587 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:29:41.154947Z", 2024-08-18T17:33:30.587 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:33:30.587 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:33:30.587 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:33:30.587 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:33:30.587 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:33:30.587 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:33:30.587 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:33:30.587 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:33:30.587 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:33:30.587 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:33:30Z" 2024-08-18T17:33:30.587 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:33:30.588 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724002313.076576_1734.1 target=1_1724002181.154947_1688.1 2024-08-18T17:33:35.589 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:33:35.590 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:33:35.590 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:33:35.645 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:33:35.645 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:33:35.679 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:33:35.678+0000 7fb107f20880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:33:35.680 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:33:35.679+0000 7fb107f20880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:33:35.694 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:33:35.695 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:33:35.695 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:33:35.695 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:33:35.695 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:33:35.695 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:33:35.695 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:33:35.695 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:33:35.695 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:33:35.695 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:33:35.695 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:33:35.695 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:33:35.695 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:33:35.696 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:33:35.696 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:33:35.696 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:33:35.696 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:33:35.696 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:33:35.696 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:33:35.696 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:33:35.696 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:33:35.696 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:33:35.696 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:33:35.696 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:33:35.696 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:33:35.696 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:33:35.696 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:33:35.697 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:33:35.697 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:33:35.697 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:33:35.697 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:33:35.697 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:33:35.697 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:33:35.697 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:33:35.697 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:33:35.697 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:33:35.697 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:33:35.697 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:33:35.697 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:33:35.698 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:33:35.698 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:33:35.698 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:33:35.698 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:33:35.698 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:33:35.698 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:33:35.698 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:33:35.698 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:33:35.698 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:33:35.698 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:33:35.698 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724002181.154947_1688.1", 2024-08-18T17:33:35.699 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:33:35.699 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:33:35.699 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:33:35.699 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:29:41.154947Z", 2024-08-18T17:33:35.699 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:33:35.699 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:33:35.699 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:33:35.699 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:33:35.699 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:33:35.699 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:33:35.699 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:33:35.699 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:33:35.699 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:33:35.699 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:33:35Z" 2024-08-18T17:33:35.700 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:33:35.700 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724002313.076576_1734.1 target=1_1724002181.154947_1688.1 2024-08-18T17:33:40.702 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:33:40.702 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:33:40.702 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:33:40.758 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:33:40.758 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:33:40.792 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:33:40.790+0000 7ffb68f32880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:33:40.793 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:33:40.791+0000 7ffb68f32880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:33:40.807 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:33:40.808 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:33:40.808 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:33:40.808 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:33:40.808 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:33:40.808 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:33:40.808 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:33:40.808 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:33:40.808 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:33:40.808 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:33:40.808 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:33:40.808 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:33:40.808 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:33:40.808 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:33:40.809 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:33:40.809 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:33:40.809 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:33:40.809 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:33:40.809 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:33:40.809 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:33:40.809 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:33:40.809 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:33:40.809 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:33:40.809 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:33:40.809 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:33:40.809 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:33:40.809 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:33:40.809 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:33:40.809 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:33:40.810 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:33:40.810 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:33:40.810 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:33:40.810 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:33:40.810 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:33:40.810 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:33:40.810 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:33:40.810 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:33:40.810 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:33:40.810 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:33:40.811 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:33:40.811 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:33:40.811 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:33:40.811 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:33:40.811 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:33:40.811 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:33:40.811 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:33:40.811 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:33:40.811 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:33:40.811 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:33:40.811 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724002181.154947_1688.1", 2024-08-18T17:33:40.811 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:33:40.811 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:33:40.812 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:33:40.812 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:29:41.154947Z", 2024-08-18T17:33:40.812 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:33:40.812 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:33:40.812 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:33:40.812 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:33:40.812 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:33:40.812 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:33:40.812 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:33:40.812 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:33:40.812 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:33:40.812 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:33:40Z" 2024-08-18T17:33:40.812 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:33:40.813 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724002313.076576_1734.1 target=1_1724002181.154947_1688.1 2024-08-18T17:33:45.814 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:33:45.814 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:33:45.815 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:33:45.870 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:33:45.870 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:33:45.903 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:33:45.902+0000 7fbc16738880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:33:45.904 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:33:45.903+0000 7fbc16738880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:33:45.919 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:33:45.919 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:33:45.919 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:33:45.919 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:33:45.920 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:33:45.920 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:33:45.920 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:33:45.920 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:33:45.920 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:33:45.920 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:33:45.920 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:33:45.920 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:33:45.920 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:33:45.920 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:33:45.920 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:33:45.920 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:33:45.921 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:33:45.921 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:33:45.921 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:33:45.921 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:33:45.921 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:33:45.921 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:33:45.921 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:33:45.921 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:33:45.921 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:33:45.921 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:33:45.921 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:33:45.921 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:33:45.922 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:33:45.922 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:33:45.922 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:33:45.922 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:33:45.922 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:33:45.922 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:33:45.922 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:33:45.922 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:33:45.922 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:33:45.922 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:33:45.922 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:33:45.922 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:33:45.923 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:33:45.923 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:33:45.923 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:33:45.923 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:33:45.923 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:33:45.923 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:33:45.923 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:33:45.923 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:33:45.923 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:33:45.923 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724002181.154947_1688.1", 2024-08-18T17:33:45.923 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:33:45.923 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:33:45.923 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:33:45.924 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:29:41.154947Z", 2024-08-18T17:33:45.924 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:33:45.924 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:33:45.924 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:33:45.924 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:33:45.924 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:33:45.924 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:33:45.924 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:33:45.924 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:33:45.924 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:33:45.924 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:33:45Z" 2024-08-18T17:33:45.924 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:33:45.926 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724002313.076576_1734.1 target=1_1724002181.154947_1688.1 2024-08-18T17:33:50.927 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:33:50.927 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:33:50.928 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:33:50.983 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:33:50.984 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:33:51.017 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:33:51.016+0000 7f072ca10880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:33:51.017 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:33:51.017+0000 7f072ca10880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:33:51.032 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:33:51.033 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:33:51.033 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:33:51.033 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:33:51.033 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:33:51.033 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:33:51.033 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:33:51.033 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:33:51.033 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:33:51.033 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:33:51.033 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:33:51.033 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:33:51.033 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:33:51.034 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:33:51.034 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:33:51.034 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:33:51.034 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:33:51.034 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:33:51.034 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:33:51.034 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:33:51.034 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:33:51.034 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:33:51.034 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:33:51.034 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:33:51.035 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:33:51.035 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:33:51.035 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:33:51.035 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:33:51.035 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:33:51.035 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:33:51.035 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:33:51.035 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:33:51.035 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:33:51.035 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:33:51.035 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:33:51.035 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:33:51.035 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:33:51.036 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:33:51.036 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:33:51.036 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:33:51.036 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:33:51.036 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:33:51.036 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:33:51.036 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:33:51.036 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:33:51.036 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:33:51.036 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:33:51.036 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:33:51.036 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:33:51.036 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724002181.154947_1688.1", 2024-08-18T17:33:51.037 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:33:51.037 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:33:51.037 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:33:51.037 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:29:41.154947Z", 2024-08-18T17:33:51.037 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:33:51.037 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:33:51.037 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:33:51.037 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:33:51.037 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:33:51.037 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:33:51.037 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:33:51.037 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:33:51.037 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:33:51.037 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:33:51Z" 2024-08-18T17:33:51.038 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:33:51.039 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724002313.076576_1734.1 target=1_1724002181.154947_1688.1 2024-08-18T17:33:56.040 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:33:56.040 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:33:56.041 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:33:56.096 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:33:56.096 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:33:56.133 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:33:56.132+0000 7f55acf27880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:33:56.134 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:33:56.133+0000 7f55acf27880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:33:56.149 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:33:56.149 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:33:56.149 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:33:56.149 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:33:56.149 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:33:56.149 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:33:56.150 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:33:56.150 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:33:56.150 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:33:56.150 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:33:56.150 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:33:56.150 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:33:56.150 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:33:56.150 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:33:56.150 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:33:56.150 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:33:56.150 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:33:56.150 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:33:56.150 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:33:56.151 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:33:56.151 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:33:56.151 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:33:56.151 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:33:56.151 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:33:56.151 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:33:56.151 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:33:56.151 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:33:56.151 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:33:56.151 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:33:56.151 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:33:56.151 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:33:56.151 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:33:56.151 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:33:56.152 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:33:56.152 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:33:56.152 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:33:56.152 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:33:56.152 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:33:56.152 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:33:56.152 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:33:56.152 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:33:56.152 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:33:56.152 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:33:56.152 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:33:56.152 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:33:56.152 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:33:56.152 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:33:56.153 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:33:56.153 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:33:56.153 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724002312.341828_1715.1", 2024-08-18T17:33:56.153 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:33:56.153 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:33:56.153 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:33:56.153 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:31:52.341828Z", 2024-08-18T17:33:56.153 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:33:56.153 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:33:56.153 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:33:56.153 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:33:56.153 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:33:56.153 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:33:56.153 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:33:56.154 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:33:56.154 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:33:56.154 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:33:56Z" 2024-08-18T17:33:56.154 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:33:56.155 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1724002313.076576_1734.1 target=1_1724002312.341828_1715.1 2024-08-18T17:34:01.155 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:34:01.156 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:34:01.156 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:34:01.210 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:34:01.211 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:34:01.245 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:34:01.244+0000 7faff6c0f880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:34:01.245 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:34:01.245+0000 7faff6c0f880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:34:01.260 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:34:01.260 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:34:01.260 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:34:01.260 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:34:01.260 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:34:01.261 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:34:01.261 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:34:01.261 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:34:01.261 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:34:01.261 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:34:01.261 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:34:01.261 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:34:01.261 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:34:01.261 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:34:01.261 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:34:01.261 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:34:01.261 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:34:01.261 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:34:01.262 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:34:01.262 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:34:01.262 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:34:01.262 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:34:01.262 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:34:01.262 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:34:01.262 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:34:01.262 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:34:01.262 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:34:01.262 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:34:01.262 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:34:01.262 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:34:01.262 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:34:01.263 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:34:01.263 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:34:01.263 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:34:01.263 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:34:01.263 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:34:01.263 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:34:01.263 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:34:01.263 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:34:01.263 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:34:01.263 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:34:01.263 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:34:01.263 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:34:01.263 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:34:01.263 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:34:01.264 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:34:01.264 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:34:01.264 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:34:01.264 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:34:01.264 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724002313.076576_1734.1", 2024-08-18T17:34:01.264 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:34:01.264 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:34:01.264 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:34:01.264 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:31:53.076576Z", 2024-08-18T17:34:01.264 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:34:01.264 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:34:01.264 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:34:01.264 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:34:01.265 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:34:01.265 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:34:01.265 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:34:01.265 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:34:01.265 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:34:01.265 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:34:01Z" 2024-08-18T17:34:01.265 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:34:01.266 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-08-18T17:34:01.266 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-08-18T17:34:01.266 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:34:01.266 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:34:01.267 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:34:01.324 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:34:01.324 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:34:01.370 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:34:01.369+0000 7f40b3412880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:34:01.371 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:34:01.370+0000 7f40b3412880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:34:01.388 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T17:34:01.388 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T17:34:01.388 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T17:34:01.389 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T17:34:01.389 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T17:34:01.389 INFO:teuthology.orchestra.run.smithi134.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:34:01.389 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:34:01.389 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:34:01.389 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T17:34:01.389 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:34:01.389 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T17:34:01.389 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:34:01.389 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:34:01.389 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:34:01.389 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:34:01.389 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 1, 2024-08-18T17:34:01.390 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:34:01.390 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:34:01.390 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T17:34:01.390 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:34:01.390 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:34:01.390 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:34:01.390 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T17:34:01.390 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:34:01.390 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:34:01.390 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:34:01.390 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:34:01.390 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:34:01.390 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:34:01.390 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T17:34:01.391 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T17:34:01.391 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:34:01.391 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:34:01.391 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:34:01.391 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T17:34:01.392 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:34:01.392 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:34:01.392 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:34:01.392 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:34:01.392 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:34:01.392 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:34:01.392 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:34:01.392 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:34:01.392 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:34:01.392 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:34:01.392 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:34:01.392 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T17:34:01.393 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:34:01.393 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:34:01.393 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1724002313.076576_1734.1", 2024-08-18T17:34:01.393 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:34:01.393 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:34:01.393 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:34:01.393 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T17:31:53.076576Z", 2024-08-18T17:34:01.393 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:34:01.393 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:34:01.393 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:34:01.393 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:34:01.393 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:34:01.393 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T17:34:01.393 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 1, 2024-08-18T17:34:01.394 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 1 2024-08-18T17:34:01.394 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:34:01.394 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T17:34:01Z" 2024-08-18T17:34:01.394 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T17:34:01.395 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-08-18T17:34:01.420 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:34:01.420 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:34:01.420 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:34:01.468 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:34:01.468 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:34:01.521 INFO:teuthology.orchestra.run.smithi134.stderr:No --period given, using current period=01050e30-179b-43be-bdbb-3bb23cf37ae8 2024-08-18T17:34:01.538 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T17:34:01.539 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:34:01.539 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:34:01.539 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T17:34:01.539 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:34:01.539 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:34:01.539 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:34:01.539 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "0.000000" 2024-08-18T17:34:01.539 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:34:01.539 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:34:01.542 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:34:01.542 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T16:42:47.672461Z" 2024-08-18T17:34:01.542 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:34:01.543 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:34:01.543 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1724002313.076576_1734.1", 2024-08-18T17:34:01.543 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T17:31:53.076576Z" 2024-08-18T17:34:01.543 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:34:01.543 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T17:34:01.544 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-08-18T17:34:01.544 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:34:01.544 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:34:01.544 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:34:01.596 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:34:01.597 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:34:01.629 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:34:01.628+0000 7f5730411880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:34:01.629 INFO:teuthology.orchestra.run.smithi193.stderr:2024-08-18T17:34:01.629+0000 7f5730411880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:34:01.644 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:34:01.644 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:34:01.644 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:34:01.644 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:34:01.644 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:34:01.644 INFO:teuthology.orchestra.run.smithi193.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:34:01.644 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:34:01.644 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:34:01.644 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:34:01.644 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:34:01.645 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:34:01.645 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:34:01.645 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:34:01.645 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:34:01.645 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:34:01.645 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 1, 2024-08-18T17:34:01.645 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:34:01.645 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:34:01.645 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:34:01.645 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:34:01.645 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:34:01.645 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:34:01.646 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:34:01.646 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:34:01.646 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:34:01.646 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "", 2024-08-18T17:34:01.646 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:34:01.646 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:34:01.646 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:34:01.646 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "0.000000", 2024-08-18T17:34:01.646 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 2 2024-08-18T17:34:01.646 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:34:01.646 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:34:01.646 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:34:01.646 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:34:01.647 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:34:01.647 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:34:01.647 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:34:01.647 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:34:01.647 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:34:01.647 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:34:01.647 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:34:01.647 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:34:01.647 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:34:01.648 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:34:01.648 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:34:01.648 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:34:01.648 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:34:01.648 INFO:teuthology.orchestra.run.smithi193.stdout: "state": 1, 2024-08-18T17:34:01.649 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "1_1724002313.076576_1734.1", 2024-08-18T17:34:01.649 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:34:01.649 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:34:01.649 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:34:01.649 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:31:53.076576Z", 2024-08-18T17:34:01.649 INFO:teuthology.orchestra.run.smithi193.stdout: "realm_epoch": 4 2024-08-18T17:34:01.649 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:34:01.649 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:34:01.649 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:34:01.649 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:34:01.650 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:34:01.650 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 1, 2024-08-18T17:34:01.650 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 1 2024-08-18T17:34:01.650 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:34:01.650 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:34:01Z" 2024-08-18T17:34:01.650 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:34:01.653 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-08-18T17:34:01.654 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-08-18T17:34:01.654 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:34:01.654 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:34:01.654 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:34:01.702 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:34:01.702 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:34:01.734 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:34:01.733+0000 7fa6a2b36880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:34:01.735 INFO:teuthology.orchestra.run.smithi134.stderr:2024-08-18T17:34:01.734+0000 7fa6a2b36880 20 RGW-SYNC:meta: read sync status 2024-08-18T17:34:01.752 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-08-18T17:34:01.753 INFO:teuthology.orchestra.run.smithi134.stdout: "sync_status": { 2024-08-18T17:34:01.753 INFO:teuthology.orchestra.run.smithi134.stdout: "info": { 2024-08-18T17:34:01.753 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "sync", 2024-08-18T17:34:01.753 INFO:teuthology.orchestra.run.smithi134.stdout: "num_shards": 4, 2024-08-18T17:34:01.753 INFO:teuthology.orchestra.run.smithi134.stdout: "period": "01050e30-179b-43be-bdbb-3bb23cf37ae8", 2024-08-18T17:34:01.754 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:34:01.754 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:34:01.754 INFO:teuthology.orchestra.run.smithi134.stdout: "markers": [ 2024-08-18T17:34:01.755 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:34:01.755 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 0, 2024-08-18T17:34:01.755 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:34:01.755 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:34:01.755 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723994364.506194_23.1", 2024-08-18T17:34:01.755 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:34:01.756 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 1, 2024-08-18T17:34:01.756 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:34:01.756 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T15:19:24.506194Z", 2024-08-18T17:34:01.756 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T17:34:01.756 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:34:01.757 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:34:01.757 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:34:01.757 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 1, 2024-08-18T17:34:01.757 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:34:01.757 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:34:01.757 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "", 2024-08-18T17:34:01.758 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:34:01.758 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:34:01.758 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:34:01.758 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "0.000000", 2024-08-18T17:34:01.758 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 2 2024-08-18T17:34:01.758 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:34:01.759 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:34:01.759 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:34:01.759 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 2, 2024-08-18T17:34:01.759 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:34:01.759 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:34:01.759 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1723999367.672461_1179.1", 2024-08-18T17:34:01.760 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:34:01.760 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:34:01.760 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:34:01.760 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T16:42:47.672461Z", 2024-08-18T17:34:01.760 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:34:01.760 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:34:01.760 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:34:01.761 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:34:01.761 INFO:teuthology.orchestra.run.smithi134.stdout: "key": 3, 2024-08-18T17:34:01.761 INFO:teuthology.orchestra.run.smithi134.stdout: "val": { 2024-08-18T17:34:01.761 INFO:teuthology.orchestra.run.smithi134.stdout: "state": 1, 2024-08-18T17:34:01.761 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "1_1724002313.076576_1734.1", 2024-08-18T17:34:01.761 INFO:teuthology.orchestra.run.smithi134.stdout: "next_step_marker": "", 2024-08-18T17:34:01.761 INFO:teuthology.orchestra.run.smithi134.stdout: "total_entries": 0, 2024-08-18T17:34:01.761 INFO:teuthology.orchestra.run.smithi134.stdout: "pos": 0, 2024-08-18T17:34:01.761 INFO:teuthology.orchestra.run.smithi134.stdout: "timestamp": "2024-08-18T17:31:53.076576Z", 2024-08-18T17:34:01.761 INFO:teuthology.orchestra.run.smithi134.stdout: "realm_epoch": 4 2024-08-18T17:34:01.761 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:34:01.761 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:34:01.762 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-08-18T17:34:01.762 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:34:01.762 INFO:teuthology.orchestra.run.smithi134.stdout: "full_sync": { 2024-08-18T17:34:01.762 INFO:teuthology.orchestra.run.smithi134.stdout: "total": 1, 2024-08-18T17:34:01.762 INFO:teuthology.orchestra.run.smithi134.stdout: "complete": 1 2024-08-18T17:34:01.762 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:34:01.762 INFO:teuthology.orchestra.run.smithi134.stdout: "current_time": "2024-08-18T17:34:01Z" 2024-08-18T17:34:01.763 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-08-18T17:34:01.764 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-08-18T17:34:01.765 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:34:01.765 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:34:01.765 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:34:01.812 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:34:01.812 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:34:01.895 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-08-18T17:34:01.895 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:34:01.895 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000052574", 2024-08-18T17:34:01.895 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T17:29:47.854757Z" 2024-08-18T17:34:01.895 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:34:01.895 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:34:01.895 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000037695", 2024-08-18T17:34:01.896 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T17:34:01.419536Z" 2024-08-18T17:34:01.896 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:34:01.896 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:34:01.896 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000060258", 2024-08-18T17:34:01.896 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T17:34:01.410750Z" 2024-08-18T17:34:01.896 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-08-18T17:34:01.897 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-08-18T17:34:01.897 INFO:teuthology.orchestra.run.smithi134.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:34:01.897 INFO:teuthology.orchestra.run.smithi134.stdout: "last_update": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:34:01.897 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-08-18T17:34:01.897 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-08-18T17:34:01.900 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-08-18T17:34:01.900 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:34:01.900 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:34:01.900 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:34:01.954 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:34:01.954 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:34:01.998 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:34:01.999 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:34:01.999 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:34:01.999 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:34:01.999 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:34:01.999 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:34:01.999 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:34:01.999 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:34:02.000 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:34:02.000 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:34:02.000 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:34:02.000 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:34:02.000 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052574", 2024-08-18T17:34:02.000 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:34:02.000 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:34:02.001 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:34:02.001 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:29:47.854757Z" 2024-08-18T17:34:02.001 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:34:02.001 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:34:02.001 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:34:02.001 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:34:02.002 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:34:02.002 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:34:02.002 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037555", 2024-08-18T17:34:02.002 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:34:02.002 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:34:02.003 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:34:02.003 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:27:37.043496Z" 2024-08-18T17:34:02.003 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:34:02.003 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:34:02.003 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:34:02.004 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:34:02.004 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:34:02.004 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:34:02.004 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000060117", 2024-08-18T17:34:02.004 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:34:02.004 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:34:02.005 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:34:02.005 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:29:47.826846Z" 2024-08-18T17:34:02.005 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:34:02.006 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:34:02.006 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:34:02.006 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:34:02.006 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:34:02.006 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:34:02.006 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:34:02.007 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:34:02.007 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:34:02.007 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:34:02.007 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:34:02.008 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:34:02.008 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:34:02.008 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:34:02.008 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:34:02.008 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:34:02.009 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:34:02.009 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:34:02.009 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:34:02.009 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:34:01Z" 2024-08-18T17:34:02.009 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:34:02.010 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000037695 target=00000000000000000000:00000000000000037555, shard=2 master=00000000000000000000:00000000000000060258 target=00000000000000000000:00000000000000060117 2024-08-18T17:34:07.011 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:34:07.011 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:34:07.011 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:34:07.069 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:34:07.069 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:34:07.122 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:34:07.123 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:34:07.123 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:34:07.123 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:34:07.123 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:34:07.123 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:34:07.123 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:34:07.123 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:34:07.123 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:34:07.123 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:34:07.123 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:34:07.123 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:34:07.124 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052574", 2024-08-18T17:34:07.124 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:34:07.124 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:34:07.124 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:34:07.124 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:29:47.854757Z" 2024-08-18T17:34:07.124 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:34:07.124 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:34:07.124 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:34:07.124 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:34:07.124 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:34:07.124 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:34:07.124 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037555", 2024-08-18T17:34:07.125 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:34:07.125 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:34:07.125 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:34:07.125 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:27:37.043496Z" 2024-08-18T17:34:07.125 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:34:07.125 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:34:07.125 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:34:07.125 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:34:07.125 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:34:07.125 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:34:07.126 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000060117", 2024-08-18T17:34:07.126 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:34:07.126 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:34:07.126 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:34:07.126 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:29:47.826846Z" 2024-08-18T17:34:07.126 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:34:07.126 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:34:07.126 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:34:07.126 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:34:07.126 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:34:07.126 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:34:07.126 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:34:07.127 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:34:07.127 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:34:07.127 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:34:07.127 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:34:07.127 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:34:07.127 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:34:07.127 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:34:07.127 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:34:07.127 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:34:07.128 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:34:07.128 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:34:07.128 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:34:07.128 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:34:07Z" 2024-08-18T17:34:07.128 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:34:07.129 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000037695 target=00000000000000000000:00000000000000037555, shard=2 master=00000000000000000000:00000000000000060258 target=00000000000000000000:00000000000000060117 2024-08-18T17:34:12.130 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:34:12.130 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:34:12.131 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:34:12.187 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:34:12.188 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:34:12.239 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:34:12.239 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:34:12.239 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:34:12.239 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:34:12.239 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:34:12.239 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:34:12.239 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:34:12.239 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:34:12.239 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:34:12.240 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:34:12.240 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:34:12.240 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:34:12.240 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052574", 2024-08-18T17:34:12.240 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:34:12.240 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:34:12.240 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:34:12.240 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:29:47.854757Z" 2024-08-18T17:34:12.240 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:34:12.240 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:34:12.240 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:34:12.240 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:34:12.240 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:34:12.240 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:34:12.241 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037555", 2024-08-18T17:34:12.241 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:34:12.241 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:34:12.241 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:34:12.241 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:27:37.043496Z" 2024-08-18T17:34:12.241 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:34:12.241 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:34:12.241 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:34:12.241 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:34:12.241 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:34:12.241 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:34:12.241 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000060117", 2024-08-18T17:34:12.241 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:34:12.242 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:34:12.242 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:34:12.242 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:29:47.826846Z" 2024-08-18T17:34:12.242 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:34:12.242 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:34:12.242 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:34:12.242 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:34:12.242 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:34:12.242 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:34:12.242 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:34:12.242 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:34:12.242 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:34:12.242 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:34:12.243 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:34:12.243 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:34:12.243 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:34:12.243 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:34:12.243 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:34:12.243 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:34:12.243 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:34:12.243 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:34:12.244 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:34:12.244 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:34:12Z" 2024-08-18T17:34:12.244 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:34:12.245 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000037695 target=00000000000000000000:00000000000000037555, shard=2 master=00000000000000000000:00000000000000060258 target=00000000000000000000:00000000000000060117 2024-08-18T17:34:17.245 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:34:17.246 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-18T17:34:17.246 DEBUG:teuthology.orchestra.run.smithi193:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-18T17:34:17.301 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:34:17.301 INFO:teuthology.orchestra.run.smithi193.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:34:17.350 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-08-18T17:34:17.350 INFO:teuthology.orchestra.run.smithi193.stdout: "sync_status": { 2024-08-18T17:34:17.350 INFO:teuthology.orchestra.run.smithi193.stdout: "info": { 2024-08-18T17:34:17.351 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "sync", 2024-08-18T17:34:17.351 INFO:teuthology.orchestra.run.smithi193.stdout: "num_shards": 4, 2024-08-18T17:34:17.351 INFO:teuthology.orchestra.run.smithi193.stdout: "instance_id": 17341079190069921619 2024-08-18T17:34:17.351 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:34:17.351 INFO:teuthology.orchestra.run.smithi193.stdout: "markers": [ 2024-08-18T17:34:17.351 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:34:17.351 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 0, 2024-08-18T17:34:17.351 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:34:17.351 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:34:17.351 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052574", 2024-08-18T17:34:17.351 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:34:17.351 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:34:17.351 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:34:17.352 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:29:47.854757Z" 2024-08-18T17:34:17.352 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:34:17.352 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:34:17.352 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:34:17.352 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 1, 2024-08-18T17:34:17.352 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:34:17.352 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:34:17.352 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000037695", 2024-08-18T17:34:17.352 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:34:17.352 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 0, 2024-08-18T17:34:17.352 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:34:17.352 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:34:01.419536Z" 2024-08-18T17:34:17.352 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:34:17.352 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:34:17.352 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:34:17.353 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 2, 2024-08-18T17:34:17.353 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:34:17.353 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:34:17.353 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000060258", 2024-08-18T17:34:17.353 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:34:17.353 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 74, 2024-08-18T17:34:17.353 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:34:17.353 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:34:01.410750Z" 2024-08-18T17:34:17.353 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:34:17.353 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:34:17.353 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-08-18T17:34:17.353 INFO:teuthology.orchestra.run.smithi193.stdout: "key": 3, 2024-08-18T17:34:17.354 INFO:teuthology.orchestra.run.smithi193.stdout: "val": { 2024-08-18T17:34:17.354 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "incremental-sync", 2024-08-18T17:34:17.354 INFO:teuthology.orchestra.run.smithi193.stdout: "marker": "00000000000000000000:00000000000000052210", 2024-08-18T17:34:17.354 INFO:teuthology.orchestra.run.smithi193.stdout: "next_step_marker": "", 2024-08-18T17:34:17.354 INFO:teuthology.orchestra.run.smithi193.stdout: "total_entries": 9, 2024-08-18T17:34:17.354 INFO:teuthology.orchestra.run.smithi193.stdout: "pos": 0, 2024-08-18T17:34:17.354 INFO:teuthology.orchestra.run.smithi193.stdout: "timestamp": "2024-08-18T17:19:10.154891Z" 2024-08-18T17:34:17.354 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:34:17.354 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-08-18T17:34:17.354 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-08-18T17:34:17.355 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:34:17.355 INFO:teuthology.orchestra.run.smithi193.stdout: "full_sync": { 2024-08-18T17:34:17.355 INFO:teuthology.orchestra.run.smithi193.stdout: "total": 83, 2024-08-18T17:34:17.355 INFO:teuthology.orchestra.run.smithi193.stdout: "complete": 83 2024-08-18T17:34:17.355 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-08-18T17:34:17.355 INFO:teuthology.orchestra.run.smithi193.stdout: "current_time": "2024-08-18T17:34:17Z" 2024-08-18T17:34:17.355 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-08-18T17:34:17.356 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-08-18T17:34:17.377 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'kibgmb-97', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:34:17.378 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'kibgmb-97', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:34:17.378 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-bucket --bucket kibgmb-97 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:34:17.433 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:34:17.434 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:34:17.516 INFO:teuthology.orchestra.run.smithi134.stdout:{"sync_policy":{"groups":[]}} 2024-08-18T17:34:17.516 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:34:17.516 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-18T17:34:17.517 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-18T17:34:17.562 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setuser ceph since I am not root 2024-08-18T17:34:17.562 INFO:teuthology.orchestra.run.smithi134.stderr:ignoring --setgroup ceph since I am not root 2024-08-18T17:34:17.631 INFO:teuthology.orchestra.run.smithi134.stdout:{"sync_policy":{"groups":[]}} 2024-08-18T17:34:17.632 INFO:tasks.rgw_multisite_tests:test_sync_single_bucket_to_multiple: ... ok 2024-08-18T17:34:17.632 INFO:tasks.rgw_multisite_tests: 2024-08-18T17:34:17.632 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-08-18T17:34:17.633 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_multi_object_delete 2024-08-18T17:34:17.633 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-08-18T17:34:17.633 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-08-18T17:34:17.633 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/virtualenv/lib/python3.10/site-packages/nose/case.py", line 170, in runTest 2024-08-18T17:34:17.633 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-08-18T17:34:17.633 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_03f4c3e85ce172201896147a7aef87ba256187f2/qa/../src/test/rgw/rgw_multi/tests.py", line 687, in test_multi_object_delete 2024-08-18T17:34:17.633 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket.name) 2024-08-18T17:34:17.633 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_03f4c3e85ce172201896147a7aef87ba256187f2/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2024-08-18T17:34:17.633 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2024-08-18T17:34:17.633 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_03f4c3e85ce172201896147a7aef87ba256187f2/qa/tasks/rgw_multisite.py", line 197, in admin 2024-08-18T17:34:17.634 INFO:tasks.rgw_multisite_tests: assert r == 0 2024-08-18T17:34:17.634 INFO:tasks.rgw_multisite_tests:AssertionError 2024-08-18T17:34:17.634 INFO:tasks.rgw_multisite_tests: 2024-08-18T17:34:17.634 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-08-18T17:34:17.634 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_sync_disable 2024-08-18T17:34:17.634 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-08-18T17:34:17.634 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-08-18T17:34:17.634 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/virtualenv/lib/python3.10/site-packages/nose/case.py", line 170, in runTest 2024-08-18T17:34:17.634 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-08-18T17:34:17.635 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_03f4c3e85ce172201896147a7aef87ba256187f2/qa/../src/test/rgw/rgw_multi/tests.py", line 1179, in test_bucket_sync_disable 2024-08-18T17:34:17.635 INFO:tasks.rgw_multisite_tests: check_buckets_sync_status_obj_not_exist(zone, buckets) 2024-08-18T17:34:17.635 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_03f4c3e85ce172201896147a7aef87ba256187f2/qa/../src/test/rgw/rgw_multi/tests.py", line 442, in check_buckets_sync_status_obj_not_exist 2024-08-18T17:34:17.635 INFO:tasks.rgw_multisite_tests: assert False 2024-08-18T17:34:17.635 INFO:tasks.rgw_multisite_tests:AssertionError 2024-08-18T17:34:17.635 INFO:tasks.rgw_multisite_tests: 2024-08-18T17:34:17.635 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-08-18T17:34:17.635 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_reshard_index_log_trim 2024-08-18T17:34:17.635 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-08-18T17:34:17.636 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-08-18T17:34:17.636 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/virtualenv/lib/python3.10/site-packages/nose/case.py", line 170, in runTest 2024-08-18T17:34:17.636 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-08-18T17:34:17.636 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_03f4c3e85ce172201896147a7aef87ba256187f2/qa/../src/test/rgw/rgw_multi/tests.py", line 1451, in test_bucket_reshard_index_log_trim 2024-08-18T17:34:17.636 INFO:tasks.rgw_multisite_tests: assert(len(json_obj_5['layout']['logs']) == 1) 2024-08-18T17:34:17.636 INFO:tasks.rgw_multisite_tests:AssertionError 2024-08-18T17:34:17.636 INFO:tasks.rgw_multisite_tests: 2024-08-18T17:34:17.636 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-08-18T17:34:17.636 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_resharded_bucket_full_sync_after_data_sync_init 2024-08-18T17:34:17.636 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-08-18T17:34:17.636 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-08-18T17:34:17.636 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/virtualenv/lib/python3.10/site-packages/nose/case.py", line 170, in runTest 2024-08-18T17:34:17.636 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-08-18T17:34:17.637 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_03f4c3e85ce172201896147a7aef87ba256187f2/qa/../src/test/rgw/rgw_multi/tests.py", line 1809, in test_resharded_bucket_full_sync_after_data_sync_init 2024-08-18T17:34:17.637 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket.name) 2024-08-18T17:34:17.637 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_03f4c3e85ce172201896147a7aef87ba256187f2/qa/../src/test/rgw/rgw_multi/tests.py", line 395, in zonegroup_bucket_checkpoint 2024-08-18T17:34:17.637 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2024-08-18T17:34:17.637 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_03f4c3e85ce172201896147a7aef87ba256187f2/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2024-08-18T17:34:17.637 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2024-08-18T17:34:17.637 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_03f4c3e85ce172201896147a7aef87ba256187f2/qa/tasks/rgw_multisite.py", line 197, in admin 2024-08-18T17:34:17.637 INFO:tasks.rgw_multisite_tests: assert r == 0 2024-08-18T17:34:17.637 INFO:tasks.rgw_multisite_tests:AssertionError 2024-08-18T17:34:17.637 INFO:tasks.rgw_multisite_tests: 2024-08-18T17:34:17.637 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-08-18T17:34:17.637 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_incremental_sync_after_data_sync_init 2024-08-18T17:34:17.637 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-08-18T17:34:17.637 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-08-18T17:34:17.638 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/virtualenv/lib/python3.10/site-packages/nose/case.py", line 170, in runTest 2024-08-18T17:34:17.638 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-08-18T17:34:17.638 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_03f4c3e85ce172201896147a7aef87ba256187f2/qa/../src/test/rgw/rgw_multi/tests.py", line 1846, in test_bucket_incremental_sync_after_data_sync_init 2024-08-18T17:34:17.638 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket.name) 2024-08-18T17:34:17.638 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_03f4c3e85ce172201896147a7aef87ba256187f2/qa/../src/test/rgw/rgw_multi/tests.py", line 395, in zonegroup_bucket_checkpoint 2024-08-18T17:34:17.638 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2024-08-18T17:34:17.638 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_03f4c3e85ce172201896147a7aef87ba256187f2/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2024-08-18T17:34:17.638 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2024-08-18T17:34:17.638 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_03f4c3e85ce172201896147a7aef87ba256187f2/qa/tasks/rgw_multisite.py", line 197, in admin 2024-08-18T17:34:17.638 INFO:tasks.rgw_multisite_tests: assert r == 0 2024-08-18T17:34:17.638 INFO:tasks.rgw_multisite_tests:AssertionError 2024-08-18T17:34:17.639 INFO:tasks.rgw_multisite_tests: 2024-08-18T17:34:17.639 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-08-18T17:34:17.639 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_resharded_bucket_incremental_sync_latest_after_data_sync_init 2024-08-18T17:34:17.639 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-08-18T17:34:17.639 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-08-18T17:34:17.639 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/virtualenv/lib/python3.10/site-packages/nose/case.py", line 170, in runTest 2024-08-18T17:34:17.639 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-08-18T17:34:17.639 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_03f4c3e85ce172201896147a7aef87ba256187f2/qa/../src/test/rgw/rgw_multi/tests.py", line 1875, in test_resharded_bucket_incremental_sync_latest_after_data_sync_init 2024-08-18T17:34:17.639 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket.name) 2024-08-18T17:34:17.639 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_03f4c3e85ce172201896147a7aef87ba256187f2/qa/../src/test/rgw/rgw_multi/tests.py", line 395, in zonegroup_bucket_checkpoint 2024-08-18T17:34:17.639 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2024-08-18T17:34:17.640 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_03f4c3e85ce172201896147a7aef87ba256187f2/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2024-08-18T17:34:17.640 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2024-08-18T17:34:17.640 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_03f4c3e85ce172201896147a7aef87ba256187f2/qa/tasks/rgw_multisite.py", line 197, in admin 2024-08-18T17:34:17.640 INFO:tasks.rgw_multisite_tests: assert r == 0 2024-08-18T17:34:17.640 INFO:tasks.rgw_multisite_tests:AssertionError 2024-08-18T17:34:17.640 INFO:tasks.rgw_multisite_tests: 2024-08-18T17:34:17.640 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-08-18T17:34:17.640 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_resharded_bucket_incremental_sync_oldest_after_data_sync_init 2024-08-18T17:34:17.640 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-08-18T17:34:17.640 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-08-18T17:34:17.640 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/virtualenv/lib/python3.10/site-packages/nose/case.py", line 170, in runTest 2024-08-18T17:34:17.640 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-08-18T17:34:17.640 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_03f4c3e85ce172201896147a7aef87ba256187f2/qa/../src/test/rgw/rgw_multi/tests.py", line 1937, in test_resharded_bucket_incremental_sync_oldest_after_data_sync_init 2024-08-18T17:34:17.641 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket.name) 2024-08-18T17:34:17.641 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_03f4c3e85ce172201896147a7aef87ba256187f2/qa/../src/test/rgw/rgw_multi/tests.py", line 395, in zonegroup_bucket_checkpoint 2024-08-18T17:34:17.641 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2024-08-18T17:34:17.641 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_03f4c3e85ce172201896147a7aef87ba256187f2/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2024-08-18T17:34:17.641 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2024-08-18T17:34:17.641 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_03f4c3e85ce172201896147a7aef87ba256187f2/qa/tasks/rgw_multisite.py", line 197, in admin 2024-08-18T17:34:17.641 INFO:tasks.rgw_multisite_tests: assert r == 0 2024-08-18T17:34:17.641 INFO:tasks.rgw_multisite_tests:AssertionError 2024-08-18T17:34:17.641 INFO:tasks.rgw_multisite_tests: 2024-08-18T17:34:17.641 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-08-18T17:34:17.641 INFO:tasks.rgw_multisite_tests:FAIL: test_sync_flow_directional_zonegroup_select: 2024-08-18T17:34:17.641 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-08-18T17:34:17.641 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-08-18T17:34:17.641 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/virtualenv/lib/python3.10/site-packages/nose/case.py", line 170, in runTest 2024-08-18T17:34:17.641 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-08-18T17:34:17.642 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_03f4c3e85ce172201896147a7aef87ba256187f2/qa/../src/test/rgw/rgw_multi/tests.py", line 2340, in test_sync_flow_directional_zonegroup_select 2024-08-18T17:34:17.642 INFO:tasks.rgw_multisite_tests: check_objects_exist(bucket, objnamesA, content) 2024-08-18T17:34:17.642 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_03f4c3e85ce172201896147a7aef87ba256187f2/qa/../src/test/rgw/rgw_multi/tests.py", line 2080, in check_objects_exist 2024-08-18T17:34:17.642 INFO:tasks.rgw_multisite_tests: check_object_exists(bucket, objname, content) 2024-08-18T17:34:17.642 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_03f4c3e85ce172201896147a7aef87ba256187f2/qa/../src/test/rgw/rgw_multi/tests.py", line 2074, in check_object_exists 2024-08-18T17:34:17.642 INFO:tasks.rgw_multisite_tests: assert_not_equal(k, None) 2024-08-18T17:34:17.642 INFO:tasks.rgw_multisite_tests:AssertionError: None == None 2024-08-18T17:34:17.642 INFO:tasks.rgw_multisite_tests: 2024-08-18T17:34:17.642 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-08-18T17:34:17.642 INFO:tasks.rgw_multisite_tests:FAIL: test_sync_different_buckets: 2024-08-18T17:34:17.642 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-08-18T17:34:17.642 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-08-18T17:34:17.642 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/virtualenv/lib/python3.10/site-packages/nose/case.py", line 170, in runTest 2024-08-18T17:34:17.642 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-08-18T17:34:17.643 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_03f4c3e85ce172201896147a7aef87ba256187f2/qa/../src/test/rgw/rgw_multi/tests.py", line 2622, in test_sync_different_buckets 2024-08-18T17:34:17.643 INFO:tasks.rgw_multisite_tests: check_objects_exist(bucket, objnamesC, content) 2024-08-18T17:34:17.643 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_03f4c3e85ce172201896147a7aef87ba256187f2/qa/../src/test/rgw/rgw_multi/tests.py", line 2080, in check_objects_exist 2024-08-18T17:34:17.643 INFO:tasks.rgw_multisite_tests: check_object_exists(bucket, objname, content) 2024-08-18T17:34:17.643 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_03f4c3e85ce172201896147a7aef87ba256187f2/qa/../src/test/rgw/rgw_multi/tests.py", line 2074, in check_object_exists 2024-08-18T17:34:17.643 INFO:tasks.rgw_multisite_tests: assert_not_equal(k, None) 2024-08-18T17:34:17.643 INFO:tasks.rgw_multisite_tests:AssertionError: None == None 2024-08-18T17:34:17.643 INFO:tasks.rgw_multisite_tests: 2024-08-18T17:34:17.643 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-08-18T17:34:17.643 INFO:tasks.rgw_multisite_tests:FAIL: test_sync_multiple_buckets_to_single: 2024-08-18T17:34:17.643 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-08-18T17:34:17.643 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-08-18T17:34:17.643 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/virtualenv/lib/python3.10/site-packages/nose/case.py", line 170, in runTest 2024-08-18T17:34:17.643 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-08-18T17:34:17.644 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_03f4c3e85ce172201896147a7aef87ba256187f2/qa/../src/test/rgw/rgw_multi/tests.py", line 2725, in test_sync_multiple_buckets_to_single 2024-08-18T17:34:17.644 INFO:tasks.rgw_multisite_tests: check_objects_not_exist(bucket, objnamesA) 2024-08-18T17:34:17.644 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_03f4c3e85ce172201896147a7aef87ba256187f2/qa/../src/test/rgw/rgw_multi/tests.py", line 2088, in check_objects_not_exist 2024-08-18T17:34:17.644 INFO:tasks.rgw_multisite_tests: check_object_not_exists(bucket, objname) 2024-08-18T17:34:17.644 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_03f4c3e85ce172201896147a7aef87ba256187f2/qa/../src/test/rgw/rgw_multi/tests.py", line 2084, in check_object_not_exists 2024-08-18T17:34:17.644 INFO:tasks.rgw_multisite_tests: assert_equal(k, None) 2024-08-18T17:34:17.644 INFO:tasks.rgw_multisite_tests:AssertionError: != None 2024-08-18T17:34:17.644 INFO:tasks.rgw_multisite_tests: 2024-08-18T17:34:17.644 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-08-18T17:34:17.644 INFO:tasks.rgw_multisite_tests:Ran 48 tests in 7974.791s 2024-08-18T17:34:17.645 INFO:tasks.rgw_multisite_tests: 2024-08-18T17:34:17.645 INFO:tasks.rgw_multisite_tests:FAILED (SKIP=1, failures=10) 2024-08-18T17:34:17.645 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/run_tasks.py", line 112, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_ceph_03f4c3e85ce172201896147a7aef87ba256187f2/qa/tasks/rgw_multisite_tests.py", line 95, in begin assert nose.run(argv=argv, config=conf), 'rgw multisite test failures' AssertionError: rgw multisite test failures 2024-08-18T17:34:17.830 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=4ca3583410c240b38baa6678943df5bf Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/run_tasks.py", line 112, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_ceph_03f4c3e85ce172201896147a7aef87ba256187f2/qa/tasks/rgw_multisite_tests.py", line 95, in begin assert nose.run(argv=argv, config=conf), 'rgw multisite test failures' AssertionError: rgw multisite test failures 2024-08-18T17:34:17.920 DEBUG:teuthology.run_tasks:Unwinding manager rgw-multisite-tests 2024-08-18T17:34:17.938 DEBUG:teuthology.run_tasks:Unwinding manager rgw-multisite 2024-08-18T17:34:17.945 DEBUG:teuthology.run_tasks:Unwinding manager rgw 2024-08-18T17:34:17.953 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2024-08-18T17:34:17.953 INFO:teuthology.orchestra.run:waiting for 300 2024-08-18T17:34:17.956 INFO:tasks.rgw.c1.client.0.smithi134.stdout:2024-08-18T17:34:17.953+0000 7f13eb858640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 39590) UID: 0 2024-08-18T17:34:17.957 INFO:tasks.rgw.c1.client.0.smithi134.stdout:2024-08-18T17:34:17.953+0000 7f13ef0b7880 -1 shutting down 2024-08-18T17:34:20.850 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2024-08-18T17:34:24.055 INFO:tasks.rgw.c1.client.0:Stopped 2024-08-18T17:34:24.055 DEBUG:teuthology.orchestra.run.smithi134:> rm -f /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock 2024-08-18T17:34:24.074 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -f /etc/ceph/vault-root-token 2024-08-18T17:34:24.147 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -f /home/ubuntu/cephtest/url_file 2024-08-18T17:34:24.216 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2024-08-18T17:34:24.216 INFO:teuthology.orchestra.run:waiting for 300 2024-08-18T17:34:24.257 INFO:tasks.rgw.c1.client.1.smithi134.stdout:2024-08-18T17:34:24.256+0000 7f8608f3b640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 92940) UID: 0 2024-08-18T17:34:24.257 INFO:tasks.rgw.c1.client.1.smithi134.stdout:2024-08-18T17:34:24.256+0000 7f860c793880 -1 shutting down 2024-08-18T17:34:26.355 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2024-08-18T17:34:26.356 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 has been restored 2024-08-18T17:34:30.319 INFO:tasks.rgw.c1.client.1:Stopped 2024-08-18T17:34:30.319 DEBUG:teuthology.orchestra.run.smithi134:> rm -f /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock 2024-08-18T17:34:30.339 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -f /etc/ceph/vault-root-token 2024-08-18T17:34:30.407 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -f /home/ubuntu/cephtest/url_file 2024-08-18T17:34:30.476 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2024-08-18T17:34:30.476 INFO:teuthology.orchestra.run:waiting for 300 2024-08-18T17:34:30.477 INFO:tasks.rgw.c2.client.0.smithi193.stdout:2024-08-18T17:34:30.476+0000 7f7086afc640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 86374) UID: 0 2024-08-18T17:34:30.477 INFO:tasks.rgw.c2.client.0.smithi193.stdout:2024-08-18T17:34:30.476+0000 7f708d00e880 -1 shutting down 2024-08-18T17:34:31.860 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2024-08-18T17:34:35.678 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2024-08-18T17:34:36.578 INFO:tasks.rgw.c2.client.0:Stopped 2024-08-18T17:34:36.578 DEBUG:teuthology.orchestra.run.smithi193:> rm -f /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock 2024-08-18T17:34:36.598 DEBUG:teuthology.orchestra.run.smithi193:> sudo rm -f /etc/ceph/vault-root-token 2024-08-18T17:34:36.668 DEBUG:teuthology.orchestra.run.smithi193:> sudo rm -f /home/ubuntu/cephtest/url_file 2024-08-18T17:34:36.736 DEBUG:tasks.rgw.c2.client.1:waiting for process to exit 2024-08-18T17:34:36.736 INFO:teuthology.orchestra.run:waiting for 300 2024-08-18T17:34:36.777 INFO:tasks.rgw.c2.client.1.smithi193.stdout:2024-08-18T17:34:36.776+0000 7f6847c4e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground (PID: 37454) UID: 0 2024-08-18T17:34:36.777 INFO:tasks.rgw.c2.client.1.smithi193.stdout:2024-08-18T17:34:36.776+0000 7f684b4a6880 -1 shutting down 2024-08-18T17:34:41.184 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~0s 2024-08-18T17:34:41.185 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2024-08-18T17:34:42.838 INFO:tasks.rgw.c2.client.1:Stopped 2024-08-18T17:34:42.839 DEBUG:teuthology.orchestra.run.smithi193:> rm -f /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock 2024-08-18T17:34:42.858 DEBUG:teuthology.orchestra.run.smithi193:> sudo rm -f /etc/ceph/vault-root-token 2024-08-18T17:34:42.928 DEBUG:teuthology.orchestra.run.smithi193:> sudo rm -f /home/ubuntu/cephtest/url_file 2024-08-18T17:34:42.997 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-08-18T17:34:43.007 DEBUG:teuthology.orchestra.run.smithi193:> sudo ceph --cluster c2 config set global mon_health_to_clog false 2024-08-18T17:34:43.340 INFO:teuthology.misc:Shutting down mds daemons... 2024-08-18T17:34:43.341 INFO:teuthology.misc:Shutting down osd daemons... 2024-08-18T17:34:43.341 DEBUG:tasks.ceph.c2.osd.0:waiting for process to exit 2024-08-18T17:34:43.341 INFO:teuthology.orchestra.run:waiting for 300 2024-08-18T17:34:43.383 INFO:tasks.ceph.c2.osd.0:Stopped 2024-08-18T17:34:43.384 DEBUG:tasks.ceph.c2.osd.1:waiting for process to exit 2024-08-18T17:34:43.384 INFO:teuthology.orchestra.run:waiting for 300 2024-08-18T17:34:43.427 INFO:tasks.ceph.c2.osd.1:Stopped 2024-08-18T17:34:43.428 DEBUG:tasks.ceph.c2.osd.2:waiting for process to exit 2024-08-18T17:34:43.428 INFO:teuthology.orchestra.run:waiting for 300 2024-08-18T17:34:43.471 INFO:tasks.ceph.c2.osd.2:Stopped 2024-08-18T17:34:43.471 INFO:teuthology.misc:Shutting down mgr daemons... 2024-08-18T17:34:43.472 DEBUG:tasks.ceph.c2.mgr.x:waiting for process to exit 2024-08-18T17:34:43.472 INFO:teuthology.orchestra.run:waiting for 300 2024-08-18T17:34:43.498 INFO:tasks.ceph.c2.mgr.x:Stopped 2024-08-18T17:34:43.498 INFO:teuthology.misc:Shutting down mon daemons... 2024-08-18T17:34:43.498 DEBUG:tasks.ceph.c2.mon.a:waiting for process to exit 2024-08-18T17:34:43.499 INFO:teuthology.orchestra.run:waiting for 300 2024-08-18T17:34:43.563 INFO:tasks.ceph.c2.mon.a:Stopped 2024-08-18T17:34:43.563 INFO:tasks.ceph:Checking cluster log for badness... 2024-08-18T17:34:43.563 DEBUG:teuthology.orchestra.run.smithi193:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/c2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'not have an application enabled' | head -n 1 2024-08-18T17:34:43.631 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-0 on ubuntu@smithi193.front.sepia.ceph.com 2024-08-18T17:34:43.631 DEBUG:teuthology.orchestra.run.smithi193:> sync && sudo umount -f /var/lib/ceph/osd/c2-0 2024-08-18T17:34:44.006 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-1 on ubuntu@smithi193.front.sepia.ceph.com 2024-08-18T17:34:44.006 DEBUG:teuthology.orchestra.run.smithi193:> sync && sudo umount -f /var/lib/ceph/osd/c2-1 2024-08-18T17:34:44.102 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-2 on ubuntu@smithi193.front.sepia.ceph.com 2024-08-18T17:34:44.102 DEBUG:teuthology.orchestra.run.smithi193:> sync && sudo umount -f /var/lib/ceph/osd/c2-2 2024-08-18T17:34:44.168 INFO:tasks.ceph:Archiving mon data... 2024-08-18T17:34:44.170 DEBUG:teuthology.misc:Transferring archived files from smithi193:/var/lib/ceph/mon/c2-a to /home/teuthworker/archive/teuthology-2024-08-16_22:40:05-rgw-reef-distro-default-smithi/7858171/data/c2.mon.a.tgz 2024-08-18T17:34:44.170 DEBUG:teuthology.orchestra.run.smithi193:> mktemp 2024-08-18T17:34:44.186 INFO:teuthology.orchestra.run.smithi193.stdout:/tmp/tmp.6Xx92fb66L 2024-08-18T17:34:44.186 DEBUG:teuthology.orchestra.run.smithi193:> sudo tar cz -f - -C /var/lib/ceph/mon/c2-a -- . > /tmp/tmp.6Xx92fb66L 2024-08-18T17:34:44.363 DEBUG:teuthology.orchestra.run.smithi193:> sudo chmod 0666 /tmp/tmp.6Xx92fb66L 2024-08-18T17:34:44.447 DEBUG:teuthology.orchestra.remote:smithi193:/tmp/tmp.6Xx92fb66L is 403KB 2024-08-18T17:34:44.544 DEBUG:teuthology.orchestra.run.smithi193:> rm -fr /tmp/tmp.6Xx92fb66L 2024-08-18T17:34:44.560 INFO:tasks.ceph:Cleaning ceph cluster... 2024-08-18T17:34:44.560 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -rf -- /etc/ceph/c2.conf /etc/ceph/c2.keyring /home/ubuntu/cephtest/c2.data /home/ubuntu/cephtest/c2.monmap /home/ubuntu/cephtest/../*.pid 2024-08-18T17:34:44.563 DEBUG:teuthology.orchestra.run.smithi193:> sudo rm -rf -- /etc/ceph/c2.conf /etc/ceph/c2.keyring /home/ubuntu/cephtest/c2.data /home/ubuntu/cephtest/c2.monmap /home/ubuntu/cephtest/../*.pid 2024-08-18T17:34:44.624 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-08-18T17:34:44.634 DEBUG:teuthology.orchestra.run.smithi134:> sudo ceph --cluster c1 config set global mon_health_to_clog false 2024-08-18T17:34:44.940 INFO:teuthology.misc:Shutting down mds daemons... 2024-08-18T17:34:44.940 INFO:teuthology.misc:Shutting down osd daemons... 2024-08-18T17:34:44.940 DEBUG:tasks.ceph.c1.osd.0:waiting for process to exit 2024-08-18T17:34:44.941 INFO:teuthology.orchestra.run:waiting for 300 2024-08-18T17:34:44.999 INFO:tasks.ceph.c1.osd.0:Stopped 2024-08-18T17:34:45.000 DEBUG:tasks.ceph.c1.osd.1:waiting for process to exit 2024-08-18T17:34:45.000 INFO:teuthology.orchestra.run:waiting for 300 2024-08-18T17:34:45.060 INFO:tasks.ceph.c1.osd.1:Stopped 2024-08-18T17:34:45.061 DEBUG:tasks.ceph.c1.osd.2:waiting for process to exit 2024-08-18T17:34:45.061 INFO:teuthology.orchestra.run:waiting for 300 2024-08-18T17:34:45.113 INFO:tasks.ceph.c1.osd.2:Stopped 2024-08-18T17:34:45.113 INFO:teuthology.misc:Shutting down mgr daemons... 2024-08-18T17:34:45.114 DEBUG:tasks.ceph.c1.mgr.x:waiting for process to exit 2024-08-18T17:34:45.114 INFO:teuthology.orchestra.run:waiting for 300 2024-08-18T17:34:45.141 INFO:tasks.ceph.c1.mgr.x:Stopped 2024-08-18T17:34:45.142 INFO:teuthology.misc:Shutting down mon daemons... 2024-08-18T17:34:45.142 DEBUG:tasks.ceph.c1.mon.a:waiting for process to exit 2024-08-18T17:34:45.142 INFO:teuthology.orchestra.run:waiting for 300 2024-08-18T17:34:45.204 INFO:tasks.ceph.c1.mon.a:Stopped 2024-08-18T17:34:45.204 INFO:tasks.ceph:Checking cluster log for badness... 2024-08-18T17:34:45.204 DEBUG:teuthology.orchestra.run.smithi134:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/c1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'not have an application enabled' | head -n 1 2024-08-18T17:34:45.272 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-0 on ubuntu@smithi134.front.sepia.ceph.com 2024-08-18T17:34:45.272 DEBUG:teuthology.orchestra.run.smithi134:> sync && sudo umount -f /var/lib/ceph/osd/c1-0 2024-08-18T17:34:45.606 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-1 on ubuntu@smithi134.front.sepia.ceph.com 2024-08-18T17:34:45.607 DEBUG:teuthology.orchestra.run.smithi134:> sync && sudo umount -f /var/lib/ceph/osd/c1-1 2024-08-18T17:34:45.695 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-2 on ubuntu@smithi134.front.sepia.ceph.com 2024-08-18T17:34:45.696 DEBUG:teuthology.orchestra.run.smithi134:> sync && sudo umount -f /var/lib/ceph/osd/c1-2 2024-08-18T17:34:45.771 INFO:tasks.ceph:Archiving mon data... 2024-08-18T17:34:45.772 DEBUG:teuthology.misc:Transferring archived files from smithi134:/var/lib/ceph/mon/c1-a to /home/teuthworker/archive/teuthology-2024-08-16_22:40:05-rgw-reef-distro-default-smithi/7858171/data/c1.mon.a.tgz 2024-08-18T17:34:45.772 DEBUG:teuthology.orchestra.run.smithi134:> mktemp 2024-08-18T17:34:45.791 INFO:teuthology.orchestra.run.smithi134.stdout:/tmp/tmp.SpNUwg4bfY 2024-08-18T17:34:45.791 DEBUG:teuthology.orchestra.run.smithi134:> sudo tar cz -f - -C /var/lib/ceph/mon/c1-a -- . > /tmp/tmp.SpNUwg4bfY 2024-08-18T17:34:45.970 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod 0666 /tmp/tmp.SpNUwg4bfY 2024-08-18T17:34:46.053 DEBUG:teuthology.orchestra.remote:smithi134:/tmp/tmp.SpNUwg4bfY is 439KB 2024-08-18T17:34:46.123 DEBUG:teuthology.orchestra.run.smithi134:> rm -fr /tmp/tmp.SpNUwg4bfY 2024-08-18T17:34:46.139 INFO:tasks.ceph:Cleaning ceph cluster... 2024-08-18T17:34:46.139 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -rf -- /etc/ceph/c1.conf /etc/ceph/c1.keyring /home/ubuntu/cephtest/c1.data /home/ubuntu/cephtest/c1.monmap /home/ubuntu/cephtest/../*.pid 2024-08-18T17:34:46.181 DEBUG:teuthology.orchestra.run.smithi193:> sudo rm -rf -- /etc/ceph/c1.conf /etc/ceph/c1.keyring /home/ubuntu/cephtest/c1.data /home/ubuntu/cephtest/c1.monmap /home/ubuntu/cephtest/../*.pid 2024-08-18T17:34:46.186 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 has been restored 2024-08-18T17:34:46.211 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2024-08-18T17:34:46.211 DEBUG:teuthology.orchestra.run.smithi134:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-08-18T17:34:46.246 DEBUG:teuthology.orchestra.run.smithi193:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-08-18T17:34:46.271 INFO:teuthology.orchestra.run.smithi134.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-08-18T17:34:46.281 INFO:teuthology.orchestra.run.smithi193.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-08-18T17:34:46.286 INFO:tasks.ceph:Archiving crash dumps... 2024-08-18T17:34:46.288 DEBUG:teuthology.misc:Transferring archived files from smithi134:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-08-16_22:40:05-rgw-reef-distro-default-smithi/7858171/remote/smithi134/crash 2024-08-18T17:34:46.289 DEBUG:teuthology.orchestra.run.smithi134:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-08-18T17:34:46.340 DEBUG:teuthology.misc:Transferring archived files from smithi193:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-08-16_22:40:05-rgw-reef-distro-default-smithi/7858171/remote/smithi193/crash 2024-08-18T17:34:46.341 DEBUG:teuthology.orchestra.run.smithi193:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-08-18T17:34:46.369 INFO:tasks.ceph:Compressing logs... 2024-08-18T17:34:46.369 DEBUG:teuthology.orchestra.run.smithi134:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-08-18T17:34:46.385 DEBUG:teuthology.orchestra.run.smithi193:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-08-18T17:38:18.553 INFO:tasks.ceph:Archiving logs... 2024-08-18T17:38:18.554 DEBUG:teuthology.misc:Transferring archived files from smithi134:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-08-16_22:40:05-rgw-reef-distro-default-smithi/7858171/remote/smithi134/log 2024-08-18T17:38:18.556 DEBUG:teuthology.orchestra.run.smithi134:> sudo tar c -f - -C /var/log/ceph -- . 2024-08-18T17:38:44.233 DEBUG:teuthology.misc:Transferring archived files from smithi193:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-08-16_22:40:05-rgw-reef-distro-default-smithi/7858171/remote/smithi193/log 2024-08-18T17:38:44.233 DEBUG:teuthology.orchestra.run.smithi193:> sudo tar c -f - -C /var/log/ceph -- . 2024-08-18T17:39:00.068 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-08-18T17:39:00.081 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-08-18T17:39:00.081 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-08-18T17:39:00.085 DEBUG:teuthology.orchestra.run.smithi193:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-08-18T17:39:00.161 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-08-18T17:39:00.171 INFO:teuthology.task.clock:Checking final clock skew... 2024-08-18T17:39:00.171 DEBUG:teuthology.orchestra.run.smithi134:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-18T17:39:00.174 DEBUG:teuthology.orchestra.run.smithi193:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-18T17:39:00.188 INFO:teuthology.orchestra.run.smithi134.stderr:bash: line 1: ntpq: command not found 2024-08-18T17:39:00.190 INFO:teuthology.orchestra.run.smithi193.stderr:bash: line 1: ntpq: command not found 2024-08-18T17:39:00.193 INFO:teuthology.orchestra.run.smithi134.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-08-18T17:39:00.193 INFO:teuthology.orchestra.run.smithi134.stdout:=============================================================================== 2024-08-18T17:39:00.193 INFO:teuthology.orchestra.run.smithi134.stdout:^* hv01.front.sepia.ceph.com 3 7 377 98 +776us[ +725us] +/- 64ms 2024-08-18T17:39:00.194 INFO:teuthology.orchestra.run.smithi134.stdout:^+ hv02.front.sepia.ceph.com 3 8 377 155 -991us[-1044us] +/- 52ms 2024-08-18T17:39:00.194 INFO:teuthology.orchestra.run.smithi134.stdout:^+ hv03.front.sepia.ceph.com 3 8 377 228 -195us[ -247us] +/- 82ms 2024-08-18T17:39:00.194 INFO:teuthology.orchestra.run.smithi134.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-08-18T17:39:00.195 INFO:teuthology.orchestra.run.smithi193.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-08-18T17:39:00.195 INFO:teuthology.orchestra.run.smithi193.stdout:=============================================================================== 2024-08-18T17:39:00.196 INFO:teuthology.orchestra.run.smithi193.stdout:^* hv01.front.sepia.ceph.com 3 7 377 31 +937us[ +955us] +/- 65ms 2024-08-18T17:39:00.196 INFO:teuthology.orchestra.run.smithi193.stdout:^+ hv02.front.sepia.ceph.com 3 7 377 106 -813us[ -795us] +/- 53ms 2024-08-18T17:39:00.196 INFO:teuthology.orchestra.run.smithi193.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 41 -67us[ -49us] +/- 85ms 2024-08-18T17:39:00.196 INFO:teuthology.orchestra.run.smithi193.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-08-18T17:39:00.197 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-08-18T17:39:00.205 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-08-18T17:39:00.206 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-08-18T17:39:00.214 DEBUG:teuthology.orchestra.run.smithi134:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-08-18T17:39:00.307 DEBUG:teuthology.orchestra.run.smithi193:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-08-18T17:39:00.380 DEBUG:teuthology.orchestra.run.smithi134:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-08-18T17:39:00.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-18T17:39:00.410 DEBUG:teuthology.orchestra.run.smithi193:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-08-18T17:39:00.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-18T17:39:00.440 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-08-18T17:39:00.450 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-08-18T17:39:00.458 INFO:teuthology.task.internal:Duration was 8921.843282 seconds 2024-08-18T17:39:00.459 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-08-18T17:39:00.467 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-08-18T17:39:00.468 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-08-18T17:39:00.470 DEBUG:teuthology.orchestra.run.smithi193:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-08-18T17:39:00.508 INFO:teuthology.orchestra.run.smithi134.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-08-18T17:39:00.521 INFO:teuthology.orchestra.run.smithi193.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-08-18T17:39:00.870 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-08-18T17:39:00.870 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi134.front.sepia.ceph.com 2024-08-18T17:39:00.871 DEBUG:teuthology.orchestra.run.smithi134:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-08-18T17:39:00.932 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi193.front.sepia.ceph.com 2024-08-18T17:39:00.932 DEBUG:teuthology.orchestra.run.smithi193:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-08-18T17:39:00.954 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-08-18T17:39:00.955 DEBUG:teuthology.orchestra.run.smithi134:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-08-18T17:39:00.975 DEBUG:teuthology.orchestra.run.smithi193:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-08-18T17:39:01.023 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-08-18T17:39:01.024 DEBUG:teuthology.orchestra.run.smithi134:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-08-18T17:39:01.042 DEBUG:teuthology.orchestra.run.smithi193:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-08-18T17:39:01.170 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-08-18T17:39:01.183 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-08-18T17:39:01.184 DEBUG:teuthology.orchestra.run.smithi134:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-08-18T17:39:01.208 DEBUG:teuthology.orchestra.run.smithi193:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-08-18T17:39:01.233 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-08-18T17:39:01.244 DEBUG:teuthology.orchestra.run.smithi134:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-08-18T17:39:01.251 DEBUG:teuthology.orchestra.run.smithi193:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-08-18T17:39:01.272 INFO:teuthology.orchestra.run.smithi134.stdout:kernel.core_pattern = core 2024-08-18T17:39:01.296 INFO:teuthology.orchestra.run.smithi193.stdout:kernel.core_pattern = core 2024-08-18T17:39:01.339 DEBUG:teuthology.orchestra.run.smithi134:> test -e /home/ubuntu/cephtest/archive/coredump 2024-08-18T17:39:01.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-18T17:39:01.362 DEBUG:teuthology.orchestra.run.smithi193:> test -e /home/ubuntu/cephtest/archive/coredump 2024-08-18T17:39:01.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-18T17:39:01.396 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-08-18T17:39:01.407 INFO:teuthology.task.internal:Transferring archived files... 2024-08-18T17:39:01.408 DEBUG:teuthology.misc:Transferring archived files from smithi134:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-08-16_22:40:05-rgw-reef-distro-default-smithi/7858171/remote/smithi134 2024-08-18T17:39:01.408 DEBUG:teuthology.orchestra.run.smithi134:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-08-18T17:39:01.441 DEBUG:teuthology.misc:Transferring archived files from smithi193:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-08-16_22:40:05-rgw-reef-distro-default-smithi/7858171/remote/smithi193 2024-08-18T17:39:01.442 DEBUG:teuthology.orchestra.run.smithi193:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-08-18T17:39:01.475 INFO:teuthology.task.internal:Removing archive directory... 2024-08-18T17:39:01.475 DEBUG:teuthology.orchestra.run.smithi134:> rm -rf -- /home/ubuntu/cephtest/archive 2024-08-18T17:39:01.478 DEBUG:teuthology.orchestra.run.smithi193:> rm -rf -- /home/ubuntu/cephtest/archive 2024-08-18T17:39:01.525 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-08-18T17:39:01.536 INFO:teuthology.task.internal:Not uploading archives. 2024-08-18T17:39:01.536 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-08-18T17:39:01.546 INFO:teuthology.task.internal:Tidying up after the test... 2024-08-18T17:39:01.547 DEBUG:teuthology.orchestra.run.smithi134:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-08-18T17:39:01.549 DEBUG:teuthology.orchestra.run.smithi193:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-08-18T17:39:01.562 INFO:teuthology.orchestra.run.smithi134.stdout: 262165 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Aug 18 17:39 /home/ubuntu/cephtest 2024-08-18T17:39:01.580 INFO:teuthology.orchestra.run.smithi193.stdout: 265688 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Aug 18 17:39 /home/ubuntu/cephtest 2024-08-18T17:39:01.581 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-08-18T17:39:01.591 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-08-18T17:39:01.615 INFO:teuthology.run:Summary data: description: rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zones supported-random-distro$/{centos_latest} tasks/test_multi} duration: 8921.84328174591 failure_reason: rgw multisite test failures flavor: default owner: scheduled_teuthology@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=4ca3583410c240b38baa6678943df5bf status: fail success: false 2024-08-18T17:39:01.615 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-08-18T17:39:01.704 INFO:teuthology.run:FAIL